MuPT: A Generative Symbolic Music Pretrained Transformer

## Abstract

Abstract In this paper, we explore the application of Large Language Models (LLMs) to the pre-training of music. While the prevalent use of MIDI in music modeling is well-established, our findings suggest that LLMs are inherently more compatible with ABC Notation, which aligns more closely with their design and strengths, thereby enhancing the model’s performance in musical composition.
To address the challenges associated with misaligned measures from different tracks during generation, we propose the development of a S ynchronized M ulti- T rack ABC Notation ( SMT-ABC Notation ), which aims to preserve coherence across multiple musical tracks.
Our contributions include a series of models capable of handling up to 8192 tokens, covering 90% of the symbolic music data in our training set. Furthermore, we explore the implications of the S ymbolic M usic S caling Law ( SMS Law ) on model performance. The results indicate a promising direction for future research in music generation, offering extensive resources for community-led research through our open-source contributions.

## 1 Introduction

Large Language Models (LLMs) have experienced remarkable advancements, leading to their broad application across numerous domains. As these models extend into multimodal areas, such as visual and auditory fields, their capability to represent and model complex information, including images (Liu et al., ) and speech (Baevski et al., ) becomes increasingly critical. However, this expansion also highlights significant challenges that must be addressed. Specifically, the development of effective tokenizers for images and videos, as well as advanced codecs for the audio domain.

In the domain of music, Large Language Models encounter inherent challenges that hinder their effective utilization. Despite achieving state-of-the-art musical performance, as demonstrated by MuseNet (OpenAI, ), these models often struggle to capture the structural symmetry essential to aesthetically pleasing music. This issue stems from the use of Musical Instrument Digital Interface (MIDI), which, while effective, poses significant challenges in terms of music’s readability and structural representation.

To tackle this issue, the integration of ABC notation offers a novel approach to overcoming the limitations of MIDI formats. Yuan et al. () advocate for this method, highlighting ABC notation’s readability and structural coherence. Their methodology involves fine-tuning the LLAMA2 model, leveraging instruction tuning to enhance the model’s musical output capabilities (Touvron et al., ; ). The research overlooks critical tokenization considerations within musical compositions.

In this paper, we aim to propose a training standard with transformer decoder-only architecture for symbolic music generation tasks, which is suitable for single / multi-track music generation. We observe that mismatches between measures can occur by employing the traditional ’next-token-prediction’ paradigm for symbolic data training. This issue arises because ABC notations are generally notated track by track, completing one track before moving on to the next. To address this challenge, we propose SMT-ABC notation to facilitate the model’s learning of how each measure is expressed across various tracks.

Furthermore, we observe that the ABC Notation model benefits from additional epochs in the training phase. This suggests that repeated data positively impacts the model’s performance. To understand this phenomenon, we introduced the SMS Law for repetitive training with symbolic music data. This law explores how scaling up the training data affects the performance of symbolic music generation models, particularly in terms of validation loss. This investigation aims to provide clear insights into the relationship between data repetition and model efficacy, offering guidance for optimizing model training strategies.

In conclusion, our contributions are highlighted as follows:

- •
We develop a Long-range Symbolic Music LLM that introduced a foundation model trained on musical notes in ABC notation, with an extended sequence length of 8192 tokens, catering to over 90% of symbolic musical scores we collected.
- •
We propose SMT-ABC notation to represent notes, significantly improving the structural integrity and quality of the generated music by maintaining consistent measures within each track.
- •
We explore the SMS Law insights for ABC notation. We demonstrate that comprehensive song modeling yields superior performance with a positive correlation between model size and metric improvement. We also reveal unique training epoch dynamics in music repetition and performance enhancement.
- •
We will release a suite of state-of-the-art long-range foundation models in the music domain, articulated in ABC notation, along with intermediate training checkpoints to foster community research and innovation in symbolic music modeling.

## 2 Related work

### 2.1 Music Pre-training

Audio pre-training through the self-supervised learning paradigm has made great progress in speech (Baevski et al., ; Hsu et al., ; Baevski et al., ; Ma et al., ; Yang et al., ; Lin et al., ), general-purpose audio (Huang et al., ; Baade et al., ; Chen et al., ; ), as well as music (Zhu et al., ; Dong et al., ; Thickstun et al., ; Ma et al., ; Li et al., ).
Two types of self-supervised music pre-training have been explored: non-autoregressive discriminative models and autoregressive generative models.
Non-autoregressive discriminative music pre-training performs mask language modelling (MLM) by constructing a pretext task.
This kind of training makes models easier to adapt to downstream understanding tasks, such as music tagging, instrument classification, and beat tracking.
Autoregressive generative music pre-training models employ a GPT-style framework to generate music, either in codec (Copet et al., ) form or in symbolic form (Thickstun et al., ; Dong et al., ).
Previous symbolic music generation models utilize MIDI to model the sequence input and output, showing the ability to generate music given conditions, or unconditional generation.
Existing models are limited by not generating long enough music (Thickstun et al., ) and limited musicality (Dong et al., ).
Therefore, long-range symbolic music generation models with data scaling and model scaling need to be explored urgently.

### 2.2 Data Representation for Symbolic Music

Symbolic music representation formats such as MIDI, Humdrum, and ABC notation offer distinct approaches for representing musical information, each with unique advantages and applicability to computational music representation. MIDI, which excels in capturing musical notes and performance, is a popular choice in the music industry and research community(Huang & Yang, ; Huang et al., ; Lu et al., ). However, the complexity and length of MIDI sequences often challenge music models, which limit the preservation of a composition’s full continuity.
In contrast, ABC notation stands out for its textual simplicity and compactness, making it particularly suited for Natural Language Processing (NLP) techniques. It can be efficiently processed and analyzed using sequence modeling and pattern recognition algorithms similar to those used in language translation and text generation, enabling automated music generation and retrieval.

ABC notation’s simplicity and broad applicability have prompted research into enhancing music retrieval and generation through deep learning and NLP. In early research, LSTM networks showed promise by producing music similar to traditional and folk styles (Sturm et al., ), using ABC notation for automated composition. Following this, TunesFormer (Wu et al., ), a tool based on the Transformer designed for Irish tunes encoded in ABC notation, utilized techniques like bar patching and introduced control codes to craft melodies that meet specific musical forms. abcMLM (Casini et al., ), a masked language model, further demonstrated how structured ABC notation can be used to create folk-like tunes, highlighting the adaptability of NLP methods and the benefits of using non-autoregressive models in music. Recent studies have started to utilize pre-trained NLP models for converting text to music (Wu & Sun, ), showing how these resources can improve the creation of symbolic music. CLaMP (Wu et al., ) introduced a unique method for learning music and text jointly, using a large collection of music-text pairs to better search for and categorize music automatically. Techniques like text dropout and bar patching are examples of how NLP and music encoding are becoming more integrated. In a significant breakthrough, ChatMusician (Yuan et al., ) introduced a new approach to incorporating music as a second language for Large Language Models (LLMs), utilizing ABC notation to seamlessly blend music and text, thereby enabling internal music creation and analysis without relying on external multimodal frameworks.

### 2.3 Scaling Law

A wide range of research underscores a significant pattern in language model performance, indicating a power-law relationship between model performance and the increases in both the number of parameters and the size of the training data (Kaplan et al., ; Hoffmann et al., ; Ghorbani et al., ). Scaling law plays a pivotal role in advancing large language models (LLMs), offering a framework to predict the optimal configurations for larger models based on the training logs of their smaller counterparts (Gao et al., ).

Further exploration into scaling laws for autoregressive generative modeling by Henighan et al. () broadens the applicability of these laws to include not just textual, but also visual and multimodal tasks, as supported by studies in Ghorbani et al. (); Hernandez et al. (); Gao et al. (). Such insights are invaluable for developing music generation models, which often blend multiple modalities such as audio, lyrics, and visual elements like album covers or artist photos. This demonstrates a consistent trajectory of performance enhancement concurrent with resource scaling.

The research by Muennighoff et al. (), which involves the repetition of the entire pre-training dataset across multiple epochs, presents promising results yet raises questions regarding its effectiveness for musical data. This uncertainty prompts a need for further research into the impact of data repetition strategy by achieving improved outcomes for models engaged in music-related tasks.

## 3 Method

### 3.1 Model Architecture

MuPT utilizes a standard Transformer model architecture (Vaswani et al., ) in a decoder-only setup. Models are trained on a context
length of 8192 tokens. We list our MuPT model parameter in Table 1 and utilize several improvements proposed after the original transformer paper. Below, we list the included improvements:

- •
SwiGLU Activation: The SwiGLU activation mechanism, represented as $(\text{Swish}(xW)\cdot xV)$, is utilized for the MLP (Multi-Layer Perceptron) intermediate activations. This approach significantly surpasses traditional activation functions such as ReLU, GeLU, and Swish in performance (Shazeer, ).
- •
RMSNorm Each transformer sub-layer, including the attention and feedforward layers, is normalized using RMSNorm as proposed by Zhang & Sennrich ()
- •
RoPE Embeddings: In contrast to positional encoding (PE) strategy, we use the Rotary Positional Encoding (RoPE) technique, as developed by Su et al. (), aimed at enhancing long-context modeling.

### 3.2 SMT-ABC Notation

ABC notation is a widely adopted system for notating music using plain text, and it offers unique advantages when used in conjunction with deep learning models.
Its well-structured text format ensures easy preprocessing, efficient data transmission, and scalability of datasets.
The diverse collection of tunes and compositions in ABC notation facilitates learning various musical structures and styles.
Moreover, ABC notation allows models to generate human-readable outputs, leading to immediate feedback and iterative refinement.
These attributes significantly enhance both the efficiency and quality of the training process.

Figure: Figure 1: Example of a multi-track tune of ABC notation.
Refer to caption: extracted/5979792/figures/abc_sample.png

An ABC file is composed of headers followed by the music notation.
The former contain metadata regarding the tune, such as its composer and tempo, while the latter defines the melody.
In ABC notation, each note is represented by a letter, and additional symbols are utilized to convey duration, rhythm, and other musical characteristics.
An example is illustrated in Figure 1.
“V:1” indicates the beginning of the first music track and the lines before it are headers.
A tune can consist of one or more tracks, each representing a distinct musical element within the composition.
The bars within each track are separated by bar line symbols like vertical lines (“$|$”), which refer to the standard bar line.

In  Yuan et al. (), ABC files without any modification are the input of models.
However, we found that the models struggle with bar alignment when dealing with multiple tracks.
Since a track represents a section or division within a musical composition, such as one of the instrumental or vocal parts in a piece of music, it is crucial for models to capture the correspondence between tracks.
Specifically, this correspondence exists in bars with the same indices, and thus, they should be treated as a series of groups.
To this end, we reorganize the tracks as depicted in Figure 2.
We concatenate music segments from bars with the same index across all tracks, including their right bar lines.
These concatenated elements from different tracks are then enclosed by a pair of a newly introduced symbol “<$|$>”, which is not part of the original ABC system.
This symbol represents the beginning or the end of a group of bars at the same stage.
In cases where a tune contains only one track, each new unit will consist of a single bar.
After processing all the bars, we obtain a synchronized version of the music notation, while the headers remain unchanged.
The length of the tracks is not always identical due to repetition or other specific musical structures, which are difficult to handle exhaustively.
Considering these special samples typically account for just a small portion (0.01% in our dataset) of the entire dataset, we simply skip them in this scenario.

Figure: Figure 2: Illustration of synchronized multiple-track ABC notation. Music segments from bars sharing the same index across all tracks, along with their right bar lines, are concatenated to guarantee alignment. The combined elements are then enclosed by a pair of a newly introduced symbol “<$|$>”.
Refer to caption: x1.png

### 3.3 Tokenizer

We chose YouTokenToMe (YTTM) (YouTokenToMe, ) framework to develop a tokenizer with a vocabulary of 50,000 tokens, leveraging the Byte-Pair Encoding (BPE) (Shibata et al., ) for ABC notation tokenization. This method is instrumental in segmenting the ABC text into manageable units, thereby enhancing the model’s ability to interpret and process the input effectively. We do not apply any normalization and dummy prefix to the training corpus, without changing its form or adding extra parts at the beginning.
Additionally, a unique symbol “<n>“is employed to denote spaces within the ABC text, ensuring accurate space recognition by the model.

**Table 1: MuPT model structure with different model size.**
| Parameters | 190M | 505M | 1.07B | 1.97B | 4.23B |
| --- | --- | --- | --- | --- | --- |
| Hidden size | 768 | 1024 | 1280 | 1536 | 2048 |
| Layers | 12 | 16 | 20 | 24 | 32 |
| Feedforward hidden dimensions | 3072 | 4096 | 5120 | 6144 | 8192 |
| Num heads | 12 | 16 | 20 | 24 | 32 |
| Head size | 256 | 256 | 256 | 256 | 256 |

### 3.4 Scaling Law

**Table 2: Notation Definition for Scaling Law.**
| Notation | Definition |
| --- | --- |
| $N$ | The number of parameters. |
| $D$ | The training tokens. |
| $U_{D}$ | The number of unique tokens used. i.e., Number of tokens in each epoch. |
| $A$, $B$, $E$, $d$, $k$, $\alpha$, $\beta$ | Term parameters requiring fitting in Scaling Laws. |
| $k_{d}$, $k_{n}$, $k_{u}$, $k_{in}$ | Parameters to fit the term designed for overfitting after early stop points. |

The Chinchilla Law, proposed by DeepMind, is a scaling law that provides insights into the training of large language models (LLMs). Our experiments reveal that the Chinchilla Law (Hoffmann et al., ) provides a good fit for general cases, where moderate models were trained with a moderate amount of data. In this section, we will list two improvements to Chinchilla Law for symbolic music scaling principles on limited training data.

#### 3.4.1 Optimizing Baseline Scaling Laws under Computational Constraints

A pivotal aspect of scaling laws is the optimization of loss within the bounds of computational feasibility. This is formalized as minimizing the valid loss $L$, subject to constraints imposed by available computational resources ($C$), specifically FLOPs, as denoted below:

$$ $\arg\min_{N,D}L(N,D)\quad\text{s.t.}\quad\text{FLOPs}(N,D)=C$ (1) $$

This framework encapsulates the trade-offs between parameters ($N$) and training tokens ($D$), and decision-making processes inherent in scaling models under resource limitations, illuminating pathways to efficiency and efficacy in LLMs training.
Notation definition is in Table 2, and more details can be found in Appendix A.1.

In this paper, we will use the Chinchilla Law(Hoffmann et al., ) and Data-Constrained law(Muennighoff et al., ) as baselines. The former is a classical baseline in LLMs’ training and the latter is crucial to address the constraints faced in scenarios where the volume of available training data does not meet the ideal requisites. This phenomenon is typical in the music domain. Please refer to A.1.2 for more information.

#### 3.4.2 Symbolic Music Scaling (SMS) Law

Figure: Figure 3: Chinchilla Law prediction and loss survey in the setting with 2.1B unique tokens.
Refer to caption: x2.png

Figure 3 demonstrates the Chinchilla prediction in yellow lines and the observed loss in blue. We can tell that the Chinchilla law does not provide good results when the data volume $D$ is small when the model just begins the pre-training stage, and when $D$ is large where repeated data provides overfitting. We proposed several modifications to address these problems.

Continuous Adaptation of the Data-Constrained Law.

The data volume $D^{\prime}$ for Data-Constrained Law Muennighoff et al. () at $n$ epoch is less then $D=n\times U_{D}$. We use its approximation of Equation 18 in Appendix A of their paper instead of the standard $D^{\prime}$ for better prediction and simpler formulas. For more information please refer to Equation 11 in Appendix A.2. We denote the modified data volume as $D^{\prime\prime}$.

Incorporation of a New Term.

We can observe that when that model parameter is small (e.g. $N=190M$), the Chinchilla underestimates the loss value and overestimates when the model size is large (e.g. $N=1072M$). This suggests that the coefficient $B$ in the Chinchilla formula $L=\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}+E$ shall be relevant to $N$ instead of a constant.

$$ $L(N,D^{\prime\prime})=\frac{d}{N^{\alpha}\cdot D^{\prime\prime\beta}}+\frac{A} {N^{\alpha}}+\frac{B}{D^{\prime\prime\beta}}+E.$ (2) $$

To address the model’s limitations in accurately capturing performance metrics for smaller data sizes, we introduce an additional term, as delineated in Equation 2. This modification aims to refine the model’s fidelity, particularly in scenarios characterized by limited data availability. Further details on this modification can be found in the Appendix A.3.1. After that, we proposed another term to predict the early stop points and overfited loss curve.

Modelling Overfitting Settings.

Crucially, previous iterations of the model fall short in predicting overfitting, particularly beyond early stopping thresholds. This gap is especially pronounced in the context of Data-Constrained environments, such as music, where open-source data is limited. To this end, we introduce a new component, $L_{overfit}$, to the model, encapsulated in Equation 3, to specifically account for overfitting losses:

$$ $L\left(N,D,U_{D}\right)=\frac{d}{N^{\alpha}\cdot D^{\prime\prime\beta}}+\frac{ A}{N^{\alpha}}+\frac{B}{D^{\prime\prime\beta}}+E+L_{overfit}$ (3) $$

where $k_{d}$, $k_{n}$, $k_{u}$ and $k_{in}$ are constants, and

$$ $L_{overfit}=GELU\left\{k_{d}\cdot D+k_{n}\cdot\log(N)-k_{u}\cdot\log(U_{D})-k_ {in}\right\}$ (4) $$

is our overfitting formulation. For comprehensive insights into the overfitting loss component, please refer to Appendix A.3.2.

Parameter Fitting and Model Integration.

Initial parameter fitting for $\{\alpha$, $\beta$, $A$, $B$, $E\}$, and $d$, subsequent linear regression analysis, focusing on the residuals between Equation 2 and empirical observations, facilitates the calibration of overfitting parameters $\{k_{d}$, $k_{n}$, $k_{u}$, $k_{in}\}$ within Equation 4. The integration of these components in Equation 3 not only predicts performance under constrained conditions but accounts for overfitting dynamics, helping to predict the true minimum of loss curve.

## 4 Experiments

### 4.1 Experimental Setup

As outlined in section 3.1, we adopt similar model architecture from LLaMA2(Touvron et al., ),
including RMSNorm(Zhang & Sennrich, ) and SwiGLU(Shazeer, ).
In the full-scale data setting, we trained models of various sizes (ranging from 190M to 4.23B parameters) on the ABC text corpus, which consists of 33.6 billion tokens derived from a diverse collection of monophonic and polyphonic musical compositions spanning various genres and styles.
For our data repetition experiments, we utilized subsets of the corpus, specifically 6.25% and 25% random sampled data.
The Adam(Kingma & Ba, ) optimizer and cosine learning rate schedules are applied throughout the training process.
All the hyperparameters are detailed in Appendix C.

### 4.2 Scaling Law

#### 4.2.1 Evaluation Metrics & Fitting Methodology

We use the $R^{2}$ value and Huber loss (with the parameter $\delta=1e-3$) between the authentic valid loss and predicted valid loss on small models (190M, 505M, 1.07B) to acquire the best scaling law. Then we use the best law to train two large models (with 1.97B and 4.23B). For more information about the two evaluation methods, please refer to Appendix A.4.

We optimized the SMS Law using the L-BFGS algorithm, the same with Chinchilla and Data-Constrained Laws. For more information, please refer to Appendix A.5.

#### 4.2.2 SMS Law are the Best on the Training Set

**Table 3: Comparison of parametric fitting performance of different scaling laws.**
| Paramatic fit | $R^{2}$ Value (train) ↑ | Huber Loss (train) ↓ | $R^{2}$ Value (test) ↑ | Huber Loss (test) ↓ |
| --- | --- | --- | --- | --- |
| Chinchilla law | 0.9347 | 0.0109 | -0.0933 | 0.0080 |
| Data-Constrained law | 0.7179 | 0.0206 | 0.1524 | 0.0071 |
| Equation 11 | 0.9075 | 0.0129 | 0.3114 | 0.0073 |
| Equation 2 | 0.9759 | 0.0102 | 0.8580 | 0.0062 |
| SMS Law | 0.9780 | 0.0085 | 0.9612 | 0.0028 |

The integration of an additional term as delineated in Equation 2, alongside the introduction of a GELU regularization component in Equation 4, collectively underpins the superior performance of the SMS Law, as empirically evidenced by its training set outcomes. This is particularly notable in the context of our parametric fitting performance comparison (see Table 3), where the SMS Law outshines other scaling laws, achieving the highest $R^{2}$ value (0.9780) and the lowest Huber loss (0.0085) on the training set.

Although Equation 11 does not eclipse the Chinchilla Law in performance metrics, it nonetheless presents a significant improvement over the Data-Constrained Law’s $D^{\prime}$ by leveraging $D^{\prime\prime}$, which is indicative of a refined approach to managing the constraints posed by data repetition. This nuanced handling of data repetition, inherent to Equation 11, suggests an enhanced generalization capability in such scenarios. Therefore, we culminate it along with other modifications, manifest in the SMS Law in order to enhance model performance and generalization at the same time. In fact, it indeed provides much better results in the test set.

#### 4.2.3 Scaled-up Performance Followed SMS Law

In our SMS Law experimentation under a computational budget of $1.2\times 10^{21}$ FLOPs, we initially aim to train a 2.10B (or 1.98B) parameter model across 2.82 epochs on the whole 33.6B dataset per epoch, achieving a loss of 0.5279 (or 0.5280). Engineering constraints necessitated a slight scale-down to a 1.97 billion parameter model, which, intriguingly, showed a minimal loss increase to 0.529 around 2.5 epochs.
Contrary to the predictions of SMS Law, the Chinchilla Law suggests optimal performance for a 990M parameter model over 6.1 epochs. Pushing boundaries, we continuously train the 1.07B parameter model and observe overfitting returns beyond 3 epochs, validating the SMS Law’s advantages in this context.
Further, we train a 4.23B parameter model that underscored the SMS Law’s predictive accuracy regarding overfitting risks, affirming its value as a strategic guide in scaling up models effectively within fixed computational constraints, beneficial for efficient model scaling decisions.

In validating the SMS Law, we analyze the performance of 1.97B and 4.23B parameter models, detailed on the right-hand side of Table 3. This comparative study highlights the SMS Law’s exceptional performance, evidenced by its unparalleled $R^{2}$ values and minimal Huber Loss on testset as well.

Unlike the Chinchilla and Data-Constrained laws, the SMS Law not only showcase superior predictive accuracy but also demonstrates its efficacy in optimizing neural network scaling within computational constraints. These results affirm the SMS Law’s value in guiding scaling strategies for symbolic music, marking a significant advancement in the field.

### 4.3 Evaluation

#### 4.3.1 Efficiency of our training strategy

Figure: Figure 4: Training Loss for different model sizes and training strategy.
Refer to caption: x3.png

To demonstrate the efficiency of our training strategies, we reference the training loss curves in Figure 4. Our comparison spans four different model sizes: 190M, 505M, 1.1B, and 2B. We observed that increasing the training input length from 4096 to 8192 significantly reduces the loss, especially noticeable in the convergence phase. The comparison shows that after aligning data, our training loss slightly decreases compared to the original ABC loss, demonstrating our method’s efficiency in improving training for various model sizes.

#### 4.3.2 Repetition Metrics

##### Repetition rate

Repetition is significant in evaluating how well-structured the music is. In Table 4, the piece-level average repetition rate of each system is calculated to reveal how often the repeat sign ${:|}$ appears in a generated set. It appears that 44.3% of the generated samples from MuPT, which is quite close to the ground truth, and much higher than GPT-4. This suggests that MuPT is able to generate more music with repetition and structure.

**Table 4: Mean value of the intra-texture similarity and repetition rate of each system. ABC notation string generated by MuPT achieves higher intra-similarity than the ground truth as well as those generated by GPT-4.**
| System | Texture similarity | Repetition Det. Rate (%) |
| --- | --- | --- |
| MuPT | 0.4288 | 44.3 |
| GT | 0.3729 | 43.5 |
| MMT | 0.1767 | - |
| GPT-4 | 0.3614 | 16.9 |

##### Intra Similarity

In addition to the naive repetition rate, we also adopt the methods introduced in Wang et al. () to calculate the intra-similarity of music in each system. Specifically, a pre-trained VAE from Yang et al. () and Wang et al. () is transferred to compute the texture latent for each music piece; the intra-similarity of a music piece is defined as the average value of its texture latent similarity matrix, excluding the diagonal. Since the texture encoder is pre-trained on MIDI data, we transform ABC notations into MIDI format via the toolkit called abc2midi (^†^†https://github.com/xlvector/abcmidi) before the latent is obtained. Table 4 shows the mean value of each system’s intra-similarity under the first-bar conditioned generation. MuPT achieves the highest score among all systems. Multitrack Music Transformers (MMT) (Dong et al., ), a MIDI-based music generation model, is also compared and its generated pieces have notably lower intra similarity than MuPT and GPT-4, both of which are ABC-based systems. This result corresponds with the intuition that score-level ABC notation is more capable of generating structured music than performance-level MIDI.

#### 4.3.3 Subjective evaluation

Human assessment should be involved to further testify the objective repetition metrics above. Following Donahue et al. () and Thickstun et al. (), we conduct a subjective listening study to measure the qualitative performance of MuPT against the ground truth (GT) and baselines consisting of GPT-4, MMT and random note sequences (Random). Listeners are asked to identify which of two musical excerpts from different sources is more ”musical” during the test process. They are also instructed to focus on two aspects of musicality: how consistently the music sounds throughout (e.g., in terms of its melodic contours, rhythmic patterns, and chord progression); and how likely it is that the development of the music follows a clear structure (e.g., verse-chorus division, repetitions). This process is similar with that in Yuan et al. () and its details are shown in the Appendix D.

**Table 5: Human evaluation of paired completions of musical excerpts generated by different sources given the first bar as the condition. The left is the matrix based on the AB test. Each row indicates the % of times listeners preferred instrumentals from that system compared to those from each system individually (N = 150). Ground truth is denoted by GT. i.e.77 means that listeners preferred MuPT over GPT-4 in 77% of cases. The right is the absolute win numbers and the corresponding p-value of each pair. P-values are reported by a Wilcoxon signed rank test.**
| Model A | Model B | Wins (A/B) | p-value |
| --- | --- | --- | --- |
| Human works | MuPT | 81/69 | 0.4237 |
|  | MMT | 109/41 | $4.2249\times 10^{-6}$ |
|  | GPT-4 | 119/31 | $6.6315\times 10^{-9}$ |
|  | Random | 138/12 | $4.4648\times 10^{-17}$ |
| MuPT | MMT | 110/40 | $4.2249\times 10^{-6}$ |
|  | GPT-4 | 115/35 | $6.6641\times 10^{-8}$ |
|  | Random | 131/19 | $1.3618\times 10^{-13}$ |
| MMT | GPT-4 | 95/55 | 0.0093 |
|  | Random | 103/47 | 0.0001 |
| GPT-4 | Random | 106/44 | $2.6691\times 10^{-5}$ |

Results for all systems are shown in Table 5. Comparing our MuPT to GPT-4, listeners preferred music from our system in 79% of cases. A Wilcoxon signed-rank test of these pairwise judgments indicates that listeners preferred music from MuPT significantly more often than MMT and GPT-4 ($p=4.2249\times 10^{-6}$ and $p=6.6641\times 10^{-8}$, respectively).

## 5 Conclusion

In this paper, we introduce the MuPT series of pre-trained models for symbolic music generation, which set the standard of training open-source symbolic music foundation models.
With 190M, 505M, 1.07B, 1.97B, and 4.23B parameters, these models have been pre-trained on the largest possible amount of ABC Notation data, including 33.6 Billion high-quality diverse symbolic music tokens.
Additionally, we dive deep into the scaling law exploration and propose SMS Law, a specialist in guiding future scaling of symbolic music foundation models.
Our results demonstrate that the MuPT series is competitive with mediocre human composers and guarantees state-of-the-art performance on symbolic music generation.
Moreover, MuPT introduces SMT-ABC, reordering the multiple-track original ABC notation format to assist pre-training of MuPT.
We believe that the open access of intermediate checkpoints of MuPT, SMS Law, and MuPT series will foster collaboration and innovation within the open-source computational music community, and open the door to the next-generation symbolic music foundation models.

## Appendix A Scaling Law

### A.1 Scaling Law Baseline

#### A.1.1 Abstracting Loss Metrics through the Chinchilla Law

In this part, we focus on the relationship of loss metrics to various resource budgets in deep learning. It is first put forward by the Chinchilla Law as illustrated in Equation 5. This law posits that both training and evaluation losses can be abstracted as a function of model capacity $N$ and training data size $D$, thus offering an insight to estimate the best combination of resources to be assigned to training.

$$ $L(N,D)=\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}+E$ (5) $$

Here, $L(N,D)$ denotes the loss metric during training or evaluation, which is assumed to exhibit a power-law dependency on $N$ and $D$. The parameters $A$, $B$, $E$, $\alpha$, and $\beta$ are determined by empirical fitting.

#### A.1.2 Data-Constrained Law

Data-Constrained Law: Scaling under Data Limitations.
Complementing the Chinchilla Law, the Data-Constrained Law shows the scaling dynamics of LLMs when facing the data scarcity problem. Here, we strictly refer to the derivation method of Muennighoff et al. ().
The goal of discovering Data-Constrained Scaling Law is to generalize the expression to multiple epochs where tokens are repeated.

Data-constrained law is defined as:

$$ $L\left(N,D,U_{D}\right)=\frac{A}{N^{\prime\alpha}}+\frac{B}{D^{\prime\beta}}+E$ (6) $$

where

$$ $\begin{split}&N^{\prime}=U_{N}+U_{N}R_{N}^{\star}\left(1-\exp\left(\frac{-R_{N }}{R_{N}^{\star}}\right)\right)\\ &D^{\prime}=U_{D}+U_{D}R_{D}^{\star}\left(1-\exp\left(\frac{-R_{D}}{R_{D}^{ \star}}\right)\right)\\ \end{split}$ (7) $$

To get a better understanding of the equation, the definitions of each of the above parameters are as follows:
Like Chinchilla Law, $N$ is defined as the number of model parameters, and $D$ is defined as the training tokens.

$U_{D}$ is defined as the number of unique tokens used. For data-constrained law, $U_{D}$ is computed as min{$D$,$D_{C}$} given a budget of unique data $D_{c}$.

$U_{N}$ is defined as the number of “unique” parameters that provide an optimal fit for $U_{D}$. According to the method mentioned in Muennighoff et al. (), given the following learned variables, $\{A,\alpha,B,\beta\,E\}$, the optimal allocation of compute(C) to $N$ and $D$ as follows:

$$ $\begin{split}&N_{\text{opt}}(C)=G\left(\frac{C}{6}\right)^{a}\\ &D_{\text{opt}}(C)=G^{-1}\left(\frac{C}{6}\right)^{b}\\ &G=\left(\frac{\alpha A}{\beta B}\right)^{\frac{1}{\alpha+\beta}}\\ &a=\frac{\beta}{\alpha+\beta}\\ &b=\frac{\alpha}{\alpha+\beta}\end{split}$ (8) $$

Thus, $U_{N}$ is equal to $\min\{N_{\text{opt}},N\}$.

$R_{D}$ is defined as the number of times the data is repeated. When training for a single epoch, $R_{D}=0$.

$R_{N}$ is the number that the ‘unique’ parameters are repeated where $R_{N}=\max\{\left(\frac{N}{U_{N}}\right)-1,0\}$.

$D^{\prime}$ is defined as the ”effective data size”: the number of unique data needed to get the same value as repeating $U$ unique tokens for $R_{D}$ repeats.The derivation process is as followed:

From a conceptual standpoint, the redundancy of data samples diminishes their incremental value in enhancing the model’s knowledge base, given the model’s prior exposure to said information. This principle underlies the hypothesis that each successive repetition of a sample contributes marginally less to the learning process, as the model has partially assimilated the information contained within the sample through prior iterations. To describe the process of training information loss, we have

$$ $D^{\prime}=U+U\sum_{k=1}^{R_{D}}(1-\delta)^{k}\ =U+(1-\delta)U\frac{(1-(1- \delta))^{R_{D}}}{\delta}$ (9) $$

where $\delta$ is defined as the ‘forgetting rate’. Each time a series of tokens is trained on a model, the model learns a $1-\delta$ fraction information from the optimization process. Assuming that the number of epochs beyond which repeating does not help, the right-hand side goes to to $\frac{(1-\delta)U}{\delta}$, since $\lim_{R_{D}\to\infty}(1-(1-\delta)^{R_{D}})=1$. We define $R_{D}^{\star}$ is defined as$\frac{1-\delta}{\delta}$, which is a learned constant.
According to Taylor expansion, if $\delta$ is small, we have:

$$ $\mathrm{e}^{\frac{-1}{R_{D}^{\star}}}\approx\ (1-\delta)$ (10) $$

Now inserting$\frac{(1-\delta)}{\delta}=R^{\star}_{D}$ and $(1-\delta)^{R_{D}}=e^{(\frac{-1}{R_{D}^{\star}})^{R_{D}}}$ into Equation9, we get our final equation representing the effective data.

As the frequency of encountering repeated tokens diminishes, the benefit gained from processing them also decreases. Hence, the derivation of the $N^{\prime}$ is similar to $D^{\prime}$. In this context, there’s no need to elaborate further. It should be pointed out that $R_{N}^{\star}$ is a learned parameter.

### A.2 Continuous Adaptation of the Data-Constrained Law.

To enhance the predictive accuracy of the Data-Constrained law
(Muennighoff et al., )
for continuous domains, we extend the original discrete formulation 11 to accommodate continuous variables, allowing for a more nuanced understanding of data constraints in varied contexts.
For an in-depth discussion on the derivation and implications of this continuous formulation, please refer to Appendix A.2.

$$ $L(N,D,U_{D})=\frac{A}{N^{\alpha}}+\frac{B}{{D^{\prime\prime}}^{\beta}}+E$ (11) $$

where $k$ is a new parameter to be fit, and
$D^{\prime\prime}$, the adjusted data size, is given by:

$$ $D^{\prime\prime}=\frac{1-k^{D/U_{D}}}{1-k}U_{D}.$ (12) $$

The definition of $D^{\prime}$ in Equation 9 is defined from a discrete version and can not be extended to the case when D is less than $U_{D}$. So we reform the Equation 9 to

$$ $\begin{split}D^{\prime}&=\frac{1-(1-\delta)^{\frac{D}{U_{D}}}}{\delta}\cdot U_ {D}\\ &=\frac{1-k_{d}^{\frac{D}{U_{D}}}}{1-k_{d}}\cdot U_{D}\end{split}$ (13) $$

where $k_{d}:=1-\delta$. This equation is equivalent to equation 10 when $D$ is a positive integer times $U_{D}$.

We implemented a formula symmetric to $N^{\prime}$ with $U_{N}$ and $k_{N}$. But the calculation results of $k_{N}\approx 0.999$. To make the formula simple, we use the original $N$ instead of $N^{\prime}$ in the following formula.

### A.3 Motivation of SMS Law

#### A.3.1 Motivation of Adding Power of “ N ⁢ D 𝑁 𝐷 ND italic_N italic_D ” Term

Figure: Figure 5: The loss curve, Chinchilla prediction, and Equation11 on 2.1B, 8.4B and 33.6B training data.
Refer to caption: extracted/5979792/figures/nd2.png

In our submission, we present an in-depth analysis of the model’s loss dynamics as illustrated in Figure 5, which juxtaposes the empirical loss trajectory (depicted through a blue line) against the theoretical predictions derived from the Chinchilla Law (illustrated by a yellow line) and further contextualized by Equation 11. This comparative study spans three distinct datasets—2.1B, 8.4B, and 33.6B data points—across models of varying capacities: 190M, 505M, and 1.07B parameters, respectively, arranged in a matrix of subfigures with datasets delineated by rows and model capacities by columns.

Observations across all data volumes reveal a nuanced interaction between model and data sizes. Specifically, for smaller datasets and model sizes (190M parameters), predictions consistently underestimate actual loss values, whereas for smaller datasets paired with larger models (1.07B parameters), predictions tend to overestimate. This discrepancy underscores a critical insight: loss reduction accelerates with increasing model size, suggesting a modified loss function, $\frac{A+\epsilon}{N^{\alpha}}$ over the simpler $\frac{A}{N^{\alpha}}$

Crucially, the term $\epsilon$ emerges as a function of a single variable $N$, ensuring variability in $\frac{\epsilon}{N^{\alpha}}$ across each unique model configuration shifting upwards or downwards without changing the shape. The ideal adjustment implies that $\epsilon$ approaches zero for large datasets, yet remains significant for smaller ones, highlighting its dependency on data volume $D$.

In addressing potential overfitting, our strategy focuses on minimizing parameter growth in line with Equation 11. A straightforward approach involves augmenting the loss $L$ into a polynomial encompassing $\frac{A}{N^{\alpha}}$ and $\frac{B}{D^{\beta}}$, with Equation 2 introducing an additional term, $\frac{d}{N^{\alpha}\cdot D^{\beta}}$, to the existing framework. This refinement, while ostensibly simple, has been shown to yield robust and promising outcomes, exemplifying the efficacy of our proposed modifications in enhancing model performance within the context of scaling laws.

#### A.3.2 Motivation of Linear Regression Term for Overfitted Residual

Figure: Figure 6: The loss curve, Chinchilla prediction, and Equation 2 (green lines) on 2.1B training data.
Refer to caption: extracted/5979792/figures/overfit.png

Figure 6 offers a detailed exposition on the fidelity of Equation 2 in capturing the loss trajectory across training sets of varied model capacities (190M, 505M, and 1.07B parameters). It is evident from the analysis that the equation adeptly mirrors the empirical loss curve across a broad spectrum of configurations, with the exception of scenarios characterized by concurrently large model sizes and token counts. A notable oversight in the literature is the scant consideration of loss dynamics beyond early stopping points, a consideration of paramount importance in music domain due to the inherent constraints on training data.

In addressing the challenges posed by modelling loss post-early stopping, our investigation delineates two distinct methodologies. The first approach involves the integration of a regularization term within $D^{\prime\prime}$, aimed at reducing its magnitude beyond the early stopping threshold. Despite its conceptual appeal, this method falls short of providing an adequate fit to the observed data. Alternatively, we explore the augmentation of the loss function $L$ with an additional term, engineered to be negligible when both $D$ and $N$ are minimal, yet incrementally assertive in influencing the loss trajectory after early stopping points. This latter strategy not only aligns more closely with empirical observations but also introduces a nuanced mechanism to accommodate the unique requirements of training in the music domain, thereby extending the utility and applicability of scaling laws within this context.

Figure: Figure 7: Residule between authentical valid loss and Equation 2 prediction (blue lines), and the linear regression results (yellow lines).
Refer to caption: extracted/5979792/figures/residule.png

As delineated in Figure 7, the analysis of residuals post the 40 billion token threshold unveils a discernible onset of overfitting, which intriguingly appears to correlate with the model size, data capacity, and the count of unique tokens processed within a single epoch. This overfitting is further characterized by a linear dependency of loss on the total number of processed tokens, coupled with a quasi-linear transition of early stopping points observed across different model capacities (as organized in rows) and magnified across columns.

The progression of model capacities—doubling across rows and quadrupling across columns—illuminates a systematic pattern, suggesting that the early stopping points and consequently, the predicted loss, might be effectively modeled through a linear regression involving dataset size $D$, the logarithm of model capacity $\log(N)$, and and the logarithm of unique tokens per epoch $\log(U_{D})$. This observation culminates in the proposition of a regularization term formulated as $k_{d}\cdot D+k_{n}\cdot\log(N)-k_{u}\cdot\log(U_{D})-k_{in}$, aimed at encapsulating and mitigating the observed overfitting dynamics.

**Table 6: Ablition study on the activation function.**
| Activation Function | $R^{2}$ (test)↑ | Huber Loss (test)↓ |
| --- | --- | --- |
| ReLU | 0.9786 | 0.0095 |
| LeakyReLU | 0.9786 | 0.0095 |
| GELU | 0.9780 | 0.0085 |
| Tanh | 0.9786 | 0.0094 |
| SELU | 0.9779 | 0.010 |
| Sigmoid | 0.6030 | 0.0700 |

In addressing the intricacies of regularization within the context of early model training, especially when considering models of smaller scale (where $U_{D}$ and $D$ are minimal while $N$ is comparatively large), it becomes imperative to ensure that the regularization term does not adopt a substantially negative value. This stipulation aims to prevent undue penalization at the onset of training, thereby necessitating the incorporation of an activation function that tempers the regularization term’s behavior. The Gaussian Error Linear Unit (GELU) function emerges as an apt choice in this scenario. GELU approximates the Rectified Linear Unit (ReLU) function for positive inputs, while also permitting slight negative values with minimal absolute magnitude, thus offering a balanced solution.

Empirical evidence, as detailed in our analysis, underscores the efficacy of applying the GELU function to the regularization term, notably achieving the lowest training loss alongside the second-highest $R^{2}$ value among the tested models. This finding is particularly salient given the broader magnitude of loss variations relative to $R^{2}$ values, thereby accentuating the GELU function’s suitability for our regularization term. Consequently, the finalized model, incorporating the GELU-modulated regularization term, is depicted through a yellow line in Figure 7.
This strategic application of the GELU function not only mitigates the potential for excessive early training penalization but also optimizes the regularization term to enhance model performance effectively.

This approach not only elucidates the linear interdependencies among critical factors influencing model performance but also presents a nuanced regularization strategy designed to enhance model generalizability. Through the integration of this regularization term, we aim to establish a more robust and theoretically informed framework for predicting and managing loss trajectories in large-scale training regimes.

### A.4 Evaluation Metrics

The R-squared value, also known as the ”Coefficient of Determination,” is a statistical measure used to evaluate the goodness-of-fit of a regression model. It is defined as:

$$ $R^{2}=1-\frac{SS_{\text{res}}}{SS_{\text{tot}}}$ (14) $$

Where $SS_{res}$ represents the Sum of Squares of Residuals, indicating the total sum of squared differences between the predicted values of the model and the actual observed values, $SS_{tot}$ represents the Total Sum of Squares, indicating the total sum of squared differences between the observed values of the dependent variable and their mean value.

The Huber loss is a type of loss function commonly employed in robust regression models. Unlike the squared error loss, which is sensitive to outliers in the data, the Huber loss is designed to be less affected by outliers. It achieves this by combining the characteristics of both the squared error loss and the absolute error loss. It is defined piecewise by:

$$ $Huber_{\delta}(y,f(x))=\begin{cases}\frac{1}{2}(y-f(x))^{2},&\text{if }|y-f(x) |\leq\delta\\ \delta(|y-f(x)|-\frac{1}{2}\delta),&\text{otherwise}\end{cases}$ (15) $$

For small residuals, it behaves like the squared error loss, whereas for large residuals, it behaves like the absolute error loss. This allows the Huber loss to provide a balance between the two, resulting in a more robust estimation procedure.

### A.5 Parameters Fitting Approach

In our study, we adopt a methodology analogous to the Chinchilla Law and the Data-Constrained Law, employing the L-BFGS algorithm—a limited-memory quasi-Newton method—for the optimization of the Huber Loss. This loss function is applied between the logarithm of the predicted loss and the logarithm of the observed (authentic) loss across multiple runs. The objective is to identify the optimal parameters (best_para) that minimize this Huber Loss, formalized as follows:

$$ $\begin{split}best_para=\min\sum_{runi}Huber_{\delta}&\left\{\log\left[\frac{d }{N^{\alpha}\cdot D^{\prime\prime\beta}}+\frac{A}{N^{\alpha}}+\frac{B}{D^{ \prime\prime\beta}}+E\right]_{i},\log(L_{i})\right\}\\ =\min\sum_{runi}Huber_{\delta}&\left\{LSE\left[\log\left(\frac{d}{N^{\alpha} \cdot D^{\prime\prime\beta}}\right),\log\left(\frac{A}{N^{\alpha}}\right),\log \left(\frac{B}{D^{\prime\prime\beta}}\right),\log(E)\right]_{i},\log(L_{i}) \right\}\\ =\min\sum_{runi}Huber_{\delta}&\left\{LSE\left[\begin{split}&\log(d)-\alpha \log(N)-\beta\log(D^{\prime\prime})\\ &\log(A)-\alpha\log(N)\\ &\log(B)-\beta\log(D^{\prime\prime})\\ &\log(E)\end{split}\right],\log(L_{i})\right\}\\ \end{split}$ (16) $$

where $LSE$ refers to the log-sum-exp a numerically stable method to compute the logarithm of a sum of exponentials of inputs. The Huber Loss parameter, $\delta$ is set to $1e-3$, reflecting a stringent criterion for switching between squared loss and absolute loss to ensure robustness in optimization. Additionally, the L-BFGS algorithm’s learning rate is configured at $1e-1$,
with an update history size of 10 to balance between computational efficiency and the capacity to capture relevant optimization trends.

### A.6 Results of Proposed Methods with Early Stops

**Table 7: Comparison parametric fitting performance of different Scaling Laws on the curve before early stop points.**
| Paramatic fit | $R^{2}$ Value (train) ↑ | Huber Loss (train) ↓ | $R^{2}$ Value (test) ↑ | Huber Loss (test) ↓ |
| --- | --- | --- | --- | --- |
| Chinchilla law | 0.9443 | 0.0073 | -0.0004 | 0.0029 |
| Data-Constrained law | 0.7216 | 0.0189 | 0.1005 | 0.0050 |
| Equation 11 | 0.8356 | 0.0151 | 0.5829 | 0.0045 |
| Equation 2 | 0.9843 | 0.0072 | 0.9866 | 0.00088 |
| SMS Law | 0.9851 | 0.0055 | 0.9864 | 0.00091 |

From the table, we can see that most of the experimental results increase after we delete the curve after the early stop points. Adding the linear regression still contributes to the performance increase on the training set but provides worse results on test set compared to Equation 2.

## Appendix B A Short Lecture Note of L-BFGS Algorithm

BFGS (Limited-memory Broyden–Fletcher–Goldfarb–Shanno) is a variant of the BFGS method, a quasi-Newton optimization algorithm used to solve unconstrained nonlinear optimization problems. It is particularly suitable for handling large-scale optimization problems by limiting the size of the stored matrices, thus reducing storage and computational costs.

The core idea of the L-BFGS algorithm is to approximate the inverse of the Hessian matrix of the objective function using historical records of function values and gradients. In contrast to traditional Newton’s method that requires storing and updating the complete Hessian matrix, L-BFGS method only needs to store and update some historical information, making it more efficient in terms of storage and computation. It iteratively constructs an approximate inverse Hessian matrix to update parameters and continuously optimize the objective function until reaching a local optimum or satisfying convergence criteria.

According to Newton-Raphson method:

$$ $\begin{split}&f:\mathbb{R}^{n}\to\mathbb{R}\\ &f(x_{t}+d)=f(x_{t})+\nabla f(x_{t})^{T}d+\frac{1}{2}d^{T}\nabla^{2}f(x_{t})d+ o(\|d\|^{2})\\ \end{split}$ (17) $$

$$ $h(d):=f(x_{t}+d)=f(x_{t})+\nabla f(x_{t})^{T}d+\frac{1}{2}d^{T}\nabla^{2}f(x_{ t})d\\$ (18) $$

$$ $\begin{split}&\hat{d}:=\arg\min_{d}h(d)\\ &\nabla h(\hat{d})=\nabla f(x_{t})+\nabla f^{2}(x_{t})\hat{d}=0\\ \end{split}$ (19) $$

$$ $x_{t+1}=x_{t}+\hat{d}=x_{t}-\nabla^{2}f(x_{t})^{-1}\nabla f(x_{t})$ (20) $$

According to BFGS:

$$ $B_{k+1}=B_{k}-\frac{B_{k}s_{k}s_{k}^{T}B_{k}}{s^{T}_{k}B_{k}s_{k}}+\frac{y_{k} y^{T}_{k}}{y_{k}^{T}s_{k}}$ (21) $$

In the BFGS algorithm, storing the approximate Hessian matrix at each iteration can be costly in terms of memory, especially in high-dimensional data scenarios. However, in practical computation, what we primarily need is the search direction. To address this issue, the L-BFGS algorithm was introduced as an improvement over the BFGS algorithm.

In L-BFGS, instead of storing the full Hessian matrix, only the most recent iterations’ information is retained, significantly reducing the memory footprint.

let $\rho_{k}=\frac{1}{y^{T}_{k}s_{k}}$, $V_{k}=I-\frac{y_{k}s^{T}_{k}}{y_{k}^{T}s_{k}}$, then $H_{k+1}$ can be represented as:

$$ $H_{k+1}=V_{k}^{T}H_{k}V_{k}+\rho_{k}s_{k}s_{k}^{T}$ (22) $$

Note that $H_{0}=I$.

$$ $\begin{split}&H_{1}=V_{0}^{T}H_{0}V_{0}+\rho_{0}s_{0}s_{0}^{T}\\ &H_{2}=V_{1}^{T}H_{1}V_{1}+\rho_{1}s_{1}s_{1}^{T}\\ &=V_{1}^{T}(V_{0}^{T}H_{0}V_{0}+\rho_{0}s_{0}s_{0}^{T})V_{1}+\rho_{1}s_{1}s_{1 }^{T}\\ &=V_{1}V_{0}^{T}H_{0}V_{0}V_{1}+V_{1}^{T}\rho_{0}s_{0}s_{0}^{T}V_{1}+\rho_{1}s _{1}s_{1}^{T}\\ &\dots\\ &H_{k+1}=(V_{k}^{T}V_{k-1}^{T}\cdots V_{1}^{T}V_{0}^{T})H_{0}(V_{0}V_{1}\cdots V _{k-1}V_{k})\\ &+(V_{k}^{T}V_{k-1}^{T}\cdots V_{1}^{T})\rho_{1}s_{1}s_{1}^{T}(V_{1}\cdots V_{ k-1}V_{k})\\ &+\cdots\\ &+V_{k}^{T}\rho_{k-1}s_{k-1}s_{k-1}^{T}V_{k}\\ &+\rho_{k}s_{k}s_{k}^{T}\\ \end{split}$ (23) $$

If only the first m steps are retained:

$$ $\begin{split}&H_{k+1}=(V_{k}^{T}V_{k-1}^{T}\dots V_{k-m}^{T})H_{0}(V_{k-m} \dots V_{k-1}V_{k})\\ &+(V_{k}^{T}V_{k-1}^{T}\dots V_{k-m}^{T})\rho_{1}s_{1}s_{1}^{T}(V_{k-m}\dots V _{k-1}V_{k})\\ &+\dots\\ &+V_{k}^{T}\rho_{k-1}s_{k-1}s_{k-1}^{T}V_{k}\\ &+\rho_{k}s_{k}s_{k}^{T}\\ \end{split}$ (24) $$

Then only $s_{k}$ and $y_{k}$ is necessary to be remained.

## Appendix C Training Details

All the models are trained using AdamKingma & Ba (), with $\beta_{1}=0.9,\beta_{2}=0.95,eps=10^{-8}.$ We use a cosine learning rate schedule, decay the final learning rate from $3^{-5}$ to $3^{-6}$, with warmup ratio of 0.1. We apply a weight decay of 0.1 and gradient clipping of 1.0. Table 8 shows other training details of each model.

**Table 8: Training Details for different ABC format and model settings.**
|  | Parameters | Context Length | Trained Tokens | Training Days | Num of GPUs |
| --- | --- | --- | --- | --- | --- |
| Original ABC | 190M | 4096 | 119B | 8.4 | 2 |
| 505M | 4096 | 97B | 8.4 | 4 |  |
| 1.07B | 4096 | 49B | 8.3 | 4 |  |
| 1.97B | 4096 | 56B | 8.4 | 8 |  |
| 190M | 8192 | 346B | 6.9 | 8 |  |
| 505M | 8192 | 322B | 4.1 | 32 |  |
| 1.07B | 8192 | 223B | 5.4 | 32 |  |
| 1.97B | 8192 | 196B | 8.1 | 32 |  |
| SMT-ABC | 190M | 8192 | 276B | 5.5 | 8 |
| 505M | 8192 | 212B | 2.7 | 32 |  |
| 1.07B | 8192 | 181B | 4.4 | 32 |  |
| 1.97B | 8192 | 272B | 11.3 | 32 |  |
|  | 4.23B | 8192 | 262B | 10.7 | 64 |

## Appendix D Human Assessment

We use webMUSHRA toolkit (Schoeffler et al., ) to conduct a web-based subjective listening AB-test. During the listening test, we ask the participants to choose the better one between a pair of music excerpts generated from two randomly selected different systems from GT, MuPT, GPT-4, MMT and Random by considering the ”Musicality” which indicates the overall perceptive quality of the music. Participants are encouraged to make a choice by refering to the guidelines below:

- $\bullet$
How consistent the music sounds as a whole (e.g., in terms of its melodic
contours, rhythmic patterns, and chord progression).
- $\bullet$
How likely the development of the music follows a clear structure (e.g.
verse-chorus division, repetitions).
- $\bullet$
If you cannot understand the two guidelines above, just choose the one from A and
B you prefer.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Aside from the questions in the weaknesses section:
- Is there any study to check if the generations are copied from the training data?
soundness: 3
strengths: The study is conducted in a systematic fashion and covers a lot of the relevant topics that LLM-related literature should talk about, such as the relationship between dataset size, model parameters, etc with the quality of generated data.
The quality of the generated music shared in the supplementary material is decent.
summary: The paper presents MuPT a pretrained music transformer for symbolic music generation. The paper is a study of the techniques from LLM literature applied to symbolic music data. The authors discuss a novel data representation for symbolic data: SMT-ABC to take issues related to misaligned multi-track data. The authors also present a scaling law appropriate for symbolic music where the dataset size is typically orders of magnitude smaller than text data that was used to establish the Chinchilla laws. The authors conduct subjective and objective evaluations of their model and compare with other similar models of symbolic music. They find that their model is heavily preferred compared to other models in human studies.
weaknesses: The authors do not share any samples from the baseline models in supplementary materials which would have helped support the results in the human study.
The repetition metrics seem to not be clearly motivated. Why would the average repetition rate be a meaningful metric. Isn't the more important metric the position of the repeats?
Some writing issues. E.g. missing reference in line 353, missing word in line 505.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: Considering the limitations of ABC notation—such as its inability to represent control information and note dynamics, and its less common use within the music community—these weaknesses could significantly impact the model’s potential for generating symbolic music for potential users (composers, musicians, etc). How do you foresee the future roles and applications of ABC and MIDI representations in music generation?

How many songs are included in the dataset, and what detailed information is provided for each song? How do the authors address intellectual property concerns?
soundness: 2
strengths: 1. Studying a foundation model for symbolic music is both interesting and meaningful work.

2. The team put significant effort into analyzing large datasets and training the foundation model.

3. The findings on scaling laws are interesting.
summary: This paper explores using large models for symbolic music pre-training, proposing ABC Notation as a more compatible format than the commonly used MIDI. To maintain track coherence in multi-track compositions, the authors introduce SMT-ABC Notation. Key contributions include models that handle up to 8192 tokens, covering 90% of their symbolic music dataset, and an examination of the Symbolic Music Scaling Law (SMS Law) on model performance. The experiments show better performance than GPT4.
weaknesses: 1. The research contribution is limited. To enhance the novelty of the approach, consider addressing research questions such as: What factors make ABC better than MIDI, and how does the uniqueness of ABC benefit large models in the feature space? Under what circumstances does the model perform better with ABC notation?

2. The comparison between ABC and MIDI is incomplete. The assumption that ABC is better than MIDI requires a detailed and careful analysis. For example, experiments exploring the following would help clarify the relative strengths and weaknesses of each format for large model training: under the same data size and model architecture, assess whether ABC consistently outperforms MIDI in terms of model efficiency and output quality. Note that MIDI can be represented in various formats, such as piano roll, matrix formats like Compound Word, or sequences like REMI. For Figure 1, further details would improve the visualization, such as comparing ABC directly with MIDI instead of only with the piano roll, as the piano roll omits much information found in MIDI. Indicate to readers what information is missing from both MIDI and ABC, what MIDI can represent (e.g., velocity, control information) that ABC cannot, and vice versa. Additionally, clarify the correspondence between ABC and MIDI, such as how "K: Bb" in ABC notation relates to MIDI, or what "|F2 z G|" represents in ABC and its MIDI equivalent in the figure.

3. The experimental design has flaws. When comparing music-related performance, the proposed model should be evaluated against symbolic music generation models rather than language models. Relevant symbolic music generation baselines include Music Transformer [1], Compound Word Transformer [2], and RWKV [3] on the same large symbolic music dataset.


4. typos such as Line 353.

---

confidence: 3
contribution: 2
initial_rating: 8
presentation: 3
questions: 1. The paper's experimental design is flawed due to insufficient comparison with other models to validate SMT-ABC's performance improvement. It's unclear if models like GPT-4 and MMT were trained on the SMT-ABC dataset, questioning the true effectiveness of SMT-ABC's innovation.

2. The Scaling Law experiments, while detailed, do not clearly show their relevance to music domain issues, suggesting they may not be significantly pertinent to the paper's objectives.

3. (1) The training data lacks metadata, potentially limiting model generalization. (2) The model may struggle with complex musical structures like polyphony. (3) There's a lack of analysis on model architectures' strengths and weaknesses in symbolic music. (4) Subjective evaluations might be biased, necessitating more diverse assessments. (5) Repeated data may improve performance but could also cause overfitting; more experimental evidence is needed to support claims on overfitting prevention.
soundness: 3
strengths: The introduction of SMT-ABC Notation represents a novel approach to enhancing the coherence of multi-track symbolic music generation. The use of ABC notation as a foundation for LLMs in music generation, instead of the more commonly used MIDI, adds an original perspective to the field. The concept of the Symbolic Music Scaling Law (SMS Law) is also a significant contribution, offering new insights into the training dynamics of symbolic music models.
The paper is well-structured, with clear explanations of the SMT-ABC Notation and the SMS Law. 
The open-sourcing of the models and intermediate checkpoints is a valuable contribution, enabling further exploration and innovation in symbolic music modeling by the broader community.
summary: The paper "MuPT: A Generative Symbolic Music Pre-Trained Transformer" explores the use of Large Language Models (LLMs) for symbolic music generation, with a focus on leveraging ABC notation over the traditional MIDI format. The authors propose the Synchronized Multi-Track ABC Notation (SMT-ABC Notation) to improve coherence across musical tracks. They present models capable of handling up to 8192 tokens and introduce the Symbolic Music Scaling Law (SMS Law) to optimize the training of these models. Their MuPT model outperforms GPT-4 and ChatMusician in both objective metrics and subjective evaluations of music generation, and they have open-sourced their models to encourage community-led research.
weaknesses: 1. The experimental design is questionable. SMT-ABC is the only methodological innovation in this paper, yet there is insufficient comparative experimentation to explore how SMT-ABC actually improves performance. Models like GPT-4 are not specifically designed for symbolic music generation, and it's predictable that the proposed model would perform better without even conducting experiments. Were the other models in the comparative experiments, such as GPT-4 and MMT, trained or fine-tuned on the SMT-ABC dataset? If not, it may not demonstrate the true effectiveness of the algorithmic innovation of SMT-ABC.

2. The Scaling Law experiments are detailed, but they primarily validate some emergent performance issues that have been verified in other domains. It does not clearly indicate whether the issues in the music domain can be resolved through the Scaling Law, which suggests that the detailed Scaling Law experiments may not be significantly relevant to the problems this paper aims to address.

3. (1) The training data lacks specific metadata, such as detailed categorization and style of music, which may limit the model's generalization ability in specific styles or types of music. (2) The paper requires adjustments for the alignment of different tracks, and the model may still have certain limitations in generating more complex musical structures, such as polyphonic music. (3) There is no in-depth analysis of the detailed differences between different model architectures, their respective strengths and weaknesses in handling symbolic music, which may make it difficult for readers to understand the specific improvements of MuPT over other models. (4) Subjective factors such as the musical background and preferences of listeners may affect the results of subjective evaluations, and more diverse assessments might provide more comprehensive conclusions. (5) Although experiments show that repeated data can improve model performance, it may lead to overfitting in some cases. Although the paper discusses early stopping and overfitting, more experimental support is needed for the effectiveness of these explanations.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What is the model scale of SMT-ABC and Original-ABC in Figure 8 in Appendix B.2?
2. Since GPT-4 is not a model for symbolic music generation, what is the motivation to compare with GPT-4?
soundness: 3
strengths: 1. This paper introduces SMT-ABC, a novel representation that improves coherence across multi-track music generation.
2. The investigation into SMS Law offers valuable insights into the relationship between data repetition and model performance, providing practical guidelines for optimizing large-scale music generation models.
summary: The paper introduces a Synchronized Multi-Track ABC to improve coherence across tracks in music generation, addressing the limitations of traditional MIDI formats. Combining with the LLM,  MuPT outperforms other models in terms of repetition rate and structure in multi-track music generation. The paper also investigates the Symbolic Music Scaling Law, showing improvements in performance as model size and training data scale up. However, the novelty of the model is limited, and the effect of the proposed SMT-ABC representation seems not very significant.
weaknesses: 1. There are some mistakes like missing reference targets (e.g. in Section 5.1 ‘please refer to ??’).
2. Although the SMT-ABC representation is interesting by adding a special token ‘<|>’ and changing the sequence of the original ABC notations, the novelty of this paper seems limited, for the model architecture is mostly based on existing works.
3. Changing the sequence of the original ABC notation from bar-sequence to track-sequence may preserve coherence across multiple musical tracks. However, what about the coherence in a track? The contents inside a bar seem to be adjusted to the very back of the input sequence because SMT ABC aligns the tracks first. Besides, from Table 5, the improvement of SMT-ABC seems very slight compared with the original ABC.

Meta-review: The paper presents a modification to ABC notation that interleaves different tracks in time based on a bar alignment. This representation was used to train an LLM model. Results were compared with other multi-track models and human evaluation. Scaling analysis of loss as a function of the number of tokens was conducted, suggesting an optimal model parameter size.

Decision: Accept (Poster)