MuPT: A Generative Symbolic Music Pretrained Transformer
Authors: Xingwei Qu
yuelin bai
Yinghao Ma
Ziya Zhou
Ka Man Lo
Jiaheng Liu
Ruibin Yuan
Lejun Min
Xueling Liu
Tianyu Zhang
Xeron Du
Shuyue Guo
Yiming Liang
Yizhi LI
Shangda Wu
Junting Zhou
Tianyu Zheng
Ziyang Ma
Fengze Han
Wei Xue
Gus Xia
Emmanouil Benetos
Xiang Yue
Chenghua Lin
Xu Tan
Wenhao Huang
Jie Fu
Ge Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Pretrained Transformer
SMT-ABC Notation
SMS Law
Primary area: foundation or frontier models, including LLMs

Abstract: In this paper, we explore the application of Large Language Models (LLMs) to the pre-training of music. While the prevalent use of MIDI in music modeling is well-established, our findings suggest that LLMs are inherently more compatible with ABC Notation, which aligns more closely with their design and strengths, thereby enhancing the model's performance in musical composition.
To address the challenges associated with misaligned measures from different tracks during generation, we propose the development of a $\underline{S}$ynchronized $\underline{M}$ulti-$\underline{T}$rack ABC Notation ($\textbf{SMT-ABC Notation}$), which aims to preserve coherence across multiple musical tracks. 
Our contributions include a series of models capable of handling up to 8192 tokens, covering 90\% of the symbolic music data in our training set. Furthermore, we explore the implications of the $\underline{S}$ymbolic $\underline{M}$usic $\underline{S}$caling Law ($\textbf{SMS Law}$) on model performance. The results indicate a promising research direction in music generation, offering extensive resources for further research through our open-source contributions.

Published as a conference paper at ICLR 2025
MUPT: A GENERATIVE SYMBOLIC MUSIC PRE- TRAINED TRANSFORMER
Xingwei Qu' 3, Yuelin Bai Yinghao Ma ?: Ziya Zhou', Ka Man Jiaheng Liu' 17 Ruibin Yuan" 13, Lejun Min .uelinn Liu'. Tianyu Zhang Xinrun Du', Shuyue Guo Yiming Liang Yizhi Li Shangda Wu Junting Zhou Tianyu Zheng' 19 Ziyang Ma 18 Fengze Han', Wei Xue Gus Xia Emmanouil Benetos", Xiang Yue', Chenghua Lin", Xu Tan 14 Wenhao Huang 19
Jie Fust, Ge Zhang' 19.
M----, University of Waterloo, HKKUST, UUiiversity of Manchester, Shenzzen Institute of Advanced Technology, CAS, Vector Institue, 'QMUL, *MBZUAI, 9 Mila, Institute of Automation, CAS, "Central Conservatory Music, Institute for AI, SJTU, M, University of Montreal, 16 CCRRMA, Stanford University." 17 NJU, " Technological University, ByyytDannce
ABSTRACT
In this paper, we explore the application of Large Language Models (LLMs) the pre-training on symbolic music. While the prevalent use of MIDI in music modeling is well-established, our findings suggest that LLMs inherently more compatible with ABC Notation, which aligns more closely with their design and strengths, thereby enhancing the model's performance in musical composition. To address the challenges associated with misaligned measures from different tracks during generation, we propose the development of Synchronized Multi-Track ABC Notation (SMT-ABC Notation), which aims to preserve coherence across multiple musical tracks. Our contributions include series of models capable of handling up to 8192 tokens, covering 90% of the symbolic music data in our train- ing set. Furthermore, we explore the implications of the Symbolic Music Scaling Law (SMS Law) on model performance. In music structure experiments on rep- etition, we outperform GPT-4 by 17% (average Intra Similarity and Repetition Rate) the full test set, and surpass the SOTA ABC-notation model ChatMusi- cian by 6% on the single-track test set. terms subject evaluation, listeners preferred music from system in 79% of cases, comparing GPT-4. The results indicate a promising direction for future research in music generation, of- fering extensive resources for community-led research through open-source contributions.
INTRODUCTION
Large Language Models (LLMs) have experienced remarkable advancements, leading to their broad application across numerous domains. As these models extend into multimodal areas, such visual and auditory fields, their capability to represent and model complex information, including images (Liu et al. (023) and speech (Baevski al.) 2020) becomes increasingly critical. However, this ex- pansion also highlights significant challenges that must be addressed. Specifically, the development of effective tokenizers for images and videos, as well as advanced codecs for the audio domain.
In the domain of music, LLMs encounter inherent challenges that hinder their effective utilization. These models often struggle to capture the consistency of long-term structural consistency music essential for pleasing music (Dai et al.) 2022 Briot & Pachet 2020 Dai et al. 2021). This issue stems from the use of Musical Instrument Digital Interface (MIDI). which, while effective, poses significant challenges in terms of music's readability and structural representation. The widely-used performance MIDI data may lack structural annotations and cannot inherently encode phenomena such music repetition, thus resulting in longer sequence lengths ('uan et al. 2024).

---

Published as a conference paper at ICLR 2025
To tackle this issue, the integration of ABC notation offers novel approach overcoming the limitations of MIDI formats, visualized in Figure [ Yuan et al. (2024) advocate for this method, highlighting ABC notation's ' readability and structural coherence. Their methodology involves fine- tuning the LLAMA2 model, leveraging instruction tuning to enhance the model's musical output capabilities (Touvron al.) 2023b The research overlooks critical tokenization considerations within musical compositions.
In this paper, we aim to propose a training standard with transformer decoder-only architecture for symbolic music generation tasks, which is suitable for single multi-track music generation. We observe that mismatches between measures can occur by employing the traditional 'next-token- prediction' paradigm for symbolic data training. This issue arises because ABC notations are gen- erally notated track by track, completing one track before moving on the next. To address this challenge, we propose SMT-ABC notation to facilitate the model's learning of how each measure is expressed across various tracks.
Furthermore, we observe that the ABC Notation model benefits from additional epochs in the train- ing phase. This suggests that repeated data positively impacts the model's performance. under- stand this phenomenon, we introduced the SMS Law for repetitive training with symbolic music data. This law explores how scaling up the training data affects the performance of symbolic mu- sic generation models, particularly in terms of validation loss. This investigation aims to provide clear insights into the relationship between data repetition and model efficacy, offering guidance for optimizing model training strategies.
We conducted both objective and subjective evaluations comparing our MuPT model with state- of-the-art models like GPT-4 ChatMusician and MMT, focusing on ABC-notation and MIDI-based approaches. Objectively, MuPT achieved the closest approximation to ground truth, with an average gap of just 0. significantly outperforming ChatMusician'' 's 0.48. This seemingly small numerical difference marks a substantial improvement in music generation quality. Notably, MuPT supports multi-track music generation, feature absent in ChatMusician, enhancing its utility in realistic settings where such complexity is common. In experiments assessing music structure, MuPT sur- passed GPT-4 by 17% and ChatMusician by 6% terms of Intra Similarity and Repetition Rate, demonstrating its superior capability in handling complex musical compositions.Subyet tions further validated MuPT's superiority, with over 70% preference ratings against both MMT and GPT-4, underscoring its appeal to human listeners.
evalua-
Our main contributions are chiefly as follows:
Models. We introduce series of long-conter: symbolic music foundation models trained on ABC notation, featuring extended sequence length of 8, 192 tokens, enabling them to accommodate over 90% of the symbolic musical data our collected dataset. This advancement significantly enhances our ability to process and generate longer, more complex musical compositions.
Method. We propose SMT-ABC notation, novel approach that seamlessly integrates the char- acteristics of auto-regressive models with the inherent nature of repeat and music structure with a high compression rate. Our extensive experimentation demonstrates that SMT-ABC not only improves the coherence of generated musical pieces but also preserves the nuanced patterns and structures inherent in music.
Scaling Law. We explore the SMS(SYmbolic Music Scaling) Law insights for music modeling based on the ABC notation. We demonstrate that comprehensive music modeling yields superior performance with a positive correlation between model size and metric improvement. We also reveal unique training epoch dynamics in music repetition and performance enhancement.
Open Source. We release a suite of statc-of-the-art long-context symbolic music foundation mod- els along with all the intermediate training checkpoints to foster community research and innova- tion in symbolic music modeling.
2 RELATED WORK
Music Pre-training Audio pre-training through the self- learning paradigm has made
great progress in speech (Baevski et al. 2020) Hsu et al., [021]] Baevski et al.] 2022; Ma et al.
2023b Yang et al. 2023 Lin al. 2023). general-purpose audio (Huang al. 2022 Baade al.
2

---

Published as a conference paper at ICLR 2025
Chen et al. 2023 2024), well as music (Zhu al. 2021 Dong et al. 2023 Thickstun et al. 2023 Ma et al. 2023a Li et al. 2023). Two types of music pre- tre-trainin have been explored: non autoregressive discriminative models and autoregressive generative models. Autoregressive generative music pre-training models employ GPT-style framework to generate music, either in codec (Copet al. 2024) form or in symbolic form (Thickstun et al. 2023 Dong et al. 2023)
2022
Data Representation for Symbolic Music Symbolic music representation formats such as MIDI, Humdrum, and ABC notation offer distinct approaches for representing musical information. Specifically, MIDI, which excels capturing musical notes and performance, a popular choice in the music industry and research community(Huang & Yang 20200 Huang et al. 2019 Lu et al. 2023). However, the complexity and length of MIDI sequences often challenge music models, which limit the preservation of composition's full continuity. MIDI sequences into shorter fragments, which limit capturing a composition 's full continuity. MIDI's encoding of performance nuances lead quantization errors and unstable rhythms when being tokenized. Besides, there are some score-level symbolic music, such as MusicXMI., MEI, and Lilypond, that are not typically used for model training due the limitation of turge-highquaaityy data and the length to representing the same music clips Ma et al. (2024) In contrast, ABC notation stands out for its textual simplicity and compactness, making particularly suited for Natural Language Process- ing (NLP) techniques. It can be efficiently processed and analyzed using sequence modeling and pattern recognition algorithms similar to those used in language translation and text generation, en- abling automated music generation and retrieval Sturm et al. 2016 Casini et al. 2023]] Yuan et al.) 2024). However, each soundtrack is recorded sequentially in typical ABC notation, which dif- ferent from music performance when each measure within every track is performed simultancously Yuan et al. (2024). This making multi-track ABC notation somehow incompatible with LLMs on next token prediction. Since if token is far from the current tokens, it becomes challenging to capture and measure the results effectively.
typically segmented
Scaling Law A wide range of research underscores a significant pattern in language model perfor- mance, indicating power-law relationship between model performance and the increases both the number of parameters and the size of the training data (Kaplan et al.) 2020 Hoffmann et al.) 2022 Ghorbani et al. 202T). Scaling law plays a pivotal role in advancing large language mod- els (LLMs), offering framework to predict the optimal configurations for larger models based on the training logs of their smaller counterparts (Gao et al., 2022). The research by MMennnghoff ct al. (2024), which involves the repetition of the entire pre- pre-training dataset across multiple epochs, presents promising results yet raises questions regarding its effectiveness for musical data. This un- certainty prompts need for further research into the impact of data repetition strategy by achieving
improved outcomes for models engaged in music-related tasks.
Data Type
Count
Pet. (%)
Avg. Tks
Genre
Count
Pet. (%)
Type
Single Track
3.5M
51.2
450
Pop
227
18.7
S/M
2 Tracks
605K
8.7
2.0K
Jazz
213
17.5
S/M
3 Tracks
412K
5.9
3.1K
Country
168
13.8
S/M
4 Tracks
632K
9.0
4.2K
Rock
99
8.1
S/M
5 Tracks
362K
5.2
5.2K
Dance
75
6.2
S/M
Tracks
248K
3.6
6.7K
Latin
22
1.8
S/M
7 Tracks
176K
2.5
8.2K
Folk
95
7.8
S/M
Tracks
149K
2.1
10.1K
R&B
111
9.1
S/M
9 Tracks
104K
1.5
10.3K
Classical
207
17.0
S/M
10 Tracks
88K
1.3
11.8L
Total
1217
100.00
11+ Tracks
633K
9.1
25.9K
Total
6.9M
100.00
4.53
Table 1: Training Set Statistics. (ct. refers to the percentage of the specific type of data. Avg. Tks refers to the average num- ber of tokens.
Table Test Set Statistics. Each genre includes both single and multi-track music pieces. Pet. refers to the percentage the specific type of music. S refers to Single track music, M refers to Multi-Track music.
3

---

Published as a conference paper at ICLR 2025
3 DATASET
The dataset used in our empirical study is divided into two parts: testing set and a training set. The testing set is derived from WIKIMT++ et al. 2023), which includes 1,010 ABC notation scores from eight music genres (e.g.. Pop, Jazz, Rock, R&B. Latin, etc.) along with 12 subjective emotions. Additionally, the test set comprises 207 multi- classical music pieces manually se- lected from Bach's compositions. Importantly. none of these pieces overlap with the training set, ensuring that the test set can effectively evaluate the model's ' performance across diverse musical genres, various emotions and in generating out-of-domain music.
The training set is built from a comprehensive collection, incorporating the Nottingham Music Datase the ABC tune book of Henrik Norbeck(Ji et al. 2020), the Irishman dataset (Wu et al., 2023), and a private dataset owned by the Central Conservatory of Music (including university li- brary corpus in ABC and other formats that can be converted to ABC like MusicXML, along with internet collections). This rich dataset spans nearly all music genres and includes diverse range of both single-track and multi-track data. Due to the unavailability of detailed genre metadata for most of the training data, we have not included genre-speciff breakdowns in the statistics. We are com- mitted to open-sourcing the training data for research purposes once all datasets are well-organized.
Tables [ provide an overview of the statistics for both the testing and training sets, illustrating the number of samples, their distribution, and average token lengths across various categories.
4 METHOD
4.1 SMT-ABC NOTATION
ABC notation is widely adopted system for notating music using plain text, and it offers unique advantages when used in conjunction with deep learning models. Its well-structured text format ensures easy preprocessing, efficient data transmission, and scalability of datasets. The diverse collection of tunes and compositions ABC notation facilitates learning various musical structures and styles. Moreover, ABC notation allows models to generate human-readable outputs, leading to immediate feedback and iterative refinement. These attributes significantly enhance both the efficiency and quality of the training process.
An ABC file is composed of headers following the music notation. The former contains metadata regarding the tune, such as its composer and tempo, while the latter defines the melody. Each note is represented by a letter, with additional symbols conveyingduration, rhythm, and other musical characteristics. Compared MIDI, another symbolic representation form of music, ABC notation offers a simple and user- fiser-frienlly text-based format that is easy to read and write, making it accessible for musicians of all skill levels. An example is shown in Figure [ "V:1" indicates the beginning of the first music track and the lines before it are headers. A tune can consist of one or more tracks, each representing distinct musical element within the composition. The bars within each track are separated by bar line symbols like vertical lines (*1"", which refer to the standard bar line.
In Yuan et al. 2024). ABC files without any modification are the input of models. However, we found that the models struggle with bar alignment when dealing with multiple tracks. Since track represents a section or division within musical composition, such as one of the instrumental or vocal parts in piece of symbolic music, it is crucial for models to capture the correspondence between tracks. Speeiffically, this correspondence exists in bars with the same indices, and thus, they should be treated as series of groups. To this end, we reorganize the tracks depicted in F We concatenate music segments from bars with the same index across all tracks, including their right bar lines. These concatenated elements from different tracks then enclosed by a pair of a newly introduced symbol "<|>", which is not part of the original ABC system. This symbol represents the beginning or the end of a group of bars at the same stage. In cases where a tune contains only one track, each new unit will consist of a single bar. After processing all the bars, we obtain a synchronized version of the music notation, while the headers remain unchanged. The length of the tracks is not always identical due to repetition or other specific musical structures, which are difficult handle exhaustively. Considering these special samples typically account for
https://i fdo. httpa////a./eeeooooooooo/ootaaaa

---

Published as a conference paper at ICLR 2025
Pann
X:1
L:1/4
Q:1/4w80
N:4/4
K:Bb
PW
d2
r2 x 02 13/2 G I... IJ
V:2
D.C.Z.
D/
V:3
8,2
A. 1.2.6..
1F,2
22003 69003
F.2 E,2 D. 2,2
Figure I: A comparison between MIDI represented by piano roll (left) and ABC notation (right) of the same music excerpt. MIDI note number represents the pitch and the square's length on the time axis represents the duration musical note.
just small portion (0.01% % in dataset) of the entire dataset, we simply skip them in this scenario. This simple and efficient measure only require 8CPU 1-2 hours to process the whole 33.6B dataset, but provide impressive results in ablation study in subsection B.3
Align Bars
V1 23 B//F/GG
<|> 23 F/F C2 A,2
V:2 26 C2 C2 C2 C2 CD
<|> g A GCIC2 C2 C2 CD G2
V3 26 A,2 I |2 F22 E,F GAI ...
F22 E,F
Illustration of synchronized multiple-track ABC notation. Music segments from bars shar- ing the same index across all tracks, along with their right bar lines, are concatenated to guarantee alignment. The combined elements are then enclosed by pair of newly introduced symbol "<|>".
Figure 2:
4.2 TOKENIZER
We chose YouTokenToMMe (YTTM) You'lokee'ToMe 2021) framework to develop tokenizer with vocabulary of 50,000 tokens, leveraging the Byte-Pair Encoding (BPE) (Shibata et al. 1999) for ABC notation tokenization. This method instrumental segmenting the ABC text into manage- able units, thereby enhancing the model's 's ability to interpret and process the input effectively. We do not apply any normalization and dummy prefix to the training corpus, without changing its form adding extra parts at the beginning. Additionally, a unique symbol "<n>""i "is employed to denote spaces within the ABC text, ensuring accurate space recognition by the model.
Parameters
190M
505M
1.07B
1.97B
4.23B
Hidden Size
768
1024
1280
1536
2048
# Layers
12
16
20
24
32
# Feedforward dims.
3072
4096
5120
6144
8192
# Heads
12
16
20
24
32
Head Size
256
256
256
256
256
Table 3: MuPT model with different model sizes.
4.3 MODEL ARCHITECTURE
MuPT utilizes standard Transformer model architecture (Vaswani et al. 2023 in decoder-only setup. Models are trained on context length of 8192 tokens. We list our MuPT model parameter in Table and utilize several improvements proposed after the original transformer paper. Below, we list the included improvements:
SwiGLU Activation: The SwiGLU activation mechanism, represented as (Swish(#W) vVV,, is utilized for the MLP (Multi-Layer Perceptron) intermediate activations. This approach signifi-
5

---

Published as a conference paper at ICLR 2025
cantly surpasses traditional activation functions such ReLU, GeLU, and Swish in performance Shazeer
2020).
RMSNorm Each transformer sub-layer, including the attention and feedforward layers, is nor- malized using RMSNorm as proposed by /Zhang Sennrich (2019)
RoPE Embeddings: contrast to positional encoding (PE) strategy, we use the Rotary Positional Encoding (RoPE) technique, as developed by Suu et al. (2023), aimed at enhancing long-conten, modeling.
4.4 SCALING LAW
The Chinchilla Law, proposed by DeepMind, is scaling law that provides insights into the training of LLMs. Our experiments reveal that the Chinchilla Law (Hoffmann al., (022) provides good fit for general cases, where moderate models were trained with a moderate amount of data. In this section, we will list several improvements to Chinchilla Law for symbolie music scaling principles on limited training data.
4.4.1 OPTIMIZING BASELINE SCALING LAWS UNDER COMPUTATIONAL CONSTRAINTS
pivotal aspect of scaling laws is the optimization of loss within the bounds computational feasibility. This is formalized minimizing the valid loss L, subject to constraints imposed by available computational resources (C"), specifically FLOPs, as denoted below:
D) arg DN s.t. F), =
This framework encapsulates the trade-offs between parameters (N) and training tokens (D), and decision-making processes inherent in scaling models under resource limitations, illuminating path- ways to efficiency and efficacy in LLMs training. More details can be found Appendix A.I
In this paper, we will use the Chinchilla Law(Hoffmann et al. 2022) and Data-Constrained law(Muunnighoff et al. 2024) as baselines. The former a classical baseline in LLMs' training and the latter crucial address the constraints faced in scenarios where the volume of available training data does not meet the ideal requisites. This phenomenon is typical in the music domain. Please refer to A.1.2 for more information.
4.4.2 SYMBOLIC MUSIC SCALING (SMS) LAW
2.15
140M
2.10 S25M
unique tobass, 1072M
Loss Vakee
Las Valee
Loss Vase
120
1.2
03
25
135
20
19
40
Figure 3: Chinchilla I Law prediction and authentic validation loss in the setting with 2. IB unique training tokens for models with 190, 505 and 1072MB.
Figure demonstrates the Chinchilla prediction in yellow lines and the observed loss in blue. We can tell that the Chinchilla law does not provide good results when the data volume D is small when the model just begins the pre-training stage, and when D is large where repeated data provides overfitting, We proposed two terms to address these problems.
Incorporation of New Term. We observe that when that model parameter is small (e.g. N 190M), the Chinchilla underestimates the loss value and overestimates when the model size large (e.g. N = 1072M). This suggests that the coefficient B in the Chinchilla formula

---

Published as a conference paper at ICLR 2025
A B E shall be relevant to D instead of constant. To cope with, incorporate term. Da After that, we proposed another term to predict the early stop points and overfited loss curve:
L(N, D) N=.Da
+
Where {A, B, d, E, are learned variables fit using the training To address the model's limitations in accurately capturing performance metrics for smaller data sizes, we introduce an ad- ditional term, delineated in Equation - This modification aims refine the model's fidelity, particularly in scenarios characterized by limited data availability. Further details on this modifica- tion can be found in the Appendix A.3.1
Modelling Overfitting Settings. Crucially, previous iterations of the model fall short in predict- ing overfitting, particularly beyond early stopping thresholds. This gap is especially pronounced in environments, such as music, where open-source data is limited. To this end, introduce new component, Liverfit the model, encapsulated in Equation to specifically account for overfitting losses:
the context of
B L. LNN,D,UD) No. + No D + F Loverfft
(3)
where
Lover füt GELU (ka- D + kn log(N)) k. log(UU) kin
(4)
overfitting formulation where {ki,, kny kus kmn} learned variables for overfitting calibra- tion. For comprehensive insights into the overfitting loss component, please refer to Appendix A.3.2
Parameter Fitting and Model Integration. Initial parameter fitting for {a,, B, A. B, E). and d. subsequent linear regression analysis, focusing on the residuals between Equation 2 and empirical observations, facilitates the calibration of overfitting parameters {ka, kn. ku kin) within Equation 4 The integration of these components in Equation not only predicts performance under constrained conditions but accounts for overfitting dynamics, helping to predict the true minimum of loss curve.
5 EXPERIMENTS
5.1
DATASET & EXPERIMENTAL. SETUP
we adopt similar model architecture from LLaMA2(Touvron Touvron et al., 2023b), including RMSNorm(Zhang & Sennrich 2019) and SwiGLU 2020p. All the hy- perparameters are detailed in Appendix B.1 In the full-scale data setting, we trained models of various sizes (ranging from 190M 4.23B parameters) on the ABC text corpus, which consists of 33.6 billion tokens derived from a diverse collection of monophonic and polyphonic musical com- positions spanning various genres and styles. And the validation set includes pop music genres and classical music, providing good generalization capability for scaling law evaluation. For more information about the corpus, plcase refer to ssusecction B.2] In data repetition experiments, we uti- lized subsets of the corpus, specifically 6.25% and 25% random sampled data. The Adam(Kingmal & Ba 2014) optimizer and cosine learning rate schedules are applied for the training process.
As outlined in section 4.3
5.2
SCALING LAW
5.2.1
EVALUATION METRICS & FITTING METHODOLOGY
We use the R² value and Huber loss (with the parameter 1e - 3) between the authentic valid loss and predicted valid loss small models (190M, 505M, 1.07B) to acquire the best scaling law. Then we use the best law to train two large models (with 1.97B and 4.23B). See Appendix A.4 for more details about the two evaluation methods.
We optimized the SMS Law using the L-BFGS algorithm, the same with Chinchilla and Data- Constrained Laws. For more information, please refer to Appendix A.5
7

---

Published as a conference paper at ICLR 2025
Paramatic fit
It Value (train)
Huber Loss (train) 4
12 Value (test) 1
Huber Loss (test)
Chinchilla law
0.9347
0.0109
-0,0933
0.0080
Data- Constraind law
0.7179
0.0206
0.1524
0.0071
Equation
0.9075
0.0129
0.3114
0.0073
Equation
0.9759
0.0102
0,8580
0.0062
SMS Law
0.9780
0.0085
0.9612
0.0028
Table 4: Comparison of parametric fitting performance of different scaling laws.
5.2.2 SMS LAW ARE THE BEST ON THE TRAINING SET
The integration of an additional term delineated in Equation 2 alongside the introduction of GELU regularization component in Equation 4 collectively underpins the superior performance of the SMS Law, empirically evidenced by its training set outcomes. This is particularly notable in the context of our parametric fitting performance comparison (see Table 4. where the SMS Law outshines other scaling laws, achieving the highest R² value (0.9780) and the lowest Huber loss (0.0085) on the training set.
Although Equation [T does not eclipse the Chinchilla Law in performance metrics, it nonetheless presents significant improvement over the Data-Constrained Law's D' by leveraging D", which is indicative of a refined approach to managing the constraints posed by data repetition. This nuanced handling of data repetition, inherent to Equation [T] suggests an enhanced generalization capability in such scenarios. Therefore, we culminate along with other modifications, manifest in the SMS Law in order to enhance model performance and generalization at the same time. In fact, it indeed provides much better results in the test set.
5.2.3 SCALED-UP PERFORMANCE USING SMS LAW
In our SMS Law experimentation under computational budget of 2 x a0a0 FLOPs, initially aim to train a 2. 10B (or 1.98B) parameter model across 2.82 epochs on the whole 33.6B dataset per epoch, achieving a loss of 0.5279 (or 0.5280). Engineering constraints necessitated a slight scale- down to a 1.97 billion parameter model, which, intriguingly, showed minimal loss increase to 0.529 around 2.5 epochs. Contrary to the predictions of SMS Law, the Chinchilla Law suggests optimal performance for 990M parameter model over 6.1 epochs. Pushing boundaries, we continuously train the 1.07B parameter model and observe overfitting returns beyond epochs, validating the SMS Law's advantages in this context. Further, we train 4.23B parameter model that underscored the SMS Law's predictive accuracy regarding overfitting risks, affirming its value as strategic guide in scaling up models effectively within fixed computational constraints, beneficial for efficient model scaling decisions.
In validating SMS Law, we analyze the performance of 1.97B and 4.23B parameter models, detailed on the right-hand side of Table 4 This comparative study highlights the SMS Law's exceptional performance, evidenced by its unparalleled R² values and minimal Huber Loss testset well.
Unlike the Chinchilla and Data Constrained laws, the SMS Law not only showcase superior predic- tive accuracy but also demonstrates its efficacy in optimizing neural network scaling within com- putational constraints. These results affirm the SMS Law's 's value in guiding scaling strategies for symbolic music, marking significant advancement in the field. For other advantage of SMT-ABC such as consistency and the subjective evaluation results, please refer to subsection B.3
5.3
EVALUATION
5.3. EFFICIENCY OF OUR TRAINING STRATEGY
To demonstrate the efficiency of our training strategies, we reference the training loss curves in Fig- ure Our comparison spans four different model sizes: 190M, 505M, I.I and 2B. We observed that increasing the training input length from 4096 to 8192 significantly reduces the loss, especially noticeable in the convergence phase. The comparison shows that after aligning data, our training loss slightly decreases compared to the original ABC loss, demonstrating our method's efficiency in improving training for various model sizes. For more discussion on the compression between MIDI, ABC and SMT-ABC, please refer to Appendix C.
8

---

Published as a conference paper at ICLR 2025
Original ABC 4K
Original ABC 8K
SMT-ABC 8K
190M models
50SM models
1.078 models
1.978 models
1.0
Train Loss
0.6
0.4
0.0
50
150
150
250
250
100
200
Training Tokens(Blllions)
Training Tokens(Billions
Training (okens(Billlons)
Training Tokens(Billion)
Figure 4: Training Loss for different model sizes and training strategy.
5.3.2 OBJECTIVE METRICS OF SYMBOLIC MUSIC ELEMENTS
Following the previous studies music generation (Dong al. 2023 Wu & Yang 2020 Mogren 2016). we adopt the pitch entropy, scale consistency and groove consistency to evaluate how well the systems generate music from the perspectives of different musical elements given the first measure. Table 5) shows the mean values of these three metrics, where MuPT achieves overall better performances than other systems compared to the ground truths. For the whole test only 51% of samples generated from GPT-4 have the correct ABC notation format. To compare MIDI representation with ABC notations, we incorporate Multitrack Music Transfomers (MMT) (Dong et al. 2023). a MIDI-based music generation model to infer the MIDI data transformed from the ABC notations by abc2mid Moreover, to compare MuPT with ChatMusician (Yuan al. 2024), another LLM pre-trained on large-scale single-track (st.) ABC notation data, we separate the single- track samples from our test set and obtain the results in Table MuPT also achieves better results.
5.3.3 REPETITION METRICS
Repetition Rate Repetition significant in evaluating how well-structured the music is. In Table 6. the piece-level average repetition rate of each system is calculated to reveal how often the repeat sign: : appears in a generated set. It appears that 43.7% of the generated samples from MuPT, which quite close to the ground truth, higher than Chatmusician in single-track data, and much higher than GPT-4. This suggests that MuPT is more likely to generate music with repetition and structure.
System GT
PE
SC (%)
GC (%)
System GT
ITS
RR (%)
2.708
96.80
93.46
0.3729
43.5
MuPT-SMT
2.631
97.48
93.45
MuPT-SMT
0.4193
43.7
MuPT-Ori.
2.621
98.09
93.36
MMT
0.1767
MMT
2.784
95.64
91.65
GPT-4
0.3614
16.9
GPT-4
2.783
97.90
95.32
GT(st.) 0,4753
59.2
GT(st.)
2.617
98.39
93.25
MuPT-SMT(st.)
0.4507
52.6
MuPT-SMT(st.)
2.612
98.20
93.39
ChatMusiciannst)
0.5260
40.1
MuPT-Ori.(st
2.619
98.16
93.49
MMT(st.) 0.2158
ChatMusician(st.)
2.664
98.55
94.47
GPT-4(st. 0.4235
23.0
(MT(st. 2.808
95.88
91.60
GPT-4(st.
2.686
99.27
95.72
Table 6: Mean value of intra-texture similarity (ITS) and repetition rate (RR) of each system. The ABC notation string generated by MuPT achieves higher intra-similarity than the GT and GPT-4.
Table 5: Mean values of pitch entropy (PE), scale consistency (SC), and groove consistency (GC) for each system. A closer value to the ground truth (GT) is considered better.
Intra Similarity In addition to the naive repetition rate, we also adopt the methods introduced in Wang et al. (2024) to calculate the intra-similarity of symbolic music each system. Specifically, pre-trained VAE from Yang et al. (2019) and Wang et al. 2020) is transferred to compute the

---

Published as a conference paper at ICLR 2025
texture latent for each music piece; the intra- of music piece is defined as the average value of its texture latent similarity matrix, excluding the diagonal. Since the texture encoder is pre- trained on MIDI data, transform ABC notations into MIDI format before the latent is obtained. shows the mean value of each system's 's intra-similarity under the first-measure conditioned generation. For the whole test set, MuPT achieves the highest score among all systems, while for the single track, its value is lower than the ChatMusician. Generated pieces of MMT have notably lower intra similarity than MuPT and GPT-4, This result corresponds with the intuition that score-level ABC notation is more capable of generating structured music than performance-level MIDI.
Table O
5.3.4
SUBJECTIVE EVALUATION
Model A
Model
Wins (A/B)
p-value
54
73
79
92
MuPT
81/69
0.4257
MuPT
46
73
87
70
Human Works
GPT-
MMT
11931
109/41
6.6315 x 10-9
4.2249 10-B
Random
138/12
4.4648 x 10
69
MMT
4.2249 10
MuPT
GPT-4
1 115/35
6.6641 10
GPT-4
21
23
37
71
Random
131/19
1.3618 x 10 18
GPT4
95/55
0,0093
Random
13
31
MMT
Random
103/47
0.0001
GT
MUFT
MMT
GPT-4
Random
GPT4
Random
106444
2.6691 10-5
Table 7: Human evaluation of paired completions of musical excerpts generated by different sources given the first bar as the condition. The left is the matrix based on the AB test. Each row indicates the % of times listeners preferred instrumentals from that system compared those from each system individually (N = 150). Ground truth is denoted by GT. i.e.77 means that listeners preferred MuPT over GPT-4 in 77% of cases. The right is the absolute win numbers and the corresponding p-value of each pair. P-values are reported by a Wilcoxon signed rank test.
Human assessment should be involved to further testify the objective repetition metrics above. Fol- lowing Donahue al. (2023) and and Thickstun al. (2023), conduct subjective listening study to measure the qualitative performance of we conduct a subjective listening study to mea- sure the qualitative performance of MuPT against the ground truth (GT) and baselines consisting of against the ground truth (GT) and baselines consisting of GPT-4, MMT and random note sequences (Random). Listeners are asked to identify which of two musical excerpts from different sources is more "musical" during the test process. They are also instructed to focus on two aspects of musical- ity: how consistently the music sounds throughout (e.g., in terms of its melodic contours, rhythmic patterns, and chord progression); and how likely it is that the development of the music follows clear structure (e.g., verse-chorus division, repetitions). This process is similar to that in and ran- dom note sequences (Random). Listeners are asked to identify which of two musical excerpts from different sources is more "musical" during the test process. They are also instructed to focus on two aspects of musicality: how consistently the music sounds throughout (e.g., in terms of its melodic contours, rhythmic patterns, and chord progression): and how likely it is that the development of the music follows clear structure (e.g., verse-chorus division, repetitions). This process is similar to that in Yuan et al.] (2024) and its details are shown in the Appendix and its details are shown in the Appendix [b. Results for all systems are shown in Table 7 Comparing MuPT to GPT-4, listeners prefer music from our system in 79% of cases. A Wilcoxon signed-rank test of these pairwise judg- ments shows that listeners preferred music from MuPT significantly more than MMT and GPT-4 (p 4.2249 x 10-6 and p 6.6641 10-$, respectively). We use Fleiss' kappaFleiss & Cohen (1733) to measure the inter-annotator agreement among 15 participants. A kappa value of 0.5807 was obtained, indicating that the participants achieved moderate agreement. This demonstrates the quality of our subjective annotators.
CONCLUSION In this paper, we introduce the MuPT series of pre-trained models trained on the largest possi- ble amount of ABC Notation data which set the standard for training open- symbolic music LLMs. Additionally, cialist in guiding future scaling of symbolic music GPTs. Our results demonstrate that the MuPT se- ries is competitive with mediocre human composers and guarantees of-te----te-artt performance on
dive deep into the scaling law exploration and propose SMS Law, spe-
10

---

Published as a conference paper at ICLR 2025
symbolic music generation. Moreover, MuPT introduces SMT-ABC, reordering the multiple-tred -rackk original ABC notation format to assist pre-training of MuPT. We conducted comprehensive evalu- ations of our MuPT model against state-of-the-art models like GPT-4k, ChatMusician and MMT. Objectively, MuPT closely approximated the ground truth, significantly outperforming ChatMusi- cian and demonstrating superior handling of complex musical compositions, including multi-track music, which is absent in ChatMusician. Subjectively, MuPT was preferred by over 70% of human listeners, outperforming both MMT and GPT-4, confirming its effectiveness in realistic music gen- eration scenarios. We believe that the open access of intermediate checkpoints of MuPT, SMS Law, and MuPT series will foster collaboration and innovation within the open- computational mu- sic community, and open the door to the next-generation symbolic music foundation models.
ETHICS STATEMENT
In designing the MuPT series, we have meticulously adhered to ethical guidelines to ensure fair- ness, transparency, and the responsible of AT in music generation. Despite these efforts, ethical challenges such as potential copyright infringement and unintended use of Al-generated music in sensitive contexts remain. We urge the research community to approach these challenges with vig- ilance and to consider ethical implications carefully when deploying similar technologies. Besides, as suggested by Ma et al. | (2024), we strongly urge people to tag music as "Al-generated" for music pieces when uploading to the website, enabling the user to choose mnon-JAl-generate music to bet- ter protect the rights of musicians. Furthermore, all 15 participants were chosen by the author of this paper, and therefore, there is no additional requirement for ethical approval for the listening test.
REPRODUCIBILITY
REFERENCES
Alan Baade, Puyuan Peng. and David Harwath. Mae-ast: Masked autoencoding audio spectrogram transformer. Proc. Interspeech, 2022.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449-12460, 2020.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, pp. 1298-1312. PMLR, 2022.
Jean-Pierre Briot and François Pachet. Deep learning for music generation: challenges and direc- tions. Neural Computing and Applicationn,, 32(4):981 2020.
Luca Casini, Nicolas Jonason, and Bob L. T. Sturm. Generating folk-like music in abc-notation with masked language models. Proceedings of the International Society for Music Information Retrieval Conference 2023 Late Breaking/Demo. ISMIR, 2023.
Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. Beats: Audio pre-tratee training with acoustic tokenizers. In International Conference Machine Learning, pp. 5178-5195 PMLR, 2023.
Wenxi Chen, Yuzhe Liang, Ziyang Ma, Zhisheng Zheng, and Xie Chen. Eat: Self-supervised pre- training with efficient audio transformer. arXiv preprint arXiv:2401.08 03449, 2024.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défossez. Simple and controllable music generation. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Shuqi Dai, Zeyu Jin, Celso Gomes, and Roger Dannenberg. Controllable deep melody generation via hierarchical music structure representation. arXiv preprint arXiv:2109.00663 2021.
Shuqi Dai, Huiran Yu, and Roger Dannenberg. What is missing in deep music generation? a study of repetition and structure in popular music. arXiv preprint 20X0::2009000882.. 2022.
11

---

Published as a conference paper at ICLR 2025
Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli, Mauro Verzetti, lan Simon, Olivier Pietquin, Neil Zeghidour, et al. Singsong: Gen- erating musical accompaniments from singing. arXiv preprint arXiv:2301.12662, 2023.
Hao-Wen Dong, Ke Chen, Shlomo Dubnov, Julian McAuley, and Taylor Berg- Kirk-Kirkpatrick. Mul- titrack music transformer. In ICASSP 2023-2023 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Joseph I. Fleiss and Jacob Cohen. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613- 619, 1973.
Leo G John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022.
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint urXiv:2203.15556 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Sclf-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. In International Conference on Learning Representations, 2019. URL https: //openreview. het.:e/netttttommmeoooaoaaaaaaa
Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances Neural Infor- mation Processing Systems, 35:28708-2877 2022.
Yu-Siang Huang and Yi-Hsuan Yang. Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions. In Proceedings of the 28th ACM international conference on multimedia, pp. 1180-1188, 2020.
Shulei Ji, Jing Luo, and Xinyu Yang. A comprehensive survey deep music generation: Multi-level representations, algorithms, evaluations, and future directions. arXiv preprint (xXiv:2011.06800) 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
GG Landis JRKoch. The measurement of observer agreement for categorical data. Biometrics, 33 (1):159174, 1977.
Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma. Xingran Chen, Hanzhi Yin, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, et al. Mert: Acoustic music understanding model with large-scale self-supervised training. arXiv preprint 2rXiv:2306.00107, 2023.
Tzu- Hua-QQann Lin, Hung-yi Lec, and Hao Tang. Melhubert: A simplified hubert on mel spectrograms. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8. IEEE, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
12

---

Published as a conference paper at ICLR 2025
Peiling Lu, Xin Xu, Chenfei Kang, Botao Yu, Chengyi Xing, Xu Tan, and Jiang Bian. Musecoco: Generating symbolic music from text. arXiv preprint arXiv:2306.001 00, 2023.
Yinghao Ma, Ruibin Yuan, Yizhi Li, Ge Zhang, Xingran Chen, Hanzhi Yin, Chenghua Lin, Em- manouil Benetos, Anton Ragni, Norbert Gyenge, et al. On the effectiveness of speech self- supervised learning for music. arXiv preprint arXiv:2307.000 20..000...6,,,, 2023a.
Yinghao Ma, Anders Øland, Anton Ragni, Bleiz MacSen Del Sette, Charalampos Saitis, Chris Don- ahue, Chenghua Lin, Christos Plachouras, Emmanouil Benetos, Elio Quinton, et al. Foundation models for music: A survey. arXiv preprint arXiv:2408. 14340, 2024.
Ziyang Ma, Zhisheng Zheng, Changli Tang, Yujin Wang, and Xie Chen. Mt4ssl: Boosting self- supervised speech representation learning by integrating multiple targets. Proc. Interspeech, 2023b.
Mary L McHugh, Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276-255 2012.
Olof Mogren. C-mnn-gan: Continuous recurrent neural networks with adversarial training. arXiv preprint arXiv:161 1.09900 .009004, 2016.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
Michael Schoeffler, Sarah Bartoschek, Fabian-Robert Stöter, Marlene Roess, Susanne Westphal, Bernd Edler, and Jürgen Herre. webmushra a comprehensive framework for web-based listening tests. Journal of Open Research Software, 6(1):8, 2018
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2000.052002 2020.
Yusuke Shibata, Takuya Kida, Shuichi Fukamachi, Masayuki Takeda, Ayumi Shinohara, and Takeshi Shinohara. Byte pair encoding: A text compression scheme that accelerates pattern matching, 09 1999.
Bob L. Sturm, João Felipe Santos, Oded Ben-Tal, and Iryna Korshunova. Music transcrip- tion modelling and composition using deep learning. CoRR, abs/1604.08723,, 2016. URL
08723
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En-
hanced transformer with rotary position embedding, 2023.
John Thickstun, David Hall, Chris Donahue, and Percy Liang. Anticipatory music transformer. arXiv preprint arXiv:2306.00 08620,860000 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ARXIV, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and finettued chat models. arXiv preprint arXiv: 2307.00 0928009888 2023b.
13

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2023.
Ziyu Wang, Dingsu Wang, Yixiao Zhang, and Gus Xia. Learning interpretable representation for controllable polyphonic music generation. arXiv preprint 20t:::2000..01222,, 2020.
Ziyu Wang, Lejun Min, and Gus Xia. Whole-song hierarchical generation of symbolic music using
cascaded diffusion models. In The Twelfth International Conference on Learning Representations, 2024. /tttpss////:peeeeeee
net/ forum? d=s.:mmmmddsssseeeyaaama
Shangda Wu, Xiaobing Li. Feng Yu, and Maosong Sun. Tunesformer: Forming irish tunes with control codes by bar patching. arXiv preprint arXiv:2301 002884, 2023.
Shih-Lun Wu and Yi-Hsuan Yang. The jazz transformer the front line: Exploring the short- comings of ai-composed music through quantitative measures. arXiv preprint 2110:2000..0100,,, 2020.
Guanrou Yang, Ziyang Ma, Zhisheng Zheng, Yakun Song, Zhikang Niu, and Xie Chen. Fast-hubert: an efficient training framework for self-supervised speech representation learning. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-7. IEEE, 2023.
Ruihan Yang, Dingsu Wang. Ziyu Wang, Tianyao Chen, Junyan Jiang, and Gus Xia. Deep music analogy via latent representation disentanglement. arXiv preprint 1906.03690 2019.
YouTokenToMe. Youtokentome: Unsupervised text tokenization library, 2021. URL https: Available online: https://github. com/
7/github. wtt. http////w/O////OO/oooo//
VKCOM/ TouTokenTMMe (accessed on March 25, 2024).
Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, al. Chatmusician: Understanding and generating music intrinsically with Ilm. arXiv preprint arXiv:2402. 16153, 2024.
Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019.
Monan Zhou, Shangda Wu, Yuan Wang. and Wei Li. Wikimt dataset card. arXiv preprint arXiv:2309.13259 2023.
Hongyuan Zhu, Ye Niu, Di Fu, and Hao Wang. Musicbert: sele-super learning of music representation. In Proceedings of the 29th ACM International Conference on Mullimedia, pp- 3955-3963, 2021.
14

---

Published as a conference paper at ICLR 2025
A SCALING LAW
A.1 SCALING LAW BASELINE
A.1.1 ABSTRACTING LOSS METRICS THROUGH THE CHINCHILLA LAW
In this part, we focus on the relationship of loss metrics to various resource budgets in deep learning. It is first put forward by the Chinchilla Law as illustrated in Equation 5 This law posits that both training and evaluation losses can be abstracted as function of model capacity N and training data size D, thus offering insight to estimate the best combination of resources to be assigned to training.
B D
L(N,D)
+
(5)
No
Here, L(N, D) denotes the loss metric during training or evaluation, which is assumed to exhibit power-law dependency on N and The parameters A, B, E, a, and B are determined by empirical fitting.
A.1.2 DATA-CONSTRAINED LAW
Scaling under Data Limitations. Complementing the Chinchilla Law, Cant-Coontrained Law shows the scaling dynamics of L.MMs when facing the data scarcity prob- lem. Here, we strictly refer to the derivation method of Muennighoff et al. (2024). The goal of discovering Pata-Constrained Scaling Law to generalize the expression to multiple epochs where tokens repeated.
Ciaa-Constr Law:
Data-constrained law defined as:
L L(,,D,UD)
E
(6)
Nio
D
where
RN
N' = UN + UNNY
1 eeexp
R's
Rp
D' = UDUDUUDR
1- exp
Rb
To get a better understanding of the equation, the definitions of each of the above parameters are as follows: Like Chinchilla Law, N defined as the number of model parameters, and defined as the training tokens.
UD is defined as the number of unique tokens used. For data-constrained law, Up is computed as min(D. D.ne given budget of umique data D..
Un is defined as the number of "unique" parameters that provide an optimal fit for Up. Accord- ing to the method mentioned in Muennighoff et al. 22244, given the following learned variables, A, a, B, 3 E), the optimal allocation of compute N and D follows:
(C) (C)= C
b
(Cp = G-1
aA
0+3
(8)
G
3B
B
a + B
a
+ B
15

---

Published as a conference paper at ICLR 2025
Thus, UN is equal to min min,
Rp is defined as the number times the data repeated. When training for a single epoch, Rp = 0.
Ry is the number that the "unique" parameters repeated where Rx max
Ux
1, 10
D' is defined as the "effective data size": the number of unique data needed to get the same value as repeating unique tokens for RD repeats. The derivation process as followed:
From a conceptual standpoint, the redundancy of data samples diminishes their incremental value in enhancing the model's knowledge base, given the model's s prior exposure to said information. This principle underlies the hypothesis that each successive repetition of a sample contributes marginally less to the learning process, as the model has partially assimilated the information contained within the sample through prior iterations. To describe the process of training information loss, have
D' D'= + DU-U+U - 87h U + (UU (1-(1-))
where ô is defined as the 'orgetting rate'. Each time a series of tokens is trained on model, the model learns 1 å fraction information from the optimization process. Assuming that the number of epochs beyond which repeating does not help. the right-hand side goes to to (1-6)U
8
since Rom (1 - (1 (1-5)(tm 5))²))= = 1. We define R8 is defined as مند which is a learned constant. According to Taylor expansion, if is small, have:
c D (1-5)
(10)
Now inserting (ق=ا) R7 and (1 8)Rp representing the effective data.
-1 R Rh
into Equatior
we get our final equation
As the frequency of encountering repeated tokens diminishes, the benefit gained from processing them also decreases. Hence, the derivation of the N' is similar to D'. In this context, there's 's no need elaborate further. It should be pointed out that Rt a learned parameter.
A.2 ABLITION STUDY ON CONTINUOUS ADAPTATION OF DATA-CONSTRAN LAW.
To enhance the predictive accuracy of the Cons-Const law MMuennighoff et al. 2024) for continuous domains, we extend the original discrete formulation to accommodate continuous variables, allowing for a more nuanced understanding of data constraints in varied contexts. For an in-depth discussion on the derivation and implications of this continuous formulation, please refer to Appendix(A.2
A
B
L(N,D,UD) =
+
(1I)
Na
D
where ls is new parameter to fit, and D" the adjusted data size, given by:
D" 1 - D/UD
1 UD.
(12)
The definition of D' in Equation 9 is defined from discrete version and can not be extended the case when D is less than Up. So we reform the Equation 9 to
1 1- (1 11--0 8)tb
P.
D'
Up
(13)
D
1
d
UD
Up
kd
where ka := 1 - 5. This equation equivalent to equation 10 when D is a positive integer times Up.
We implemented a formula symmetric to N' with UN and kx. But the calculation results of ky ss 0.999. To make the formula simple, use the original N instead of N' in the following formula.
16

---

Published as a conference paper at ICLR 2025
A.3
MOTIVATION OF SMS LAW
A.3.1 MOTIVATION OF ADDING POWER OF DN TERM
2:38 dat.. 196M EoroIn
2:36 CoEa SOSH PATDRIS
2:38 CAFA, 1072H PUrsIES
chechilla pred 6128 peci
chinchlleppe prod
Chinciila
3:45 (EC). 13EM Earores
5:45 COEO, 5039 p3ra015
6.45 (MEA, 1072M PuEarI
chiechila mare poot
chicchill wan pood
clindill caw prod
33.02 data, 1331 parama
33. datA 305H
32, 2072M
Vallec rane prod
Low Wiler sen
pond
Lakens murderr
Figure 5: The loss curve, Chinchilla prediction, and Equation 11 on 2.1B, 8.4B and 33.6B training data.
In our submission, we present an in-depth analysis of the model's loss dynamics as illustrated in Figure 5 which juxtaposes the empirical loss trajectory (depicted through a blue line) against the theoretical predictions derived from the Chinchilla Law (illustrated by a yellow line) and further contextualized by Equation TT This comparative study spans three distinct datasets- IB, 8.4B, and 33.6B data points-accne across models of varying capacities: 190M, 505M, and 1.07B parameters, respectively, arranged in a matrix of subfigures with datasets delincated by rows and model capaci- ties by columns.
Observations across all data volumes reveal a nuanced interaction between model and data sizes. Specifically, for smaller datasets and model sizes (190M parameters), predictions consistently un- derestimate actual loss values, whereas for smaller datasets paired with larger models (1.07B pa- rameters), predictions tend to overestimate. This discrepancy underscores a critical insight: loss reduction accelerates with increasing model size, suggesting a modified loss function, Atc simpler A N=
over the
"No"
Crucially, the term e emerges as a function of single variable N, ensuring variability in va across each unique model configuration shifting upwards or downwards without changing the shape. The ideal adjustment implies that approaches zero for large datasets, yet remains significant for smaller ones, highlighting its dependency on data volume D.
In addressing potential overfitting, our strategy focuses on minimizing parameter growth in line with Equation []] A straightforward approach involves augmenting the loss L into a polynomial encompassing - and a with Equation introducing an additional term, Da to the existing framework. This refinement, while ostensibly simple, has been shown to yield robust and promising outcomes, exemplifying the efficacy of our proposed modifications in enhancing model performance within the context of scaling laws.
B
17

---

Published as a conference paper at ICLR 2025
A.3.2 MOTIVATION OF LINEAR REGRESSION TERM FOR OVERFITTED RESIDUAL
1013
13
Figure 6: The loss curve, Chinchilla prediction, and Equation 2 (green lines) on 2.1B training data.
Figure 6 offers a detailed exposition on the fidelity of Equation 2 in capturing the loss trajectory across training sets of varied model capacities (190M, 505M, and 1.07B parameters). It is evident from the analysis that the equation adeptly mirrors the empirical loss curve across a broad spectrum of configurations, with the exception of scenarios characterized by concurrently large model sizes and token counts. A notable oversight in the literature the scant consideration of loss dynamics beyond early stopping points, a consideration of paramount importance in music domain due to the inherent constraints on training data.
In addressing the challenges posed by modelling loss post-early stopping, our investigation delin- eates two distinct methodologies. The first approach involves the integration of a regularization term within D aimed at reducing its magnitude beyond the early stopping threshold. Despite its conceptual appeal, this method falls short of providing an adequate fit to the observed data. Alter- natively, we explore the augmentation of the loss function I. with an additional term, engineered to be negligible when both D and N are minimal, yet incrementally assertive in influencing the loss trajectory after early stopping points. This latter strategy not only aligns more closely with empirical observations but also introduces a nuanced mechanism to accommodate the unique requirements of training in the music domain, thereby extending the utility and applicability of scaling laws within this context.
cata, with 1900 pararss
inos data, with SCrM pararss
lnns data,, with 107/2M partars
Los
Vilce
Lesa Veioe
pace
ente pind
number
103731
3.2
Figure 7: Residule between authentical valid loss and Equation 2 prediction (blue lines), and the linear regression results (yellow lines).
As delineated in Figure 7 the analysis of residuals post the 40 billion token threshold unveils a discernible onset of overfitting, which intriguingly appears to correlate with the model size, data capacity, and the count of unique tokens processed within a single epoch. This overfitting is fur- ther characterized by linear dependency of loss the total number of processed tokens, coupled with a quasi- quasi-linear linear transition of carly stopping points observed across different model capacities (as organized in rows) and magnified across columns.
18

---

Published as a conference paper at ICLR 2025
The progression of model capacities doubling across rows and quadrupling across columns-iiluminate illuminattss a systematic pattern, suggesting that the early stopping points and con- sequently, the predicted loss, might be effectively modeled through linear regression involving dataset size D. the logarithm of model capacity log( N)), and and the logarithm of unique tokens per epoch log(Up). This observation culminates in the proposition of regularization term formulated as Ka- D + kn log((N) ky log(Up) overfitting dynamics.
pooubiies-dduulingg
Kins aimed at encapsulating and mitigating the observed
Activation Function
It_ (test)f
Huber Loss (test)
ReLU
0.9786
0.0095
LeakyReI.U
0.9786
0.0095
GELU
0.9780
0,0085
Tanh
0.9786
0.0094
SELU
0.9779
0.010
Sigmoid
0.6030
0.0700
Table 8: Ablition study on the activation function.
In addressing the intricacies of regularization within the context of early model training, especially when considering models of smaller scale (where Un and D are minimal while N is comparatively large), it becomes imperative to ensure that the regularization term does not adopt a substantially negative value. This stipulation aims to prevent undue penalization at the onset of training, thereby necessitating the incorporation of an activation function that tempers the regularization term's 's be- havior. The Gaussian Error Linear Unit (GELU) function emerges as an apt choice in this scenario. GELU approximates the Rectified Linear Unit (ReLU) function for positive inputs, while also per- mitting slight negative values with minimal absolute magnitude, thus offering a balanced solution.
Empirical evidence, as detailed in our analysis, underscores the efficacy of applying the GELU function to the regularization term, notably achieving the lowest training loss alongside the second- highest R2 value among the tested models. This finding is particularly salient given the broader mag- nitude of loss variations relative to R² values, thereby accentuating the GELU function's 's suitability for our regularization term. Consequently, the finalized model, incorporating the GELU-modulated regularization term, is depicted through yellow line in Figure 7 This strategic application of the GELU function not only mitigates the potential for excessive early training penalization but also optimizes the regularization term to enhance model performance effectively.
This approach not only elucidates the linear interdependencies among critical factors influencing model performance but also presents a nuanced regularization strategy designed to enhance model generalizability. Through the integration of this regularization term, we aim establish a more robust and theoretically informed framework for predicting and managing loss trajectories in large- scale training regimes.
A.4 EVALUATION METRICS
The R-squared value, also known as the "Coefficient of Determination," is a statistical measure used to evaluate the goodness-of-fh of regression model. It defined as:
SSSess
R²
(14)
SSm
Where SSres represents the Sum of Squares of Residuals, indicating the total sum of squared differ- ences between the predicted values of the model and the actual observed values, S...o represents the Total Sum of Squares, indicating the total sum of squared differences between the observed values of the dependent variable and their mean value.
The Huber loss is type of loss function commonly employed in robust regression models. Unlike the squared error loss, which is sensitive to outliers in the data, the Huber loss is designed to be less affected by outliers. It achieves this by combining the characteristies of both the squared error loss and the absolute error loss. It is defined piecewise by:
[2 f(z)) 16(1y- - f(x) |-]] 20),, otherwise
if f()) <
H Hubers f(x))
(15)
19

---

Published as a conference paper at ICLR 2025
For small residuals, it behaves like the squared error loss, whereas for large residuals, it behaves like the absolute error loss. This allows the Huber loss to provide a balance between the two, resulting in more robust estimation procedure.
A.5 PARAMETERS FITTING APPROACH
In our study, we adopt methodology analogous to the Chinchilla Law and the Data-Constraine Law, employing the L-BFGS algorithm a limited-memory quasi-Newton method-foo the opti- mization of the Huber Loss. This loss function is applied between the logarithm of the predicted loss and the logarithm of the observed (authentic) loss across multiple runs. The objective to identify the optimal parameters (best_para) that minimize this Huber Loss, formalized as follows:
best.para min Σ
Hubers
log
A
+
B
+
E
log(L,) }
Na
D
Na
D
rumi
A
B DMg
min
Σ
Hubers
LSE
log
.log
log
.lo((E)
.og(LL)
Na D
Na
runi
log(d) a log(VV) log (")
log( A) a log(N))
min Σ
Hubers
LSE
log(Le)
log( B) B Blgg("" log(E)
(16) where LSE refers to the sum-sum-exp numerically stable method to compute the logarithm of exponentials of inputs. The Huber Loss parameter, ō is stringent criterion for switching between squared loss and absolute loss to ensure robustness in optimization. Additionally, the L-BFGS algorithm's learning rate is configured at 1e with an update history size of 10 to balance between computational efficiency and the capacity to capture relevant optimization trends.
of
le 3, reflecting
A.6 RESULTS OF PROPOSED METHODS WITH EARLY STOPS
Paramatic
7t Value (train) 1
Huber Loss (train)
H Value (lest)
Huber Lass (test)
Chinchilla low
0,9443
0.0073
-0.0004
0:0029
Data-Contrained
0.7216
0.0189
0.1005
0.0050
Equation 1
0.8356
0.0151
0.5829
0.0045
Equation
0.9843
0.00088
SMS Law
0.9851
0.0055
0.9864
0,00091
Table 9; Comparison parametric fitting performance of different Scaling Laws on the curve before early stop points.
From the table, we can see that most of the experimental results increase after we delete the curve after the early stop points. Adding the linear regression still contributes to the performance increase on the training set but provides worse results on test set compared to Equation
20

---

Published as a conference paper at ICLR 2025
B TRAINING DETAILS & DATASET
B.1 TRAINING DETAILS
All the models are trained using Adam Kingma & Ba 2014), with 31 0.9, 0.95, eps = 10
We cosine learning schedule, decay the final learning rate from 3-5 3- with warmup ratio of 0.1. We apply a weight decay of 0.1 and gradient clipping of 1.0. Table I0 shows other training details of each model.
Table 10: Training Details for different ABC format and model settings.
Parameters
Context Length
Trained Tokens
Training Days
Num of GPUs
190M
4096
119B
8.4
2
505M
4096
97B
8.4
4
1.07B
4096
49B
8.3
4
1,97B
4096
56B
8.4
8
Original ABC
190M
8192
346B
6.9
8
505M
8192
322B
4.1
32
L.07B
8192
223B
5.4
32
1.97B
8192
196B
8.1
32
190M
8192
276B
5.5
8
505M
8192
212B
2.7
32
SMT-ABC
1,07B
8192
181B
4.4
32
1.97B
8192
272B
11.3
32
4.23B
8192
262B
10.7
64
B.2
ADDITIONAL INFORMATION ON TRAINING SET
Metadata for the training dataset The dataset includes total of more than 1.8 million songs. For the subset of data with genre metadata, the approximate genre distribution is as follows.
Genre
Number of Songs
Pop
256k
Jazz
107k
Country
49k
Rock
217k
Disco
6k
World Music (including Latin)
47k
Folk
1 118k
R&B, Funk & Soul
63k
Classical
466k
Genre metadata in Training Set. Each song can have more than one genre tag, and can also have no tag information.
Table 11:
Due to the lack of comprehensive metadata for the entire dataset, cannot provide precise genre statistics for all songs. Nevertheless, we ensured that the training data is diverse and representative of wide range of musical styles, enabling robust and comprehensive model training. Besides, to address intellectual property concerns, the datasets are sourced from publicly available repositories, ensuring ethical usage and proper citation. The authors explicitly state that the data is used strictly for research purposes, complying with licenses and copyright laws. The dataset is not redistributed, ensuring adherence the terms set by the original sources.
B.3 ABLATION STUDIES ON (SMT-)ABC REPSENTATIONS FOR TRAINING
To validate the effect the SMT-ABC Notation training strategy, which has previously shown advantages in reduced training loss 5.3.1 and higher consistency rate 5.3.2 conduct two exper-
21

---

Published as a conference paper at ICLR 2025
iments: the first evaluates measure consistency in multi-track notations, and the second involves subjective evaluations.
1.0
Original ABC
Measures Consistency Rate
0.8
0.6
0.4
0.2
2k
4k
6k
10k Training Iterations
16k
18k
20k
22k
Figure 8: Measure consistency of SMT-ABC and ABC models in different training iterations. Measure Consistency To assess the measure consistency in generated ABC music sequences, we measure the proportion of sequences where all tracks contain equal number of measures. Fig- ure X illustratee that the sequences generated by the SMT-ABC model demonstrate a significantly higher consistency rate compared those generated by the model trained on Original-ABC nota- tion. This suggests that the SMT-ABC notation facilitates models to maintain structural uniformity across different tracks, which is critical for ensuring the coherence and usability of the generated compositions in practical applications.
Objective and Subjective Evaluation Table MuPT-SMT and MuPT-Ori. represent the SMT- ABC notation and Original- notation respectively. The results show that mostly SMT-ABC performs better than Original-ABC. Meanwhile, we also conduct the AB test of all multi-track samples in the test between these systems and it shows listeners prefer music from SMT- ABC in 53% of cases than Original-ABC. (p = 2. 2.7265 x 10-6).
C FURTHER DISCUSSION ON SYMBOLIC MUSIC TOKENIZATION
While there are many representations for symbolic music besides ABC notation and MIDI, including some adaptations of MIDI such as REMI, etc. (Ma al.] 2024). the following discussion focuses on the vanilla setting of MIDI, ABC notation and our proposed method SMIT-ABC. We leave the ablation study on different symbolic music tokenizers the future work
C.I THE ALIGNMENT AND DIFFERENCE BETWEEN ABC NOTATION AND MIDI
ABC is the music staff/stave in terms natural language notations, including tune info (e.g. title, composer, meter, key, pitches, rhythms, bar lines & repeats, instrument, etc. The "K: Bb" you mentioned in ABC means the key of the music piece is Bb, which is the same as the key of the MIDI note in the piano roll and is related to the overall pitch of the MIDI note in the piano roll. And " F2 pitch F, an highthnnoteqqaaee
z G corresponds to one music bar that includes a quarter rest, and eighth-note at pitch G. MIDI start and stop times are absolute time slots in seconds and do not include quantized beat/har information. This given measure corresponds two MIDI note rectangles the chart that correspond to the pitch and time (the rest is not shown).
The velocity, control information, and also sound effects that are included MIDI are not included in ABC notations. ABC notation includes bar and note quantization information (how many beats v.s. how many seconds per note), key, and music/tune structure such as repeat sign, etc., that MIDI does not include. In general, MIDI can keep more information the styles of performers with acoustic information, and ABC notations are better with modelling repeats and structures, leading to better compression rate and more suitable for LIM training.
22

---

Published as a conference paper at ICLR 2025
C.2 ADVANTAGE OF ABC NOTATION
ABC notation offers significant advantages over MIDI in terms of compression ratio, which directly impacts model efficiency and reduces training costs. As highlighted in ChatMusician, ABC notation achieves an average of 288.2) tokens per song and 5.16 tokens per second, requiring roughly 38% fewer tokens than various MIDI-based representations. This remarkable reduction token count stems from ABC's ability to efficiently encode symbolic musical repetition using concise notations such as repeat signs () and : I) These symbols effectively represent patterns spanning several seconds to minutes. By reducing sequence length, ABC not only lowers computational overhead but also simplifies the learning complexity, making it an ideal format for music-related tasks.
It is undeniable that MIDI offers more detailed dynamic information compared to leadsheet formats like ABC. However, a major drawback of MIDI is that, within a limited context length, it may not be possible to encode an entire song. While hierarchical modeling methods, such as Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models, have been proposed to address this issue, scaling up such approaches is compatible with natural language symbols and naturally encoded structures like repetition, ben- efitting training efficiency, convergence speed, and modelling the overall structure of song with NLP codebase. Moreover, ABC notation integrates seamlessly with text tokenizers, making it an efficient choice. After carefully weighing the pros and cons of both formats and considering the characteristics of our collected dataset, we decided to use ABC notation.
be challenging. In contrast, ABC notation
C.3 COMPARSION BETWERN SMT-ABC AND ABC
Our proposed SMT-ABC notation effectively addresses the alignment challenges in multitrack ABC notation, significantly improving training loss and overall model performance.
One major issue is the measure consistency problem. As shown in Appendix Figure 8, this fig- ure illustrates the measure consistency scores for SMT-ABC and standard ABC notation under the same training iterations and model structures. The results demonstrate that as training progresses, SMT-ABC quickly achieves consistent measures across tracks, indicating effective alignment and coherence. In contrast, models trained solely with ABC notation struggle to maintain measure con- sistency across tracks, ABC in addressing alignment issues and improving overall musical structure.
training iterations increase. This highlights the advantage of SMT-
The concepts of bar coherence and track coherence useful but challenging evaluate directly. Instead, we propose the concept of the closeness of related tokens. Since this is a causal model for pretraining, the later tokens in the sequence are less influenced by earlier tokens as the tokens gap increases. The purpose of designing the SMT-ABC notation is to ensure that bars played in each track at the same time are generated as closely as possible. Thus, focus on discussing the token gaps between different tracks.
In our training dataset, which aligns with the natural distribution of ABC notation, the average tokens per bar is 3.38 tokens, while the average bars per track is 77.40 bars. This means that the benefits of reducing token distances between tracks significantly outweigh those of reducing token distances within a track.
For example, we define the most related bars as: the bar with the same index in different track, and the next bar in the same track. In a 4-track piece of music: the token distance between Track 1, Bar and Track 4, Bar in standard ABC notation is approximately 77.4 x (4 1) = 232.2 tokens. And the token distance to the next bar within the same track is much smaller (3.38 tokens) ).
In contrast, with SMT-ABC notation: the token distance between Track 1, Bar and Track 4, Bar I is reduced to: 3.38 (4 - 1) = 10.14 tokens. However, the token distance to the next bar in the same track increases slightly to 3.38 x 4 = 13.52 tokens This design in SMT-ABC notation brings related bars closer together while slightly increasing the distance for less related bars. This shift in token distribution improves the closeness of related tokens, enhancing the overall coherence and ultimately boosting the model's final performance.
23

---

Published as a conference paper at ICLR 2025
C.4 APPLICATIONS OF ABC NOTATION
Though MIDI is widely used in the music industry, ABC has a strong potential for application in music generation. Leadsheet-based multitrack symbolic music generation is a promising yet un- derexplored area. Leadsheets effectively compress the semantic information of music, facilitating whole-song-lever modeling. Besides, ABC notation is designed with natural language symbols that can be easily combined with natural language. This approach could guide the text-to-music gener- ation models similar to Suno's framework integrating multimodal capability easily. Furthermore, is also worth noting that audio can be generated from ABC leadsheets by using rendering model in various style. For example, Seed-Music: Unified Framework for High Quality and Controlled Music Generation demonstrates this approach, enabling the generation of high-quality audio based on ABC notation. This allows for the inclusion performance characteristics and timbre while maintaining the capability to model entire songs with ease.
D SIMILARITY BETWEEN TRAININGSET AND INFERENCE SAMPLES
We conducted experiments to investigate whether the music generated by the model exhibits any copying from the training data. Specifically, sampled 750 pieces of music with 1-15 tracks and 100 pieces with more than 15 tracks from the training set. For these samples, we used their headers and the first bar as prompts and calculated the similarity between the model-generated music and the original music from the training set. The similarity was measured using three metrics: n-grams (character-based segmentation), n-gram-music (note/chord-basee segmentation), and the longest common subsequence(LCCS) between the two music sequences. The results are shown in the table below.
N- N-gram(n=4) N-gram-Music(n=44) LCS
0.19
0.08
0.29
Table 12: Similarity between Training set samples and model generation.
The results suggest that the model is capable of generating novel sequences rather than simply copying training data. The low n-gram and n-gram-music scores, paired with the moderate LCS score, indicate that the model retains stylistic or structural influences from the training set without overfitting to exact examples.
E HUMAN ASSESSMENT
E. ADDITIONAL INFORMATION OF HUMAN ASSESSMENT
We use WebMUSHRA toolkit (Schoeffler et al. 2018) to conduct a web-based subjective listening AB-test. About the music background of participants, 30% of them beginners, 40% are interme- diates, 25% are advanced and 5% are experts. During the listening test, we ask the participants to choose the better one between pair of music excerpts generated from two randomly selected dif- ferent systems from GT, MuPT, GPT-4, MMT and Random by considering the "Musicality" which indicates the overall perceptive quality of the music. Participants are encouraged to make choice by refering to the guidelines below:
How consistent the music sounds as a whole (e.g., in terms of its melodic contours, rhyth- mic patterns, and chord progression).
How likely the development of the music follows clear structure (e.g. verse-chorus divi- sion, repetitions).
If you cannot understand the two guidelines above, just choose the one from A and you prefer.
We use Fleiss' kappa (McHugh 2012) value to measure the inter-annotatos agreement among 15 participants. A kappa value of 0.5807 was obtained, indicating that the participants least achieved moderate agreement (if the value is 1, the perfect agreement is obtained) by referencing Table [3] with only two participants. This demonstrates acceptable quality of our subjective annotators.
24

---

Published as a conference paper at ICLR 2025
Table 13: Fleiss' kappa metrics interpretation
Subjective example: only for two annotators, on two classes (Landis JRKoch 1977).
Poor agreement
0.01 0.20 Slight agreement
0.21 - 0,40 Fair agreement
0.41 0.60 Moderate agreement
0.61 0.80 Substantial agreement
0.81 1.00 Almost perfect agreement
E.2 CASE STUDY ON GENERATION RESULT OF MUPT AND GPT-4
We show two pairs of examples generated by MuPT and GPT-4, Each pair of examples are generated given the same first bar. For all outputs in ABC notation, we export the corresponding human- readable music sheets here. The single-track samples of MuPT and GPT-4 are shown in Figure ? and Figure and the multi-track samples are shown in Figure [1 and Figure [2] Compared with GPT-4, the MuPT generates a better-structured single-track sample with more harmonious chords and repetition symbols in separated theme and multi-track sample with closer relationship and more similar progression.
140
Intro
A Dm
Dm
Gm
Dm
A
A7
B Dm
Dm
Dm
Gm
Gm
A7
Dm
Figure 9: MuPT single-track example.
J= 140
Intro
Dm
A7
Gm
Dm
A7
Figure 10: GPT-4 single-track example.
F LIMITATIONS
In this paper, introduce the MuPT series, comprising pre-trained models dedicated to symbolic music generation. These models set new standard for training open-source symbolic music GPT. However, models primarily accept input in ABC notations and lack the capability for interactive generation based on human instructions, unlike systems such as Chat Musician Yuan et al. (2024)..
25

---

Published as a conference paper at ICLR 2025
c
c
c
6
6
Dm
Dm
8.
C
A
D7
Figure 11: MuPT multi-trade example.
26

---

Published as a conference paper at ICLR 2025
Am/B
C/G
Dm
FAm
Dm
C
E
Am
F/G
6
Dm
E
Am
E
Dm
Em/G
67/8
9
40
C
Dm/F Dm A
Bi
Fdim/GF
c
Gsus4
G
G/B
C
Am/FG/D
6
2
be
9
Figure 12: GPT-4 multi-trad example.
27


Reviews:
[{"review_id": "cZwG45LH5p", "reviewer": "Reviewer_FKmb", "summary": "The paper presents MuPT a pretrained music transformer for symbolic music generation. The paper is a study of the techniques from LLM literature applied to symbolic music data. The authors discuss a novel data representation for symbolic data: SMT-ABC to take issues related to misaligned multi-track data. The authors also present a scaling law appropriate for symbolic music where the dataset size is typically orders of magnitude smaller than text data that was used to establish the Chinchilla laws. The authors conduct subjective and objective evaluations of their model and compare with other similar models of symbolic music. They find that their model is heavily preferred compared to other models in human studies.", "questions": "Aside from the questions in the weaknesses section:\n- Is there any study to check if the generations are copied from the training data?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The study is conducted in a systematic fashion and covers a lot of the relevant topics that LLM-related literature should talk about, such as the relationship between dataset size, model parameters, etc with the quality of generated data.\nThe quality of the generated music shared in the supplementary material is decent.", "weaknesses": "The authors do not share any samples from the baseline models in supplementary materials which would have helped support the results in the human study.\nThe repetition metrics seem to not be clearly motivated. Why would the average repetition rate be a meaningful metric. Isn't the more important metric the position of the repeats?\nSome writing issues. E.g. missing reference in line 353, missing word in line 505.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RwBKLN3682", "reviewer": "Reviewer_nWKA", "summary": "This paper explores using large models for symbolic music pre-training, proposing ABC Notation as a more compatible format than the commonly used MIDI. To maintain track coherence in multi-track compositions, the authors introduce SMT-ABC Notation. Key contributions include models that handle up to 8192 tokens, covering 90% of their symbolic music dataset, and an examination of the Symbolic Music Scaling Law (SMS Law) on model performance. The experiments show better performance than GPT4.", "questions": "Considering the limitations of ABC notation—such as its inability to represent control information and note dynamics, and its less common use within the music community—these weaknesses could significantly impact the model’s potential for generating symbolic music for potential users (composers, musicians, etc). How do you foresee the future roles and applications of ABC and MIDI representations in music generation?\n\nHow many songs are included in the dataset, and what detailed information is provided for each song? How do the authors address intellectual property concerns?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Studying a foundation model for symbolic music is both interesting and meaningful work.\n\n2. The team put significant effort into analyzing large datasets and training the foundation model.\n\n3. The findings on scaling laws are interesting.", "weaknesses": "1. The research contribution is limited. To enhance the novelty of the approach, consider addressing research questions such as: What factors make ABC better than MIDI, and how does the uniqueness of ABC benefit large models in the feature space? Under what circumstances does the model perform better with ABC notation?\n\n2. The comparison between ABC and MIDI is incomplete. The assumption that ABC is better than MIDI requires a detailed and careful analysis. For example, experiments exploring the following would help clarify the relative strengths and weaknesses of each format for large model training: under the same data size and model architecture, assess whether ABC consistently outperforms MIDI in terms of model efficiency and output quality. Note that MIDI can be represented in various formats, such as piano roll, matrix formats like Compound Word, or sequences like REMI. For Figure 1, further details would improve the visualization, such as comparing ABC directly with MIDI instead of only with the piano roll, as the piano roll omits much information found in MIDI. Indicate to readers what information is missing from both MIDI and ABC, what MIDI can represent (e.g., velocity, control information) that ABC cannot, and vice versa. Additionally, clarify the correspondence between ABC and MIDI, such as how \"K: Bb\" in ABC notation relates to MIDI, or what \"|F2 z G|\" represents in ABC and its MIDI equivalent in the figure.\n\n3. The experimental design has flaws. When comparing music-related performance, the proposed model should be evaluated against symbolic music generation models rather than language models. Relevant symbolic music generation baselines include Music Transformer [1], Compound Word Transformer [2], and RWKV [3] on the same large symbolic music dataset.\n\n\n4. typos such as Line 353.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bycaEA2JeY", "reviewer": "Reviewer_f4e7", "summary": "The paper \"MuPT: A Generative Symbolic Music Pre-Trained Transformer\" explores the use of Large Language Models (LLMs) for symbolic music generation, with a focus on leveraging ABC notation over the traditional MIDI format. The authors propose the Synchronized Multi-Track ABC Notation (SMT-ABC Notation) to improve coherence across musical tracks. They present models capable of handling up to 8192 tokens and introduce the Symbolic Music Scaling Law (SMS Law) to optimize the training of these models. Their MuPT model outperforms GPT-4 and ChatMusician in both objective metrics and subjective evaluations of music generation, and they have open-sourced their models to encourage community-led research.", "questions": "1. The paper's experimental design is flawed due to insufficient comparison with other models to validate SMT-ABC's performance improvement. It's unclear if models like GPT-4 and MMT were trained on the SMT-ABC dataset, questioning the true effectiveness of SMT-ABC's innovation.\n\n2. The Scaling Law experiments, while detailed, do not clearly show their relevance to music domain issues, suggesting they may not be significantly pertinent to the paper's objectives.\n\n3. (1) The training data lacks metadata, potentially limiting model generalization. (2) The model may struggle with complex musical structures like polyphony. (3) There's a lack of analysis on model architectures' strengths and weaknesses in symbolic music. (4) Subjective evaluations might be biased, necessitating more diverse assessments. (5) Repeated data may improve performance but could also cause overfitting; more experimental evidence is needed to support claims on overfitting prevention.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The introduction of SMT-ABC Notation represents a novel approach to enhancing the coherence of multi-track symbolic music generation. The use of ABC notation as a foundation for LLMs in music generation, instead of the more commonly used MIDI, adds an original perspective to the field. The concept of the Symbolic Music Scaling Law (SMS Law) is also a significant contribution, offering new insights into the training dynamics of symbolic music models.\nThe paper is well-structured, with clear explanations of the SMT-ABC Notation and the SMS Law. \nThe open-sourcing of the models and intermediate checkpoints is a valuable contribution, enabling further exploration and innovation in symbolic music modeling by the broader community.", "weaknesses": "1. The experimental design is questionable. SMT-ABC is the only methodological innovation in this paper, yet there is insufficient comparative experimentation to explore how SMT-ABC actually improves performance. Models like GPT-4 are not specifically designed for symbolic music generation, and it's predictable that the proposed model would perform better without even conducting experiments. Were the other models in the comparative experiments, such as GPT-4 and MMT, trained or fine-tuned on the SMT-ABC dataset? If not, it may not demonstrate the true effectiveness of the algorithmic innovation of SMT-ABC.\n\n2. The Scaling Law experiments are detailed, but they primarily validate some emergent performance issues that have been verified in other domains. It does not clearly indicate whether the issues in the music domain can be resolved through the Scaling Law, which suggests that the detailed Scaling Law experiments may not be significantly relevant to the problems this paper aims to address.\n\n3. (1) The training data lacks specific metadata, such as detailed categorization and style of music, which may limit the model's generalization ability in specific styles or types of music. (2) The paper requires adjustments for the alignment of different tracks, and the model may still have certain limitations in generating more complex musical structures, such as polyphonic music. (3) There is no in-depth analysis of the detailed differences between different model architectures, their respective strengths and weaknesses in handling symbolic music, which may make it difficult for readers to understand the specific improvements of MuPT over other models. (4) Subjective factors such as the musical background and preferences of listeners may affect the results of subjective evaluations, and more diverse assessments might provide more comprehensive conclusions. (5) Although experiments show that repeated data can improve model performance, it may lead to overfitting in some cases. Although the paper discusses early stopping and overfitting, more experimental support is needed for the effectiveness of these explanations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gd1yLK1Hxt", "reviewer": "Reviewer_PkBU", "summary": "The paper introduces a Synchronized Multi-Track ABC to improve coherence across tracks in music generation, addressing the limitations of traditional MIDI formats. Combining with the LLM,  MuPT outperforms other models in terms of repetition rate and structure in multi-track music generation. The paper also investigates the Symbolic Music Scaling Law, showing improvements in performance as model size and training data scale up. However, the novelty of the model is limited, and the effect of the proposed SMT-ABC representation seems not very significant.", "questions": "1. What is the model scale of SMT-ABC and Original-ABC in Figure 8 in Appendix B.2?\n2. Since GPT-4 is not a model for symbolic music generation, what is the motivation to compare with GPT-4?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces SMT-ABC, a novel representation that improves coherence across multi-track music generation.\n2. The investigation into SMS Law offers valuable insights into the relationship between data repetition and model performance, providing practical guidelines for optimizing large-scale music generation models.", "weaknesses": "1. There are some mistakes like missing reference targets (e.g. in Section 5.1 ‘please refer to ??’).\n2. Although the SMT-ABC representation is interesting by adding a special token ‘<|>’ and changing the sequence of the original ABC notations, the novelty of this paper seems limited, for the model architecture is mostly based on existing works.\n3. Changing the sequence of the original ABC notation from bar-sequence to track-sequence may preserve coherence across multiple musical tracks. However, what about the coherence in a track? The contents inside a bar seem to be adjusted to the very back of the input sequence because SMT ABC aligns the tracks first. Besides, from Table 5, the improvement of SMT-ABC seems very slight compared with the original ABC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)