Byte Pair Encoding for Symbolic Music
Authors: Nathan Fradet
Nicolas Gutowski
Fabien Chhel
Jean-Pierre Briot
Venue: EMNLP 2023 Main (2023)
Keywords: Symbolic music
BPE
Music generation
MIR
TL;DR: The paper shows that when applied to symbolic music, Byte Pair Encoding improves the results of generation and classification tasks while increasing the inference speed, and leading to better the learned embeddings / token representations.

Abstract: When used with deep learning, the symbolic music modality is often coupled with language model architectures. To do so, the music needs to be tokenized, i.e. converted into a sequence of discrete tokens. This can be achieved by different approaches, as music can be composed of simultaneous tracks, of simultaneous notes with several attributes. Until now, the proposed tokenizations rely on small vocabularies of tokens describing the note attributes and time events, resulting in fairly long token sequences, and a sub-optimal use of the embedding space of language models. Recent research has put efforts on reducing the overall sequence length by merging embeddings or combining tokens. In this paper, we show that Byte Pair Encoding, a compression technique widely used for natural language, significantly decreases the sequence length while increasing the vocabulary size. By doing so, we leverage the embedding capabilities of such models with more expressive tokens, resulting in both better results and faster inference in generation and classification tasks. The [source code is shared on Github](https://github.com/Natooz/bpe-symbolic-music), along with a [companion website](https://Natooz.github.io/BPE-Symbolic-Music). Finally, BPE is directly implemented in [MidiTok](https://github.com/Natooz/MidiTok), allowing the reader to easily benefit from this method.

Byte Pair Encoding for Symbolic Music
Nathan Fradet 122 Nicolas Gutowski', Fabien Chhel4.3, Lean-Pierry Briot
Sorbonne University, CNRS, LIP6, F-75005 Paris
2 Aubay, Maulogne-BBilancouus France
3 University of Angers, LERIA, 49000 Angers, France
*ESEO, ERIS, 49100 Angers, France
Abstract
punctuation mark. For symbolic music, tokens usu- ally represent note attributes or time events, such as pitch or duration. Tokenizing music, i.c., con- verting raw data into tokens, can be achieved by several ways, as music can be composed of simul- taneous tracks, of simultaneous notes with several attributes such their pitch and duration. Multiple approaches exist represent these features.
When used with deep learning, the symbolic music modality often coupled with language model architectures. To do so, the music needs to be tokemized, i.. converted into a sequence of discrete tokens. This can be achieved by dif- ferent approaches, as music can be composed of simultaneous tracks, of simultaneous notes with several attributes. Until now, the proposed lokenizations rely on small vocabularies of to- kens describing the note attributes and time events, resulting in fairly long token sequences, and a sub-optimal use of the embedding space of language models. Recent research has put efforts on reducing the overall sequence length by merging embeddings or combining tokens. In this paper, show that Byte Pair Encod- ing, compression technique widely used for natural language, significantly decreases the se- quence length while increasing the vocabulary size. By doing so, we leverage the embedding capabilities of such models with more expres- sive tokens, resulting in both better results and faster inference generation and classification tasks. The source code is shared Github!, along with a companion website". Finally, BPE directly implemented MidiTok allowing the reader easily benefit from this method.
Recently, the token representation of symbolic music has been studied (Fradet al., 2023), with the goal improve 1) the results, e.g. the quality of generated music or the accuracy of classification tasks, and; 2) the efficiency of the models. The former is tackled with more expressive represen- tations (Huang and Yang, 2020; Kermarec et al.. 2022; von Rütte et al., 2023; Fradet et al., 2021), and the latter by representations based either token combinations (Payne, 2019; Donahue et al., 2019). or embedding pooling (Hsiao al., 2021; Zeng et al., 2021; Ren al., 2020; Dong et al., 2023), which reduce the overall sequence length.
Still, these tokenizations are based on tokens only representing the values of time events and note attributes. This comes with big limitation: these tokens do not carry much information by them- selves. We can assume that their embeddings does not either. By analogy natural language, these tokens closer the character level than word level. Yet, a powerful feature of LMs is their ability to learn (embedding) representations of discrete ele- ments such as tokens, and leverage this information for reasoning and downstream tasks. For natural language, LMs are usually coupled with vocabular- ies containing up to 50k tokens, represented on a few hundreds dimensions (often between 512 and 2048). Using vocabulary containing fewer tokens than the number of dimensions used represent them appears as a suboptimal usage of such mod- els. Moreover, the expressive information carried by music is deduced from the combinations its notes and their attributes. Considering the infinite
1 Introduction
When used with deep learning, the symbolic music modality mostly represented as discrete and used with language models (LM) such as Transformers (Vaswani et al., 2017). These models receive se- quences of tokens input, convert them to learned embedding vectors representing their semantic fea- tures continuous space, and process these em- beddings for the task at hand. A token a distinct element, known within finite vocabulary. For nat- ural language, a token can be a word, subword or
'https: /ottps://Natooz.gg. io//oee "https://gitee httpa//...oom/atoo///iiiiii

---

possible music arrangements, we can suppose that using solely nole attribute embeddings imposes to LMs a heavier modeling effort than embeddings of potential whole note successions that would be more expressive and explicit.
REMI
Pitch D3
22
Dur 7
Pitch A3
Vel. 24
Pos. 8
Pitch G3
Vel. 26
Bar
Pos.
Pos
Bar
Ve1
REMI + BPE
BPE 2.0
Vel: 22
BPE 20M
BPE W.HUZ
UPE. D- 90.3.24
In this paper, we show that Byte Pair Encoding (BPE, described in Section 3) applied to symbolic music allows address the two goals just men- tioned, while outperforming the previous methods and making the model learn better distributed em- beddings. To the best of our knowledge, BPE has yet not been studied for the symbolic music modal- ity, although can be applied on top of any music tokenization that does not perform embedding pool- ing. This work aims at closing this gap by shedding light on the results and performance gains of using BPE:
Bar
Bar
Pos.
289
Embedding pooling
Bar
Pos. 0
Pos.
Pos.
Pitch D3
Pitch A3
Pitch G3
Vel. 22 Vel. 24
Vel. 26
Dur. 7
Dur.
Dur. 3
Figure 1: Three tokenizations of the same three notes. Tokens are ordered from left to right, the numbers put below are their integer ids. Top row REMI (Huang and Yang. 2020), middle correspond to the top row with BPE applied to some tokens, bottom row a tokenization similar to (Zeng al., 2021; Dong et al., 2023) where the embeddings are merged (pooled).
We experiment on four public datasets (Wang et al., 2020b; Kong et al., 2021; Ens and Pasquier, 2021; Hung et al., 2021), with two base tokenizations, on which BPE learned with several vocabulary sizes, on generation and classification tasks;
2.2 Sequence length reduction strategies
More recent works put efforts on the efficiency of the models. As most of them rely on the Trans- former architecture (Vaswani et al., 2017) and the attention mechanism, their time and space complex- ity grows quadratically with the input sequence length. This bottleneck led researchers to work more efficient attention estimations (Tay et al., 2021), 1) down to linear complexity. In the field of symbolic music specifically, researchers worked on strategies to reduce the sequence length order to increase 1) the efficiency of the models; 2) the scope of the attention mechanism; 3) the quality the generated results. These strategies can be split in two categories:
We compare BPE with other sequence reduc- tion techniques introduced in recent research;
We study the geometry of the learned embed- dings, and show that BPE can improve their isotropy and space occupation;
We show some limits of BPE, such the proportion of sampled tokens, and that the vocabulary size has be carefully chosen.
2
Related work
2.1
Tokenization of symbolic music
Most deep learning models using symbolic mu- sic generation
embedding pooling such as Compound Word (Hsiao et al., 2021) (CPWord), Octuple (Zeng et al., 2021), PopMag (Ren et al., 2020), Sym- phonyNet (Liu et al., 2022) or MMT (Dong et al., 2023). Embeddings of several tokens merged with pooling operation. This is often done by concatenating the embeddings and projecting the vector, resulting in an ag- gregated embedding of fixed size.
specific music tokenization. Early research introduced representations specif- ically tied the training data being used, such as DeepBach (Hadjeres et al., 2017), FolkRNN (Sturm et al., 2015) or BachBot (Liang et al., 2017). Non-autoregressive models such as MuseGAN (Dong et al., 2018) often represent music pi- anoroll matrices.
Since, more universal representations have been studied, allowing to convert any sequence of (si- multaneous) notes into tokens (Oore et al., 2018; Huang and Yang. 2020; Hadjeres and Crestel, 2021; Fradet et al., 1). Some of them are depicted in Figure I.
token combination such in MuseNet (Payne, 2019), LakhNES (Donahue al., 2019) or other recent works (Liu et al., 2022: Thickstun et al., 2023), which consists of us- ing a vocabulary of tokens representing sev-

---

eral attributes, e.g.. Pitch-x_Dur-y repre- senting both the pitch and velocity informa- tion. BPE can be seen as learned token combination technique.
3 Byte Pair Encoding
Byte Pair Encoding (BPE) (Gage, 1994) is data compression technique. It converts the most current successive bytes in a corpus into newly created ones. For instance, in the character se- quence aabaabaacaa, the sub-sequence aa occurs three times and the most recurrent one. Learn- ing and applying BPE on this sequence would re- place aa with a new symbol, e.g., d, resulting in compressed sequence dbdbdcd. The latter can be reduced again by replacing the db subsequence, giving eedcd. In practice BPE is learned on cor- pus until the vocabulary reaches target size. BPE learning described by the pseudo-code of Algo- rithm I.
2.3 Limitations
One of the main limitation of the previous work is the suboptimal usage of the embedding space of LMs. Most of them use models with embed- dings represented from 512 to 1024 dimensions, for vocabularies of less than 500 tokens. As the model contextually learns to gather embeddings along dimensions representing learned features, us- ing a number of dimensions larger than the num- ber of elements to represent causes embeddings not take advantage all the space of the embed- ding dimensions, which will stay unoceupied for a large proportion. For comparison, the same models, when trained on natural language data, use to learn up to 50k embeddings on 512 to 1024 dimensions. The sequence length reduction strategies just introduced also have big limitations. Embedding pooling: 1) requires specific model input and out- put modules, which can break compatibility with popular software libraries; 2) requires multiple losses at training, which increases the complex- ity: 3) for generation, inferring from such model can be seen sampling from a multivariate distri- bution, which can be very delicate, 4) the results can easily degenerate if the pooling does not yield semantically rich embeddings that represent the underlying tokens. On the other hand, token com- binations of entire types of tokens can lead to large vocabularies with unused tokens and potentially non-optimized or unbalanced token distributions. To the best of our knowledge, no work has been conducted on applying BPE (introduced in Sec- tion 3) to symbolic music generation. Although (Liu et al., 2022) introduced method which they named MusicBPE, this technique links weakly with BPE and has a limited scope. It adds to the vo- cabulary new tokens for recurrent chords. These tokens represent pitch combinations for simulta- neous notes having the exact same velocity and duration. It can only be used for limited pro- portion of notes (and in turn tokens), actually less than quarter when strong downsampling is ap- plied (Appendix D). As it does not apply on token successions, it cannot capture the contextual and probability relations between them, including time dependencies. For these reasons, we do not com- pare it with BPE as would not be relevant.
Algorithm 1 Learning of BPE pseudo-code
Require: Base vocabulary V, target vocabulary size N, dataset X' while V N do
Find m {41 (2), the most recurrent
ken succession from X
3:
V- + [4 : m]
4: Substitute occurrences of m in X' with fpe
5: end while
6: return V
BPE is nowadays largely used in the NLP field as it allows to encode rare words and segment- ing unknown or composed words as sequences of sub-word units (Sennrich et al., 2016). Other to- ken aggregation, or vocabulary building techniques exist. The two other most commonly used are Uni- gram (Kudo, 2018) WordPiece (Wu et al., 2016), which operations share similarities with BPE.
For natural language, bytes are the distinct char- acters composing the text. For symbolic music, the distinct note and time attributes can be used as the "bytes" to merge. In this context, BPE can allow to represent a note, or even a succession of notes, that is recurrent in the dataset, as a single token. For instance, note that would be tokenized as the succession of tokens Pitch D3, Velocity_60 Duration could be replaced by a single new one. Rare note (and attributes) can still be tok- enized as non-BPE tokens. The same logic applies to time tokens, that can also be associated to note tokens.

---

4 Experimental settings
Doyg toleen comb.
12
8 15 12
4.1 Models and training
As specifically focus LMs, experiment with the state of the art deep learning architecture for most NLP tasks at the time of writing, the Trans- former (Vaswani et al., 2017). We use the GPT2 (Radford et al., 2019) and BERT (Devlin et al.. 2019) implementations of the transformers library (Wolf ct al., 2020) for respectively music genera- tion and classification. They are made of 12 layers, embedding sizes of 512, eight attention heads and layers of 2048. They count approx- imately 40M learned parameters. The generators are trained with teacher forcing with the target se- quence being the input shifted by one to the left. The classifier are first pretrained to retrieve random- ized tokens, and then finetuned to classify the input sequences. More details on the training procedure can be found in Appendix A.
Mird din PF1
Vacabulary
Vecabulary
(a) Avg. nb. combinations (b) Max. nb. of combinations
Figure 2: Average and maximum number of token com-
binations of tokens learned with BPE in function of the vocabulary size.
slor
Inlenns Beot (U)
Tsk. Gime (c)
time (1)
Steategy
No BPE
140
162
18.5
0.174
0.031
(8(68.9))
69(633993
0,53
PVa
(122
138::377455)
0.134
0.04
20185
28198
(353.55
8.6)5488)
0.119
0.25
CP
Octuple
241
(.2(-72
Table 1: Vocabulary size, average tokens per beat ratios, and average tokenization decoding times second using MidiTok (Fradet et al., 2021) and the Hugging Face tokenizers'Iibrant on the Maestro dataset.
All models receive sequences of 256 to 384 to- kens, beginning with special BOS (Beginning of Sequence) and ending EOS (End of Sequence) to- kens. We split datasets in three subsets: one only used for training to update the models, one for vali- dation during training, one used to test the models after training. The last two represent respectively 10% and 15% of the dataset for classification and 2% and 5% for generation.
distinctive. We detail our downsampling strategy in Appendix C.
We choose to experiment with five vocabulary sizes: without BPE, 1k, 5k, 10k and 20k tokens.
Finally, we compare BPE with other sequence reduction strategies introduced in Section 2.2. We experiment with merging Pitch and Velocity ty to- kens (PVm), and Pi Pitch, Velocity and Duration together (PVDm). PVm is similar to the strategy used with MuseNet (Payne, 2019). We also experi- ment with embedding pooling strategies CPWord (Hsiao et al., 2021) and Octuple (Zeng et al., 2021) that we group with REMI in our experiments they represent time similarly. We use the same pooling strategy, and sample independently from the logits of each output modules. All embeddings have the same size than the model dimension.
4.2 Tokenization
We experiment with REMI (Huang and Yang. 2020) and TSD (for Time Shift Duration) as base tokeniza- tions, on top of which BPE applied. Both tok- enizations describe notes as a succession of Pitch, Velocity ty and Duration tokens. REMI represents time with Bar and Posi tion tokens, which respec- tively indicates when a new bar is beginning and at which position within the time is. TSD represents time with TimeShif ft tokens, indicating explicit time movements. For the multitrack MMD dataset, we prepend a Program token before the Pi tch to- ken of each note to represent its instrument.
BPE learning
BPE allows to significantly reduce the sequence length. As shown in Figure 2, the ratio of average number tokens representing a beat can be reduced up to more than 50%. As BPE replaces recurrent pair of bytes in the data, the average number of byte combinations of the vocabulary tends to first quickly increase, then more slowly grow. The in- verse tendency can be observed the tokens per
When tokenizing symbolic music, continuous characteristics are usually downsampled to discrete sets of values (Huang and Yang, 2020; Oore et al., 2018; Hadjeres and Crestel, 2021). For instance, velocities can be downsampled from 128 32 values. These sets should be sufficiently precise to keep the global information. Downsampling these characteristics helps models to learn more easily, as the values of the reduced sets will be more
chttps://gite htw.attww.o.omuugggggg////teeeee

---

beat ratios shown in Table 1, while showing that BPE increase only slightly the tokenization time. The maximum number of byte combinations varies depending on the data. Here, the MMD dataset allows to learn much more combined tokens. This shows that the Maestro dataset contain much di- verse token successions, which is not surprising considering that it is made of classical music while MMD contains many genres, among which some with very repetitive patterns. The tokenization time with BPE naturally increases, but stays relatively close to the basclines without.
at same time, or to not go back in time. Successions of incorrect token types can be inter- preted as errors of prediction. These errors can help us to measure a model has efficiently learned the music representation and if it can yield coherent results, or not otherwise. From this motivation, we introduce new metric we called Tokenization Syntax Errors (TSE).
We distinguish five categories of errors:
TSEtype the predicted token is of an invalid type regarding the previous one;
Appendix E complements this analysis by shed- ding light on the types of the underlying tokens represented by the newly learned tokens.
TSEtime predicted Posi tion value is in- ferior or equal to the current one, making the time goes backward;
6 Music generation
TSEdupa (duplicated note): when the model predicts a note that has already been played at the current moment (by the same instrument);
Music generation is a popular application of deep learning models (Briot et al., 2020; Briot, 2021). We ought to experiment on this task to demon- strate the benefits of BPE music modeling. For this task, we choose to use the Maestro dataset (Hawthorne ul., 2019), which is made of Ik pairs of audio and MIDI files of classical piano perfor- mances. Each MIDI file is made of one piano track, with dynamic melodies and complex harmonies. We generate autoregressively the next 512 tokens of input prompts from the test subsets of the Gi- antMIDI dataset, filtering the logits by keeping the top p 0, 95 probability mass (nucleus sampling (Holtzman et al., 2020)) and top 15 token probabil- ities (top-k sampling (Fan al., 2018)).
TSE (no NoteOff): when using NoteOn and NoteOff, and that NoteOn token has been predicted with no NoteOff later to end it, or too distant in time;
TSEnonn (no NoteOn): when NoteOff to- ken is predicted but the corresponding note has not been played.
For a given sequence of tokens, TSE measures the ratio, scaled between 0 and 1, of errors for these five categories. A TSE of 0 means that there no error in the sequence, while a ratio of means only errors were predicted. Our experiments are not concerned by the last two categories as we do not use NoteOff tokens.
Evaluation of symbolic music is still an open issue (Yang and Lerch, 2020). In the absence of automatic metrics measuring the distances between subsets of data, most works evaluate generated re- sults with human surveys along with feature simi- larity metries. The latter however cannot capture the quality of music, and subject to irregularities in case of model over or underfitting. We decide here to replace them with an metric measuring the errors of prediction of the models.
Finally, we should mention that most of these errors can be avoided by ruled-based sampling. When predicting a token, possible to track the time, notes played and token types to automatically exclude invalid predictions. In practice, this can be achieved by setting the invalid indices of the predicted logits to - oc before softmax.
6.2 Human evaluations
6.1 Tokenization syntax error
For both TSD and REMI tokenizations, we selected about 130 prompts of 4 bars from the test subset, and generated continuations of 512 tokens with all models. We gathered nine participants, among which seven are musicians, to evaluate the results. They were asked to open the MIDI files with Dig- ital Audio Workstation (DAW) softwares such as Logic Pro or Ableton, play each track individually
Every music tokenization has an underlying syn- tax of token type and value successions, that can normally happen. For instance, if the last token of an input sequence is of type Pitch, some tokeniza- tion could imply that the next token to be predicted must be of type Velocity ty. We could also expect a model to not predict more than once the same note

---

TSEepe
TSEdupo()
TSEmel()
Hum. Fidelity (1)
Hum. Correctness (1)
Hum. Diversity (1)
Hum. Overall (1)
Strategy
TSD
REMI
TSD
RFMI
TSD
REMI
TSD
RFMI
TSD
REMI
TSD
REMI
TSD
REMI
No BPE
1.53
1.34
4.19
5.59
28.93
4.9%
4.0%
2.0%
2.0%
1.0%
0.0%%
4,8%
0.0%
BPE 1k
0.62
3.60
4.16
34.65
13.6%
11.9%
11.8%
14.9%
10.8%
6.8%
8.6%
8,
BPE 5k
0.31
0.38
3.28
4,10
39.25
21.4%
31.7%
20.6%
21.8%
11.8%
11.7%
20.0%
18. 1%
BPE 10k
0.49
1.04
3.83
6.39
48.16
23.3%
20.8%
29.4%
22.8%
18.6%
20.45
22.9%
29.5%
BPE
0.38
0.64
4.09
3.60
52.00
29.1%
19.8%
29.4%
24.8%
36.3%
34.0%
30.5%
30.5%
PVm
2.45
2.99
16.90
16.35
36.31
2.9%
2.0%
2.9%
0.0%
7.8%
2.9%
4.8%
1.0%
PVDm
0.63
6.32
2.84
10.64
46.75
4.9%
9.9%
3,9%
11.9%
13.7%
21.4%
8.6%
12.4%
CPWord
6.15
28.55
62.15
0.0%
2.0%
2.9%
0.0%
Octuple
244.11
305.43
0.0%%
0.0%
0.0%
0.0%
Table 2: Metries of generated results. TSE results are all scaled at e 3 for better readability. Hum stand for human, for non-concerned (i.e. 0).
tok/iec (f)
beat/see (1)
notolsee (1)
Vor. sampled (1)
We gathered total of 814 human preferences, with a bit more than 100 for each criteria for TSD and REMI. There is a clear preference for results
Strategy
TSD
RFMI
TSD
RFMI
TSD
RFMI
TSD
REMI
No
4.5
9.9
10.6
10.9
1005%
100%
BPE Ik
73.5
67.0
13.0
17.9
20.8
16.8
100%
99.9%
BPE
12.8
26.7
with BPE, especially with vocabularies of 10k and 99.9%
BPE 10k
97.5
85.4
12.5
26.0
26.3
21.3
99.9%
12.9
31.5
20k tokens. Baselines without BPE still accounts for a few preferences for the fidelity and correctness criteria, but are less preferred overall, especially with REMI. We note that the PVDm baselines show competitive preferences with BPE baselines, espe- cially for diversity. Octuple and CP Word perform poorly on the other hand, which is not surprising as they are not 100% autoregressive, and the sense of the combinations of tokens sampled uncondition- ally is likely to degenerate, especially when time and notes are handled all at once. Overall, BPE helps models to generate more natural and pleasant music. The new contextually learned embeddings may represent richer and more explicit information, helping model the musical information.
PVm
59.3
58.1
8.2
12.2
15.9
14.9
99.3%
99.0%
PVDm
11.4
CPWord
75.8
15.2
19.0
36.7%
Octuple
58.5
Table 3: Inference speeds on a V100 GPU and a batch size of 64 and ratio of vocabulary sampled during gener- ation. For tok/sec, the results account for "basic" tokens of note attributes and time. Tok/sec for Octuple is not calculated as the equivalent number of base tokens is not clearly deducible.
and select the best one on four criteria: 1) fidelity on pitch scale and rhythm regarding the prompt: 2) correctness, i.e. featuring good note succession and harmony; 3) coherent diversity, i.e. featuring diverse correct melodies and harmonies; 4) their overall subjective preference. The advantage of using DAW software is twofold: it allows to con- veniently listen the different tracks, and compare them by also visualizing them as pianorolls. You find more details the human evaluations in Appendix F, and all the generated samples used on the demo website (Section l).
Besides results quality, the second big advantage of BPE is the inference speed increase. We reported three inference metries tokens, beat and note per second in Table 3, along with the proportion of the vocabulary ever sampled by the models.
We first highlight that models with BPE, up to the maximum vocabulary size tested here, do use most of the tokens of the vocabulary, with a slight decrease as the vocabulary grows. This also sup- ports that the vocabulary can easily be scaled while keeping tokens that are still used by the models.
6.3 Results and analysis
The TSE and human preferences results are re- ported in Table 2.
As BPE creates new tokens that combine several types and values, it also increases the overall com- plexity of music modeling when using these tokens. Thus, we initially expected the generative models predict higher ratios of errors. But surprisingly. it decreases these ratios, except for the time errors with REMI. These results show that the models eas- ily capture the information of the tokens, and that the vocabulary can be scaled consequently.
BPE increases all inference speeds measured by at least two times, even with small vocabularies. We note that the increase of beat/sec does not in- crease linearly with the vocabulary size, which also indicates that the models predict higher number of notes per beat. CP Word, despite having low to- kens per beat ratios (Table 1), yields lower tokens per second generation speeds, due to the additional input and several sampling steps.

---

Genre (1)
Artist (1)
Con/ Macste
PL. PL/MDD
Gea Mecirr
PL HMM
Gen/ Moctto
PL. MM/DD
Strategy
REMI
REMI
No DPE
0,209
0.883
01225
0.330
62
66
344
5.2
8.1
7.9
No BPE
0.836
0.796
0,907
0.876
BPE 16
0.110
0.553
0091
0.586
100)
55
115
102
2.3
6.1
15.5
12.2
BPE 1k
0.882
0.871
0,934
0.920
381:
01982
5989
0.985
148
119
18.7
13.1
BPE 5k
0,901
0.875
0,933
0.925
RPE
6.903
8.95
197
157
165
PY
01999
0061
0.883
71
74
139
14.1
BPE 10k
0,904
0.869
0.937
0.922
PVTn
0.655 0:517
0.88
0
58
14
57
44
44
26.1
226
BPE 20k
0,851
0.877
0,909
0.923
PVm
0,853
0,810
0,905
0.886
Table 5: Isoscore, and intrinsic dimension (ID) estima- tions. Gen. corresponds to the causal generative models, Pt. to the pretrained bidirectional models.
PVDm
0.875
0.818
0,914
0.893
Octuple
0.923
0.941
Table 4: Average accuracy of classification models.
8 Learned embedding spaces
7 Classification
We have shown so far that BPE improves the results of music modeling on the generation and classifi- cation tasks. Our assumption is that, non-only the reduced sequence length allows to pack more in- formation (longer music piece) within the same number of tokens, but mostly the vocabulary can be scaled while making the model efficiently learn and use the embedding representations of the newly created tokens with BPE.
For our classification task, we experiment with the MMD dataset (Ens and Pasquier, 2021). It is, to our knowledge, the biggest MIDI dataset publicly available. It features more than 430k MIDI files of all genres of music with multiple tracks. Each piece is matched to Spotify and MusicBrainz ids, allowing to link them with a wide variety of in- formation such as artist or music genre. In order to get more quality training corpus, we perform a preprocessing step which deduplicates the files of the same music and keeps only the best. This is explained in Appendix B. We also merged the tracks of the instruments of the same class in order to reduce the overall complexity (Appendix C).
The embedding space, i.e. the way LMs learn to represent tokens into continuous space Rd of d dimensions, has recently been studied (Gao et al., 2019; Bis al., 2021: Cai al., 2021). More specifically, it has been shown that most LMs learn anisotropic embedding distributions (Ethayarajh. 2019; Reif et al., 2019), despite that their isotropy have been linked to improved performances on downstream tasks (Gong et al., 2018; Wang et al., 2020a; Biś et al., 2021; Liang et al., 2021; Rajaee and Pilehvar, 2022).
To handle multiple tracks, we placed Program tokens before each Pitch token of cach note to specify its instrument. This strategy is similar to REMIPlus (von Rütte et al., 2023).
We perform genre and artist classification, from the 40 and 100 most present genres and artist in the MMD dataset. The results, reported in Table show that BPE improves the models performances compared to the baselines without, and outperform the other token combination techniques. The mod- els seem to benefit from larger vocabulary sizes. It however shows limits, as the accuracy does not in- crease from a vocabulary of 10k to 20k tokens. The Octuple baseline outperforms the others. Here, the model is bidirectional (no attention mask) and do not sample from multiple distributions. Our as- sumption is that the reduced sequence length allows to carry more information within a same number of tokens, allowing the models to capture more easily the global melody, harmony and music structure and directly improving their performances.
Isotropy is measure of the uniformity of the space occupied by a manifold across all dimen- sions. A high isotropy is associated with a uniform variance of the distances between the points of the manifold across all dimensions. In our case the manifold is the collection of contextually learned embeddings X Vxd where V is the vocabu- lary size and d the model/embedding dimension. An isotropic embedding space can be viewed as a space where the embeddings are uniformly spaced with uniform densities.
Isotropy is often estimated by different ways: singular value decomposition (SVD) (Bis et al., 2021; Gao et al., 2019; Liang et al., 2021; Wang et al., 2020a), intrinsic dimension (Cai et al., 2021), partition function (Arora et al., 2016; Mu and Viswanath, 2018), average cosine similarity (Lihh- yarajh, 2019). We chose the two firsts, along with IsoScore (Rudman et al., 2022) which alle- viates some of their shortcomings, to have results
concurs with our results, and is explored in the next section by analyzing the geometry of the learned embeddings.

---

Noare B3
No Stt
872 24
8 36
Singular value
Sinqutor value
B75
0.0
Sinquler
SPC 1k
Pr
Pim
0.4
FV3n
0.4
P/Dm
PiEm
0.2
0.0
107
10
18
10
30
10
10
10
12
17
10
10'
Dimension
Dimension
Dimension
Dimension
Pretrained + TSD
Pretrained REMI
Generator + TSD
Generator + REMI
Figure 3: Normalized singular values of the embedding matrices. Pretrained refers to the (bidirectional) classification models after pretraining, and generators to the (causal) models for generation after training.
that complement themselves. We did measure isotropy on models using embedding pooling, as it would untractable considering the very large number of possible embeddings, and that the low frequency of the majority of them would result in unreliable results.
more uniformly distributed.
We also note that similarly to models trained on natural language (Ethayarajh, 2019), our bidi- rectional models learn more isotropic embeddings than causal (generative) ones. Appendix G depicts UMAP representations of the embedding. showing the narrow cones and clusters they form.
SVD, for which results are plotted in Figure 3, allows to visualize the relative domination of some dimensions. Baselines without BPE and PVm and PVDm show quicker singular value decays, indi- cating that fewer dominate, whereas baselines with BPE show more uniformly distributed values.
Conclusion
We showed that BPE can increase the quality of results of Transformer models for symbolic music generation, and classification tasks, while signif- icantly improving their efficiency and inference speed and making better use of their embedding spaces. BPE can be applied on top of any tok- enization. The tokenization and decoding times almost not affected by this extra step, when performed by a well-optimized Rust code. Con- sidering the considerable benefits and low require- ments of this technique, we advise anyone using Transformer models with symbolic music to use BPE.
The intrinsic dimension is an estimation of the minimal number of dimensions required to repre- sent manifold in Rd n. It links with isotropy in the sense that isotropic manifold occupies all the dimensions, hence its intrinsic dimension is close to d. We chose to estimate with the Prin- ciple Component Analysis (PCA) (Fukunaga and Olsen, 1971) and FisherS (Albergante et al., 2019) methods as they are insensitive to redundancy, fo- cus on common similarities and can scale to large number of points and dimensions. As the embed- ding matrix is often initialized with stochastic method around the origin, simple method can estimate high intrinsic dimensions even though the points coordinates have been very little or not even optimized. This can be the case when a large num- ber of tokens has low frequencies or are absent from the training data, such as with PVDm. The in- trinsic dimensions results are reported in Section 8, along with the IsoScores. In all cases, as the vo- cabulary grows with BPE, the intrinsic dimension increases, the embeddings occupy more space.
There are still questions that remain uncovered. We showed that 40M parameters models can handle well vocabularies up 20k tokens with medium- size datasets. We however do not know what are the limits in vocabulary and dataset sizes over which the results might not improve. Moreover, we ex- perimented with BPE, but other common vocab- ulary building techniques exist, such as Unigram (Kudo, 2018) and WordPiece (Wu et al., 2016). Recent work on natural language showed that Uni- gram yielded higher model performances than BPE (Bostrom and Durrett, 2020), it might also be the case for symbolic music. Future research will study these questions and hopefully find optimal tokenization guidelines to improve model perfor- mances under more various settings.
IsoScore is an estimation of isotropy based on the distance of the covariance matrix of a Principle Component Analysis (PCA) and the identity matrix, and is normalized between and 1. As for the intrinsic dimension, the isoscore grows with the vocabulary size, indicating that the embeddings are

---

Limitations
American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 5117-5130, Online. Association for Computa- tional Linguistics.
BPE allows to build vocabulary based on data. Hence, in case the data has specific token distribu- tions, a model trained with this vocabulary might not generalize and perform well on data with oppo- site token distributions.
Kaj Bostrom and Greg Durrett. 2020. Byte pair encod- ing is suboptimal for language model pretraining. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 46 4617-4624, Online. Association for Computational Linguistics.
BPE implies additional step during data tok- enization. In Table I we showed that the impact on tokenization time is very limited. The impact on decoding time is however more substantial.
Jean-Pierre Briot. 2021, From artificial neural networks to deep learning for music generation: history, con- cepts and trends. Neural Computing and Applica- tions, 33(1))39-65.
Finally, although we experimented with two pub- lic datasets, two tokenizations and two tasks, we did not find a "limit" vocabulary size above which the results might not increase with. More research must be performed in order to find such limit.
Jean-Pierre Briot, Gaätan Hadjeres, and François-Dave Pachet. 2020. Deep Learning Techniques for Music Generation. Computational Synthesis and Creative Systems. Springer International Publishing.
Ethies Statement
Xingyu Cai, Jiaji Huang. Yuchen Bian, and Kenneth Church, 2021. Isotropy in the contextual embedding space: Clusters and manifolds. In International Con- ference on Learning Representations.
We believe that open science and open sourcing code and model parameters ensure an equal ac- cess to the latest research to everybody. Neverthe- less, we acknowledge that generative models can be used in harmful ways to artists and copyright owners. Generative models can be used to create new content, that can be conditioned on human prompt such as text description. Malevolent users might control them to copy. alter or use content of artist without their approval. Moreover, such model can represent an unfair competitive tool to music creators, which is a time of writing an open issue and subject to ethic considerations.
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Chris Donahue, Huanru Henry Mao, Yiting Ethan Li, Garrison W. Cottrell, and Julian McAuley. 2019. Lakhnes: Improving nulti-instrummetal music gener- ation with cross-domain pre-training. In Proceedings of the 20th International Society for Music Informa- tion Retrieval Conference, ISMIR 2019, Delft, The Netherlands, November 4-8, 2079, pages 685-692.
Acknowledgements
This work was partially funded by Aubay. We would like to thank Eric Remilleret for is helpful support.
Hao-Wen Dong, Ke Chen, Shlomo Dubnov, Julian McAuley, and Taylor Berg-Kirkpatriek. 2023. Mul- titrack music transformer. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
References
Luca Albergante, Jonathan Bac, and Andrei Zinovyew. 2019. Estimating the effective dimension of large bi- ological datascts using fisher separability analysis. In International Joint Conference on Neural Networks (IJCNN), pages 1-8.
Hao- Wen Dong, Wen-Yr Yi Hsiao, Li-Chia Yang, and Yi- Hsuan Yang. 2018 Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016. A Latent Variable Model Approach to PMI-based Word Embeddings. Transac- tions of the Association for Computational Linguis- tics, 4:385-399.
Jeffrey Ens and Philippe Pasquier. 2021, Building the metamidi dataset: Linking symbolic and audio musi- cal data. In Proceedings of 22st International Con- ference on Music Information Retrieval, ISMIR.
Kawin Ethayarajh. 2019. How contextual are contextu- alized word representations? Comparing the geom- etry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
Daniel Bis, Maksim Podkorytov, and Xiuwen Liu. 2021. Too much in common: Shifting of embeddings in transformer language models and its implications. In Proceedings of the 2021 Conference of the North

---

9th International Joint Conference on Natural Lan- guage Processing (EMNLP- -JCNLP,, pages 55-65, Hong Kong, China. Association for Computational Linguistics.
Wen-Yi Hsiao, Jen-Yi Liu, Yin-Cheng Yeh, and Yi- Hsuan Yang. 2021. Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs. Proceedings of the AAAI Con- ference on Artificial Intelligence, 35(1):178-186.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Yu-Siang Huang and Yi-Hsuan Yang. 2020. Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions. In Proceedings of the 28th ACM International Conference on Mul- timedia, MM '20, page 1180-1 188, New York, NY, USA. Association for Computing Machinery.
Nathan Fradet, Yean-Pierre Briot, Fabien Chhel, Amal EI Fallah Seghrouchni, and Nicolas Gutowski. 2021. MidiTok: A python package for MIDI file tokeniza- tion. In Extended Abstracts for the Late-Breaking Demo Session of the 22nd International Society for Music Information Retrieval Conference.
Hsiao-Tzu Hung, Joann Ching. Seungheon Doh, Nabin Kim, Juhan Nam, and Yi-Hsuan Yang. 2021. EMMPPА: A multi-modal pop piano dataset for emo- tion recognition and emotion-base tion. In Proceedings of the 22nd International So- ciety for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021, pages 318 325.
music genera-
Nathan Fradet, Nicolas Gutowski, Fabien Chhel, and Jean-Pierre Briot. 2023. Impact of time and note du- ration tokenizations on deep learning symbolic music modeling. In Proceedings of the 24th International Society for Music Information Retrieval Conference, Milano, Italy.
Mathieu Kermarec, Louis Bigo, and Mikaela Keller. 2022. Improving tokenization expressiveness with pitch intervals. In Extended Abstracts for the Late- Breaking Demo Session of the 23nd International Suciety for Music Information Retrieval Conference.
Fukunaga and D.R. Olsen. 1971. An algorithm for finding intrinsic dimensionality of data. IEEE Trans- actions Computers, C-20(2): 1611116....1
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Philip Gage. 1994, A new algorithm for data compres- sion. Users J., 12(2):23-38
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. Representation degeneration proh- lem in training natural language generation models. International Conference on Learning Representa- lions.
Qiuqiang Kong, Bochen Li, Jitong Chen, and Yuxuan Wang. 2021. Giantmidi-pianoo A large-scale midi dataset for classical piano music. In Transactions of the International Society for Music Information Retrieval, volume 5, pages 87-98.
Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. 2018 Frage: aeqqueecc-agnostie word representation. In Advances Neural Informa- tion Processing Systems, volume 31. Curran Asso- ciates, Inc.
Taku Kudo. 2018 Subword regularivation: Improv- ing neural network translation models with multiple subword candidates. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia, Association for Computational Linguistics.
Gaëtan Hadjeres, François Pachet, and Frank Nielsen. 2017. DeepBach: a steerable model for Bach chorales generation. In Proceedings of the 34th In- ternational Conference Machine Learning, vol- ume 70 of Proceedings of Machine Learning Re- PML.R. search, pages 1362-1371. PMLR.
Feynman T. Liang, Mark Gotham, Matthew Johnson, and Jamie Shotton. 2017. Automatic stylistic compo- sition bach chorales with deep LSTM. In Proceed- ings of the 18th International Society for Music Infor- mation Retrieval Conference, ISMIR 2017, Suzhou, China, October 23-27, 2017, pages 449 456.
Gaëtan Hadjeres and Léopold Crestel. 2021. The piano inpainting application.
Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon. Cheng-ZM Zhi Anna Huang. Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck. 2019, Enabling factorized piano music modeling and gener- ation with the MAESTRO dataset. In International Conference on Learning Representations.
Yuxin Liang. Rui Cao, Jie Zheng, Jie Ren, and Ling Gao. 2021. Learning to remove: Towards isotropic pre-trained bert embedding. In Artificial Neural Net- works and Machine Learning - ICANN 2021: 30th International Conference on Artificial Neural Net- works, Bratislava, Slovakia, September 14-17, 2021, Proceedings, Part V. page berg. Springer-Verlag.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learning Representations.
Berlin, Heidel-

---

Jiafeng Liu, Yuanliang Dong, Zehua Cheng, Xinran Zhang. Xiaobing Li, Feng Yu, and Maosong Sun. 2022. Symphony generation with permutation in- variant language model. In Proceedings of the 23rd International Society for Music Information Retrieval Conference, Bengalore, India. ISMIR
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Anmual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. 2018. Umap: Uniform manifold ap- proximation and projection. The Journal of Open Source Software, 3(2)::661.
Bob L. Sturm, João Felipe Santos, and Iryna Kor- shunova. 2015. Folk music style modelling by recur- rent neural networks with long short-term memory units. In Extended abstracts for the Late- Breaking Demo Session of the 16th International Society for Music Information Retrieval Conference.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gre- gory Diamos, Erich Elsen, David Garcia, Boris Gins- burg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed precision training. In International Conference on Learning Representations.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. Long range arena : benchmark for efficient transformers. In International Conference on Learning Representa- tions.
Jiaqi Mu and Pramod Viswanath. 2018. All-but-the-op:: Simple and effective postprocessing for word repre- sentations. International Conference on Learning Representations.
John Thickstun, David Hall, Chris Donahue, and Percy Liang. 2023. Anticipatory music transformer.
Sageev Oore, Ian Simon, Sander Dieleman, Douglas Bek, and Karen Simonyan. 2018. 'Tii time with feeling:
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Learning expressive musical performance. Neural Computing and Applications, 32:955-967.
Christine Payne. 2019. Musenet.
Ofrr Press and Lior Wolf. 2017. Using the output em- bedding improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Vol- ume 2, Short Papers, pages 157-163, Valencia, Spain. Association for Computational Linguistics.
Dimitri von Rütte, Luca Biggio, Yannic Kilcher, and Thomas Hofmann. 2023, FIGARO: Controllable mu- sic generation using learned and expert features. In The Eleventh International Conference on Learning Representations.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Lingxiao Wang, Jing Huang. Kevin Huang, Ziniu Hu, Guangtao Wang. and Quanquan Gu. 2020a. Improv- ing neural language generation with spectrum control. In International Conference on Learning Representa- tions.
Sara Rajacc and Mohammad Taher Pilehvar. 2022. An isotropy analysis in the multilingual BERT embed- ding space. In Findings of the Association for Com- putational Linguistics: ACL 2022, pages 1316, Dublin, Ireland. Association for Computational Lin- guistics.
Ziyu Wang. Ke Chen, Junyan Jiang, Yiyi Zhang, Mao- ran Xu, Shuqi Dai, Guxian Bin, and Gus Xia. 2020b. Pop909: A pop-song dataset for music arrangement generation. In Proceedings 21st International Con- ference on Music Information Retrieval, ISMIR.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: Sf natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Yi Ren, Jinzheng He, Xu Tan, Tao Qin, Zhou Zhao, and Tie Yan Liu. 2020. Popmag: Pop music accom- paniment generation. In Proceedings of the 28th ACM International Conference on Multimedia, page 1198-1206. Association for Computing Machinery.
William Rudman, Nate Gillman, Taylor Rayne, and Carsten Eickhoff 2022. IsoScore: Measuring the uniformity of embedding space utilization. In Find- ings of the Association for Computational Linguis- fies: ACL 2022, pages 3325-3339, Dublin, Ireland. Association for Computational Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi. Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff

---

Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang. Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's 's neural machine translation system: Bridging the gap between human and machine trans- lation.
Li-Chia Yang and Alexander Lerch. 2020. On the evalu- ation generative models in music. Neural Comput. Appl. 32(9):4773-47844
Mingliang Zeng, Xu Tan, Rui Wang. Zeqian Ju, Tao Qin, and Tie- Yan Liu. 2021. MusicBERT: Symbolic music understanding with large-scale pre-training. In Findings of the Association for Computational Lin- guisties: ACL-IJCNLP 2021, pages 100-000 Online. Association for Computational Linguistics.

---

1.75
Datase:
GiantMIDI
GiantMIDI
Macstro
Moestro
eoratt
1.00
EMOPAA
POPSO9
density
0.03
EMOPIA
0.75
0.02
0.01
0.00
0.00
20
40
60
100
120
duration (beal))
Velodity
Figure 4: Distributions of the note durations and velocities of five popular MIDI datasets. The duration axis is
limited beats better readability.
A Model training
The generator and classifiers are respectively trained and pretrained on 100k steps. For classifiers pretraining, we use the same objective than done with BERT (Devlin et al., 2019): 15% of each input sequences is masked with special MASK token, 10% of these masked tokens randomized, and the loss is computed on the capacity of the model to recover the original tokens. Additionally each sequence is divided into two equal parts separated with special SEP token, and half of the batch sequences have non-related parts. The model predicts the second part is the next part of the first. The input embedding and output pretraining module weights are tied to improve the performances (Press and Wolf, 2017).
The classifiers are then finetuned on 10k steps on the downstream tasks. We feed the output hidden state of the first position (BOS token) to an output fully connected layer, to train the model to classify the input sequence.
Trainings are performed on V100 and RTX2080ti GPUs, each time in distributed setups of pairs of the same GPU model, for total batch size of 128. All trainings are done with automatic mixed- precissio (Micikevicius al., 2018), the Adam optimizer (Kingma and Ba, 2015) with 31 = 0.9, B3 0.999 and c = 10 8 and dropout, weight decay and gradient clip norm respectively 10 10 2 and 3. We use a one cycle learning rate scheduler: the initial learning rate close to O and gradually grows for the 30% first steps to le - 4 for the generators and classifier pretraining and 3e - 5 for the classifier fine-tuning, then slowly decreases down to O. The model parameters are saved when the validation loss is the lowest ever observed, and after training the last version saved is used for testing.
B MMD preprocessing
With more than 436k MIDI files, the MMD dataset contains many duplicated songs, corrupted files and poor quality performances. In order to train our models with a well balanced dataset composed pieces of good quality, we perform a preprocessing step to deduplicate each song, and keep the best versions. Each MIDI file has matching score with audio files linked to Spotify and MusicBrainz ids. Hence. with several different ids, and Spotify MusicBrainz id can have have high matching scores with several different MIDI files.
each MIDI file can have high matching
In order to deduplicate the songs, we represented the matching scores as a weighted bipartite graph, and computed its matching. To build the graph, we first read each MIDI file, add it to the graph if it is not corrupted, has a time signature and has at least three tracks. The opposite nodes are the Spotify ids, and the edges (weights) are the MIDI-audio matching scores. When the graph is complete, we compute its matching in order to have the maximum sum of the weights between pairs of distinct and unique MIDIs and Spotify ids. After matching, end up with 30k distinct MIDI files.
Data downsampling
Figure 4 shows the distributions of velocity and duration values of the notes from the two datasets we use, along with the POP909 (Wang et al., 2020b), Maestro (Hawthorne et al., 2019) and EMOPIA (Hung et al.,

---

POP909
Maestro
GiantMIDI
MMD
EMOPIA
Ticks
0.014
0.000
0.002
0.143
0.002
Preprocessed (32nd)
0.124
0.129
0.182
0.203
0.124
Preprocessed 616hh
0.175
0.229
0.236
0.222
0.145
Table 6: Ratio of notes played simultaneously with the same velocity. Preprocessed rows means that the onset and offset times ticks of the notes have been aligned, to the corresponding portion bar. For fair comparison, results for POP909 are for all tracks being merged, and those for MMD are for the unprocessed (vanilla) dataset.
2021) datasets which are commonly used the music information retrieval community. As there a larger proportion of low note durations (below two beats), we decided to downsample the Durat in and TimeShift tokens with different resolutions: those up to one beat are downsampled to samples per beat (spb), those from one to two beats to 4 spb, those from two to four beats to 2 spb, and those from four to eight beats to spb. This way, short notes are represented more precisely than longer ones, reducing the vocabulary size. For REMI, Position tokens are downsampled to 8 spb, resulting in 32 different tokens only consider the time signature. This allows represent the 16th note. We only consider pitches within the recommended range for piano (program 0) specified in the General MIDI 2 specifications". 21 108. We then deduplicate all duplicated notes. Velocities are downsampled additional token (e.g., Chord, Tempo) is used.
distinct values. No
We finally apply a downsampling on the instruments for the MMD dataset. The General MIDI 2 protocol features 128 instruments, called programs. In practice, many programs are very similar in their sounds and the way they are played. A model could struggle to capture the differences between these similar programs, especially considering that the program choices were made by humans and potentially subject to bias subjective preferences. In order reduce alleviate this complexity, tracks with programs of the same category, for the twelfth first categories (programs from 0 to 95) except ensembles (programs 48 55), and drums, ending up with twelve distinct programs.
merge the
Proportion of simultaneous notes in common datasets
Table 6 shows the ratios of notes being played simultaneously (having the same onset and offset times), with the same velocity, for the datasets used in this paper, well as POP909 (Wang et al., 2020b), GiantMIDI (Kong et al., 2021) and EMOPIA (Hung al., 2021).
The proportion of simultaneous note low, even with a strong downsampling of their attributes, onset and offset times. Hence, the scope of token aggregation techniques such in SymphonyNet (Liu et al., 2022) is arguably limited.
"Available on the MIDI Manufacturers Association website

---

Pit-Vel-ur
Vil Dur Pos
Vel Dur
Pos Pie Dur
0.6
Proportion
0.3
Mii-Du--Pt
Propartion
Pos-Pit
T5 S.-P2
Misdel-Dur-PPe Vei--D
0.4
Other
0.1
0.2
0.0
0.0
1k
5k
10k
20k
1k
5k
10k
20k
Vocab size
Vocab size
(a) Maestro and TSD
(b) Maestro and REM/
Dur Png Pit
Pos Png Pi-Vet Dur
0.5
Prg.Ptt
Vel Dun Pt:
0.4
Pivvel
Pos Png Fit
Drr-Dur-Prg
Pos-Ping
Proportion
0.4
T5-PrrgPh
Vei--Dur
0.3
Vel- Pos
VeilDur
0.3
0.2
0.1
0.1
0.0
10K
Vacab size
Vocab size
MMD and TSD
(d) MMD and REMI
Figure 5: Normalized distributions of the token types of the BPE tokens, per vocabulary size. Abbreviations in the legend stand for: Pit: Pitch; Vel: Velocity; Dur: Duration; Pos: Position; TS: TimeShift; Prg: Program.
E Types of learned byte pairs
Figure shows the distribution token types combinations of the learned BPE tokens. The majority the learned combinations represent single notes all cases, except for the case of MMD when tokenized with TSD. In this latter case, most BPE tokens begin with Velocity base tokens, indicating that the dataset contains lot of recurrent Velocity Duration token successions. With REMI however, the Position token seems to be more recurrent, showing that the notes have more common onset positions, which is not surprising considering that the MMD dataset features many music of genre known to have repeating patterns, As the vocabulary grows, the combinations tend be more diverse.
F Human evaluations
We report here the human evaluation instructions given to the participants to assess the generative models:
Each MIDI file contains several music tracks generated from different Deep Learning models, that are the continuations of the same 4-bars prompt. For each file, you have choose the best track on several criteria:
Fidelity: the track with the best fidelity (coherent) relative the prompt, from a tonal and rhythm point of view;
Correctness: the track with the most correct note successions and harmonies, contrarily to tracks with dissonant notes unnatural melodies:
Diversity: the track with the best coherent diversity, i.e. featuring diverse correct melodies, contrarily to music that would repeat the same note patterns. A "bad" or uncertain music (i.e. non-correct) cannot be consider as diverse;
Overall preference: the track that you overall prefer subjectively;
Do not answer to all for all the files, as the evaluations can time-consuming. Fix yourself number of files to evaluate, and randomly pick them from the list.

---

how
Bral
Figure 6: Example of MIDI file given to participants for human evaluations, opened with the Logic Pro DAW.
You will find generated results than can be very similar, even identical sometimes. As such, you might feel uncertain or unable to decide. In such cases, do not answer for all criteria and just skip to the next file. There is no good or wrong answer, you just have to answer subjectively. Trust yourself and trust your musical instinct.
An example of MIDI file open with the Logic Pro DAW is shown in Figure 6.
G Learned embedding space
UMAP (Mclnnes et al., 2018) representations shown in Figure 7, Figure 8, Figure 9 and Figure 10 show the embeddings of the models of the paper, computed with the official UMAP Python package with default parameters. For each figure, only 1k randomly sampled points are represented in order to keep them in vector format without adding too much weight in this file document. We encourage the reader to visualize them on our demo website for more convenient readability.
The models learn clusters of embeddings of the same type. The embeddings do occupy the space uniformly, but rather have preferred directions following their type and value. We still that bi- directional (pretrained) models tends to occupy more space than causal (generative) ones. This especially noticeable for the PVm and PVDm models. For generative models, the embeddings tends to be oriented towards common dimensions, and slightly spread towards orthogonal one.
Pretrained bi-directional models learn more isotropic embedding representations. The embeddings are spread more uniformly across all directions.

---

RG
14
Waney
WOIN
-22 42 00
000 123
-00
TSD
ISD BPE
TSD BPE
7SD BPE 10k
15.
Seucel
A
J
12
TSD BPE 20k
TSD PVm
TSD PVDm
320
300
Eards
Octer
REMI No BPE
REMI BPF 1k
REMI BPR 5k
REMI BPF 10k
Spmual
Tralis
di REMI + BPF 20k
25
REMI + PVm
REMI + PVDm
Figure UMAP 2d representations of the embeddings of the generators, trained with the Maestro dataset.
Abbreviations in legend stand for: Pi: Pitch: Ve: Velocity; Du: Duration; Po: Position; TS: Shif.

---

TSD No BPE
TSD BPE 1k
TSD BPE 5k
TSD + BPE 10k
TreDeh:
Fogen
22
200 L1
&
TSD BPE
TSD PVm
TSD + PVDm
21
10
Swria Pooren Dustar
Toche
100
REMI No BPE
REMI BPE 1k
REMI BPE 5k
REM! BPE 10k
Soeds Rooa
Scede Puscan
Par
FIOKOS
REMI + BPF 20k
REMI PVm
REMI PVDm
Figure UMAP 2d representations embeddings of the pretrained bidirectional models, trained with the
Maestro dataset. Abbreviations in legend stand for: Pit: Pitch; Ve: Velocity; Du: Duration; Po: Position; TS:
TineShift; Pr: Program.

---

10
TSD No BPE
TSD BPE
TSD BPE 5k
TSD BPE 10k
TSD + BPE 20k
TSD + PVm
TSD + PVDm
611
15
-50
REMI No BPE
REMI BPE 1k
REMI BPE 5k
REMI BPE 10k
POWN
FARROS
Poder
Mhnlly
10
REMI + BPE 20k
REMI + PVm
REMI + PVDm
Figure 9: UMAP 3d representations of the embeddings of generative models, trained with the Maestro dataset.
Abbreviations in legend stand for: Pi: Pitch: Ve: Velocity; Du: Duration; Po: Position: TS: Shif.

---

325
15
TSD
TSD
TSD 5k
TSD BPE 10K
Paara
TSD BPE 20k
TSD PVm
TSD PVDm
100
REMI
REMI
REMI Sk
REMI 10K
Dave Dr
REMI BPE 20k
REMI PVm
REMI PVDm
Figure 10: UMAP representations of embeddings of pretrained bidirectional models, trained with the MMD
dataset. Abbreviations in legend stand for: Pi: Pitch; Ve: Velocity; Du: Duration; Po: Position; TS: TimeShift; Pr:
Program.


Reviews:
[{"review_id": "a20tcOIZYz", "reviewer": "Reviewer_SeWr", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would be happy to receive explanations about how the research \"appears to be somehow experimental\", and why is it a reason to reject it. We assume that this point motivated your soundness score. Having more details would help up to understand better your decision, and how we can improve the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The average token number can be reduced by 50% by utilizing the BPE on top of the normal tokenizer while achieving better performance. This feature can make future studies on music generation and classification less computationally expensive, and the representation can be easier to modelling on.\n- The proposed method can expand the vocabulary set and make better use of the embedding dimension of the language model.", "reasons_to_reject": "- Despite the experiment being interesting and showing promising result, the research mostly build upon the well-studied modules and appears to be somehow experimental.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies the representation of symbolic music in generation and classification tasks. The author proposes to utilize Byte Pair Encoding (BPE) to reduce the sequence length for generation and classification. The paper compares the proposed method with several dimension reduction baselines as well as the tokens without BPE. Finally, the limitation of using BPE is discussed in the paper.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KZb7H71pUU", "reviewer": "Reviewer_heEW", "summary": "", "questions": "A: What exactly would be problematic if the embedding dimension is larger than the vocabulary size?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Reasons to reject\n\nThank you for your suggestions, which are now applied to the paper.\n\nOn the significance of the classification and embedding results: indeed, as we were short on space, we resorted to write concisely and may have left some ambiguity.\nOur assumption for why Octuple outperforms other baselines is that: 1) We are using a bi-directional model (no attention mask) for which the modeling task is more easily performed than with generative models with causal attention masks; 2) The sequence length is the most reduced with Octuple, which is suitable and takes advantage of the embedding space; 3) The finetuned model is trained on a classification task with a single set of labels, unlike the generation which implies several output modules / distributions / losses for octuple which are computed unconditionally from each other and adds a lot of instability. This last point explains most the differences of results for Octuple between generation and classification.\n\nOn the vocabulary size / embedding size: see response below.\n\nOn the end-to-end generation speed: indeed, the BPE brings a significant end-to-end generation speedup. Here is a benchmark we ran, the times are for loading MIDI --> tokenization --> generation of 512 tokens --> detokenization, with several batch sizes on a V100 GPU.\nWe observe the same speed gains, with all batch sizes. With higher batch sizes, the generation is faster as the GPU takes advantage of parallelization, and case easily handle more samples for the same execution time.\nWe also must note that for a given batch, the tokenization and detokenizations are performed sequentially. We could further reduce the duration by using multithreading, and encoding / decoding BPE in batch.\nWe will include these results in the appendix.\n\n|Tokenization |Nb beats - 1 - TSD|Nb beats - 1 - REMI|Nb beats - 8 - TSD|Nb beats - 8 - REMI|Nb beats - 64 - TSD|Nb beats - 64 - REMI|Nb base tokens - 1 - TSD|Nb base tokens - 1 - REMI|Nb base tokens - 8 - TSD|Nb base tokens - 8 - REMI|Nb base tokens - 64 - TSD|Nb base tokens - 64 - REMI|Nb notes - 1 - TSD|Nb notes - 1 - REMI|Nb notes - 8 - TSD|Nb notes - 8 - REMI|Nb notes - 64 - TSD|Nb notes - 64 - REMI|\n|-------|------------------|-------------------|------------------|-------------------|-------------------|--------------------|------------------------|-------------------------|------------------------|-------------------------|-------------------------|--------------------------|------------------|-------------------|------------------|-------------------|-------------------|--------------------|\n|No BPE |14.0              |22.1               |53.4              |95.9               |130.6              |238.0               |95.0                    |98.8                     |410.4                   |415.3                    |1011.1                   |1046.2                    |24.9              |24.6               |106.0             |103.6              |262.6              |261.0               |\n|BPE 1k |18.9              |46.1               |108.1             |179.6              |274.1              |451.0               |176.4                   |146.8                    |734.3                   |647.1                    |1771.5                   |1591.8                    |47.7              |36.6               |195.0             |161.8              |469.0              |395.6               |\n|BPE 5k |23.4              |86.2               |97.1              |285.2              |257.3              |706.2               |221.3                   |196.7                    |950.6                   |793.7                    |2307.2                   |2020.8                    |59.4              |46.8               |255.9             |195.2              |620.3              |499.9               |\n|BPE 10k|24.8              |43.5               |117.3             |252.0              |295.1              |615.0               |243.8                   |195.0                    |955.0                   |845.5                    |2379.5                   |2115.4                    |65.8              |50.4               |258.1             |212.6              |638.0              |531.8               |\n|BPE 20k|26.3              |52.4               |125.8             |221.0              |284.6              |595.1               |269.0                   |230.2                    |1026.5                  |868.6                    |2614.4                   |2207.7                    |72.6              |58.4               |277.9             |216.6              |707.5              |547.3               |\n|PVm    |15.8              |35.4               |82.2              |119.8              |206.3              |314.1               |137.2                   |133.9                    |575.0                   |573.7                    |1470.4                   |1478.9                    |36.8              |33.3               |153.9             |148.6              |392.4              |380.9               |\n|PVDm   |23.0              |43.8               |104.5             |181.4              |305.4              |488.5               |220.9                   |209.9                    |883.7                   |866.4                    |2350.2                   |2330.3                    |61.4              |55.4               |241.3             |230.4              |642.0              |618.3               |\n|CPWord |                  |28.9               |                  |136.8              |                   |338.8               |                        |191.8                    |                        |784.5                    |                         |1922.0                    |                  |49.3               |                  |198.6              |                   |484.9               |\n|Octuple|                  |23.3               |                  |97.1               |                   |250.8               |                        |95.5                     |                        |400.1                    |                         |                          |                  |95.5               |                  |400.1              |                   |1027.5              |\n\n\n### Questions\n\n**A:** an embedding size larger than the size of the vocabulary is not problematic in the sense that it will not degrade the performances. However it is suboptimal, or oversized as you are making the model learn to find the best localization of $V$ tokens, onto $E$ dimensions, with $E > V$.\nAn example: let's say we have a set of 10k sentences, all of them with different meanings and semantic information, and we want to automatically place them in a continuous space $\\mathbb{R}^E$, for instance to gather those with similarities or to measure their distances. In this case, we would want the $E$ dimensions of this space to represent some feature learned from the sentences. When choosing $E$, it makes sense to choose a value < 10k, as the 10k sentences share common features around which we want to localize them. If we set a higher $E$ value, a model can still learn and perform well, but a large number of dimensions will simply be unused.\nIn our case, models trained with small vocabularies (< to embedding size) still perform well, but the model could have learned a much larger number of token / embeddings, while performing as well or better as reported by our results in tables 2 to 5. Table 3 shows that the generative models still use (i.e. outputs high probabilities during inference) most of the tokens (99% with 30k tokens) when the vocabulary grows.\n\nWe thank you for these constructive questions / remarks that will undoubtedly strengthen the quality / affirmations of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Their research idea is innovative and impactful -- combining pre-tokenised inputs with BPE leads to large improvements for symbolic music generation models according to their human evaluation. Also, it positively impacts inference speed. Thus, it is worth sharing in the NLP community.", "reasons_to_reject": "While the logical structure of the paper is sound, the writing is often sloppy and should be made more clear to boost the perceived trustworthiness of the paper:\n- Often, The Transformer is mentioned in a way that suggests to the reader that it would be a specific model even though it is a model architecture (e.g. in the first sentence of the abstract and the introduction)\n- The paper sometimes uses \"assumption\" where it should be \"hypothesis\" (e.g. line 505/506).\n- While the examples are very helpful in Section 3 to understand BPE, not all symbols used in Algorithm 1 are explained, which might confuse a reader who doesn't know the algorithm yet. The section before the algorithm doesn't explicitly state what the base vocabulary consists of, which might again confuse an unfamiliar reader.\n\nA challenge of the paper is that the significance of the results is not discussed. At least for the classification models and the embeddings' analysis, it would be worthwhile to evaluate models across different random seeds to get an idea of the variation of the results. \n\nThe paper sometimes makes claims that would need more explanations in my opinion:\n- There is no explicit discussion why the Octuple model outperforms the BPE models on music classification\n- It is mentioned multiple times in the paper that it would be problematic if the embedding dimension was larger than the vocabulary size without giving an explicit reason nor citing a source.\n\nSome minor comments:\n- I'd personally reorganise the discussion of the speed advantages of integrating BPE into the music generation pipeline. Intuitively (and as shown by the authors) tokenisation and detokenisation is slower (Table 1) while inference speed increases (Table 3). As the inference speed dominates the speed of an end-to-end system, I suspect that the method provides substantial end-to-end speed-ups, but unless I misunderstand the authors' analyses, speed is never benchmarked in an end-to-end fashion. I think such an analysis would make the paper even more convincing.", "questions_to_authors": "A: What exactly would be problematic if the embedding dimension is larger than the vocabulary size?", "paper_topic_and_main_contributions": "This paper proposes to use BPE as a second tokenisation step in the preprocessing pipeline of symbolic music models. The authors demonstrate better performance of models trained on data that was preprocessed with this additional step (on generation tasks) and link performance gains to improved token representations (compared to token representations learned without BPE). Furthermore, their method leads to faster inference. They also discuss symbolic music classification for which their method performs comparable (but slightly worse).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 60: does -> do\nline 195: Unfortunately -> Unfortunately\nline 585: I'd reference directly to Table 5 instead of to Section 8."}, "extra_text": {}}, {"review_id": "jXjAQ6NzHa", "reviewer": "Reviewer_ntvA", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recommendations, we will reformulate the typos and formulations.\nWe found better classification results with Octuple, however is Octuple is actually more constraining. The reasons are:\n\n* It requires multiple input heads, hence to either modify model implementations (e.g. from Hugging Face transformers, or even the PyTorch Transformer implementation) or create a model from scratch. If you are using a library to train the model, or to generate from it, you might need to modify it too as the input / output tensors have now an additional dimensions that needs to be handled `(batch_size, sequence_length, sub_tokens)`. During pretraining, you also have multiple output heads, the model implementation must be modified too and weight tying [1] must be adapted. This was our case using the HF transformers trainer and generate methods, and we had to spend time to make the adaptations required;\n* Pretraining takes more time (as using multiple losses, one per output module). We did not emphasize on this downside in the paper, as our experiment is quite short (about 10 hours to pretrain), but we can report the training times in the paper;\n* It can break the traceability of models (torchscript);\n* Octuple cannot represent additional tokens such as chords, sustain pedal, pitch bend or control change effects. In theory, one could adapt the tokenizer to represent these features by adding more input/output heads, but this would also increase the complexity of the model and training time, and good results are not guaranteed. Its expressivity is hence limited.\n\nThese are the reasons for our recommendation to use BPE, as it allows competitive results, out of the box, on top of any tokenization and without having to deal with these limitations / inconveniences. But we must admit that these reasons were not clearly stated in the paper, that the sentence is not clearly supported, and we thank you for having pointed out this. We plan to add them in the paper, but we are also open to simply reformulate the sentence.\n\n[1] Press, O., & Wolf, L. (2017). [Using the Output Embedding to Improve Language Models.](https://aclanthology.org/E17-2025/) In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers (pp. 157–163). Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written, the methodology is solid. There are enough experiments and the method shows to be simple yet effective. The authors provide a nice website with the code and audio samples. The results in generation are impressively better than other methods", "reasons_to_reject": "The claim on classification performances need to be clearer. The BPE method is not better nor less constraining than Octuple. Especially this sentence needs to be reformulated: \n\"We advise anyone using Transformer models with symbolic music to use BPE.\" This would not be true if the goal is to perform classification.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose to use BPE tokenizer to train a transformer causal to generate and classify symbolic music. The method is said to outperform other tokenization methods on both classification and generation tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In general you should use less adjectives and be more rigorous. Please reformulate the following:\n\nline 072: seems like a big shortfall\n\nline 167: The same models, when used with natural language, use to learn up to 50k embeddings on a range of 512 to 1024 dimensions. \n\nline 165: One can easily see here..."}}, {"review_id": "5OoLkYftim", "reviewer": "Reviewer_ErZ1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We agree that our paper does not make substantial modifications to the BPE algorithm, we still emphasize that our contribution lies in the application of this established technique to the realm of music scenarios. By doing so, we show it allows significant performance (inference speed) and results quality gains, at the cost of short preprocessing step. Considering these benefits, we strongly encourage researchers and engineers to use BPE with symbolic music, and so believe that our paper adds valuable insights to the field. Moreover, we make the use of BPE for symbolic music easy for everyone by releasing the code in a friendly and easy way to use, which would encourage researchers to adopt it.\n\nWe would be happy to receive your feedback on how we could improve the paper, and in turn convince you to increase your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors introduce byte pair encoding into symbolic music.\nThe authors compare BPE with other sequence reduction techniques introduced in recent research.\nThe authors study the geometry of the learned embeddings, and show that BPE can improve their isotropy and space occupation;\nThe authors show some limits of BPE.", "reasons_to_reject": "This paper hardly makes any modifications to the BPE algorithm for music scenarios. Therefore, the novelty of this paper is limited.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The topic of this paper is symbolic music representation.\nAuthors propose byte pair encoding into symbolic music.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an innovative application of Byte Pair Encoding (BPE) in the realm of music scenarios. The proposed algorithm significantly enhances inference speed and result quality, only with a short preprocessing step. This research holds valuable insights for researchers and engineers engaged in symbolic music.

The reviewers have agreed on the innovation and impact of this research. By reducing the average token count by 50% while achieving improved performance and faster inference speed, the proposed method has the potential to reduce computational overhead in future studies on music generation and classification, making representation modeling more accessible. Furthermore, the paper is well-written, providing a comprehensive package with code, audio samples, and results on a dedicated website.
Decision: Accept-Main