Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation
Authors: Botao Yu
Peiling Lu
Rui Wang
Wei Hu
Xu Tan
Wei Ye
Shikun Zhang
Tao Qin
Tie-Yan Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: music composition
music generation
music structure
Transformer
attention
efficient Transformer
sparse attention

Abstract: Symbolic music generation aims to generate music scores automatically. A recent trend is to use Transformer or its variants in music generation, which is, however, suboptimal, because the full attention cannot efficiently model the typically long music sequences (e.g., over 10,000 tokens), and the existing models have shortcomings in generating musical repetition structures. In this paper, we propose Museformer, a Transformer with a novel fine- and coarse-grained attention for music generation. Specifically, with the fine-grained attention, a token of a specific bar directly attends to all the tokens of the bars that are most relevant to music structures (e.g., the previous 1st, 2nd, 4th and 8th bars, selected via similarity statistics); with the coarse-grained attention, a token only attends to the summarization of the other bars rather than each token of them so as to reduce the computational cost. The advantages are two-fold. First, it can capture both music structure-related correlations via the fine-grained attention, and other contextual information via the coarse-grained attention. Second, it is efficient and can model over 3X longer music sequences compared to its full-attention counterpart. Both objective and subjective experimental results demonstrate its ability to generate long music sequences with high quality and better structures.

Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation
Botao Yu', Peiling Lu', Rui Wang',
Wei Hu Xu Tan'", Wei Yes, Shikun Zhang Tao Qin Tin Liu
State Key Laboratory for Novel Software Technology, Nanjing University, China
Microsoff Research Asia
9 National Engineering Research Center for Software Engineering, Peking University, China btyu@foxmail.. com, (peil,ruiver xuta, taoqin, tyliu)@miee www.s/oft.tfotooommmmmmmmmommo.e com, whu@nju. edu. cn, {wye, zhwngskknnkkree edu. cn
Abstract
Symbolic music generation aims to generate music scores automatically. A recent trend is to use Transformer or its variants in music generation, which is, how- ever, suboptimal, because the full attention cannot efficiently model the typically long music sequences (e.g., over 10,000 tokens), and the existing models have shortcomings in generating musical repetition structures. In this paper, we pro- pose Museformer, a Transformer with a novel fine- and coarse- coarse-grained attention for music generation. Specifically, with the fine- tne-grained attention, token of specific bar directly attends to all the tokens of the bars that are most relevant music structures (e.g., the previous 1st, 2nd, 4th and 8th bars, selected via similarity statistics); with the coarse-grand grained attention, a token only attends to the summarization of the other bars rather than each token of them computational cost. The advantages are two-fold. First, it can capture both music structure-related correlations via the fine-grained attention, and other contextual information via the coarse-grained attention. Second, it is efficient and can model over 3x longer music sequences compared to its full-attention counterpart. Both objective and subjective experimental results demonstrate its ability to generate long music sequences with high quality and better structures
reduce the
Introduction
Symbolic music generation aims at generating music scores automatically and has drawn more and more attention in recent years [. Since music can be represented in organized sequences of discrete tokens just like text, Transformer-base
models, which have been demonstrated to work well on text generation 4 [5), are incressingly applied in music generation [ 6-11 and have made great success. While the self-attention mechanism empowers Transformer to capture the complex correlations in music, there are two ubiquitous challenges to solve for this task: 1) Long sequence modeling. Music sequences are typically very long, especially for multi-instrument polyphonic music where the lengths can usually exceed 10,000. The quadratic complexity of full attention limits its scalability to that length. 2) Music structure modeling. Music has its unique structures, where piece can usually repeat some patterns of a previous piece, occasionally with some variations, after either a short or long distance (see Figure I for example). Successfully generating reasonable structures would make the music more realistic just like human-man made music.
Wei Hu and Xu Tan are the corresponding authors. This work was partially done while the first author was interning Microsoft Research Asia.
'The generated music samples can be found at atttttss:/aiimuuiiii wtt.:mmmuiiiiiiiiii.. github........ htw.://wwww...iii.i.ii.teeteeeee hommmbefmmmeeeeeerr code can be found at httph::///// http::///iithan com comm/microsoff/muzie
The source
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

120
Tempo
Beat(1)
Inst (Synth))
Pitch(C4)
Dur (4th)
Vel(64)
Inst(Piano)
Piano
Pitch(C3)
Dur (2thh
Vel(58)
10
Beat(2).
Inst(Synth)
8ymth.
Pitch(C4)
Dur (4thh)
Vel(64)
Poo.
Bar(end)
Figure 1: The music score of Twinkle, Twinkle, Little Star and its corresponding token representation. Every two consecutive bars on the Syuth. track have the same rhythmic pattern, and the 9th 12th bars repeat the 1st - 4th bars with an interval bars. Structures embodied as repetitions and variations common in music.
Although many Transformer variants in natural language processing (NLP) [2] have been proposed to handle long sequences (the first challenge), they cannot well model the music structures (the second challenge). According to the basic principles of these models, we can classify them into two types. The first type is called local focusing. Models of this type, e.g., Transformer-XI. [13] and Longformer [1]. mainly focus on part of the input sequence, and drop the rest tokens to reduce the cost. However, the parts that they focus cannot contain many of the essential ranges important to music structures, and directly dropping the rest tokens may lead losing some important information. The second type is called global appreximation. Models of this type such as Linear Transformer [IS] utilize linearized attention or sequence compression over the whole input sequence to approximate the token pair-wise attention. While the approximation effectively reduces the complexity, they cannot accurately capture the correlations between related parts and accordingly are inadequate in generating repetition structures. We will review more existing long-sequence Transformers in $2 Directly applying these models to music generation is suboptimal, and it is desirable to design an efficient model that can well model long music sequences as well as their structures.
In this paper, we propose to unify the above two types of models, which can well fit the characteristies of music. Our motivation is based on the observation that the importance is not uniformly distributed over the music sequence, and thus we do not need treat all the tokens equally. Intuitively, to generate music with repetition structures, the most important information the model should directly refer to when generating music bar, lies those bars that tend be repeated in the current bar. We call these bars structure- struture-related bars. For the other bars that are less important, approximation should do the trick. To this end, the proposed fine- and coarse-graines attention takes two different schemes towards different bars: with the fine-grainee attention, tokens of the structure-related bars are directly attended to, to well learn the structure-related correlations; with the coarse-grained attention. information of the other bars are captured via summarization (16), rather than attending to each token of them, to decrease the computation and space complexity, and meanwhile retain the necessary information of these bars. The related bars selected according to the statistics on human-made music. They are not necessarily the most recent and consecutive bars but include distant ones, to cater to long-term structures in music. Our main contributions in this paper are summarized as follows:
We propose Museformer, a Transformer model with novel fine- and coarse-grained attention for music generation. It captures the correlations of structure-related bars via fine-grained attention for learning the music structures, as well as the necessary and concentrated information of the other bars through their summarization via coarse-grained attention.
We propose to select the structure-related bars based on similarity statistics human-made music, which can help decrease the perplexity and generate music that exhibits better structures.
The computation complexity and space complexity are reduced greatly to nearly linear in practice, which enables Museformer to scale up long music sequences.
Experimental results show that Museformer can generate music of full-song length with high quality and better structures.

---

2 Related Work
2.1 Symbolic Music Generation
Symbolic music gemeration aims exploit machines compose music scores automatically. It has been attracting more and more people to work on it, and the solutions continuously evolve from to deep learning models 6-11.
rule-based models M to probabilistic models 17 [8] 2
In recent years, the Transformer- based models have achieved great success in many text generation tasks [4 thus have also been increasingly applied in the similar music generation tasks. Huang et al. B] apply Transformer in symbolic music generation for the first time and show that can achieve better performance compared to previous deep learning models such as recurrent neural networks. While Transformer shows promising results in music generation, the quadratic complexity of the attention mechanism limits its applications on the typically long music sequences. To resolve this problem, researchers on symbolic music generation come up with different solutions. One popular solution is to design a new representation method to represent musical information in fewer tokens, such as compound word [9] and OctupleMIDI 20 While this solution indeed decreases the lengths of music sequences to certain extent, full-song music sequences still too long for Transformer to handle. Another solution is to use a senneee Transformer variant as the backbone. For example, Huang and Yang (2l) use Transformer-X XL [13), and Hsiao et al. [19] use Linear Transformer ([5). Although these models can process long sequences, they are initially designed for NLP tasks and cannot well model the music structures. We will introduce and discuss about the ung-seeuence Transformers in the following section.
2.2 Long-Sequence Transformers
Many Transformer variants have been proposed tackle sengeee tasks [12]. which in general
can be categorized into the following types. 1) Recurrent Transformer, which encodes sequences chunk by chunk [3
22
23
2) Sparse attention, which sparsifies the attention layout with either
16 26], or with content-based patterns 31 3) Linearized attention, which replaces the exponent of the inner product of features with the multiplication of feature maps [15
predefined patterns 14
24
25
attention, which reduces the number of queries or key-value pairs by compressing the contextual representations (35 In addition, some models also try to combine multiple techniques, such as Compressive Transformer [22] that combines recurrent and compression- based methods, Poolingformer (36] and Transformer-LLS attention and compression-bast methods.
32 4) Compression
38) that combine sparse
Existing works on music generation directly adopt some of those long-sequence Transformers to process long music sequences, but it is suboptimal due to the unique structures of music. In general, music has many repeating or similar pieces, many of which are distant from each other, and the distance is measured by time units such as bar or beat instead of the number of tokens. Therefore, the receptive fields of the existing recurrent Transformers or sparse attention methods cannot cover the many ranges of structure-related content. Although the linearized or compression-basee attention can cover the whole sequence, they do not precisely capture the correlation between each pair of tokens and may have shortcomings in generating repeating or similar music pieces.
3 Museformer
In order to well model long music sequences as well as their music structures, we propose Museformer. The general idea is that we do not need to focus on the whole sequence with the same importance level given that the complexity of pair-wise full attention is unacceptably high, but instead different attention schemes fine- aine-grained attention for the structure-related bars, and coarse coarse-grained attention for the other bars. Museformer basically follows the original Transformer architecture [39], and a novel fine- and coarse coarse-graine grained attention (FC-Attention) is designed to replace the original self-attention module to tackle the challenges in long music sequence modeling.
combine two
Museformer takes a music token sequence X = XI,...,,4 X, as input, where denotes the number of music bars, and the i-th bar X; Ii,,, .fi.l..... , Ti,X, contains [X,l tokens. For the i-th bar where i = 1,....,, b, we insert a summary token 8; after to facilitate local information aggregation in FC- FC-Attention. The summary token sequence is denoted by S = .i.....8 Sb. After the insertion, the
3

---

21.
22
bar
bar 2
bar 3
X
X
X 33
X
X
S
X
S
X
N22
X
S2
X
S
S
S
(a) Information flow. dashed/solid lines depict summariza- tion'aggregation step. The dark lines represent the information flow towards T3.2-
(b) Attention layouts. The bottom/top part corresponds to the summariza- tion/aggregation step.
Figure 2: The fine- and coarse-grained attention. It shows toy example of 3 bars, where for each bar, only the previous 1st bar is regarded as the structure-related bar.
overall token sequence becomes X1, 81 .... X,, Sb. The embedding layer embeds the overall token sequence into a vector space, and concatenates bar embeddings and beat embeddings as the positional information [10], followed by linear projection. The Museformer layers then model the contextual representation, and the hidden state output by the last layer is fed into softmax classifier to predict next token.
In the following part, we first give a brief formulation of the attention mechanism $3. Then, we introduce the details of FC-Attention ($3.2), followed by the selection of the structure-related bars (3.3). Finally, we discuss the merits of Museformer $3.4)
3.1 Preliminary: Attention
The attention mechanism [39 is the basis of FC-Attention. It receives two sequential inputs, namely source X E R xd and target X' E Xd where nir and nig are the sequence lengths of source and target respectively, d is the embedding dimension. It computes contextual representation for each a, E TR. 1ld in target X' as
x W/ T X
Attn Att X)) softmax
(I)
Vd
where Wo, WK. Wy Rdxd are the trainable parameters. In practice, we employ the multi-head version of attention, but omit from the equation for simplicity.
3.2 Fine- and Coarse-Grained Attention
The basic idea of FC-Attention is that, instead of directly attending to all the tokens which causes the quadratic complexity, token of specific bar only directly attends the structure-related bars that are essential for generating structured music (fine-grained attention), and for the other bars, the token only attends to their summary tokens obtain concentrated information (coarse- -rerinedgraeedddd attention). To achieve this, we first summarize the local information of each bar through the summarization step, and then aggregate the fine-grained and coarse- grained information through the aggregation step. Figure visualizes the process.
In the summarization step, we aggregate the information of each bar into the corresponding summary token. For the i-th bar, given the representation of summary token E R lxd and that of the music tokens X; = " 1 E TR xxxx the summarization of this bar is
Summarization
8: Attn 8,, [ s:]),
where Attn(-) defined in Equation (1)
the concatenation operation. In this step, each summary token attends music tokens of its corresponding bar as well as the summary token itself.

---

14
10
2
24
Interval (bar)
Figure 3: The similarity distribution of the training used in our experiments. is calculated over the melody track because melody best shows the structure of song. We omit the similarity when t because the similarity between bar and itself always 1.0.
Aggregation In the aggregation step, aggregate the information of the tokens belonging to the structure-related bars or within the current bar, as well the summarization of the other bars, so as to update the contextual representations of music tokens. The updated representation for zij is
Ei
Attn Zidi
[X n()) X.nsj
(3)
where R(ii) the the indices of the structure-related bars with respect to the i-th bar, and R(i) is the set of the indices of other previous bars. X X(()) is the matrix formed by stacking (X, R(i)). Similarly, RR()) is the matrix formed by stacking {&, l R(i)). X the matrix formed by stacking {æt, It. E X,, k S j}- other words, the music token æ4, only attends those music tokens that belong to the structure-relate bars, and its previous tokens within the current bar. For the other bars, only attends to their summary tokens.
3.3 Straature-Related Bar Selection
Considering that the aruuttre-related bars are expected to contain those bars that are most likely to be repeated by the current bar be generated, propose pinpoint them by similarity statistics, and the bars with high similarities should be selected. Specifically, for each song in the training set, calculate the similarity between each pair of the bars, which is defined
(N( NNGI [N UNO)
4.j
where N(i) is the set of music notes within the i-th bar, and two notes are considered equal when their pitches, duration, and onset positions within their bars are all the same. The value of 4., ranges from 0.0 to 1.0. If two bars are exactly the same, this value equals 1.0. We then calculate the average similarity of those bar pairs whose intervals are f over the training set D, which formulated as
L: Mean ΣΣ 4.))
D jEit
In this paper, we call the distribution of the similarity with respect to the bar intervals the similarity distribution, and show that the training data in Figure 3] As we can see, it shows an obvious periodical pattern music bar tends to be more similar to its previous 2 bars, and also to the previous 4-th bar or its multipliers in most cases. We further conduct the similarity statistics on different datasets involving music of various genres and styles. The results shown in Appendix ijinterestingly indicate that this pattern is universally applicable to the music of the great diversity. We believe that general rule applicable to most music in daily life.
it can be regarded
Based on the statistical results, we carefully select bars - the previous 1st, 2nd, 4th, 8th, 12th, 16th, 24th, and 32nd bars, the default structured-related bars, since they can cover the most similar bars in most cases. The number of selected structure-nelated bars trade-off between efficiency and information richness. One can select more or other bars according to the computation resources and

---

the specific similarity distribution of the used dataset. We have to admit that the selected bars cannot always cater to any song, but as long as it covers most cases, can already enable Museformer to generate music with better structures.
3.4 Merits of Museformer
We explain why Museformer is suitable for the music generation task.
First, the receptive fields of the fine-grained attention comply with the characteristics of music and can cover most of the structure-related information. Unlike previous models that adopt fixed token-level patterns [4 38 or learn content-based patterns 27 31, which may not cover the most important ranges to be referred to for generating music structures, we directly use the structure- derived from human-made music. These bars can include both neighboring ones and distant ones, which enables Museformer to generate both short-term and long-term music structures.
bars
Second, the coarse gaarse-rrined attention can preserve necessary information for generating better music. In general, FC-Attention can be categorized into sparse attention, but unlike conventional sparse attention that simply drops a large amount of information, which may limit the model's s capability, especially for music where mutual connections abound, the bearse-grained attention preserves the information of the other bars so as to provide rich clues for generation.
Third, the combination of the fine- and coarse-grained attention enables Museformer handle long music sequences efficiently. Compared to the number of all the music tokens, the number of tokens in the structure-related bars and the summary tokens much smaller, so the memory consumption and the running time are greatly reduced. It is essential for this task where sequences are quite long.
4 Experiments
4.1 Experiment Settings
We give brief introduction to the experiment settings. Please refer to Appendix B for more details.
Dataset and Music Representation We conduct our experiments on the widely used Lakh MIDI (LMD) dataset 40), which contains multi-instrument music in the format of MIDI. After preprocess- ing, the final dataset contains 29,940 songs (1,727 hours), each song contains 95 bars on average. Following [10]. various instruments are merged into the 6 basic ones namely square synthesizer, piano, guitar, string, bass, and drum, with the square synthesizer playing the melody. We use REMI-like representation method [21] to transfer MIDIs into token sequences, where the musical information (instrument, bar line, note position, pitch, duration, etc.) represented in separate tokens (see (see Figure 1] for an example). Each song is represented by 15,042 tokens on average, and each bar is represented by 158 tokens on average. We randomly split all the songs by 8/1/1 for training/validaaiionness respectively.
Implementation Museformer is implemented with Py Py and fairsed For the efficient compu- tation of FC-Attention, we write CUDA kernels to construct the attention layouts for each sample, and then transfer the layouts into a blocksparse form and compute with SparTA
3
Model and Training Configurations The core parameters of Museformer include layer number 4. hidden size- = 512, number of attention heads = 8, and FFN hidden size = 2,048. During training, the batch size is set to 4 songs. Following [39]. we use the Adam optimizer with 3, 0.9, B2 = 0.98 and = = 10 9 and the learning rate is warmed up over the first 16,000 steps linearly to peak value of 5 10 -4 and then decreases based on the inverse square root of the steps. The L2 weight decay is set to 0.01. During inference, we use top-k sampling with = 8, Generation continues until the end-of-sentence token is generated or it reaches the maximum length of 20,480.
https://pytorcch.or
httpe://gthhbb.cm//tttttraaaaaae

---

Similarity Error (SE): the error between the similarity distribution of training data and generated music, to evaluate the models' ability to generate music with realistic structures. is defined as Compared Models We compare Museformer with 4 representative Transformer-based models, most of which have been adopted music generation:
Music Transformer 3 vanilla Transformer with memory-efficient "skewing" relative position
embedding implementation.
Transformer-XI K3
recurrent Transformer that encodes the sequence chunk by chunk and uses the gradient-stopped representations of previous chunks as the memory.
Longformer model with sliding window sparse attention.
Linear Transformer 15 model that kernel-based attention of linear complexity.
All the compared models are set with comparable hyper-parameters as Museformer. For Music Transformer that uses the full attention and thus cannot model long sequences once due to memory limit, we chunk each song into multiple samples during training, and apply the model to generate long sequences during validation and inference to its generalization on long music sequences.
Objective Evaluation We use the following objective metrics to evaluate the models:
Perplexity (PPL): common metric measure whether generative model correctly predict future tokens. The smaller, the better. To see the models' performances on different lengths, it is calculated the first 1,024, 5,120, or 10,240 tokens of each sample.
SE T Σ L - Lel,
where f, and Lt are the average similarities (defined in Equutiin (5) of the generated music and the training data respectively. We set T = 40 in our experiments, and L, is calculated on 100 generated music pieces for each model. The smaller the value is, the more similar the structures of the generated music are to muman-made music.
Subjective Evaluation The most canonical way to evaluate a music generation model is the human listening test. We apply each model to randomly generate 100 music pieces, and invite 10 people, where 7 of them have music-related learning experiences, to score these music pieces. Specifically, for each participant, are generated by Museformer and the 4 compared models, respectively. Participants are asked to score these music pieces from (lowest) to 10 (highest) over the following subjective metrics:
randomly construct groups, where each group contains 5 music pieces that
Musicality: whether is pleasant and interesting, and real enough just like human-man made music.
Short-term structure: whether shows good structures in neighboring content, such good repetitions and reasonable development.
Long-term structure: whether it shows good structures in long distances, such as song-level repetitions and long-distance connections.
Overall: overall score. We also calculate preference score based on this overall score, which is defined as the ratio of winning times (obtaining the highest overall score within a group).
4.2 Comparison with Previous Models
Table shows the results of the objective evaluation. As we can see: 1) Music Transformer achieves comparable PPL as other models on short music sequences (1,024 tokens) but undergoes severe deterioration on longer sequences, which implies that model trained on short music sequences cannot well generalize long music sequences. Accordingly, an appropriate long- long-sequence Transformer model is needed for modeling the long music sequences. 2) Although the receptive field of Linear Transformer can cover the whole sequence, PPLs show no superiority compared to other models, which may be because the kernel-based attention cannot accurately capture the complex correlations of music. 3) The proposed Museformer can consistently achieve the best PPLs on various sequence lengths, especially on larger length, which demonstrates the effectiveness of Museformer on the
7

---

Table The results of objective evaluation and ablation study. The numbers in the parentheses for PPLs are sequence lengths.
PPL (1,024)
PPL (5,120)
PPL (10,240)
SE (%)
Music Transformer
1.66
1.77
2.55
2.49
Transformer-XL.
1.64
1.45
1.43
15.66
Longformer
1.65
1.46
1.45
5.25
Linear Transformer
1.86
1.67
1.64
1.97
Museformer (ours)
1.64
1.41
1.35
0.95
w/o coarse-grained
1.65
1.42
1.38
1.08
w/o bar selection
1.65
1.43
1.39
6.39
Table 2: The results of subjective evaluation. ST and LT stand for short-term and long-term re- spectively. For all the subjective metrics, mean standard deviation is reported. Pref stands for preference score.
Musicality
ST structure
LT structure
Overall
Pref
Music Transformer
6.00 2.21
6.90 1.76
5.30 + 2.58
5.90 + 1.90
0.20
Transformer-XI.
6.10 = 2.19
7.40 + 1.81
6.26 + 2.78
6.44 2.01
0.34
Longformer
6.46 = 1.81
7.60 1.47
6.18 + 2.54
1.74
0.24
Linear Transformer
6.06 = 1.99
6.92 + 2.03
5.78 2.64
6.30
0.24
Museformer (ours)
6.88 1.95
7.86 1.51
6.72 2.74
7.12 + 1.81
0.46
music generation task. 4) The results SE demonstrate that the music generated by Museformer has structures that are most similar to the human-made music. We provide similarity distribution for each model and more discussion in Appendix C
Furthermore, we present the results of the subjective evaluation in Table 2,
which show that Muse- former achieves the best performance. Specifically, 1) Museformer gets the highest scores on all the metrics. 2) On the structure-related metrics, especially the long-term structure, Museformer exceeds other models by a large gap, indicating that the proposed FC-Attention can empower the model to capture the correlations distant bars.
We further do pairwise comparisons over the subjective evaluation results. Table 3 shows the number of wins/tiees/osses based the overall scores, well the p-values of the Wilcoxon signed rank test. Museformer obtains more wins than the compared models, and the p-values indicate that Museformer achieves statistically significant improvements < 0.05).
4.3 Ablation Study
We verify the effectiveness of the Museformer components as follows: 1) see the effectiveness of the coarse-grained attention, we compare Museformer with the setting w/o coarse-grained, where each music token attends to the music tokens the structure-related bars and its previous ones the current bars, and no summary token of any bar. 2) To the effectiveness of the structure-related bar
Table The results of pairwise comparisons based on the subjective overall scores.
Wins
Ties
Losses
p-value
Museformer VS Music Transformer
33
7
10
0.0003
Museformer VS Transformer-XI.
25
14
11
0.0375
Museformer VS Longformer 28
8
14
0.0424
Museformer VS Linear Transformer
29
7
14
0.0128

---

25
CPU monmory (68)
24
20
20
10
Altention
FC Altention
4096 8192 12288 16364 20480 24576 28672 32768 36664
4096
8192 12288 16384 20480 24576 28672 32768 36664 Maximum sequance length
Maximum sequence length
(a) GPU memory consumption.
(b) Running time per iteration.
Figure 4: Memory consumption and running time of Museformer with FC-Attention compared to its full attention counterpart.
selection for the fine-grained attention, we compare Museformer with the setting w/o bar selection, where we use the most recent 8 bars for the fine-grained attention instead of the structure-related bars selected via the similarity statistics.
we can observe that: 1) Museformer consistently achieves better performances on both PPLs and SE compared the two ablation settings, which demonstrates the effectiveness of the coarse-grained attention and the structure-relatd bar selection. 2) The coarse-graine grained attention preserves the necessary information of the bars other than the structured-relate bars and can consistently help decrease the PPLs. 3) The contribution of the structure- related bar selection increases as the sequences get longer. This is reasonable because that in longer music sequence, there tends to be more bars and more long-term structures, and fine-grained attention can directly capture the correlations
From the results presented in the bottom block of Table 1
the distant structure-related bars, which helps make more accurate predictions. 4) Embodied on SE, the bar selection is contributory to the structures generated music. Please refer to Aopppeddix Cf more details. In addition to the above two settings, have also tried disabling the fine- grained attention for previous bars, where music token only directly attends to its previous tokens within its bar and attends to all the previous bars via the summary tokens. The result indicates that only summarized information and no precise teven-level information for the previous bars is insufficient for the model to generate coherent music.
4.4 Complexity Analysis
Suppose the sequence length is n, the average bar number is b, the average bar length is m. and the number of selected structure-related bars is k. For FC-Attention, the time complexity of the summarization step is (n), and that of the aggregation step is O(kkm + b)n). so the overall complexity is O(km + b)n) = O + n/m)n). Although the complexity is still proportional to the square of n, the typically large divider m (mostly exceeds 100) greatly reduces the complexity. Thus, the specific complexity is between linear and quadratic. Note that the computation of FC- Attention depends on the input content, i.c., the number of bars and the number of tokens in each individual bar, the complexity cannot be precisely formulated, and the efficiency in real applications more noteworthy.
To see the efficiency of FC-Attention in real applications, we train Museformer and its full attention counterpart on the validation set, and record their memory consumption and running time. With batch size = we increase the maximum sequence length until we use up the 32GB memory of an Nvidia V100 GPU, and record the peak memory consumptiorf and the running time one-epoch training. Figure 4 shows the results. We observe that the memory consumption of Museformer increases at a nearly linear rate with respect to the maximum sequence length, and can process over 3x longer music sequences than its full attention counterpart, making it capable of generating full-song music. Museformer also achieves faster training time when the maximum sequence length is greater than around 5,000, which is common for music sequences.
*Obtained by torch.cuda.max memory memory_allocateer allocated().

---

4.5
Case Study
Figure shows a snippet of song generated by Museformer. As we can see, on the Strings track, the 13th 16th bars repeat the 9th 12th bars with an interval of 4 bars, which is a short-term structure. The 25th 32nd bars repeat the 9th 16th bars with an interval of 16 bars, and there are reasonable variations in the 27th bar compared to the 1 1tth bar, which shows long-term structure. This case demonstrates that Museformer can generate music with both short-term and long-term structures, and not only exact repetitions but also some variations. In addition to the two exhibited tracks, other tracks, such as piano and drum, have more variations on the two similar segments, which creates reasonable development of music. Please refer to our demo page for more information.
14
15
Synth.
8 0
8
Strings
9
0
26
26
Synth.
8
8
Strings
9
Figure 5: A snippet of generated song.
5
Conclusion
To solve the long sequence modeling and music structure modeling challenges in symbolic music generation, we propose Museformer with novel fine- and coarse-grained attention. The fine-grained attention is applied to the structure-related bars for learning the structure-related correlations, and the coarse-grained attention is applied to the summarization of the other bars for getting a sketch of them. We propose to select the atructure-relate bars via bar- bar-pair similarity statistics. Experimental results show that Museformer is efficient and can generate music with good quality and structures.
Museformer is not perfect yet. and we would like to discuss about its limitations and possible future explorations. First, since Museformer takes random samplings during inference and does not receive manual control, it can hardly ensure that every generated music piece well-structured in an expected way. Techniques to enhance its reliability and controllability can be further explored. Furthermore, the musicality and the creativity of the generated music are still behind those of human- made music, which remains problem for all the existing music generation models. We believe that more sophisticated music representation and large models trained on large-scale data can help alleviate this problem. Finally, we anticipate Museformer's 's adaptation to more tasks and domains. can be easily and reasonably applied to music understanding tasks. For NLP tasks, since natural languages do not have the periodical patterns music, how to determine the semantically related sentences, paragraphs, or documents is an interesting challenge to solve.
Acknowledgments
This work is supported by National Natural Science Foundation of China (No. 61872172). We thank the scorers in the subjective evaluation, the SparTA group at Microsoft Research Asia, and many other people from the Websoft group at Nanjing University and Microsoft for their kind help.
References
[1] H. A. G. Salas, A. F. Gelbukh, H. Calvo, and F. G. Soria, "Automatic music composition with simple probabilistic generative grammars," Polibits, vol. 44, pp. 39-65, 201
[2] A. K. Yanchenko and Mukherjee, "Classical music composition using state space models," arXiv preprint 11w.11708..322...
10

---

[3] C.-Z. A. Huang, A. Vaswani, J. Uszkoreit, I. Simon, C. Hawthorne, N. Shazeer, A. M. Dai, M. D. Hoffman, M. Dinculeseu, and Eck, "Music transformer: Generating music with long- structure," in Proceedings of International Conference on Learning Representations (ICIR), 2018.
[4] A. Radford, Narasimhan, T. Salimans, Sutskever al., "Improving language understanding by generative pre- wre-training." OpenAl, 2018.
[5] J. Li, T. Tang, W. X. Zhao, and J.-R. Wen, "Pretrained language model for text generation: A survey," in Proceedings of International Joint Conference on Artificial Intelligence (ICCAI), 2021, pp. 4492 4499.
[6] C. Donahue, H. H. Mao, Y. F. Li, G. W. Cottrell, and J. McAuley, "Lakhnes: Improving multi- instrumental music generation with cross-domain pre-training," in Proceedings of International Society for Music Information Retrieval Conference (ISMIR), 2019, pp. 685-692.
[7] A. Muhamed, L. Li, X. Shi, S. Yaddanapudi, W. Chi, D. Jackson, R. Suresh, Z. C. Lipton, and
A. J. Smola, "Symbolic music generation with transformer-gans," in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), vol. 35, 2021, pp. 408-417.
[8] A. Roberts, J. Engel, C. Raffel, C. Hawthome, and D. Eck, "A hierarchical latent vector model for learning long-term structure in music," in Proceedings of International Conference Machine Learning (ICML), vol. 80, 2018, pp. 4364-4373.
[9] Z. Ju, Lu, X. Tan, R. Wang, C. Zhang, S. Wu, K. Zhang, X. Li, T. Qin, and T.-Y. Liu, "Telemelody: Lyric-to-melody generation with cemplate-based two-stage method," arXiv preprint wxxiv:209.0061/ 2021.
[10] Y. Ren, J. Hc, X. Tan, T. Qin, Z. Zhao, and T.-Y. Liu, "Popmag: Pop music accompaniment generation," in Proceedings of ACM International Conference Multimedia (MM), 2020, pp. 1198-1206.
[11] Z. Sheng, K. Song, X. Tan, Ren, W. Ye, S. Zhang, and T. Qin, "Songmass: Automatic song writing with pre- training and alignment constraint," in Proceedings of the AAAI Conference Artificial Intelligence (AAAI), 2021, pp. 13 798-13880 805.
[12] T. Lin, Y. Wang, X. Liu, and X. Qiu, "A survey of transformers," arXiv preprint DxXiv:216604453 2021.
[13] Z. Dai, Z. Yang. Y. Yang, J. Carbonell, Q. Le, and R. Salakhutdinov, "Transformer-XI: Attentive language models beyond fixed-length context," " in Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL), 2019, pp. 2978-2988 2988.
[14] I. Beltagy, M. E. Peters, and A. Cohan, "Longformer: The long- documentt transformer," arXiv preprint arXiv:2004.051550 2020.
[15] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, "Transformers are RNNs: Fast autore- gressive transformers with linear attention," in Proceedings of International Conference on Machine Learning (ICML), 2020, pp- 5156-5165.
[16] Z. Ye, Q Guo, Q. Gan, X. Qiu, and Z. Zhang, "Bp-transformer: Modelling long-range context via binary partitioning," arXiv preprint arXiv: 1.04070, 2019.
[17] M. M. Farbood and B. Schöner, "Analysis and synthesis of palestrina-style counterpoint using markov chains," in Proceedings International Computer Music Conference (ICMC), 2001.
[18] M. Allan, "Harmonising chorales in the style of johann sebastian bach," Master's. 's, 2002.
[19] W.- Hsiao, J.-Y. Liu, Y.-C, Ych, and Y.-H. Yang, "Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs." in Proceedings of the AAAI Conference Artificial Intelligence (AAAI), vol. 35, 2021, pp. 178-186.
[20] M. Zeng, X. Tan, R. Wang. Z. Ju, T. Qin, and T.-Y. Liu, "MusicBERT: Symbolic music understanding with large-scale pre-training," in Proceedings of Findings of the Association for Computational Linguistics (ACL Findings), 2021, pp. 791-800.
11

---

[21] Y.-S. Huang and Y.-H. Yang, "Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions." in Proceedings of ACM International Conference on Multimedia (MM), 2020, pp. 1180-1188.
[22] J. W. Rae, A. Potapenko, S. M. Jayakumar, C. Hillier, and T. P. Lilllcrap, "Compressive transformers for long-range sequence modelling," in Proceedings of International Conference on Learning Representations (ICLR), 2020.
[23] S. Ding. J. Shang. S. Wang. Y. Sun, H. Tian, H. Wu, and H. Wang. "ERNIE-Doc: A retrospective long-document modeling transformer," in Proceedings of Annual Meeting of the Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL-IJCNLP), 2021, pp. 2914-2927.
[24] R. Child, S. Gray, A. Radford. and I. Sutskever, "Generating long sequences with sparse transformers," " arXiv preprint arXiv:1904. 10509, 2019.
[25] M. Zaheer, G. Guruganesh, K. A. Dubey. J. Ainslie, C. Alberti, S. Ontanon, Pham, A. Ravula, Q Wang, L. Yang, and A. Ahmed, "Big bird: Transformers for longer sequences," in Pro- ceedings of Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp.
17 272-177222 297.
[26] N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran, "Image transformer." in Proceedings of International Conference on Machine Learning (ICML), vol. 80, 2018, pp. 4055 4064.
[27] N. Kitaev, I.. Kaiser, and A. Levskaya, "Reformer: The efficient transformer," in Proceedings of International Conference on Learning Representations (ICLR), 2020.
[28] A. Roy, M. Saffar, A. Vaswani, and D. Grangier, "Efficient content- content-based sparse attention with routing transformers." " Transactions of the Association for Computational Linguistics (TACL), vol. 9, pp. 53-68, 2021.
[29] X. Li, Y. Meng. M. Zhou, Q. Han, F. Wu, and Li, "Sac: Accelerating and structuring self- attention via sparse adaptive connection," " in Proceedings of Advances in Neural Information Processing Systems (NeurIPS), H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33, 2020, pp. 16 997-19 008.
[30] Y. Tay, D. Bahri, L. Yang. D. Metzler, and D.-C. Juan, "Sparse Sinkhorn attention," in Proceed- ings of International Conference on Machine Learning (ICML), vol. 119, 2020, pp.
[31] C. Wu, F. Wu, T. Qi, B. Jiao, D. Jiang, Y. Huang, and X. Xie, "Smart bird: Learnable sparse attention for efficient and effective transformer," arXiv preprint arXiv:2 108.0993,,, 2021.
[32] K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, D. Belanger, L. Colwell et al., "Masked language modeling for proteins via linearly scalable long- long-context transformers." arXiv preprint arXiv:2006.0355: 035555 2020.
[33] K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser al., "Rethinking attention with performers," arXiv preprint arXiv:2009. 14794, 2020.
[34] H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. Smith, and L. Kong. "Random feature attention," " in Proceedings of International Conference on Learning Representations (ICLR), 2021.
[35] S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma, "Linformer: Self-attention with linear complexity," arXiv preprint arXiv:2006. 04768, 2020.
[36] H. Zhang, Y. Gong, Y. Shen, W. Li, J. Lv. N. Duan, and W. Chen, "Poolingformer: Long document modeling with pooling attention," " in Proceedings of International Conference on Machine Learning (ICML), vol. 139, 2021, pp. 12437-12446.
[37] A. Vyas, A. Katharopoulos, and F. Fleuret, "Fast transformers with clustered attention," " in Proceedings of Advances in Neural Information Processing Systems (NeurIPS), H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33, 2020, pp. 21 665-216727 674.
12

---

[38] C. Zhu, W. Ping, C. Xiao, M. Shoeybi, T. Goldstein, A. Anandkumar, and B. Catanzaro, "Long- short transformer: Efficient transformers for language and vision," in Proceedings of Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 17 773-117736.
[39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. ukasz Kaiser, and
I. Polosukhin, "Attention is all you need," in Proceedings of Advances in Neural Information Processing Systems (NIPS), 2017, pp- 5998-599900 6008.
[40] C. Raffel, Learning-bused methods for comparing sequences, with applications to audio-to-midd alignment and matching. Columbia University, 2016.
[41] N. Zheng, B. Lin, Q. Zhang, Ma, Y. Yang, F. Yang. Y. Wang, M. Yang, and L. Zhou, "Sparta: Deep-learning model sparsity via wwtp---tthhhssst--aaateee Symposium on Operating Systems Design and Implementation (OSDI), 2022, pp. 213-232.
in Proceedings of USENIX
[42] J. Liu, Y. Dong, Z. Cheng. X. Zhang, X. Li, F. Yu, and M. Sun, "Symphony generation with permutation invariant language model," arXiv preprint arXiv:2205.05445 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We do not any potential negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "qiy_zPTevyC", "reviewer": "Reviewer_Lgcu", "summary": "This paper presents Museformer, a music-inspired transformer architecture which uses fine-grained and coarse-grained attention to be able to model very long sequences of music while being able to maintain structure-awareness. The proposed architecture reduces the memory and computation cost from quadratic in standard transformers to linear which helps with scaling to long sequences. The transformer architecture involves learning at individual note-level as well as bar-level. The authors determine so-called structure related bars for every bar in an input piece. These are determined using prior knowledge of music and are chosen as the previous 2 bars (for short-term context) along with previous bars at a distance of 4, 8, 16, 24 and 32. This is obtained by analyzing the corpus which is a pop music corpus and most songs use standard 4/4 rhythms. The fine-grained attention attends to each token of all structure-related bars, whereas the coarse-grained attention attends to a summary token for every other previous bar. Finally a combination step combines the information from the current bar, the fine-grained attention, and the coarse-grained attention using another attention layer.\n\nThe evaluation includes comparison against various Transformers for long sequences. The authors use Perplexity as an objective metric and also conduct a small scale user study with 10 participants. The Museformer architecture outperforms all other baselines on all metrics. An ablation study also shows the benefits of coarse-grained attention and structure-related bar selection. The authors also demonstrate the time and space complexity benefits against vanilla transformers for varying sequence lengths.", "questions": "- Why do the authors choose to make the model causal during training? I get that the inference process has to be causal, but during training wouldn't it be better to also attend to future bars?\n- Please see weakness section for question about evaluation of generated structure.", "limitations": "The authors briefly talk about the complex implementation of dynamic sparse transformers as being a limitation. I would like the authors to also possibly discuss some limitations of the generated results; for example, I find that the drums were very unnatural. I'm sure the authors also have some thoughts about the generation quality which readers might be interested to hear about.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe museformer architecture uses domain-knowledge of music to reduce the complexity of Transformers to be able to generate long sequences of symbolic music. There are other papers that attempt to address the issue of maintaining long-term structure in music, but most attempts are unable to consistently maintain long-term structure. The demo samples from the authors demonstrate repetitions at large gaps as well as short term repetitions. While I do find the samples to still lack quality, the work is a step in the right direction. Additionally, the authors choose strong baselines for long-sequence learning and conduct the relevant ablation studies to show the benefits of their components.\n\nWeaknesses:\nI find the evaluation to be a little weak. The authors focus on long-term structure but do not perform any objective comparison of the generated structures to structures from real music. Some kind of self similarity matrix comparison might be interesting to look at. Some statistical analysis as well. I get the feeling that the museformer really focuses on producing almost exact copies of structure-related bars and real music might have more variation. Some visual comparison of these SSMs might be interesting. The writing could also be improved a bit but that is a minor weakness. There are a few phrasing issues.\n\ncomment:\nL231: please fix the commas in the numbers. This took me a while to figure out that perplexity is measured over 1024, 5120, and 10240 tokens. I got it after looking at the tables but was very confused at first glance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DbVZxv2ZG2P", "reviewer": "Reviewer_kVqg", "summary": "The paper proposes a form of structured, sparse self-attention designed to work well for symbolic pop music generation\nwith transformer networks.  Instead of uniformly attending to all previous tokens in a sequence, the proposed method\nreduces computation by only attending to 1) per-bar summary tokens (coarse attention) and 2) full-token sequences _only_\nwithin a small fixed set of \"structure-related bars\" (sparse fine-grained attention).\n\nThis structure incorporates domain-specific knowledge (and empirical analysis) of common musical structure to identify\nthe set of \"structure-related bars\" and results in high quality generation with 1) reduced computation compared to\nfull-context attention models and 2) improved long-term structure compard to several baselines, including long-term\nrepetitions/recapitulations commonly found in western pop music.\n", "questions": "- How are bar lines determined?  Are bar boundaries explicitly represented in all songs of the Lakh MIDI dataset, or do\n  they need to be inferred somehow?\n\n- Sec 3.2, lines 159-166: Would this distribution differ in other corpora or genres?  Seems like enforcing such a\n  constrained bar relation pattern would make it more difficult to model music with unusual structures, a limitation of\n  the proposed approach.\n  - Also, according to Sec A.1, only 4/4 music is considered.  Is the proposed structure not suitable for other\n    time-signatures?  Was time-signature generalization considered empirically at all?\n\n- line 172: $\\bar{N(i)}$ notation is not defined.  Does this simply indicate non-structure bars?\n\n- line 187: It seems that this method could easily be be considered a form of sparse attention with a predefined\n  structure?\n\n- line 193: how much smaller is \"much smaller\"? I'd expect an average bar length of something like 8 or 16 notes?  line\n  202 claims 158 tokens per bar on average.  Why is this so much higher?  It would be helpful to future readers to\n  elaborate on this.  Presumably it is related to the REMI encoding shown in Fig 1 which uses 5+ tokens per note\n  (separating the note identity, pitch, velocity, duration, instrument, etc)?\n\n- Sec 4.4. lines 279-281: The claim that FC-attention is linear feels a bit overstated. The number of bars $b$ will\n  scale linearly with $n$ since bar lengths are generally independent of overall sequence length, i.e., $b = n / K$\n  assuming K token per bar.  So the overall complexity would be $O(b n) = O(n^2/K) = O(n^2)$, remaining quadratic in\n  sequence length.  Due to the large discount factor $K$ it is clearly much sparser than full attention, but the\n  *asymptotic* complexity remains quadratic even if it is reduced by a large factor.\n\n- Fig 5: Why aren't other methods included in this figure?  E.g., transformer XL?", "limitations": "As described above, it would be helpful for the authors to address the limitations of the proposed approach across musical genres, time-signatures, etc.  Is the set of  \"structure-related bars\" used in the experiments thought to be generally applicable across musical corpora, or should they be empirically determined using an analysis similar to the \"Structure-Related Bar Selection\" procedure on a dataset-by-dataset basis?\n\nFurthermore, any exposition on the potential applicability of similar ideas to other domains, e.g. long-form text generation, would be interesting to see.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Straightfoward, well-motivated approach, works well on a challenging area of generating long sequences (10k+ tokens)\n  with coherent long-form musical structure.\n- Well rounded evaluation, including nice demos.  I especially appreciate the inclusion of subjective evaluations given\n  the lack of objective metrics for realistic musical structure.\n- The text is well written and clear.\n\nWeaknesses\n- Very specific application in music, perhaps somewhat narrowly at that (see below).\n  - It would be most interesting to hear whether the authors imagine similar ideas applying to NLP, e.g., by incorporating\n    sentence-level summary tokens to cheaply expand the potential context and/or explicitly attending to, say, the first\n    and last sentence of nearby paragraphs, analogous to the proposed structure-related bars.\n- The repeated characterization of the proposed method reducing the computation complexity of attention from quadratic\n  to linear (e.g., lines 69-71 and elsewhere in the text) is, at best, overstated.  Since the ratio of bars to tokens is\n  a roughly fixed proportion, the proposed model still requires quadratic attention across bar-summary tokens, albeit\n  scaled down by a large factor corresponding to the average number of tokens per bar (~150 according to the paper).\n  The text should be updated to clarify this claim.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9E8nXqs3En", "reviewer": "Reviewer_MUNC", "summary": "The paper proposes Museformer, a Transformer-based music generation model which can handle long music sequences via tailored design of attention mechanism, called Fine- and coarse-grained attention. Specifically, the long music sequences are broken into bars and attention are designed to connect with both local tokens and structurally-important bar summary tokens. Results and analysis show the model perform better in long sequence generation compared to existing Transformer-based models and computes attention in linear time complexity.", "questions": "1. The major concern for this paper is about the novelty of the proposed method. Music structure is a hard problem because simple assumptions fail. I believe it is okay to simplify the assumption to a certain extent, as the authors made in this paper. However, it needs to be proved in the experiment where the method can be generalized to a representative dataset. Lakh MIDI dataset shares the same assumption with POP909 so it is not proper. But it could also be the case that I did not get it. So I would like for more explanation why the proposed paper is not a trick but a systematic method for music.\n\nBelow are the questions related to experiments:\n\n2. In table 2 and the relevant text, what test is used and how is p-value computed?\n\n3. If I understand it correctly, in Table 3, \"Museformer w/o coarse-grained attention\" is almost the same as \"Music Transformer\". (They might be different in the implementation of bar summary.) So, why is the PPL so different for Music Transformer and \"Museformer w/o coarse-grained attention\"? Should we conclude \"bar summary\" is a more fundamental improvement compared to FC-attention?\n\nBelow are the questions related to demo:\n\n4. Lakh MIDI dataset has multiple tracks, however in each track, the music is relatively simple (monophonic or chords with simple texture). How is the model on a dataset where we have less track but more complicated? (E.g., a dataset like Maestro)\n\n5. The demos have very different instrumentation and beginning. Are all these samples generated from scratch?\n\n6. The demos have clear repeats and actually sometimes too many repeats. However, grasping the whole song structure and keeping the song interesting is still difficult. I wonder if the claim for music structure is too strong. Also, I noticed the melody is not so human-like compared to existing music generation algorithms. ", "limitations": "Limitation of this paper is discussed in the weakness section and question section. My suggestion is to consider claim the model to achieve a weaker goal, such as, a music generation algorithms for pop music which consider structural repetition.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Although there are many studies on automatic music generation, few study consider the music generation under the scope of the whole song. This work proposes an adaptation of the powerful Transformer model to the music domain.\n* The assumption of the paper (the subset of music structure considered) is very clear and addressed and evaluated throughout the paper.\n* The adaptation to music is both scientifically efficient and makes some sense in music.\n* implementation detail is clear.\n\nWeakness:\nMusic structure is a difficult problem. The main weakness of this paper is some too strong assumptions made in this paper about music structure (e.g., long-term structure is regarded as 2, 4, ..., 32 bars ahead). Such assumption cannot be tested in the paper because the model is evaluated on a dataset where such assumption prevails. In other words, such assumption is not general for music. It is a novel *trick* for certain styles of music, however it remains to be tested whether it is a *method* (for all music styles. \n* Although FC-attention is powerful in the given experiment setting, the method is actually quite not novel. In a nutshell, if attention can be done to a subset of *fixed number of tokens*, the computation is of course done in linear complexity. The fundamental hard problem is how to determine a subset. However, in this paper, it assumes the subset is the multiples-of-two-bars ahead. Such estimate can be generally true for pop music, but definitely not true for other styles. The dataset for the estimate is also quite arbitrary. \n* I am skeptical about Table 1 - Table 3. Please see the questions below.\n* Figure 6 is not well-formatted for score reading. Maybe there are better examples, but the current one is not musical enough by just reading this example. Please note that repetition is not always good. Rigid repetition can be bad.\n* Demos are in general not musical enough. I have listened to the proposed demo and have some concern. Please see the questions below.\n\nOther issues:\n* The method is clear but the writing is a bit wordy and sometimes explain things illogically. I have to understand the meaning with empathy. For example, \n    * ll. 112-115 is illogical. The first sentence is not the cause of the second sentence. \n    * ll. 146-150 is illogical. The clause after \"instead of\" is only part of the \"basic idea of FC-Attention\", as it only discusses fine-grained attention.\n    * ll. 134-136: what is $d$?\n    * ll. 159 - 164: \"music tends to repeat every 4 bars...\" is not true from the previous experiment. It is true that \"music from the POP909 dataset tends to repeat every 4 bars\". \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "27f1v7K0v-z", "reviewer": "Reviewer_RLny", "summary": "This paper proposed a new transformer-type model that considers long-range musical structures while maintaining linear computational capacity. By separating event-level and bar-level, short-range information is encoded between local tokens while long-range information is encoded in attention in selected bars. As a result, the trained model generated consistent musical structure and achieved better perplexity and human evaluation.", "questions": "- Is there any extension of this approach from bars to higher structure, such as motifs, phrases, and so on? If more higher (abstract, sparse) structure does not work, what will be the source of the problem?\n\n- Will the model able to encode musical information that spans longer than 32 measures? \n", "limitations": "- What is key difference between linear attention-type transformers (such as Linformer)? As far as fixing structure of bar-level attention, overall model will effectively work as similar as linear transformer with limited attention view of 32 measures, if data is sufficiently prepared.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n- This work suggested noble transformer architecture considering musical structures.\n- Recorded better perplexity and human evaluation compared to previous models.\n- Showed consistency in long phrases in audio examples.\n\nCons\n- This approach is not a fundamental improvement of the quadratic attention problem.\n- Fits for limited musical structure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept