Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes Museformer, a Transformer-based music generation model which can handle long music sequences via tailored design of attention mechanism, called Fine- and coarse-grained attention. Specifically, the long music sequences are broken into bars and attention are designed to connect with both local tokens and structurally-important bar summary tokens. Results and analysis show the model perform better in long sequence generation compared to existing Transformer-based models and computes attention in linear time complexity.
strengths_and_weaknesses: Strengths:
* Although there are many studies on automatic music generation, few study consider the music generation under the scope of the whole song. This work proposes an adaptation of the powerful Transformer model to the music domain.
* The assumption of the paper (the subset of music structure considered) is very clear and addressed and evaluated throughout the paper.
* The adaptation to music is both scientifically efficient and makes some sense in music.
* implementation detail is clear.

Weakness:
Music structure is a difficult problem. The main weakness of this paper is some too strong assumptions made in this paper about music structure (e.g., long-term structure is regarded as 2, 4, ..., 32 bars ahead). Such assumption cannot be tested in the paper because the model is evaluated on a dataset where such assumption prevails. In other words, such assumption is not general for music. It is a novel *trick* for certain styles of music, however it remains to be tested whether it is a *method* (for all music styles. 
* Although FC-attention is powerful in the given experiment setting, the method is actually quite not novel. In a nutshell, if attention can be done to a subset of *fixed number of tokens*, the computation is of course done in linear complexity. The fundamental hard problem is how to determine a subset. However, in this paper, it assumes the subset is the multiples-of-two-bars ahead. Such estimate can be generally true for pop music, but definitely not true for other styles. The dataset for the estimate is also quite arbitrary. 
* I am skeptical about Table 1 - Table 3. Please see the questions below.
* Figure 6 is not well-formatted for score reading. Maybe there are better examples, but the current one is not musical enough by just reading this example. Please note that repetition is not always good. Rigid repetition can be bad.
* Demos are in general not musical enough. I have listened to the proposed demo and have some concern. Please see the questions below.

Other issues:
* The method is clear but the writing is a bit wordy and sometimes explain things illogically. I have to understand the meaning with empathy. For example, 
    * ll. 112-115 is illogical. The first sentence is not the cause of the second sentence. 
    * ll. 146-150 is illogical. The clause after "instead of" is only part of the "basic idea of FC-Attention", as it only discusses fine-grained attention.
    * ll. 134-136: what is $d$?
    * ll. 159 - 164: "music tends to repeat every 4 bars..." is not true from the previous experiment. It is true that "music from the POP909 dataset tends to repeat every 4 bars". 


questions: 1. The major concern for this paper is about the novelty of the proposed method. Music structure is a hard problem because simple assumptions fail. I believe it is okay to simplify the assumption to a certain extent, as the authors made in this paper. However, it needs to be proved in the experiment where the method can be generalized to a representative dataset. Lakh MIDI dataset shares the same assumption with POP909 so it is not proper. But it could also be the case that I did not get it. So I would like for more explanation why the proposed paper is not a trick but a systematic method for music.

Below are the questions related to experiments:

2. In table 2 and the relevant text, what test is used and how is p-value computed?

3. If I understand it correctly, in Table 3, "Museformer w/o coarse-grained attention" is almost the same as "Music Transformer". (They might be different in the implementation of bar summary.) So, why is the PPL so different for Music Transformer and "Museformer w/o coarse-grained attention"? Should we conclude "bar summary" is a more fundamental improvement compared to FC-attention?

Below are the questions related to demo:

4. Lakh MIDI dataset has multiple tracks, however in each track, the music is relatively simple (monophonic or chords with simple texture). How is the model on a dataset where we have less track but more complicated? (E.g., a dataset like Maestro)

5. The demos have very different instrumentation and beginning. Are all these samples generated from scratch?

6. The demos have clear repeats and actually sometimes too many repeats. However, grasping the whole song structure and keeping the song interesting is still difficult. I wonder if the claim for music structure is too strong. Also, I noticed the melody is not so human-like compared to existing music generation algorithms. 
limitations: Limitation of this paper is discussed in the weakness section and question section. My suggestion is to consider claim the model to achieve a weaker goal, such as, a music generation algorithms for pop music which consider structural repetition.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a form of structured, sparse self-attention designed to work well for symbolic pop music generation
with transformer networks.  Instead of uniformly attending to all previous tokens in a sequence, the proposed method
reduces computation by only attending to 1) per-bar summary tokens (coarse attention) and 2) full-token sequences _only_
within a small fixed set of "structure-related bars" (sparse fine-grained attention).

This structure incorporates domain-specific knowledge (and empirical analysis) of common musical structure to identify
the set of "structure-related bars" and results in high quality generation with 1) reduced computation compared to
full-context attention models and 2) improved long-term structure compard to several baselines, including long-term
repetitions/recapitulations commonly found in western pop music.

strengths_and_weaknesses: Strengths
- Straightfoward, well-motivated approach, works well on a challenging area of generating long sequences (10k+ tokens)
  with coherent long-form musical structure.
- Well rounded evaluation, including nice demos.  I especially appreciate the inclusion of subjective evaluations given
  the lack of objective metrics for realistic musical structure.
- The text is well written and clear.

Weaknesses
- Very specific application in music, perhaps somewhat narrowly at that (see below).
  - It would be most interesting to hear whether the authors imagine similar ideas applying to NLP, e.g., by incorporating
    sentence-level summary tokens to cheaply expand the potential context and/or explicitly attending to, say, the first
    and last sentence of nearby paragraphs, analogous to the proposed structure-related bars.
- The repeated characterization of the proposed method reducing the computation complexity of attention from quadratic
  to linear (e.g., lines 69-71 and elsewhere in the text) is, at best, overstated.  Since the ratio of bars to tokens is
  a roughly fixed proportion, the proposed model still requires quadratic attention across bar-summary tokens, albeit
  scaled down by a large factor corresponding to the average number of tokens per bar (~150 according to the paper).
  The text should be updated to clarify this claim.

questions: - How are bar lines determined?  Are bar boundaries explicitly represented in all songs of the Lakh MIDI dataset, or do
  they need to be inferred somehow?

- Sec 3.2, lines 159-166: Would this distribution differ in other corpora or genres?  Seems like enforcing such a
  constrained bar relation pattern would make it more difficult to model music with unusual structures, a limitation of
  the proposed approach.
  - Also, according to Sec A.1, only 4/4 music is considered.  Is the proposed structure not suitable for other
    time-signatures?  Was time-signature generalization considered empirically at all?

- line 172: $\bar{N(i)}$ notation is not defined.  Does this simply indicate non-structure bars?

- line 187: It seems that this method could easily be be considered a form of sparse attention with a predefined
  structure?

- line 193: how much smaller is "much smaller"? I'd expect an average bar length of something like 8 or 16 notes?  line
  202 claims 158 tokens per bar on average.  Why is this so much higher?  It would be helpful to future readers to
  elaborate on this.  Presumably it is related to the REMI encoding shown in Fig 1 which uses 5+ tokens per note
  (separating the note identity, pitch, velocity, duration, instrument, etc)?

- Sec 4.4. lines 279-281: The claim that FC-attention is linear feels a bit overstated. The number of bars $b$ will
  scale linearly with $n$ since bar lengths are generally independent of overall sequence length, i.e., $b = n / K$
  assuming K token per bar.  So the overall complexity would be $O(b n) = O(n^2/K) = O(n^2)$, remaining quadratic in
  sequence length.  Due to the large discount factor $K$ it is clearly much sparser than full attention, but the
  *asymptotic* complexity remains quadratic even if it is reduced by a large factor.

- Fig 5: Why aren't other methods included in this figure?  E.g., transformer XL?
limitations: As described above, it would be helpful for the authors to address the limitations of the proposed approach across musical genres, time-signatures, etc.  Is the set of  "structure-related bars" used in the experiments thought to be generally applicable across musical corpora, or should they be empirically determined using an analysis similar to the "Structure-Related Bar Selection" procedure on a dataset-by-dataset basis?

Furthermore, any exposition on the potential applicability of similar ideas to other domains, e.g. long-form text generation, would be interesting to see.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents Museformer, a music-inspired transformer architecture which uses fine-grained and coarse-grained attention to be able to model very long sequences of music while being able to maintain structure-awareness. The proposed architecture reduces the memory and computation cost from quadratic in standard transformers to linear which helps with scaling to long sequences. The transformer architecture involves learning at individual note-level as well as bar-level. The authors determine so-called structure related bars for every bar in an input piece. These are determined using prior knowledge of music and are chosen as the previous 2 bars (for short-term context) along with previous bars at a distance of 4, 8, 16, 24 and 32. This is obtained by analyzing the corpus which is a pop music corpus and most songs use standard 4/4 rhythms. The fine-grained attention attends to each token of all structure-related bars, whereas the coarse-grained attention attends to a summary token for every other previous bar. Finally a combination step combines the information from the current bar, the fine-grained attention, and the coarse-grained attention using another attention layer.

The evaluation includes comparison against various Transformers for long sequences. The authors use Perplexity as an objective metric and also conduct a small scale user study with 10 participants. The Museformer architecture outperforms all other baselines on all metrics. An ablation study also shows the benefits of coarse-grained attention and structure-related bar selection. The authors also demonstrate the time and space complexity benefits against vanilla transformers for varying sequence lengths.
strengths_and_weaknesses: Strengths:
The museformer architecture uses domain-knowledge of music to reduce the complexity of Transformers to be able to generate long sequences of symbolic music. There are other papers that attempt to address the issue of maintaining long-term structure in music, but most attempts are unable to consistently maintain long-term structure. The demo samples from the authors demonstrate repetitions at large gaps as well as short term repetitions. While I do find the samples to still lack quality, the work is a step in the right direction. Additionally, the authors choose strong baselines for long-sequence learning and conduct the relevant ablation studies to show the benefits of their components.

Weaknesses:
I find the evaluation to be a little weak. The authors focus on long-term structure but do not perform any objective comparison of the generated structures to structures from real music. Some kind of self similarity matrix comparison might be interesting to look at. Some statistical analysis as well. I get the feeling that the museformer really focuses on producing almost exact copies of structure-related bars and real music might have more variation. Some visual comparison of these SSMs might be interesting. The writing could also be improved a bit but that is a minor weakness. There are a few phrasing issues.

comment:
L231: please fix the commas in the numbers. This took me a while to figure out that perplexity is measured over 1024, 5120, and 10240 tokens. I got it after looking at the tables but was very confused at first glance.
questions: - Why do the authors choose to make the model causal during training? I get that the inference process has to be causal, but during training wouldn't it be better to also attend to future bars?
- Please see weakness section for question about evaluation of generated structure.
limitations: The authors briefly talk about the complex implementation of dynamic sparse transformers as being a limitation. I would like the authors to also possibly discuss some limitations of the generated results; for example, I find that the drums were very unnatural. I'm sure the authors also have some thoughts about the generation quality which readers might be interested to hear about.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposed a new transformer-type model that considers long-range musical structures while maintaining linear computational capacity. By separating event-level and bar-level, short-range information is encoded between local tokens while long-range information is encoded in attention in selected bars. As a result, the trained model generated consistent musical structure and achieved better perplexity and human evaluation.
strengths_and_weaknesses: Pros
- This work suggested noble transformer architecture considering musical structures.
- Recorded better perplexity and human evaluation compared to previous models.
- Showed consistency in long phrases in audio examples.

Cons
- This approach is not a fundamental improvement of the quadratic attention problem.
- Fits for limited musical structure.
questions: - Is there any extension of this approach from bars to higher structure, such as motifs, phrases, and so on? If more higher (abstract, sparse) structure does not work, what will be the source of the problem?

- Will the model able to encode musical information that spans longer than 32 measures? 

limitations: - What is key difference between linear attention-type transformers (such as Linformer)? As far as fixing structure of bar-level attention, overall model will effectively work as similar as linear transformer with limited attention view of 32 measures, if data is sufficiently prepared.
review_title: null

Meta-review: metareview: This paper presents a novel attention selection scheme for modeling the long-term structure of symbolic music. Modeling multi-track music with repetitive structure is a challenging task, especially due to the quadratic memory problem of the transformer. The authors propose measure-limited fine-grained attention and coarse-grained attention that attends to summarizations of measure, not individual tokens. The experiments show that the proposed method can achieve better perplexity compared to the other transformer variants in multi-track symbolic music generation. 

One of the main questions raised by the reviewers was the musical quality of the generated examples, such as how the repetitive structure was generated or how it would work in other types of dataset. The authors responded with many additional results that can answer those questions. Though one of the reviewers was not fully satisfied with the evaluation, which is valuable criticism that can improve this paper, the majority of reviewers have agreed that the paper has a clear contribution to be introduced in NeurIPS. 

## Strengths
- The paper proposes a nicely working solution for the widely-known but critical limitation of long-sequence modeling of a transformer. The idea is clear, and the implementation is sound. 
- The experiment result and the presented generation examples show that the proposed attention strategy can help model long symbolic music sequences.

## Weaknesses
- The evaluation lacks musical analysis of the generated results. It is complicated to evaluate the quality of generated music samples only using the NLL and listening tests with limited examples and subjects. After the reviewer's feedback, the authors presented additional results, including repeat statistics and self-similarity matrices of each track. Since the criticized weaknesses can be largely improved by these additional results, I hope the authors can figure out a good solution to include these contents somehow in the revised paper. 
- Even though it can be applied to many kinds of music, the selection of structure-related bar (1,2,4,8, 16) is a strong assumption. It is a reasonable choice if one has to select it rule-based, but there is a clear limitation of this approach, as the authors also agreed.

## Recommendation 
- BP-transformer for natural language has a similar concept to the proposed coarse-grained attention in terms that each token attends to the summary of far-located tokens. 
  - Ye, Zihao, et al. "BP-Transformer: Modelling long-range context via binary partitioning." arXiv preprint arXiv:1911.04070 (2019)
- As one of the reviewers criticized, a detailed analysis of generated examples is necessary. The authors might consider asking music experts to do a qualitative analysis on the generated examples. Current subjective evaluation has many limitations as a musical background of subjective can be very important to evaluate the quality of music structure. Even though it would be difficult to include those detailed analyses in the main paper, it would be beneficial for the authors and readers to understand the characteristics of Museformer truly. The following paper is a good example of analyzing machine-generated music. 
  - Sturm, Bob L., and Oded Ben-Tal. "Taking the models back to music practice: Evaluating generative transcription models built using deep learning." Journal of Creative Music Systems 2 (2017): 32-60.
- The result also suggests the possibility that coarse-grained bar-level summarization is more important than fine-grained attention with structure-related bars. It is not clear which part is more essential to model long-term structures. Therefore, I recommend authors do an ablation study not only just with perplexity score but with detailed analysis of the actual generated examples from two different models (coarse-grained only, and fine-grained only).
confidence: Less certain
Decision: Accept