Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models
Authors: Ziyu Wang
Lejun Min
Gus Xia
Venue: ICLR 2024 spotlight (2024)
Keywords: Cascaded generative models
Diffusion models
Symbolic Music Generation
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose a hierarchical music language together with a cascaded diffusion model for whole-song generation of symbolic music.

Abstract: Recent deep music generation studies have put much emphasis on long-term generation with structures. However, we are yet to see high-quality, well-structured **whole-song** generation. In this paper, we make the first attempt to model a full music piece under the realization of *compositional hierarchy*. With a focus on symbolic representations of pop songs, we define a hierarchical language, in which each level of hierarchy focuses on the semantics and context dependency at a certain music scope. The high-level languages reveal whole-song form, phrase, and cadence, whereas the low-level languages focus on notes, chords, and their local patterns. A cascaded diffusion model is trained to model the hierarchical language, where each level is conditioned on its upper levels. Experiments and analysis show that our model is capable of generating full-piece music with recognizable global verse-chorus structure and cadences, and the music quality is higher than the baselines. Additionally, we show that the proposed model is *controllable* in a flexible way. By sampling from the interpretable hierarchical languages or adjusting pre-trained external representations, users can control the music flow via various features such as phrase harmonic structures, rhythmic patterns, and accompaniment texture.

Published as a conference paper at ICLR 2024
WHOLE-SONG HIERARCHICAL GENERATION OF SYM- BOLIC MUSIC USING CASCADED DIFFUSION MODELS
Ziyu Wang"? ? Lejun Min", Gus Xia²
CCmppter Science Department, NYU Shanghai, "Machine Learning Department, MBZUAI riyu, wangenyu. (lejun.min, min, gus. vus.xts xiia)mmbuuai ac. ae
ABSTRACT
Recent deep music generation studies have put much emphasis on long-term gen- eration with structures. However, we are yet to see high-quality, well-structured whole-song generation. In this paper, make the first attempt to model a full music piece under the realization compositional hierarchy. With a focus symbolic representations of pop songs, define hierarchical language, in which each level of hierarchy focuses on the semantics and context dependency at certain music scope. The high-level languages reveal whole-song form, phrase, and cadence, whereas the low-level languages focus on notes, chords, and their local patterns. cascaded diffusion model trained to model the hierarchical language, where each level is conditioned on its upper levels. Experiments and analysis show that our model is capable of generating full-piece music with recog- nizable global verse-chorus structure and cadences, and the music quality is higher than the baselines. Additionally, we show that the proposed model is controllable in flexible way. By sampling from the interpretable hierarchical languages adjusting pre-trained external representations, users can control the music flow via various features such phrase harmonic structures, rhythmic patterss, and accompaniment texture.
INTRODUCTION
In recent years, we have witnessed lot of progress in the field of deep music generation. With significant improvements on the quality of generated music (Copet et al., 2023; Thickstun et al., 2023) on short segments (typically ranging from a measure up to phrase), researchers start to put more emphasis on long-term structure as well as how to control the generation process in a musical way. The current mainstream approach of structural generation involves first learning disentangled latent representations and then constructing a predictive model that can be controlled by the learned representations external labels (Yang et al., 2019; Wang et al., 2020b; Wei et al., 2022; Chen et al., 2020). However, generating an entire song remains an unresolved challenge. As composi- tions extend in length, the number of involved music representations and their combinations grow exponentially, and therefore it is crucial to organize various music representations in a structured way.
We argue that compositional hierarchy of music is the key to the solution. In this study, we focus on symbolic pop songs, proposing a computational hierarchical music language and modeling such language with cascaded diffusion models. The proposed music language has four levels. The top- level language describes the phrase structure and key progression of the piece. The second-level language reveals music development using reduction of the melody and a rough chord progression, focusing on the music flow within phrases. The third-level language consists of the complete lead melody and the finalized chord progression, which is usually known as a lead sheet, further detailing the local music flow. At the last level, the language defined as piano accompaniment. Intuitively, the language aims to characterize the intrinsic homophonic and tonal features of most pop songs- a verse-chorus form, chord-driven tonal music flow, and a homophonic accompaniment texture.
We release the complete source code and model checkpoints https: / /gg thub. com com/x2wangg/ whol e- song- gen. The demo page is available at https: ctctp:://vooeesongeen .g gitb.u. io.

---

Published as a conference paper at ICLR 2024
We represent all levels of the symbolic languages multi-channel images and train four layers of image diffusion models a cascaded fashion, one for each level of the music language. The generation scope of the first layer is full-song and up to 256 measures, the scope of the second layer is 32 measures, and the third and the fourth layer each has a scope of 8 measures. Additional autoregressive controls added to the low-level diffusion models strengthen long-term temporal coherency. Experimental results show that our model is capable of generating well-structured full- piece music with recognizable verse-chorus structure and high music quality.
Moreover, at each level, optional external conditions can be added via the cross-attention mechanism of diffusion models control the generation process at each level of the hierarchy. As demonstra- tion, we add long-term control of chord progression, local control of rhythmic and accompaniment pattern to the corresponding levels of the hierarchy. All the external controls uses pre-trained la- tent codes from existing music representation learning models. We show that these controls can effectively guide hierarchical generation a more customizable way.
In summary, the contribution of the paper is as follows:
We achieve high-quality and well-structured whole-song generation with cascaded dif- fusion models. Objective and subjective measurement show that both monophonic lead sheets and polyphonic accompaniment generated by our model have more identifiable phrase boundaries, better-structured phrase development in similarity and contrast, and higher music quality compared to basclines.
We propose computational hierarchical music language structural inductive bias, making the training process decomposable and efficient in terms of data and computing power utillization. Also, the hierarchical languages can be extracted automatically without manual annotation of music structure.
Our model enables flexible and interpretable controls, with not only our proposed hier- archical language but also with external pre-trained latent representations, such as chord, melodic rhythm and accompaniment texture.
RELATED WORK
In this section, we first review music structure in musicology in Section 2.1, followed by music struc- ture modeling in deep music generation approaches in Section 2.2. Finally, in Section 2.3 we review the state-of-the-art deep generative methods relevant to the problem of whole-song generation.
2.1 MUSIC STRUCTURE MODELING
Traditional music theory focuses on the analysis of music structure in terms of counterpoint (Clementi et al., 2010), harmony (Schoenberg. 1983), forms (Koch, 1787), etc. In the early 20th century, more comprehensive theory, Schenkerian analysis (Schenker, 1979), emerged with a fo- cus on the generative procedure of music. The theory introduces compositional hierarchy of music, aiming to show how piece is composed from its background, the normal form music, to its middle ground, where music form and rough music development are realized, and finally to the foreground, the actual composition.
Nowadays, compositional hierarchy is still prevalent in modern musicology. Notable developments include Tagg (1982), a general compositional hicrarchy for pop music, and Generative Theory of Tonal Music (GTTM) (Lerdahl & Jackendoff, 1996), a theory focusing the definition and analysis of formal musical syntax. From a computational perspective, these studies provide more formal music features and computer-friendly generative processes (Hamanaka al., 2015; 2016). The focus of this paper is to further leverage the compositional hierarchy of music to develop fully computable language and model it with deep neural networks.
2.2 STRUCTURED DEEP MUSIC GENERATION
Recent advances in deep generative models have greatly improved music generation quality, pri- marily by more effective modeling of the local musical structure in two ways: implicit and explicit.
2

---

Published as a conference paper at ICLR 2024
Implicit approaches, exemplified by models such Music Transformer (Huang al., 2019), Muse- BERT (Wang& & Xia, 2021), and Jukebox (Dhariwal et al., 2020). learn structures by predicting and filling musical events, often revealing context dependencies via attention weights. Explicit ap- proaches leverage domain knowledge to define music features or extract interpretable music repre- sentations, allowing the learning of structures like measure-level pitch contour and accompaniment OYang et al., 2019; Wang et al., 2020b; Dai et al., 2021; Zhao et al., 2023). This study aims to combine both explicit and implicit approaches and further model phrase and whole-song structures. The explicit modeling lies in our definition of computational hierarchical music language, and the implicit modeling of the structure lies in the cascaded diffusion models.
2.3 DIFFUSION AND CASCADED MODELING FOR MUSIC GENERATION
Diffusion models, after their success in image and audio domains, have very recently been applied to music generation (Mittal et al., 2021; Li Sung, 2023; Min et al., 2023). Besides high sample quality, diffusion models naturally lead to coherent local structures with the innate inpainting method Lugmayr et al. (2022), ie., by generating music segments conditioned on surrounding contexts. As for long-term structures, we recently saw the design of cascaded diffusion modeling in Moûsai Schneider et al. (2023), which generates high-fidelity audios using multi-scale sampling.
In this study, focus is on symbolic music and adopt the idea of multi-scale generation with cascaded models. Additionally, we integrate the cascaded process with the proposed hierarchical music language, so that each layer of the diffusion model focuses certain interpretable aspect of music composition. In particular, all levels of music languages are defined as image-like rep- resentations. Inspired by sketch- and stroke-basee image synthesis Cheng et (2023), we model hierarchical music generation by regarding high-level and low-level music languages as the back- ground and foreground "strokes", respectively.
METHODOLOGY
Our model for whole-song generation is realization of the music compositional hierarchy. In this section, we first introduce the definition of our hierarchical music languages in Section 3.1. Then, we discuss how model these languages via cascaded diffusion models, where each level of the language is conditioned on its upper levels. We show the data representation of these languages in Section 3.2. The training and inference of the model are discussed Section 3.3 and Section 3.4, respectively.
3.1 DEFINITION OF HIERARCHICAL MUSIC LANGUAGES
We define a hicrarchical music language with four levels to reveal the generative procedure of music, as shown in Table 1. The highest level, Form, includes music keys and phrases. This is followed by Reduced Lead Sheet, which contains reduced melody and simplified chords. The third level, Lead Sheet, includes the lead melody and chords. The final level, Accompaniment, consists of the piano accompaniment. The key idea behind this hierarchical design lies in the relationship among the four levels- more abstract music concepts higher levels realized by stylistic specifications at lower levels. For example, a lead sheet is an abstraction implying many possible ways to ar- range the accompaniment that share the same melodic and harmonic structure, while an instantiated accompaniment is one of the possible realizations showing the accompaniment structure in more detail.
Note that for Form, Lead Sheet and Accompaniment, there are established music information re- trieval algorithms for labeling. In contrast, the Reduced Lead Sheet is a unique design, which we refer the readers to Appendix A for details.
3.2 DATA REPRESENTATION
While music scores are inherently symbolic, we transform them into continuous, image-like piano- roll representations for better compatibility with diffusion models. Specifically, languages at all levels are represented by multi-channel images (examples are shown in Appendix A). The image width represents sequence length under different resolutions, and the height represents 128 MIDI
3

---

Published as a conference paper at ICLR 2024
Table 1: Definition of the four-level hierarchical music language. We use m for measure, b for beat, for step, to represent the temporal resolution. M denotes the number of measures in a piece, denote the number of beats in a measure, and 8 denotes the number of steps in beat.
Languages (res.)
Specification
Data Representation
Structural Focus
Key changes
Form (m)
X E 18x
Music form
Phrases
Reduced Lead Sheet (b)
Melody reduction
X² E R2xxMx128
Phrase similarity, phrase development & cadence
Simplified chord
Lead melody
Melodic pattern, similarity & coherence
Lead Sheet (s)
Chord
Accompaniment (s)
Accompaniment
x
E M2x6yMX122
Acc. pattern, similarity & coherence, Mel-ace relations
pitches or 12 pitch classes. We denote the piece length to be M measures, each measure containing y beats, and each beat containing ō steps. In this paper, we consider {3, 4) and 8 = 4.
The language Form is a sequence of keys and phrases under the resolution of one measure. Keys represented by K E 2 where tonic information and scale information are stored on the two channels with binary values. For phrases, we use P E IR 6x Mx1, where six channels correspond six phrase types (e.g.. verse and chorus, see Table 3 for more detail), and the pixel values indicate measure countdown. Formally, let ma. ..., m2g + L - 1 be the indices of a L-measure phrase of type to. then for my S m < mg + L.
Pli, == 1 {i= m mo
(1)
L
We broadcast P to match the pitch-axis of K and define the first-level language Form as X concat(K, P) IXxMx12 The other levels of languages use piano-roll representa- tion. Reduced Lead Sheet is represented by X² E 122xyMx under the resolution of one beat, where two channels correspond to note onset and sustain. Both melody reduction and simplified chord progression share the same piano-roll using different pitch registers. Similarly, Lead Sheet uses X3 E 2xx 128 to represent the actual melody and chords, and Accompaniment uses X E M2x67Mxx22 to represent the accompaniment, both in the same resolution of one step.
Note that for the four levels A 1,....,4, 4, X have different shapes. In the following sections, we write [ C 2, 3 to denote the concatenation along the channel axes with possible broadcasting and repetition operations. For example, X can be expanded y times in width and repeated II times in height to be concatenated with X², resulting in tensor X E I 1010x x128 Additionally, we write the time-series expression X to denote [ for simplicity.
3.3 MODEL ARCHITECTURE
Whole-song music generation is achieved by generating the four levels of hierarchical music lan- guages one after another in top-down order (as shown in Figure 1). For each level, we train a diffusion model to realize the current-level language based on the existing upper-level languages. The actual scopes (image widths) of these diffusion models are generally the same, yet the music scopes vary significantly since the resolution in lower-level languages is finer. In this paper, for level = 1,....,, 4, we set the actual scope by to be by = 256 and $2.4 = 128, which means the music scopes for these levels are 256 measures, 128 beats, 128 steps, and 128 steps, respectively. In the usual setting when = 5 = 4, the music scopes of the models are 256 measures, 32 measures, measures, and 8 measures, respectively. Consequently, except that the first layer is unconditional generation of the whole sequence, the generation at all the other layers are essentially conditional generation of music segments sliced from the entire sequences.
The generation of a music language slice X Ex at level k * 1 can be conditioned on multiple resources inside and outside the defined hierarchy. In this study, our model designed take in three sources of structural conditions:
Background condition. We regard the generation as realization of existing higher-Ievel languages at the corresponding scope <k bit+b,
where the higher-level language segments like sketch images
4

---

Published as a conference paper at ICLR 2024
Stage Generation
Stage Reduced Lead Sheet Generation
Hierarchical Music Languages
Form
128 beats
Up measures
Sheet
Lead
Accompaniment
Current Generation Segments
XN
Genarated/ Genarated/Known Known Segments
Stage 3: Lead Sheet Generation
Stage 4: Accompaniment Generation
Gaussian Nolse
128 steps
External Controls
Diffusion Models
DM
Backbone model
Cross attin, autoregressive control
DM
Cress-attn, for external loptional)
Figure 1: The diagram of cascaded diffusion models for hierarchical symbolic music generation.
directly guiding the current generation. Background condition is applied by concatenating the input with X br along the channel axis.
Autoregressive condition. The segment should not only realization of the background condi- tion, but also coherent with prior realizations X <k For example, the realization of a verse phrase <t
in the end of the composition is usually similar to the realization in the beginning. In our model, we make an autoregressive assumption that X* are known. We select Sş relevant music segments <t
prior to based on defined similarity metric on X These music segments are encoded into latent representations and being cross-attendes in the diffusion models.
External condition. Besides the compositional hierarchy, music generation can controlled by other external conditions. These conditions can be stylistic controls of multiple scopes (Wei & Xia, 2021; Yang et al., 2019; Wang et al., 2020b), or cross-modality controls of text (Zhang et al., 2020) audio (Wang al., 2022). As an illustration of our model compatibility, we pre-trained latent representations of chords, rhythmic pattern, and accompaniment texture as the control for Reduced Lead Sheet, Lead Sheet, and Accompaniment generation, respectively. At each level k, we denote the array of external latent codes by Z by which are cross-altended in our diffusion models.
For all four levels, we adopt 2D-UNet with cross-attention similar to Min et al. (2023) as the backbone neural architecture and make several modifications. First, the input channels are increased allow background condition. Second, autoregressive and external conditions are fed through cross-attention layers with classifier-free guidance (Baykal et al., 2023). In Appendix B we include more details on the model architecture and training. Mathematically, we use diffusion model the conditional probability of multiple levels of music segments. Let the backbone model at level k be denoted by
602 (Ii, n. y y yxx))
where 9 is the model parameter, n = 0. O.., N the diffusion step, Fn is the input image mixed with Gaussian noise diffusion step n, and bb yat and ex are background, autoregressive, and external control, respectively. Our training objective to model the probability
pe (X X
x Sk z
(3)
et
under the loss function
L((O)) = E X.1
Lo (X iitbb
x x
Z
Z
(4)
batb,
20:
where
to, (x. bs nx yxx) Be llle ea, (Fn yb bs yar yy )1
(5)
5

---

Published as a conference paper at ICLR 2024
3.4 THOLE-SONG GENERATION ALGORITHM
At the inference stage, we leverage the conditional probability (see Equation 3) achieve whole- song generation by autoregressively inpainting the generated segments. Inpainting is a commonly- used method in diffusion models for image editing, and is developed method for sequential generation (Min et al., 2023). In our algorithm, we use a hop length of hy = by 2 for inpainting, and the algorithm shown in Algorithm I. Note that in training, we zero-pad X to 256 measures, so during inference, we derive the actual song length by finding the first all- entries of the generated X This process denoted by INFERSONGLENGTH Here, we use
quasi-autoregressive
X be peo (1x tug
ck bi
Zettbe
(6)
indicate the distribution of the second half of the sequence conditioned on the first half via in- painting, together with the background, autoregressive, and external conditions.
Algorithm Whole-song generation algorithm.
Constants: Resolution factor for each level Fi = 1,72 = 7,73 = 87 Input: External control z S ls I 4) (optional)
1: X 1 ~ Pe; (-10, 0. 0)
2: M <- INFERSONGLENCT((X"
3: for k = ......440
4:
X -po (-
Dibe 0, Zobb))
5: lo hn, 2h1 ... M by do
6: X Xitheetbb pa, (1x tit+
X <k
X ck Z
bi+bb*
7: end for
8: end for
9:
return
{x*11 545
ANALYSIS OF STRUCTURAL MUSIC GENERATION
In this section, we show example of whole-song music generation of 40 measures in Figure 2. The given Form of the piece has simple verse-chorus structure with 4-measure verse and 8-measure chorus phrases appearing multiple times.
The generated music shows a clear music structure. The melodies of three verses all consist of syncopated rhythm in narrow pitch range, while the melodies of two choruses are both relatively lyrical with broader pitch range (indicated by shaded rectangles). The accompaniment pattern pre- dominantly features eighth notes in verses and sixteenth notes in choruses. Moreover, the cadences at phrase boundaries are clearly indicated by the tonic or dominant chords and the "fill" in the ac- companiment (indicated by dotted red rectangles). Furthermore, second half is stronger than in the first half, realized by more active pitch and rhythm movements and higher pitches (indicated by shaded rectangles with dotted borders). Such intensity changes make the composition go to a climax point before ending, showing a well-formed chronological structure.
notice the music intensity in the
In Appendix C, we break down the hierarchical generation process and show examples of structural controllability of each level. More generation results are available at the demo page.
5 EXPERIMENTS
We focus our experiments on the generation of Lead Sheet and Accompaniment, the two lower levels of languages. The rationale is that the information of higher-level languages is difficult to evaluate directly, and they are implied at the lower levels. In this section, we first evaluate the structure of full pieces based structure and quality on music segments (8- (8-measure) and whole-song samples (32 measures). For more evaluations, we refer the readers the appendices.
proposed objective metric, and then subjectively evaluate both

---

Published as a conference paper at ICLR 2024
Fm
D5
Abadd2
Fm
D5
Ebsusi
A4 Fm
D6
Fm
Ebb
Ab
R
A4
T>;
Cm
Fm
Bam
Rb
Ab
B8
12
Fm
136
Eb
Ab
Fm
D6
15
A6
Fm
D;
b4
Fm
Me
Eb
Ab
A4
Fm/As
Db
Cm
Fm
Bbm
E5
Ab
B8 Fm
Eb
Ab
Fm
Db
Eb
Ab
Fm
Db
Eb
Fm
88
13b
Bmm
Eb
Ab
04
Ab
Fm
13
Al47
Fm
13b
Eb
Figure 2: An example of whole-song generation of 40 measures under given Form (Ab major key and " 4444448bb444B8o4 phrases). The three staves (from top bottom) show the generated Reduced Lead Sheet, Lead Sheet, and Accompaniment. Here, rectangles with colored background are used to indicate the appearance of the same motifs verse and chorus sections. Dashed border rectangles with colored background indicate a variation of motifs. We use red dotted rectangles to show where the generated score show a strong implication phrase boundary or cadence. The generated chord progressions in Reduced Lead Sheet and Lead Sheet are identical, shown by the chord symbols.

---

Published as a conference paper at ICLR 2024
5.1 DATASET
We use the POP909 dataset to train our model (Wang et al., 2020a). POP909 is pop song dataset of 909 MIDI pieces containing lead melodies, secondary melodies, piano accompaniment tracks, key signatures, and chord annotations. We pad each song to 256 measures to train Stage 1, and segment each song into corresponding time scopes (128 beats for stage 2 and 128 steps for Stage 3 and 4) with hop size of one measure. 90% of the songs used for training and the rest 10% are used for testing. Training samples are transposed to all 12 keys. In Appendix A, we introduce more details on data processing.
5.2 BASELINE SETTINGS
We construct two baseline models for whole-song Lead Sheet and Accompaniment generation tasks. The two models are modified from two state-of-the-ae phrase-level generation models: diffusion- based one, and a Transformer-basee
one.
Diffusion-based (Polyff- +ph.I.). We add phrase label control to Polyffusion (Min et al., 2023) as the external condition, and use the iterative inpainting technique to generate full pieces. We train two separate diffusion models for Lead Sheet and Accompaniment generation on POP909, both adopting the same data representations as the proposed method. This also serves as an ablation study on the effectiveness of our cascaded model design.
Transformer-based (TFxl( REMM)) Naruse et al. (2022) enables phrase label control on the REMI representation (Huang & Yang, 2020) with Transformer-XL. as the model backbone. Simi- larly, we train two versions for Lead Sheet and Accompaniment generation on POP909.
5.3 EVALUATION
Objective evaluation. For whole-song well-structuredness, we design the Inter-Phrase Latent Sim- ilarity (ILS) metric measure music structure based on content similarity. The metric encourages similarity between music with same phrase labels (e.g.. two verses), and distinctiveness between music with different phrase labels (e.g.. verse and chorus). We leverage pre-trained disentangled VAEs that encode music notes into latent representations and compare cosine similarities in the la- tent space. Given a similarity matrix showing pairwise similarity of 2-measure segments within song, ILS is defined as the ratio between average similarity between phrases of the same types and average similarity between all phrases, and therefore higher values indicate better structure.
Table 2: Objective evaluation of music structure via the proposed inter-phrase latent similarity.
Lead Melody
Chord
Accompaniment
ILS* +
ILS f
IL_ehhd
ILS
Ground Truth
2.28 0.14
2.30 + 0.13
1.42 0.07
1.68 0.09
Cas. Diff. (ours)
2.05 = 0.14
1.32
1.19 : 0.06
Polyff. + ph.l.
0.60 + 0.12
0.76 0.05
0.32 + 0.06
0.61 + 0.04
TFx ph.1.
1.89 0.15
1.71 0.13
0.68 + 0.06
0.71 + 0.04
We compute ILS on lead melody, chord, and accompaniment. Using pre-trained VAEs from Yang et al. (2019) and Wang et al. (2020b), we compute the latent representations of pitch contour and rhythm (i.e., Zp: 2₂) for lead melody, latent Schd for chord, and latent texture 2ext for accompaniment. We pre-define four types of common phrases and let models generate 32 samples for each phrase type, resulting in 128 full songs in total. IISS 0 E {p, chd, txt) are calculated for each song, and we show their mean and standard deviation in Table outperforms basclines the phrase content similarity of chord and accompaniment, indicating its effectiveness in preserving long-term structure.
The results show our model significantly
Subjective evaluation. We design double-blind online survey that consists of two parts: short- term (8 measures) evaluation of music quality, and whole-song (32 measures) evaluation of both music quality and sell-structureddess. Participants rate Creativity, Naturalness, and Musicality for short-term music segments. For whole-song evaluation, we drop Creativity but introduce two more criteria: Boundary Clarity and Phrase Similarity to focus on the structure of the generation. All
8

---

Published as a conference paper at ICLR 2024
(a) Short-term lead sheet gencration
Short-term accompeniment generation
DXE: (oum)
Pilyf!
Dit: (mmm)
Polytt
TP
Paly
Polyt:-ph..l
Creativity
Naturalnoss
Musicality
Creativity
Naturalneas
Musicality
(c) Whole-song lead sheet generation
(d) Whole-sope accompanimemt generation
Cas.tt (ou5)
TEMREED
Cts.itt (ours
TPXIREMM)
TPR//EMD-phi.
PolyI--ppi.
TFAi
PolyIZ-ph.ll
2
Boundary Clarity
Musicality
Boundary Clarity
Musicality
Similarity
Similarity
Subjective evaluation results on music quality and well-structureenesss. GT indicates ground truth samples composed by humans.
Figure 3:
metrics are rated based on a 5-point scale. We constrain whole-song length to 32 measures so that the participants can better memorize the samples and the survey can have a reasonable duration. These generated pieces still preserve a condensed pop-song structure by specifying Form to contain intro, outro and repetitive verses or choruses.
Additionally, for short-term evaluation, we use two more reference models: Polyff and TFx((EEMI), two baseline models without phrase label conditions. This is investigate whether the introduction of phrase control causes degradation in music quality. For each model, we select three samples for both short-term and whole-song evaluation as well as both lead sheet and accompaniment genera- tion, resulting in 3 x x 2 12 groups of samples. Each group of samples shares the same prompt (2 measures for 8-measure samples, and measures for 32-measure samples) and phrase labels (for whole-song evaluation). In the survey, both the group order and the sample order are randomized.
total of people participated in our survey, and the evaluation result is shown in Figure 3. The bar height shows the mean rating, and the error bar shows the 95% confidence interval (computed by wuthin-subject ANOVA). We show that our model significantly outperforms basclines in the structural metrics of whole-song generation, especially in accompaniment generation. Our model consistently outperforms baselines in terms music quality both short-term and whole-song generation, proving that the introduction of compositional hierarchy does not hinder the generation quality.
CONCLUSION
In conclusion, we contribute the first hierarchical whole-song deep generative algorithm for sym- bolic music. The current study focuses on the pop music genre, and experimental results demon- strate that our model consistently generates more structured, natural, and musical outputs compared to bascline methods, both at the whole-song and the phrase scales. Additionally, our model offers extensibility, allowing flexible external controls via pre-trained music embeddings. Our approach relies on two key components: hierarchical music language that balances human interpretability with computational tractability, and a cascaded diffusion architecture that effectively captures the hierarchical structure of entire compositions through both top-down and context-depender mecha- nisms. It demonstrates that a strong structural inductive bias can lead to more effective and efficient learning for deep music generative models, and such methodology is potentially useful for other domains as well. In the future, we plan to extend our hierarchical language and generation approach to both multi-track symbolic music and music audio.

---

Published as a conference paper at ICLR 2024
REFERENCES
Gulcin Baykal, Halil Faruk Karagoz, Taha Binhuraib, and Gozde Unal. Protodiffusion: Classifier- free diffusion guidance with prototype learning. CORR, abs/2307.01922 2023. doi: 10.48550/ arXiv.2307.0199 URL https: / /ow. wrg/oorgg//........ooo..oo...... 10. 1....45550 48550/ arxiv 2307.01924.
Ke Chen, Cheng-i Wang, Taylor Berg- Kirkpairka, and Shlomo Dubnov. Music sketchnet: Con- trollable music generation via factorized representations of pitch and rhythm. In Julie Cumming, Jin Ha Lee, Brian McFee, Markus Schedl, Johanna Devaney, Cory McKay, Eva Zangerle, and Timothy de Reuse (eds.), Proceedings of the 21th International Society for Music Information Retrieval Conference, ISMIR 2020, Montreal, Canada, October 11-16, 2020, pp. 77-84, 2020. URL http: //archaiiieieeeeeeeeeeeeeeaaaaaa -smir net/ /isssssm r2000 Saaper/000140e
Shin-I Cheng, Yu-Jie Chen, Wei-Chen Chiu, Hung- Yu Tseng, and Hsin- Ying Lee. Adaptively- realistic image generation from stroke and sketch with diffusion model. In IEEE/CVF Winter Conference on Applications of Computer Vision WACV 2023, Wuikoloa, HI, USA, January 2- 7, 2023, pp. 4051, IEEE, 2023. doi: D,.109/WACV56600 (ENVV66688202200000 2023.00404. URL https: //doi.orgi 10. 1101
20t..111///AV55588080 WACV56588. 2023. 00404.
Muzio Clementi, Carl Tausig, and Karl Friedrich Weitzmann. Gradus ad parnassum. Peters, 2010.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défossez. Simple and controllable music generation. arXiv preprint arXiv:2306.0522 052844 2023.
Shuqi Dai, Huan Zhang, and Roger B Dannenberg. Automatic analysis and influence of hierarchical structure on melody, rhythm and harmony in popular music. In Proceedings of the 2020 Joint
Conference on Al Music Creativity (CSMC-MuMe), 2020.
Shuqi Dai, Zeyu Jin, Celso Gomes, and Roger B. Dannenberg. Controllable deep melody gener- ation via hierarchical music structure representation. In Jin Ha Lee, Alexander Lerch, Zhiyao Duan, Juhan Nam, Preeti Rao, Peter van Kranenburg, and Ajay Srinivasamurthy (eds.). Proceed- ings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021, pp. 143-150, 2021. URL https: peattpps//archivessssss http/aacchhiiiieeeeeeeaaaaaeaeee ismir .net/ ismmi
000017 pdf.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. CoRR, 2as/2005500034/ 2020. URL https::/aaxaa
org/abs// 2205.00. 00341.
Masatoshi Hamanaka, Keiji Hirata, and Satoshi Tojo. aGTTM III: Learning-based time-span tree generator based on pefg. In International Symposium on Computer Music Multidisciplinary Re- search, pp. 387-404. Springer, 2015.
Masatoshi Hamanaka, Keiji Hirata, and Satoshi Tojo. deepGTTM-II. Automatic generation of met- rical structure based on deep learning technique. In 13th Sound and Music Conference, pp. 221- 249, 2016.
Cheng-ZI ZZi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:://openeeeeee
net SetemmmPeddrreeShhhShteee
Yu-Siang Huang and Yi-Hsuan Yang. Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions. In Chang Wen Chen, Rita Cucchiara, Xian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann (eds.), MM '20: The 28th ACM International Conference on Multimedia, Virtual Event / 2020, pp. 1180-1188. ACM, 2020. doi: 01114//394171.3441833 org/ 10. 1ee.1111433999
Seattle, WA, USA, October 12-16, URL https:/ //oi.
3413671.
Heinrich Christoph Koch. Versuch einer Anleitung zur Composition, volume 72. bey Adam Friedrich Böhme, 1787.
10

---

Published as a conference paper at ICLR 2024
Carol L. Krumhansl. Cognitive Foundations of Musical Pitch. Oxford University Press, 11 2001.
ISBN 9780195148367. doi: 10110831appo00000777119144443843 0tt000/9951114777001100000000000 URL https:// wttp::o::ooo/9888/ http://///ooo///////////1//1
001 0001.
Fred Lerdahl and Ray S Jackendoff. A Generative Theory of Tonal Music, reissue, with new preface. MIT press, 1996.
Shuyu Li and Yunsick Sung. Melodydiffusion: Chord-conditioned melody generation using a transformer-basee diffusion model. Mathematics, 11(8):1915, 2023.
Andreas Lugmayr, Martin Danelljan, Andrés Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In JEEE/CVE Conference on Computer Vision and Puttern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 11451-11461. IEEE, 2022. doi: 10. 1..100//VVP52668, 202..01177 URL https:
///ooi.or/////// 10.
2022.0111...111.1110010011.1.1.. 01117.
Lejun Min, Junyan Jiang, Gus Xia, and Jingwei Zhao. Polyffusion: A diffusion model for poly- phonic score generation with internal and external controls. In Proceedings of the 24th Interna- lional Society for Music Information Retrieval Conference, ISMIR, 2023.
Gautam Mittal, Jesse H. Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. In Jin Ha Lee, Alexander Lerch, Zhiyao Duan, Juhan Nam, Preeti Rao, Peter van Kranenburg, and Ajay Srinivasamurthy (eds.), Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 1, pp. 468- 475, 2021. URL https:tpp:.... archi smii r. net / smi r2021 pwwpppppppp..0.....0 pdf.
Daiki Naruse, Tomoyuki Takahata, Yusuke Mukuta, and Tatsuya Harada. Pop music genera- tion with controllable phrase lengths. In Preeti Rao, Hema A. Murthy, Ajay Srinivasamurthy, Rachel M. Bittner, Rafael Caro Repetto, Masataka Goto, Xavier Serra, and Marius Miron (eds.), Proceedings of the 23rd International Society for Music Information Retrieval Confer- ence, ISMIR 2022, Bengaluru, India, December 4-8, 2022, pp. 125-131, 2022. URL https: //archives ismir net/ seeet/ismi2222200 httpa//22222/ppee/0000.... httppppppp/po00. pdd..
Yi Ren, Jinzheng He, Xu Tan, Tao Qin, Zhou Zhao, and Tie- Yan Liu. Popmag: Pop music accom- paniment generation. In Chang Wen Chen, Rita Cucchiara, Xian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann (eds.), MM '20: The 28th ACM International Conference on Multimedia, Virtual Event Seattle, WA, USA, October 12-16, 2020, pp. 1198- 1206. ACM, 2020. doi: 11145//399411.3413722 URL https: /tteps::/ccii http://t...o.....11111111 3394 171 3413721.
Heinrich Schenker. Free Composition (Der freie Satz). Longman, New York, 1979. Translated and edited by Ernst Oster.
Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf. Moûsai: Text-to-music generation with long-context latent diffusion. CoRR, abs/2301.11757, 2023. doi: 10.485500a2Xiv.3311.111 URL https::///// /oii.ii..///// org/10. http:/....550//// (2550//rxXiv www.rxivv.........
Arnold Schoenberg. Theory of harmony. Univ of California Press, 1983.
Philip Tagg. Analysing popular music: theory, method and practice. Popular music, 2:37-67, 1982.
John Thickstun, David Hall, Chris Donahue, and Percy Liang. Anticipatory music transformer.
arXiv preprint 206.023300000600 2023.
Ziyu Wang and Gus Xia. Musebert: Pre-training music representation for music understanding and controllable generation. In Jin Ha Lee, Alexander Lerch, Zhiyao Duan, Juhan Nam, Precti Rao, Peter van Kranenburg, and Ajay Srinivasamurthy (eds.), Proceedings of the 22nd Interna- tional Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021, pp. 722-729, 2021. URL https:// /etes///areee arch vees. smmi r comir.et///.......mmm smi r2021 222011pppppp / 000090.0 pdf.
Ziyu Wang, Ke Chen, Junyan Jiang, Yiyi Zhang, Maoran Xu, Shuqi Dai, Guxian Bin, and Gus Xia. Pop909: A pop-song dataset for music arrangement generation. In Proceedings of 2 International Conference Music Information Retrieval, ISMIR, 2020a.

---

Published as a conference paper at ICLR 2024
Ziyu Wang. Dingsu Wang, Yixiao Zhang, and Gus Xia. Learning interpretable representation for controllable polyphonic music generation. In Julie Cumming, Jin Ha Lee, Brian McFee, Markus Schedl, Johanna Devaney, Cory McKay, Eva Zangerle, and Timothy de Reuse (eds.). Proceedings of the 21th International Society for Music Information Retrieval Conference, ISMIR 2020, Mon- freal, Canada, October 11-16, 2020, pp. (62-669, 2020b. URL www.httt///rreee ves ssmir. pdf.
wwwpprappppp... / net /tt/simiii22200000
Ziyu Wang, Dejing Xu, Gus Xia, and Ying Shan. Audio-to-symbolic arrangement via cross-modal music representation learning. IEEE International Conference on Acoustics, Speech and Sig- nal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, pp. 181-185. IEEE, 2022. doi: 10. 00.100//CASSP49220000.9999333 ICASSP43922 2tt...2..2......... 9747884.
URL https: /ttp://ci...ooo/////////// wttp/o..1//.......1111...
Shiqi Wei and Gus Xia. Learning long-term music representations via hierarchical contextual con- straints. In Jin Ha Lee, Alexander Lerch, Zhiyao Duan, Juhan Nam, Preeti Rao, Peter van Kra- nenburg, and Ajay Srinivasamurthy (eds.), Proceedings of the 22nd International Society for Mu- sic Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021, pp. 738-745, 2021. URL https: archives i ismir. net smi r2021 httpp/aapppp/aaae/eeee 000092 pdf. htt.mm//wweppp///////am
Shiqi Wei, Gus Xia, Yixiao Zhang, Liwei Lin, and Weiguo Gao. Music phrase inpainting using long-term representation and contrastive loss. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, pp. 186- 190. IEEE, 2022. doi: 10.1 (0.1009//CASSP4492222227.. 1109 SSAS/IICASPPPPP 4322 2022. 20T..22222...2.2.2 9747817.
URL https: /oii org/10.
Ruihan Yang, Dingsu Wang. Ziyu Wang, Tianyao Chen, Junyan Jiang, and Gus Xia. Deep music analogy via latent representation disentanglement. In Arthur Flexer, Geoffroy Peeters, Julián Ur- bano, and Anja Volk (eds.), Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, Delft. The Netherlands, November 4-8, 2019, pp. 596-603, 2019. URL http: http///rrcvvvvseeeee www.s..nnttiiee.eeeeem.meeeeeeee penett/smii20111 https///pppe//eeee 000000.000.2
Yixiao Zhang, Ziyu Wang, Dingsu Wang, and Gus Xia. BUTTER: A representation learning framework for bi-directional music-sentence retrieval and generation. Proceedings of the 1st Workshop NLP for Music and Audio (NLP4MusA), pp. 54-58 Online, 16 October 2020. Association for Computational Linguistics. URL https: http///aaaaanttoooa.a/a wrw.tthhoooooo....22....... org/2020. 11p4musa-1.11
Jingwei Zhao, Gus Xia, and Ye Wang. Accomontage-3: Full-band accompaniment arrangement via sequential style transfer and multi-track function prior. CoRR, abs/2310.16334, 2023. doi: 10. 48550/ARXIV.2300.66 URL https: /ooi www.oiooorg/000 48550/ /a///////rrivv./ arwiv.2311.... 16334.
12

---

Published as a conference paper at ICLR 2024
APPENDICES
The appendices included this paper are designed to provide more technical details, generation samples, and discussions about our study. They are organized into three parts. Appendices A-B introduce more detailed information about data data representation and processing, model archi- tecture, and training; Appendix C provides some more generation samples; and Appendices D-G include more discussions on the external control efficacy, model's memorization v.s. generation ability, model efficiency, and limitations.
A DATA REPRESENTATION AND PROCESSING DETAILS
In this section, introduce the methods to extract the four levels of music languages defined in Section 3.1. We begin by quantizing MIDI files from the POP909 dataset using the provided beat annotations. For the Form language, include only six phrase types (see Table 3 for reference), and extract keys by pitch profile matching, similar the method described in Krumhansl (2001). In the Reduced Lead Sheet language, melody reduction is computed through shortest path reduction algorithm, and chords are downsampled by merging consecutive chords that share the same triad structure (root, third, and fifth) within one measure. The Lead Sheet language uses the vocal melody (i.e.. "MELODY" track) and the provided chord annotations. The Accompaniment language combines the secondary melody (i.e., "BRIDGE" track) with the accompaniment (i.e., "PIANO" track).
simplify the phrase laheling from Dai al. (2020) to
As discussed in Section 3.2, the processed data is converted to image-like data representation. A visualization of the data representation is provided in Figure 4.
Table 3: Definition of phrase type
Phrase Type
Channel ID)
Meaning
"A"
Verse section phrases
g
Chorus section phrases
"x
2
Other phrases with lead melody
Intro section phrases Outro section phrases
"o"
"b"
Bridge section phrases
B MODEL ARCHITECTURE AND TRAINING DETAILS
In this section, we elaborate on the model architecture and training. We first discuss the three condi- tioning methods introduced in Section 3.3 more detail. The configurations of the four generation stages are summarized in Table 4.
Detail on background condition. At each level >>> 1, the background condition x sented by an image having the same width and height as the diffusion output. Thus, the background condition can be concatenated with the input along channel axis at each step of the diffusson process. The background condition will be set to all -1.0 under the probability Puncond 0.2, following classifier-fre guidance (Baykal et al., 2023).
tit--by is repre-
Detail on autoregressive condition. At each level > the generation of X Eitb, is dependent
on past generation X Here we select top-S; past segments of length bg based on their phrase type similarity to the current segment. These music segments are embedded using a 3-layer 2d- convolutional network and fed to the backbone diffusion models by cross-attention mechanism. In our implementation, we set S2 = 3, 1/2 = 32, S3 S4 = 2, and 0/ = b/4 = 64. The autoregressive condition will be set all - 11.0 under the probability Puncond 0.1.
Detail on external condition. The condition for Reduced Lead Sheet is four 8-measure latent codes of chord progression encoded from the chord encoder in Min al. (2023). The condition for Lead
Code for melody reduction is available http:///thhooo////////nnnaanyyee
13

---

Published as a conference paper at ICLR 2024
W
1
oo
bb
Tonic
(12-d)
Scale
(12-d)
Time (Meosure)
(a) The first-level language Form. The six channels of phrases shown the top and brightness indicates phrase countdown (see Equation 1). The two channels of keys are shown at bottom and brightness indicates one-hot tonic and multi-ho halti-hotten scale.
Pitch (128-d)
Time (Seat)
(b) The second-level language Reduced Lead Sheet. Brightness indicates onset and sustain channels.
Marting Maiio
Pitch (128-d)
(Step)
(c) The third-level language Lead Sheet. Brightness indicates onset and sustain channels.
Pitch
(128-d)
Time (Step)
(d) The fourth-leve language Accompaniment. Brightness indicates and sustain channels.
Pitch (128-d)
Time (Step)
Time (Step)
Time (Step)
(e) Zoom-in views of the segments in Figure 4b (left), Figure 4e (middle), and Figure 4d (right) marked with rectangles.
Figure An example data representation of our proposed hierarchical music language.
14

---

Published as a conference paper at ICLR 2024
Sheet is four 2-measure latent codes of rhythmic pattern encoded from the EC encoder in Yang et al. (2019). The condition for Accompaniment is four 2-measure latent codes of accompa- niment texture encoded from the texture encoder in Wang al. (2020b). These latent codes fed to the backhone diffusion models by cross-attention mechanism and set to all -1.0 under the probability Puncond 0.2.
The diffusion models for all four stages use the same noise schedule and training methods. Similar Min al. (2023), the backbone model 2D-UNet model, the encoder and decoder of which contain 4 layers of 2d-convolution with spatial attention at the third and fourth layers. We summarize these common details Table
Table 4: Configuration the conditioning methods in four stages the proposed model.
Form
Red. Lead Sheet
Lead Sheet
Accompaniment
Time scope
256 measures
128 beats
128 steps
128 steps
Output shape
(6, 256, 12)
(2, 128, 128)
(2, 128, 128)
(2, 128, 128)
Background cond.: shape
N/A
(6, 128, 128)
128, 128)
(10, 128, 128)
Background cond.: Puncond
N/A
0.2
0.2
Autoreg. cond.; # of seg.
N/A
3 (8,32, 128)
2
2
Autoreg. cond.: shape
N/A
(10, 64, 128)
(12, 64, 128)
Autoreg. cond.: Puncond
N/A
0.1
0.1
0.1
Ext. cond.: # of latent codes
N/A
4
Ext. cond.: latent dimension
N/A
512
128
256
Ext. cond.: Puncond
N/A
0.2
0.2
0.2
Table 5: The hyperparameter configuration of diffusion model training. The listed attributes are the same across all four stages.
Hyperparameter
Configuration
Diffusion Steps (N)
1000
Noise Schedule
Lincar from 1 to le -
UNet Channels
64
UNet Channel Multipliers
1,2,4,4
Batch Size
16
Attention Levels
3,4
Number of Heads
4
Learning Rate
5e
5
C MORE EXAMPLES ON STRUCTURAL GENERATION
In this section, break down each level of the hierarchical language and show generation examples. For each level, we fix the upper level, and demonstrate a variety of generation results under the upper level control. We also show generation samples that are controlled by external conditions.
Form generation. Below shows examples of Form generated by our model:
(i8) (A3B1628B16) (b6) (B14) (o2)
(i12) (A4A4B12) (b4b4) (A4A4B12) (b4b4 (B16) o4ol)
(i44 (A4A484X5) (b4) (A4b584X4X5) (o2)
(14) (A8B9A8B9X18) (o2o1)
(18) (А444B4В4) (b7) (4444B4B4B4B4) (o4)
(18) (A444B4B4) (b7) (А444B4В4В4В4) (c4)
(18) (ABB8X8X8) (b4b4) (A8BеX8X8X4) (o6o1)
(i4) (A4A4B9) (b3) (A4B1CB9X1) (o2)
(i4) (A4A4B9) (b3) (A4B10B9X1) (o2)
(112) (A16R16) 6) (b4b4b4) 6A1 6816) (c10o1)
15

---

Published as a conference paper at ICLR 2024
Here, use parentheses to manually group music sections for better readability. The results show the model captures verse-chorus form of pop songs: the composition usually starts with intro and ends with outro; verse and chorus appear multiple times with bridge phrases in between. Phrases are usually 4 or 8 measures long, similar to real music samples.
K
Floes?
Al-77
Ab
Ri47
AL-7
Fm?
B67
b)
I5
K+
Km
Fm
Gm
6m
Ont
Pm
Rjxne2
F5
Ab6
Cm7
Ro
B6
Cmilg
Risnel
11
Gm
Pm
Cn7
Um
K5
Cm
Pm
16
hm
Bd47
Bkg
1 B
E
1
Figure 5: Examples of generated Reduced Lead Sheet of "A8" phrase in Eb major. The samples marked with # are controlled by the external condition meaning "unchanging chord progression".
Eisodd
On
On
A6
Gm
Cnn
b)
0)
Eonnd
Cm
E
Ab
A5
B5
On
Ftn
Gn
B6
d)
Eivosd
Gn
On
E5
Ab
A6
Gn
Ot
F1s
Ot
0
Eimud
Gm
Un
Un
g)
On
h*)
14)
Gn
Figure 6: Examples of generated Lead Sheet of "A8" phrase in Eb major given the upper-level Reduced Lead Sheet Figure 6a. The samples marked with * are controlled by the external condition meaning "sixteenth-note rhythm".
16

---

Published as a conference paper at ICLR 2024
114
Figure 7: Examples of generated Accompaniment of "A8" phrase in Eb major given the upper-level Lead Sheet Figure 7a. The sample marked with are controlled by the external condition meaning "Alberti bass".
MC
(a) The external music sample used to control the gen- cration of Figure 6h-i.
(b) The external music sample used to control the gen- cration of Figure 7c.
Figure 8: The music samples for external control.
17

---

Published as a conference paper at ICLR 2024
Reduced Lead Sheet generation with external harmony control. Figure 5a-f show examples of 8-measure generation of the Reduced Lead Sheet level. The results are all controlled by the same Form: 8-measure verse phrase in Ey major key. The generated samples show many ways to develop the melody (different contour and melodic climax positions) and the harmony (different chord types and harmonic rhythm). Moreover, each of the samples has consistent style and usually ends in a tonic or dominant chord indicating the ending of phrase. Moreover, we also apply the external control of "unchanging chord" to generate Figure 5g-h. Such control is achieved by encoding the latent chord representation of a sequence of all Eb: :aa) chords using the pre- trained VAE encoder (Min al., 2023). The results have fewer changes in harmony and the melody reduction alters accordingly.
Lead Sheet generation with external rhythm control. Figure 6b-g show examples of 8- 8-measure Lead Sheet generation controlled by the same Reduced Lead Sheet, shown in Figure 6a. The gener- ated samples follow the pitch contour the melody reduction and differ in local pitch and rhythm patterns. At this level, we also use latent control of "sixteenth-note rhythm" to generate Figure 6h-i. Such a control is achieved by encoding the latent rhythm representation of the sample in Figure Sa using the pre- pre-trained VAE encoder (Yang et al., 2019). The generation examples show melody realization with more frequent onsets accordingly.
Accompaniment generation with external texture control. Figure 7b-d show examples of 8- measure Accompaniment generation controlled by the same Lead Sheet, shown in Figure 7a. The generated samples mainly use arpeggios but different in the exact patterns. Some of the gener- ation has a "fill" in the fourth and eighth measures to indicate phrasing. At this level, we also use latent control of "Alberti bass" to generate Figure 7e, Such a control is achieved by encoding the latent texture representation of the sample in Figure 8b using the pre-trained VAE encoder (Wang al., 2020b). The generation adopts the Alberti bass figure and makes reasonable variations.
D EVALUATION OF EXTERNAL CONTROL EFFICACY
In this section, we evaluate the efficacy of external controls. These controls are achieved by feed- ing pre-trained representations as external condition to each layer of the cascaded diffusion model (introduced in Section 3.3). Specifically, we evaluate three scenarios: (1) chord control in Reduced Lead Sheet generation (Stage two), (2) rhythm control in Lead Sheet generation (Stage three), and (3) texture control in Accompaniment generation (Stage four). In this section, we let zet denote the external control in one of the three scenarios and let ext denote the actual observation from which 2xt is encoded from. Let nut denote the conditional generation results.
Efficacy of control can be evaluated by computing the similarity between the input control and the generation result. We propose a rule-based metric and a latent metric. The rule-based metric directly computes the distance between out and ext in terms of the corresponding features. In particular, for chord control, we compute the &z distance between the given chord condition and the generated chord at each time step; and for rhythm texture control, we compute the l distance of note onsets between the given control and the generated lead sheets or accompaniments. Such distance metric has previously been used to evaluate control efficacy in Ren et al. (2020) and Min et al. (2023). In the latent metric, we encode the generation nut back to the latent code 2 out using the same pre-trained encoders and measure the cosine similarity between out and nxt
There are two reference methods for comparison. First, we use the unconditional mode of our method to serve as baseline where control ineffective. Second, we generate samples by sampling from the Variational Autoencoders (VAEs) that the pre-trained encoders belong to. Specifically, we sample ext from the VAE posterior distribution and sample the rest of the latent codes from Gaussian prior to decode results. By the well-disentangled property shown in the original paper (Yang et al., 2019; Wang et al., 2020b), this reference method indicates the maximum attainable level of controllability.
For each of the three scenario, we randomly selected 32 versions of external control from the test set and generate 128 music segments for each methods. In Table 6, we show the rule-based distance (denoted by dist and latent similarity (denoted by sim for the three generation stages. Experi- mental results show that the use of external condition significantly yields controllability for all three scenarios.
18

---

Published as a conference paper at ICLR 2024
Table 6: Objective evaluation of external control efficacy of chord, rhythm and texture in the three diffusion stages. dis denotes the rule-based distance-basce metric and sim it denotes the latent similarity-baan -baee metric.
Stage I: Chord
Stage 2:
Rhythm
Stage 3:
Texture
dis
sim
dis
sim
dis
sim
Cas. (uncond)
2.09 0.80
0.37
0.09
2.27
0.53
0.14 0.23
3.94 1.46
0.02 0.11
VAB Sampling
0.19 0.47
0.97 0.07
0.14 0.42
0.96 0.04
0.33
0.59
0.90 0.06
Dif. (cond)
1.73 = 1.02
0.48 + 0.14
1.10 4:00:00 0.74
0.75 + 0.16
0.87 + 0.80
0.89 = 0.06
E DOES THE MODEL JUST COPY THE TRAINING DATA?
In generative modeling, critical consideration is whether the model overfits and the generation copies the training data. This section includes a quantitative evaluation focused on the similarity between generated segments and the entire training set. We primarily focus melody similarity, most recognizable aspect of music composition.
Our goal is to measure the Degree of Copying (DoC) with respect set of generated samples. Let æ be a two-measure melody segment from a generated piece, we define Similarity to the Training Set of segment as:
S(x) max sim x'),
where T denotes the training set, æ' is a two-measure segment from the training set, and sim (a, .') computes the similarity between æ and æ'. Here S(x) E [0, 1], and a larger S(x)) shows higher degree of copying, The DoC can be represented by the histogram of S(x). In our experiments, report the mean and standard deviation the histogram. We consider rule-based similarity metric and latent similarity metric follows:
Ruse- similarity metric. We compute the note-wise similarity between two segments by matching the exact onsets and pitches. Let Rwn'w denote the number of notes that appear in both and x' with the same pitch class and onset, and let n and nr denote the number of notes in æ and æ', respectively. The rule-based similarity metric is defined as:
2m
sim (x, x²)
(8)
Latent similarity metric. We also measure the melodic similarity in the latent space because rule- based methods cannot detect indirect copying (e.g., same pitch contour or rhythm). We leverage the pre-trained -CC-VVE (Yang et al., 2019), which learns semantically meaningful and disentangled latent space of pitch contour and rhythmic pattern. We extract the latent code of pitch (denoted as = and rhythm (denoted as x,") of melody segments and compute the cosine similarity terms of both pitch and rhythm:
(2 20
sim" It (xx x²):
(9)
||
||
sim (ar, x²): := (2 200
(10)
||
||27
For both of the metrics, the samples in the training set are transposed to 12 keys to account for relative pitch similarity. Segments that only contain rests are discarded beforehand.
In Table 7, we show the mean and standard deviation of S(x) on the data samples generated using our proposed methods and other baselines used for whole-song generation. We compute the statistics of the test set of POP909 as a reference for no risk of copying. since no song in the training set (or their cover-song versions) appears in the test set. We also design two copy-bots as references for potential risk of copying. The first copy-bot copies a different part of the training set at each measure, which emulates direct copying behavior. The second copy-bot encodes the melodies from the
19

---

Published as a conference paper at ICLR 2024
Table Evaluation on whether the generative models copy the training data. The highlighted data in red indicate potential copying the training set.
Similarity Metric
Sample Source
Sample Size
sim 1b $
sim!
sim,"
Test set (no plag.)
pieces
0.6567 + 0.11411
0.8637 1 0.0486
0.8320 + 0.0680
Copy-bot (plag.)
128 pieces
0.7108 + 0.1159
0.8616 + 0.0526
0.8276 = 0.0699
Copy-bot 2 (plag.)
128 pieces
0.6888 = 0.1628
0.9086 = 0.0340
0.8555 = 0.0411
Diff. (ours)
128 pieces
0.6530 = 0.1321
0.8743 = 0.0491
0.8180 0.0710
Polyff. ph.l.
128 pieces
0.6117 = 0.1162
0.8639 + 0.0487
0.8424 = 0.0622
TFxI(RMM)) + ph.l.
128 pieces
0.6088 + 0.1053
0.8599 + 0.0446
0.8154 + 0.0642
training set and adds noise to the latent representation before reconstruction, which emulates an indirect copying behavior. Experimental results show that our proposed method (as well as the baseline whole-song generation methods) have similar DoC compared to that of the test set. Also, the proposed metrics successfully detect both direct and indirect copying behaviors as the DoCs of copy-bots are noticeably higher. Thus, conclude that our model has very low risk of copying the training set.
F DISCUSSION: EFFICIENCY OF THE CASCADED METHOD
In this section, we conduct theoretical comparison between the efficiency of our cascaded diffusion models and an alternative end-to-end approach, which generates full-piece music with single diffusion model. We evaluate the time and model parameter complexities of both methods. As summarized in Table 8, we demonstrate that end-to-end approaches exhibit quadratic complexities both time and model parameters relative to data length, whereas cascaded approach achieves linear time complexity and logarithmic model parameter complexity.
We base our analysis on a diffusion architecture with a UNet backbone, identical to the proposed architecture. We assume the sequential data has length T' (corresponding to image width in UNct), and the dimension each time step is D (corresponding to image height in UNet). The UNet will first embed the input image have shape (C,T, D) regardless of the number of input channels, where C' is called the base channel size. Additionally, our computation assumes a typical UNet configuration follows:
1. In the contracting path of the UNet, the number of channels each layer doubles, and the width of the feature maps are halved due to max pooling.
2. The expanding path of the UNet mirrors the contracting path.
3. The number of the levels (or model depth) My. scales logarithmically with the input quence length T', i.e., My log, T + co. where co is constant.
Complexity of end-to-end approach. At each level i = 0, ..... My log, T + co. the width of the feature map is I.1 the number of input channels is C' 2', and the number of output channels is The time complexity of the convolution per layer can be computed as:
C 21+1
T -)- DC- (C. (C (f+ O C² 22..222212 OT = 2
(1I)
Equation 11, we regard D and as constants, therefore removing from the complexity term. Summing Equation 11 over all levels results in the total time complexity:
Mr Σ
- = -
(12)
Similarly, at each level i = 0,..., Mr, the number of model parameters can be computed as:
O(C 2¹) (C.2++1) O(C2 2 (
(13)
20

---

Published as a conference paper at ICLR 2024
Table 8:
Theoretical comparison of time and model parameter complexity between cascaded ap- proach and end-to-end approach. 17 denotes the resolution scaling factor. L. denotes the time scope (i.e., receptive field) of cascaded models.
Cascaded Approach
End-to-End Approach
Time Complexity
( O O log, 1)
O
Model Parameter Complexity
resulting in the total model parameter complexity:
Mr
Σ =0(22-1)=0(1
(14)
In conclusion, the time and model parameter complexity for end-to-end approach are both O(T2).
Complexity of cascaded approach. The cascaded models proposed in this paper is tailored for music data. For the analysis in this section, we define a theoretical cascaded approach as follows. To generate sequential data of length T. define K-level compositional hierarchy, and the resolution at each level is scaled by a factor 7/. The top-level (A: = 1) has resolution of L, and for each level k ...... K, the resolution is L 1/"-1, such that the final level K, the resolution is L 1n 17n-1 = T. We train K diffusion models in total, all having the same generation scope L. Thus, the diffusion model level
generates the whole sequence, and the diffusion models for level k = 2, ..... K generate only a slice of the sequence. The model parameter complexity of a single level can be computed by substituting My with Mr, Equation 14:
log, +co
O(T- 0(13).
(15)
Summing up the parameters in K separate models results in the total model parameter complexity:
O(L K) = O log,, T).
(16)
Similarly, for time complexity, each model call is O(L2) (see Equation 12); and at all levels, the generation requires (2n 1) autoregressive iterations (see Algorithm 1). So, the total time complexity is:
= A=1
(17)
Therefore, the cascaded approach has time complexity of (I.2T) and a model parameter com- plexity of O(L log., 7"). The computation implies that the efficiency of cascaded approach will be more significant when >> L. Theoretically, if L a constant (e.g., bounded by the computational resources available), the time and model parameter complexity will become O(T) and Olog,, T), respectively.
G LIMITATIONS AND FUTURE PLAN
Our current model supports a maximum generation scope of 256 measures, typically adequate for pop songs, but insufficient for other genres (e.g., classical music), which may require longer lengths. While our model can generate irregular phrase lengths and theoretically supports both 3/4 and 4/4 time signatures, it does support meter change, and the capability of 3/4 song generation is lim- ited, since the proportion of 3/4 songs in the dataset is pretty low. Additionally, we observe that the generated endings are sometimes not idcal, such as failing resolve on the tonic harmony. We suspect the issue is related to imprecise quantization in the POP909 dataset's 's outro sections. More- over, there is room for improvement overall music quality, as the model occasionally produces obvious flawed samples, such blank measures. To address these issues, we plan to enhance model performance with a more refined architecture and more extensive data training.
21


Reviews:
[{"review_id": "IeRA4KeNfx", "reviewer": "Reviewer_QD5q", "summary": "This paper proposed a new symbolic music generation system that can generate a full pop song. By defining four levels of music representations, the system generates piano roll from coarse to fine using a cascade diffusion model. The system is trained on POP909 and evaluated using objective metrics and a subjective listening test. The objective metrics show that the system can generate music with better long-term structural consistence. The listening test results show that the proposed system generates music of better quality.", "questions": "- (Section 1) \"and therefore we need to organize various music representations in a structured way.\" -> I cannot understand this sentence. Why do you mean by \"organize representations\"?\n- (Section 3.2) \"continuous\" -> I'm not sure what \"continuous\" means here. Are you using binary or real-valued piano rolls?\n- (Section 3.2) \"Both melody reduction and simplified chord progression ...\" -> How were these achieved? A pointer to the Appendix would be helpful here.\n- (Section 3.2) \"13 times\" -> Why 13 times? Isn't it 128/12 = 11 times?\n- (Section 3.3) \"We select Sk relevant music segments\nprior to t based on a defined similarity metric on X<k.\" -> Is this X^k or X^<k? The descriptions in this paragraph are somewhat confusing. From Figure 1, it seems like it's both  X^k and X^<k. Please clarify this.\n- (Algorithm 1) Isn't the song length M also an input?\n- (Section 4) \"40 measures\" -> How do you determine the number of measures to be generated?\n- (Section 5.1) \"... and segment them into 8-bar musical segments with a 1-bar hopping size.\" -> What is this segmentation step for?\n- (Section 5.3) \"Using pre-trained VAEs from Yang et al. (2019) and Wang et al. (2020).\" -> Are these the models you used to extract autoregressive controls?\n- (Section 5.3) How did you select the test inputs for the evaluation?\n\nHere are some other comments and suggestions:\n\n- (Section 1) \"(typically ranging from a measure up to a phrase)\" -> I don't think \"phrase\" has a strict definition.\n- (Section 2.3) It's good to scope this section properly as only symbolic music models are discussed here.\n- (Section 3.1) The musical compositional hierarchy discussed in this paper seems to be constrained to pop music. It would be help to discuss this limitation somewhere in the main text.\n- (Section 3.1) \"counterpoint\" -> I personally find this term confusing as we have melody and harmony here -- it sounds more like a \"reduced/simplified lead sheet\" to me.\n- (Section 3.3) \"The time scopes (image widths) of these diffusion models are more or less the same\" -> What do you mean by \"more or less\" the same? Please avoid such wordings.\n- (Section 4) Having this qualitative analysis before introducing the dataset is somewhat misleading. I don't even know what I should expect. Please consider rearranging the sections.\n- (Section 4) \"In Appendix A.4, ...\" -> I would love to see Figure 4 in the main text rather than Figure 2. The sheet music might be hard to understand for ICLR readers.\n- (Section 4) \"long (32 measures)\" -> This is still far from whole song evaluation.\n- (Section 5.3) \"whole-song (32 measures)\" -> I think 32 measures is still far from whole song generation.\n- (Figure 3) It would be great to also include the ground truth for 32-measure generation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is a timely and significant contribution to the field of symbolic music generation. As far as I know, this is the first deep music generation model that can generate a full structural song given high-level structural hints.\n- The demo is impressive! Some of the samples are too good that I wonder if there is some overfitting issue (some nearest neighbor analysis might help clear these doubts).\n- The proposed model is clever. Figure 4 clearly shows how the musical compositional hierarchy imposes a proper inductive bias to the system.", "weaknesses": "- A discussion on the limitations of the proposed system is missing. I see two main limitations: First, the musical compositional hierarchy adopted here is constrained to pop music. Second, the high-level form and structures still need to be provided.\n- One thing missing in the evaluation is some nearest neighbor analysis to check if the model is returning part of the training data directly.\n- The evaluation doesn't really measure the capability of whole-song generation, but there is no proper baselines as far as I know, so it's fine.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R8CnjLBgSP", "reviewer": "Reviewer_JZXS", "summary": "The authors showcase a system to generate complete pop songs in pianoroll format. The approach consists in splitting the generation into a hierarchy of four stages. The representation for each of this stage can be computed directly from the original pianoroll and a bespoke algorithm termed Tonal Reduction Algorithm is introduced to compute the so-called \"Conterpoint\" representation.\nThe different level of the hierarchy are then generated iteratively by conditioning on the preceding stages using a diffusion model.", "questions": "The diffusion model used auto regressively\n\nWhat are the pretrained models for chord progression, rhythmic pattern, accompaniment texture?\n\n\"Conterpoint\" term for the second stage is not appropriate as this stage describes the sketch of the melody and harmony t. \"Draft\" or \"Sketch\" stage may be more suitable?\n\nLeadsheet encoded as melody and chords? Are chords in string format?\n\nWhat is the sampling time for a whole song?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This article features a very well-engineered system. The generated pop songs are convincing and seem to capture well the style of the POP909 dataset.\nThe presentation website features lots of interesting examples that sound well.\nIt shows that this model is able to cover many use cases beyond generation from-scratch: from accompaniment generation based on texture to leadsheet generation.", "weaknesses": "The main weaknesses seem the lack of details concerning the diffusion model and the sampling procedure.\nThis is even more problematic as it seems that the modeling process is not standard, with the diffusion model used to generate chunks of music in an autoregressive manner. \n\nIt is also unclear how the data is represented as it seems at first sight that the diffusion model used would be a Discrete diffusion model.\nAs such, it is not very accessible for people knowing the standard literature on diffusion models.\n\"We represent key by K ∈ R2×M×12, where tonic information and scale information are stored on the two channels\"\nIt may be interesting to emphasize on the non standard points .\n\n\nVery custom Tonal Reduction Algorithm. Seems very close from Polyffusion.\nThe results shown here may not be of interest for the broad ICLR community as the main contributions are mostly about symbolic music generation in a pianoroll format.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bhcmrZPmop", "reviewer": "Reviewer_JPv8", "summary": "This paper proposes to learn to generate a full pop song with piano accompaniment as a hierarchical generation process. This paper defines a music language or representation of 4 levels. The generation process is then defined into four stages: form, counterpoint, lead sheet and accompaniment. In each stage, a diffusion model is used as the backbone generative model, generating the image-like representation at that level.", "questions": "I wonder how this method applies to a more general MIDI-like dataset. Or does it rely heavily on the POP909 dataset?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a novel hierarchical representation of symbolic music.\n2. This paper proposes a novel task formulation of generating full-song symbolic music and has a good qualitative result and offers a wide range of controlability.", "weaknesses": "The system, including its condition input, appears complex. The reliance on multiple pretrained models (as referenced in Section 3.3) might be cumbersome. It would enhance the paper's credibility if the authors could provide ablation studies both on the model's architecture and the efficacy of the control input.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QVOWwMUusP", "reviewer": "Reviewer_EK1V", "summary": "This paper proposes a hierarchical strategy for symbolic music generation. Specifically, their approach involves a cascade of conditional diffusion models which iteratively generate a series of interpretable representations in a coarse-to-fine fashion. The authors compare their proposed approach to strong baselines through both quantitative metrics and a qualitative user study, demonstrating promising performance.", "questions": "Can the authors provide quantitative or qualitative evidence that the model is not overfit to the training data? If so, I would consider raising my score from a 6 to an 8.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a nice paper overall. Among its virtues are (1) **the quality of the results**, (2) **simplicity of the approach**, (3) **usefulness for controllable generation**, and (4) **clarity of the writing**.\n\n**Result quality**. The proposed method achieves impressive results in both the quantitative evaluation and subjective tests, especially compared to strong baselines. Moreover, the included sound examples are quite compelling, and the contribution of the proposed hierarchical approach to the final outputs is immediately apparent.\n\n**Simplicity**. While the design of the hierarchical approach is somewhat complex, the proposed generative modeling approach is satisfyingly simple, with each stage using the same basic setup despite their structural differences, and later stages adding in well-motivated mechanisms to address clear issues (e.g., autoregressive component to generate locally but with global consistency).\n\n**Usefulness for control**. The proposed hierarchy (both in the data representation and modeling) is helpful for enabling long-form generation w/ global structure. However, it has an additional benefit of enabling interpretable manipulation of intermediate representations. While the authors don’t specifically explore interaction, it is clear that this aspect of the approach could be very powerful for users.\n\n**Well-written**. This paper is extremely clear and well-written, especially relative to the median paper on music generation. Symbolic music generation, especially work that focuses on interpretability, tends to be a very messy subject with lots of in-the-weeds details that often manifest as confusing and poorly-written papers. All symbolic music gen papers tend to require substantial music expertise to fully understand, but this paper does a fantastic job of both minimizing the expertise needed and being exceptionally clear in overall formulation.", "weaknesses": "There are two primary weaknesses with this work: (1) **unclear if model is copying**, and (2) **impact is limited by data availability**\n\n**Unclear if model is copying**. This model is trained on a very small amount of data, just 909 songs. Despite this, subjectively speaking, the results from the proposed model are quite good. It seems quite plausible that the model is overfit to the training data and producing copies or near-copies. Can the authors provide any evidence that this is not happening?\n\n**Impact limited by data availability**. A broader issue is that extracting the proposed hierarchical representation requires rich annotations aligned with the raw notes: chords, melody, key, phrase boundaries, etc. Some of this the authors extract (e.g. key / phrase boundaries) and some of this comes from human labels (e.g. chords and melody). This limits the overall applicability of the approach to music datasets with such rich labels (POP909), and prevents application to much larger symbolic music datasets (Lakh) which cover more styles.\n\nLow-level comments:\n\n- Table 1 could describe M, γ, δ in the caption\n- Table 1 would be well-complemented by a Figure 1 showing the languages visually and their relationships to music scores / one another\n- Sound examples page could have qualitative examples from baselines\n- It’s a shame that there is no “overall structure” music in the whole-song subjective study, so users can rate if models produce music with clear long-form structure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)