Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling

Authors: jingwei  zhao
gus  xia
ziyu  wang
ye  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 15:28:50

Abstract: In the realm of music AI, arranging rich and structured multi-track accompaniments from a simple lead sheet presents significant challenges. Such challenges include maintaining track cohesion, ensuring long-term coherence, and optimizing computational efficiency. In this paper, we introduce a novel system that leverages prior modelling over disentangled style factors to address these challenges. Our method presents a two-stage process: initially, a piano arrangement is derived from the lead sheet by retrieving piano texture styles; subsequently, a multi-track orchestration is generated by infusing orchestral function styles into the piano arrangement. Our key design is the use of vector quantization and a unique multi-stream Transformer to model the long-term flow of the orchestration style, which enables flexible, controllable, and structured music generation. Experiments show that by factorizing the arrangement task into interpretable sub-stages, our approach enhances generative capacity while improving efficiency. Additionally, our system supports a variety of music genres and provides style control at different composition hierarchies. We further show that our system achieves superior coherence, structure, and overall arrangement quality compared to existing baselines.

Main paper: 
 introduction: 
 Representation learning techniques have enabled new possibilities for controllable generative modelling. By learning implicit style representations, which are often hard to explicitly label (e.g., timbre of music audio [21] , texture of music composition [39] , and artistic style in paintings [20] ), new music and artworks can be created via style transfer and latent space sampling. These learned style factors can also serve as external controls for downstream generative models, including Transformers [18, 36] and diffusion models [42] . However, applying style factors to long-term sequence generation remains a challenging task. Existing approaches rely on style templates specified manually or by heuristic rules [36, 42, 51] , which are impractical for long-term generation. Moreover, when structural constraints are imposed, misaligned style factors can result in incoherent outputs.To address these challenges, we aim to develop a novel sequence generation framework leveraging a global style planner, or prior, which models the conditional distribution of style factors given the model input's content factors. Both style and content factors are sequences of compact, structurally aligned latent codes over a disentangled representation space. By infusing the style back to the content, we can recover the observational target with globally coherent style patterns.In this paper, we study style prior modelling through the task of multi-track accompaniment arrangement, a typical scenario for long-term conditional sequence generation. We assume the input of a piano accompaniment score, which typically carries a verse-chorus structure. Our target is to generate corresponding multi-track arrangements featuring band orchestration. We start by disentangling a band score at time t into piano reduction c t (content factor) and orchestral function s k t (style factors for individual tracks k = 1, 2, • • • , K). On top of this, we model the prior of finding appropriate functions to orchestrate a given piano score, or formally p(s1:K 1:T | c 1:T ). To model dependencies on both time (T ) and track (K) directions, we develop a multi-stream Transformer with interleaved time-wise and track-wise layers. The track-wise layer allows for flexible control over the choice of instruments and the number of tracks, while the time-wise layer ensures structural alignment through cross-attention to the piano reduction. Decoding the inferred s 1:K 1:T with c 1:T , we can address accompaniment arrangement in a flexible multi-track form with extended whole-song structure.Experiments show that our method outperforms existing sequential token prediction approaches and provides better multi-track cohesion, structural coherence, and computational efficiency. Additionally, compared to existing designs of multi-stream language models, our model handles flexible stream combinations more effectively with enhanced generative capacity.To summarize, our contributions in this paper are three-folded:• We propose style prior modelling, a hierarchical generative methodology addressing both long-term structure (via style prior at high level) and fine-grained condition/control (via representation disentanglement at low level). Our approach moves beyond the limitation of manual specification of style factors, providing a flexible, efficient, and self-supervised solution for long-term sequence prediction and generation tasks.• We propose a novel layer interleaving architecture for multi-stream language modelling. In our case, it models parallel music tracks with a flexible track number, controllable instruments, and manageable computation. To our knowledge, it is the first multi-stream language model with tractable generalization to flexible stream combinations.• Integrating our previous study on piano texture style transfer [39, 50] , we present a complete music automation system arranging an input lead sheet (a basic music form with melody and chord only) via piano accompaniment to multi-track arrangement. The entire system is interpretable at two composition hierarchies: 1) piano texture and 2) orchestral function, and demonstrates state-of-the-art arrangement performance for varied genres of music. 1  
related works: 
 In this section, we overview three topics related to our study. Section 2.1 reviews existing studies on representation disentanglement. Section 2.2 summarizes prior modelling methods in music generation. Section 2.3 reviews the current progress with the task of accompaniment arrangement.Representation disentanglement is a popular technique in deep generative modelling [3, 16, 48, 49] .In the music domain, this approach has proven valuable by learning compositional factors related to music style and content. By manipulating these factors through interpolation [32] , swapping [39] , and prior sampling [46] , it provides a self-supervised and controllable pathway for various music automation tasks. Recent works leverage disentangled style factors as control signals for long-term music generation [36, 42] . However, these approaches typically treat style representations as fixed condition sequences during training, requiring manual specification or additional algorithms for control during inference. In contrast, we model the prior of the style to apply conditional on the given music content, which is a more generalized and flexible approach.In sequence generation tasks (e.g., music and audio), learning a prior sampler over a compact, latent representation space is often more efficient and effective. Jukebox [7] models the latent codes encoded by VQ-VAEs [34] as music priors, which can further reconstruct minutes of music audio. More recently, MusicLM [2] and MusicGen [4] learn multi-modal priors for generating music from text 
method: 
 We develop a model that takes a piano reduction as input and outputs an orchestrated multi-track arrangement. Using an autoencoder, we disentangle a multi-track music score into its piano reduction (content factor) and orchestral function (style factor). We then design a prior model to infer orchestral functions given the piano reduction. The autoencoder operates at segment level, while the prior model works on the whole song. The entire model can operate as an orchestrator module in a complete arrangement system. In this section, we introduce our data representation in Section 3.1, autoencoder framework in Section 3.2, and prior model design in Section 3.3.We summarize our data representations in Table 1 . Let x be a K-track arrangement score. We split it into T segments and represent x k t -each segment track -as a matrix of shape P × N . Here P = 128 represents 128 MIDI pitches and N is the time dimension of a segment. This matrix representation aligns with the modified piano roll in [39] , where each non-zero entry (p, n) > 0 indicates a note onset and its value indicates the note duration. In this paper, we primarily focus on music pieces in 4/4 time signature with 1/4-beat resolution. Duration values range from 1 (for sixteenth notes) to 32 (for double whole notes). We consider 1 segment = 8 beats (2 bars) and derive N = 32, which is a proper scale for learning music content/style representations [37, [39] [40] [41] 46] . The piano reduction of x is notated as pn[x]. It is approximated by downmixing all K tracks into a single-track mixture similar to [8] . When concurring notes are found across tracks, we keep the one with the largest duration (i.e., track-wise maximum). Segment-wise, pn[x] t is also a P × N matrix. It preserves the overall music content while discarding the multi-track form.The orchestral function of x is notated as fn[x]. It describes the rhythm and grooving patterns [45] of each segment track, which serves as the "skeleton" of a multi-track form. Formally,EQUATIONwhere indicator function 1 Our autoencoder consists of two components as shown in Figure 1 . A VQ-VAE submodule (right of Figure 1 ) encodes orchestral function fn[x] k t . A VAE module (left of Figure 1 ) encodes piano reduction pn[x] t and reconstructs individual tracks x 1:K t leveraging the cues from fn[x] 1:K t . During training, both inputs pn[x] and fn[x] are deterministic transforms from the output x and the entire model is self-supervised. We see similar techniques for representation disentanglement in [39, 41, 46] .The VQ-VAE consists of Function Encoder Enc f and Decoder Dec f . Encoder Enc f contains a 1-D convolutional layer followed by a vector quantization block. Our intuition for applying a VQ-VAE is that orchestral function commonly consists of rhythm patterns (such as syncopation, arpeggio, etc.) that can naturally be categorized as discrete variables. In our case, each segment track is encoded into 8 discrete embeddings on a 1-beat scale, indicating the flow of orchestration style. Formally,EQUATIONwhere s k τ is the latent orchestral function code for the k-th track at the τ -th beat. We encode fn[x] k t at a finer 1-beat scale (instead of segment) to preserve fine-grained rhythmic details. The new scale is re-indexed by τ = 1, 2 EQUATIONwhere Track Separator Sep is a Transformer encoder. In this process, each s k t queries c t to recover the corresponding track (k), while they also attend to each other to maintain the dependency among parallel tracks. Learnable instrument embeddings [51] are added to each track based on its instrument class. We provide details of the autoencoder architecture in Appendix A.1.The VQ-VAE in Section 3.2 derives latent codes s 1:K 1:8T for orchestral function as a multi-stream time series. Herek = 1, 2, • • • K is the stream (track) index and τ = 1, 2, • • • , 8T is the time (beat) index.The purpose of style prior modelling is to infer orchestral function given piano reduction so that the former can be leveraged to orchestrate the latter into multi-track music. We design our prior model as shown in Figure 2 . It is an encoder-decoder framework that models parallel tracks/streams of orchestral function codes conditional on the piano reduction.The decoder module (right of Figure 2 ) has alternate layers of Track Encoder and Auto-Regressive Decoder. Track Encoder is a standard Transformer encoder layer [35] and it aggregates inter-track information along the track axis. Auto-Regressive Decoder is a Transformer decoder layer (with self-attention and cross-attention) and it predicts next-step orchestral function codes on the time axis. By orthogonally stacking two types of layers, we can model track-wise and time-wise dependencies simultaneously with a manageable computational cost. Compared to sequential tokenization methods in previous studies [9, 30, 36] , our method brings down the complexity from O(N 2 T 2 ) to O(max(N, T ) 2 ). Moreover, we support a flexible multi-track form (N being variable) with a diverse instrumentation option. We add instrument embedding [51] and relative positional embedding [15] to the track axis, where 34 instrument classes [25] are supported. We add music timing condition [7] to the time axis, which encodes the positions in a training excerpt as fractions of the complete song, helping the model capture the overall structure of a song.The encoder module (left of Figure 2 ) of our prior model is a standard Transformer encoder, which takes piano reduction c 1:T as global context. It is connected to the decoder module via cross-attention and maintains the global phrase structure. During training, both c 1:T and s 1:K 1:8T are derived from the same multi-track piece and the entire model is self-supervised. Let p θ be the distribution of orchestral function codes fitted by our prior model θ, the training objective is the mean of negative log-likelihood of next-step code prediction:EQUATIONWe provide more implementation details of the prior model in Appendix A.2. We note that there is a potential domain shift from our approximated piano reduction to real piano arrangements. To prevent overfitting, we use a Gaussian noise ϵ to blur c 1:T while preserving its high-level structure. During training, ϵ is combined with c 1:T using a weighted summation with noise weight γ ranging from 0 to 1. It encourages a partial unconditional generation capability. At inference time, γ is a parameter that can balance creativity with faithfulness. An experiment on γ is covered in Appendix C. 
whole-song multi-track accompaniment arrangement: 
 We finalize a complete music automation system by applying style prior modelling at two cascaded stages. As shown in Figure 3 on top of chord/texture representation learning [39] , for lead sheet to piano arrangement. Given a lead sheet, the first stage generates a piano accompaniment, establishing the rough whole-song structure.Our system then orchestrates the piano accompaniment into a complete multi-track arrangement with band instrumentation. This two-stage approach mirrors musicians' creative workflow [1] and allows for control at both composition levels. In particular, we provide three control options: We showcase an arrangement example by the complete system in Figure 4 . The system input is a lead sheet shown by the Mel staff. The final output is the accompaniment in the rest staves. Notably, the lead sheet consists of 60 bars in an structure of i4A8B8B8x4A8B8B8O4 (using notations by [5] ).Here, i4, x4, and O4 each denote a 4-bar intro, interlude, and outro. A8 and B8 represent an 8-bar verse and chorus, respectively. Figure 4 shows the arrangement result for the first and third choruses, spanning from bar 13 to 41. We leverage control option 2 to customize the instrumentation as celesta, acoustic guitars (2), electric pianos (2), acoustic piano, violin, brass, and electric bass in a total of K = 9 tracks. The complete arrangement score is included in Appendix E.In Figure 4 , we observe some multi-track arrangement patterns that are common in practice. Purple blocks highlight a counterpoint relation between guitar track A.G.1 and electric piano track E.P.1.Green blocks show two guitar tracks with complementary orchestral functions: one melodic (A.G.1) and the other harmonic (A.G.2). Yellow blocks illustrate the metrical division between the string (Vlns.) and the brass (Br.) sections, with strings on the downbeat and brass on the offbeat. These patterns demonstrate a natural and cohesive multi-track arrangement by our system. Moreover, we see consistent accompaniment patterns echoing in both chorus parts that span over 30 bars (shown by red dotted boxes), while the latter adds a piano arpeggio track (Pno.) to enhance the musical flow. This demonstrates structured whole-song arrangement over extended music contexts. 
experiment: 
 In this section, we evaluate the performance of our multi-track accompaniment arrangement system. Given that existing methods primarily focus on lead sheet to multi-track arrangement, we ensure a fair comparison by using the two-stage approach discussed in Section 4. In Section 5.1, we present the datasets used and the training details of our model. In Section 5.2, we describe the baseline models used for comparison. Our evaluation is divided into two parts: objective evaluation, detailed in Section 5.3, and subjective evaluation, covered in Section 5.4. For the single-stage piano to multitrack (Stage 2) and lead sheet to piano (Stage 1) arrangement, we perform additional comparisons with various ablation architectures in Section 5.5 and 5.6, respectively.We use two datasets to train the autoencoder and the style prior, respectively. The autoencoder is trained with Slahk2100 [25] , which contains 2K curated multi-track pieces with 34 instrument classes in a balanced distribution. We discard the drum track and clip each piece into 2-bar segments with 1-bar hop size. We use the official training split and augment training samples by transposing to all 12 keys. The autoencoder comprises 19M learnable parameters and is trained with batch size 64 for 30 epochs on an RTX A5000 GPU with 24GB memory. We use Adam optimizer [19] with a learning rate from 1e-3 exponentially decayed to 1e-5. We use exponential moving average (EMA) [29] and random restart [7] to update the codebook with commitment ratio β = 0.25.We use Lakh MIDI Dataset (LMD) [28] to train the prior model. It contains 170k music pieces and is a benchmark dataset for training music generative models. We collect 2/4 and 4/4 pieces (110k after processing) and randomly split LMD at song level into training (95%) and validation (5%) sets. We further clip each piece into 32-bar training excerpts (i.e., T = 16 at maximum) with a 4-bar hop size.Our prior model has 30M parameters and is trained with batch size 16 for 10 epochs (600K iterations) on two RTX A5000 GPUs. We apply AdamW optimizer [22] with a learning rate of 1e-4, scheduled by a 1k-step linear warm-up followed by a single cycle of cosine decay to a final rate of 1e-6.For model inference and testing, we consider two additional datasets: Nottingham [10] and WikiMT [44] . Both datasets contain lead sheets (in ABC notation) that are not seen during training or validation. Moreover, they cover diverse music genres including folk, pop, and jazz. When arranging a piece, we leverage control option 2 to set up the instrumentation. Without loss of generality, this control choice is randomly sampled from Slakh2100 validation/test sets. To arrange music longer than the prior model's context length (32 bars), we use windowed sampling [7] , where we move ahead our context window by 4 bars and continue sampling based on the previous 28 bars. We apply nucleus sampling [13] with top probability p = 0.05 and temperature t = 6.We compare our system with three existing methods: PopMAG [30] , Anticipatory Music Transformer (AMT) [33] , and GETMusic [23] . PopMAG and GETMusic generate multi-track accompaniments from an input lead sheet based on a Transformer and a diffusion model, respectively. AMT is Transformer-based and it continues the accompaniment part from an input melody with starting accompaniment prompt. We provide detailed configurations in the following.PopMAG is an encoder-decoder architecture based on Transformer-XL [6] . It represents multi-track music by sequential note-level tokenization and is fully supervised. The encoder takes a lead sheet as input and the decoder generates multi-track accompaniment auto-regressively. Since the model is not open source, We reproduce it on LMD with lead sheets extracted by [24] (melody) and [17] (chord). It introduces an "anticipation" method, where conditional tokens (melody and starting prompt) and generative tokens (accompaniment continuation) are interleaved to train a conditional generative model. Since our testing dataset does not provide ground-truth accompaniment, the starting prompt is given by the generation result (first 2 bars) of our system. We use the official implementation of the AMT model, 2 which is also trained on LMD.GETMusic represents multi-track music as an image-like matrix resembling score arrangement, based on which a denoising diffusion probabilistic model is trained with a mask reconstruction objective. Given a lead sheet, it supports generating 5 accompaniment tracks using piano, guitar, string, bass, and drum or their subsets. In our experiment, we generate all 5 accompaniment tracks. We use the official implementation of the GETMusic model, 3 which is trained on internal data.We introduce four metrics to evaluate multi-track arrangement performance: chord accuracy [23, 30] , degree of arrangement (DOA), structure awareness [42] , and inference latency [30] . Among them, chord accuracy measures the multi-track harmony that reflects the fitness of the accompaniment to the lead sheet; DOA measures inter-track tonal diversity that reflects the creativity of the instrumentation. Both metrics demonstrate music cohesion at local scales. On the other hand, structure awareness measures phrase-level content similarity that reflects long-term structural coherence of the whole song. Finally, we use inference latency (in second/bar) to evaluate computational efficiency of each method. The detailed computation of each metric is provided in Appendix B. In Table 2 , we compute ground-truth DOA using 1000 random pieces from LMD. We compute ground-truth structure awareness using 857 pieces in 4/4 from POP909 Dataset [38] .We randomly sample 50 pieces in 4/4 time signature from Nottingham and WikiMT respectively (100 pieces in total) to conduct experiment. The length of each piece ranges from 16 to 32 bars. We run our method and baseline models at each piece in 3 independent rounds, deriving 300 sets of multi-track arrangement samples. In Table 2 , we report the evaluation results with mean value, standard error of mean (sem), and statistical significance computed by Wilcoxon signed rank test [43] . We find significant differences (p-value p < 0.05) between our method and all baselines for each metric. In particular, our method outperforms in chord accuracy, structure awareness, and DOA, indicating the capability of arranging harmonious, structured, and creative accompaniments. The diffusion model outperforms in inference latency as it applies only 100 diffusion steps. Our method's efficiency is on par with it, while being 10 times faster than vanilla note-level auto-regression.We also conduct a double-blind online survey to test music quality. Our survey consists of 5 evaluation sets, each containing an input lead sheet followed by 4 arrangement samples by our method and each baseline. Each sample is 24-32 bars long and is synthesized to audio at 90 BPM (~1 minute per sample). Both the set order and the sample order in each set are randomized. We request participants to listen to 2 sets and evaluate the musical quality based on a 5-point Likert scale from 1 to 5. ,QVWUXPHQWDWLRQ 6WUXFWXUH &UHDWLYLW\ 0XVLFDOLW\ 2XUV 3DUDOOHO 'HOD\ 5DQGRP The evaluation is based on 5 criteria: 1) Harmony and Texture Coherency, 2) Long-Term Structure, 3) Naturalness, 4) Creativity, and 5) Overall Musicality.A total of 23 participants (8 female and 15 male) with diverse musical backgrounds have completed our survey. The average completion time is 22 minutes. We show the mean ratings and standard errors computed by within-subject (repeated-measures) ANOVA [31] in Figure 5 . Our method outperforms all baselines regarding each criterion, which echoes the objetive evaluation results. Particularly, we report a significant result (p-value p < 0.05) in Creativity and Musicality.We now validate our design with the prior model by exclusively evaluating on the piano to multi-track arrangement task. Our prior model is based on a unique layer interleaving design, which enables multi-stream time series modelling with explicit stream-wise attention. We compare it with two other multi-stream architectures: 1) Parallel: summing up parallel code embeddings for joint language modelling [18] , and 2) Delay: leveraging a 1-step delay code interleaving to catch implicit streamwise dependency [4] . Both Parallel and Delay are trained under the same setup as our model. We additionally introduce 3) Random: a naive prior based on random template retrieval. The templates are sampled every 2 bars with shared instrumentation from the validation/test sets of Slakh2100.We introduce two metrics to evaluate piano to multi-track arrangement: faithfulness and degree of arrangement (DOA). Faithfulness measures if the generated arrangement faithfully reflects the original content from the piano. It computes the similarity between i) the input piano, and ii) the piano reduction of the generated multi-track arrangement. In our case, we compute cosine similarity over two features: a statistical pitch class histogram [45] and a latent texture representation [39] , which emphasize tonal and rhythmic similarity, respectively. DOA measures the arrangement creativity as defined in Section 5.3. We also report the NLL loss for our model, Parallel, and Delay.We conduct experiments using the test set of POP909 [39] , which consists of 88 piano arrangement pieces. In our experiment, we use the first section of each piece, which contains 2 to 4 complete phrases totally spanning 24 to 32 bars. We use control option 3 to prompt our model, Parallel, and Delay with the same 2-bar orchestral function template (sampled from Slakh2100) and see how it is developed. We report mean value, standard error of mean (sem), and statistical significance in Table 3 and find significant differences in both faithfulness and DOA. We also conduct a subjective evaluation in the same setup as Section 5.4. We consider one additional criterion: Instrumentation. Results are reported in Figure 6 with a significant difference (p-value p < 0.05) in Instrumentation. Overall, Parallel and Delay both fall short in performance because they assume a preset stream combination, while in our case, both track numbers and choices of instruments are flexible. By explicitly modelling stream-wise attention, our model fits well to this generalized scenario.Now we validate our choice for the lead sheet to piano arrangement module on the first stage of our two-stage system. Our choice is a piano texture prior as covered in Section 4. We conduct an ablation study by replacing it with the Whole-Song-Gen model [42] , which, to our knowledge, is the only existing alternative that can handle a whole-song structure. The ablation study is conducted in the same setup as Section 5.3. In Table 4 , we report chord accuracy, structure awareness, and DOA regarding the final multi-track arrangement results. We further compare our piano texture prior with Whole-Song-Gen exclusively on the piano accompaniment arrangement task. In Table 5 , we report chord accuracy and structure awareness regarding piano arrangement for both models.By comparing Table 4 and Table 5 , we can see that a higher-quality piano arrangement generally encourages a more musical and creative final multi-track arrangement result. Specifically, the piano arrangement on Stage 1 lays the groundwork for (at least) chord progression and phrase structure for Stage 2, both of which are important for capturing the long-term structure in whole-song multi-track arrangement. Moreover, we see that our piano texture prior outperforms existing alternatives and guarantees a decent piano quality, thus being the best choice for our system. 
conclusion: 
 To sum up, we contribute a music automation system for multi-track accompaniment arrangement.The main novelty lies in our proposed style prior modelling, a generic methodology for structured sequence generation with fine-grained control. By modelling the prior of disentangled style factors given content, we build a cascaded arrangement process: from lead sheet to piano texture style, and then from piano to orchestral function style. Our system first generates a piano accompaniment from a lead sheet, establishing the rough whole-song structure. It then orchestrates the piano accompaniment into a complete multi-track arrangement with band instrumentation. Extensive experiments show that our system generates structured, creative, and natural multi-track arrangements with state-of-the-art quality. At a higher level, we elaborate our methodology as interpretable modular representation learning, which leverages finely disentangled and manipulable music representations to tackle complex tasks with a compositional hierarchy. We hope our research brings new perspectives to broader domains of music creation, sequence data modelling, and representation learning. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [Yes] Justification: The human study in our experiment is based on online crowdsourcing, which bears minimum risk. Participants are informed that participation in our study is entirely voluntary and that they may choose to stop participating at any time without any negative consequences. No personally identifying information is collected in the human study. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 AutoencoderThe autoencoder consists of a VQ-VAE submodule and an overarching VAE. The encoder of the VQ-VAE consists of a 1-D convolutional layer of kernal size 4, stride 4, and 16 output channels, followed by a vector quantization block with codebook size 128. The decoder takes the concatenated latent codes and leverages two fully-connected layers (shape 128 × 256 and 256 × 32) for reconstruction. In the overarching VAE, Piano Encoder and Track Decoder are adapted from PianoTree VAE [40] . The encoder first applies a pitch-wise bi-directional GRU to summarize concurrent notes at time step n and then applies a time-wise GRU to encode the full representation. The decoder mirrors the encoder structure with time-and pitch-wise uni-directional GRUs to reconstruct individual tracks. We use hidden size 256 in a single layer for pitch GRUs and 512 for time GRUs. The Track Separator is a 2-layer Transformer encoder with 8 attention heads, 0.1 dropout ratio, and GELU activation [12] . The hidden dimensions of self-attention d model and feed-forward layers d ff are 512 and 1024, respectively.The autoencoder is trained with joint reconstruction loss for orchestral function (MSE) and individual tracks (cross entropy). The VQ-VAE is additionally regularized with latent loss and commitment loss with commitment ratio β = 0.25. The VAE is regularized with KL loss over all continuous factors (c t and z 1:K t ) based on KL annealing [41] with a ratio exponentially increasing from 0 to 0.5.The prior model consists of a 12-layer Context Encoder and a 12-layer Auto-Regressive Decoder. The latter is interleaved with another 12 track-wise Track Encoder layers. For each layer, we apply 8 attention heads, 0.1 dropout ratio, and GELU activation. We apply layer normalization before self-attention (i.e., norm first). The hidden dimensions of self-attention d model and feed-forward layers d ff are 256 and 1024, respectively. We apply relative positional embedding [15] to Track Encoder so that two tracks initialized with identical instruments can still generate different content.Our prior model is trained on the latent codes c 1:T and s 1:K 1:8T inferred by a well-trained autoencoder on LMD. For discrete code s, we take the codebook indices and learn a new embedding.In multi-track arrangement, parallel tracks typically play a unique role to each other in the overall arrangement. We are interested in capturing the diversity and creativity inherent in these roles.To achieve this, we consider the pitch class histogram [45] as a probability distribution P . Let P t,k be the distribution of the t-th bar in track k, and P pn t be that of the t-th bar in the piano reduction. Recall that in this paper we approximate the piano reduction of a multi-track piece by downmixing all tracks. Both P t,k and P pn t are 12-D vectors, describing tonality of individual tracks and the overall arrangement, respectively. We compute the KL divergence of each track to the piano reduction:EQUATIONwhere T is the total number of bars.Interpreting d k in terms of KL divergence, we see it as the "excess surprise" from the overall arrangement (piano reduction) when track k is played in isolation. A large d k indicates that track k possesses a unique quality, such as a bass track playing the root or a counter-melody track focusing on tensions. Conversely, a small d k suggests that track k serves as a foundational element in the arrangement, such as string padding that establishes the harmonic foundation.If all d k values are small, it implies homogeneity across tracks and thus a low degree of arrangement.Conversely, if all d k values are high, it suggests a composition dominated by counterpoints, a scenario less common in pop music. A well-orchestrated piece typically exhibits a diverse range of d k values, encompassing both foundational and unique decorative tracks. We thus define the degree of arrangement DOA as the standard deviation of d k for k = 1, 2, • • • , K across all tracks:EQUATIONwhere d is the mean. K is the total number of tracks. To establish a reference point, we calculate the ground-truth DOA = 0.333 based on 1000 randomly selected pieces from the LMD dataset. Within this context, a higher DOA signifies a more creative arrangement.We introduce Inter-phrase Latent Similarity (ILS) from [42] to measure the structural awareness of long-term arrangement. ILS calculates the content similarity among same-type phrases (e.g., chorus) versus the whole song. It leverages pre-trained disentangled VAEs that encode music notes into latent representations and then compare cosine similarities in the latent space. In our case, we compute ILS over the piano reduction of a generated arrangement since it contains the overall content. We apply the texture VAE [39] and obtain a latent texture representation c txt t for every 2-bar segment. For odd-numbered phrases, we repeat its final bar and pad it to the end of the phrase. Suppose there are M different types of phrases in one piece and let I m be the set of segment indices in the type-m phrase, ILS is defined as the ratio between same-type phrase similarity and global average similarity:EQUATIONwhere | • | is the cardinality of a set. T is the number of 2-bar segments. When applying ILS, we use [5] to automatically lable the phrase structure of a piece. To establish a reference point, we calculate the ground-truth ILS = 1.980 based on the POP909 dataset (with phrase annotation by human). Within this context, a higher ILS signifies saliency with long-term phrase-level structure.We introduce chord accuracy from [30] to measure if the chords of the generated arrangement match the conditional chord sequence in the lead sheet. It reflects the harmonicity of the generated music and is defined as follows:EQUATIONwhere N chord is the number of chords in a piece; C i is the i-th chord in the (ground-truth) lead sheet; and Ĉi is the aligned chord in the generated arrangement.The original formulation in [30] considers chord accuracy for individual tracks. Given our system's capability to accommodate a variable combination of tracks, we opt for a broader evaluation for the overall arrangement. In our case, we extract the chord sequence of a generated arrangement with [17] and compare it in root and quality with ground-truth at 1-beat granularity, which is more rigorous.We measure the faithfulness of orchestration by the similarity between i) the input piano and ii) the piano reduction of the generated multi-track arrangement. Let e in t and e pn t be vector features derived from the t-th segment of the input and the reduction, respectively. Orchestration faithfulness OF is defined as follows:OF = 1 T T t=1cos(e in t , e pn t ),where cos(•, •) is cosine similarity. T is the number of segments.In our work, we select two options for vector feature e. One is a statistical pitch class histogram [45] , which is a 12-D vector describing pitch class distribution. The other is a latent 256-D texture representation learned by a pre-trained VAE [39] . Both features are general descriptors of the musical content with respective focus on tonal harmony and rhythmic grooves.Continuing from Section 3.3, we compare different γ values and see their impact to the model performance. When applying our model to piano to multi-track arrangement, γ balances the force of a noisy factor added to the piano, which encourages a partial unconditional generation capability. The experimental settings are the same as Section 5.5. We evaluate the results based on faithfulness and DOA. In Table 6 , we report mean value, standard error of mean (sem), and statistical significance computed by Wilcoxon signed rank test. By varying the γ value, we observe a controllable balance between faithfulness and creativity. Specifically, a larger γ encourages creativity (higher DOA) at the cost of faithfulness. If not mentioned otherwise, we use γ = 0.25 for experiments in this paper. We distribute our survey via SurveyMonkey. 4 Our survey consists of 5 sample sets for both the lead sheet to multi-track and the piano to multi-track arrangement tasks (10 sets in total). Each sample is 24-32 bars long and is synthesized to audio at 90 BPM using BandLab5 with the default soundfont. Each participant listens to 2 sets (in random order) and the mean time spent is 22 minutes. Figure 7 shows the sample pages of our survey with instructions to the participants. We demonstrate an example of accompaniment arrangement by our proposed system. The input lead sheet is a complete pop song shown in Section E.1. Our system first arranges a piano accompaniment for the whole song, which is shown in Section E.2. The piano score is then orchestrated into a multi-track arrangement with customized instrumentation, which is shown in Section E.3.We use our system to arrange for Can You Feel the Love Tonight, a pop song by Elton John. As shown in Figure 8 , the entire song is 60 bars long and it presents a structure of i4A8B8B8x4A8B8B8O4, where i, x, O, A, and B each refer to intro, interlude, outro, verse, and chorus.                                                                                                                                                                                                                                             =The piano arrangement result is shown from Figure 9 to Figure 10 . It roughly establishes a whole-song structure and lays the groundwork for band orchestration at the next stage. Demo audio for the piano score is available at https://zhaojw1998.github.io/structured-arrangement/. Pno.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             =EQUATIONThe multi-track arrangement is shown from Figure 11 to Figure 15 . We customize the instrumentation as celesta, acoustic guitars (2), electric pianos (2), acoustic piano, violin, brass, and electric bass in a total of K = 9 tracks. We can see that the structure of the accompaniment follows the lead sheet. Demo audio is available at https://zhaojw1998.github.io/structured-arrangement/. More detailed analysis on this arrangement demo is covered in Section 4.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       = 65 A8 i4 E♭/G F/A F/A B♭/D B♭ B♭ Cm7 Cm7 B♭/D B♭/D E♭ E♭ B♭/D B♭/D E♭ F/A B♭/F E♭                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              B8 Gm F E♭ Gm B♭/D A♭ C/E E♭ C7/E B♭ Cm/E♭ B♭/D F B♭/D E♭ Cm7 E♭ B♭ E♭ F/A E♭ Gm F E♭ B♭/D                          EQUATION                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    B8 E♭ Cm7EQUATIONWe propose a two-stage system for whole-song, multi-track accompaniment arrangement. In the context of this paper, we acknowledge that our current system exclusively supports tonal tracks in quadruple meters while disregarding triple meters, triplet notes, and drums. However, we perceive this as a technical limitation rather than a scientific challenge. We also acknowledge that our current system primarily emphasizes the composition level, thereby omitting the modelling of MIDI velocity, dynamic timing, and MIDI control messages. Consequently, the generated results do not encompass performance MIDI and may lack expressive qualities. Nevertheless, we believe that our composition-centric work serves as a solid and vital foundation for further advancements in those specific areas, thus facilitating the development of enhanced techniques and features. As a pioneering work, our system is the foremost accomplishment in solving whole-song multi-track accompaniment arrangement, characterized by flexible controllability on track number and choice of instruments.Our multi-track accompaniment arrangement system, which incorporates style to generate accompaniment, is designed to enhance originality and creativity. It serves as a platform for human-AI co-creation, where the user provides content-based material (in our case, lead sheet) that remains fundamentally original, while the AI agent infuses style, enriches the form, and enhances creativity.Our system therefore empowers musicians to explore new musical ideas and expand their creative boundaries. This approach also allows for rapid mock-up with different styles and arrangements, fostering an environment where innovation and artistic expression can thrive.However, we acknowledge the need to address potential risks. The accessibility of our system may inadvertently lead to excessive reliance on automation, potentially impeding the development of fundamental skills among musicians. Additionally, widespread adoption of the system may contribute to the homogenization of music, threatening the distinctiveness and individuality that are crucial to artistic expression. We recognize that our datasets predominantly features contemporary Western music, which introduces a cultural bias that could limit the diversity of generated compositions.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.Demo and more resources: https://zhaojw1998.github.io/structured-arrangement/https://github.com/jthickstun/anticipationhttps://github.com/microsoft/muzic/tree/main/getmusichttps://www.surveymonkey.comhttps://www.bandlab.com/ 


Decision: **Summary**: The paper introduces a new system for generating multi-track accompaniments from a lead sheet using a style prior modelling approach. The method involves a two-stage process: generating a piano arrangement and then orchestrating multiple tracks. The system leverages vector quantization and a multi-stream Transformer to model orchestration styles, enhancing generative capacity and providing flexible control.

**Strengths**: The paper presents a unique method for disentangling piano reduction and orchestral function, providing a novel element of controllability. The proposed model is technically robust and has a well-executed evaluation. The authors have documented experimental parameters thoroughly and provided code for reproducibility. The method addresses a challenging task in music AI, making significant contributions to the broader generative AI community and the niche music-generation community.

**Weaknesses**: The reviewers noted that the paper could benefit from clearer explanations of concepts and better organization of the model description (e.g., what are interleaving layers). Some baselines used for comparison are not strictly comparable, which should be clarified in the paper. Some high-level design choices, such as discrete embeddings and Gaussian noise, need better justification.

**Recommendation**: The reviewers unanimously agree that the paper should be accepted. As such, it is recommended for acceptance conditioned on minor revisions being implemented to address the presentation issues and provide clearer justifications for high-level design choices. The unique approach and state-of-the-art performance make it a valuable contribution to the field of music AI.

Reviews: review: paper_summary: paper_summary: The authors introduce a new model for generating multi-track accompaniment given a lead sheet. While being strictly a two-stage method (first generate a piano arrangement from the lead sheet and *then* generate the accompaniment), the authors' contribution is focused on the 2nd stage, and they rely on existing modules for the 1st stage. Their proposed method features two key components: **first**, a VQ-VAE submodule whose aim is to learn (quantized) representations of "orchestral functions" (orchestral functions being 1-D time-series representations of a track, with each element representing the sum of notes active in each segment, more on that in weaknesses), and, **second**, a transformer-based "mixing" VAE which combines the representations of the VQ-VAE (i.e., the representations of the orchestral functions) with learnt representations of "piano reductions" of each song (derived by averaging all individual tracks) to generate latent representations which are used to synthesize the full song. A key novelty of their model is the "mixing" part of the VAE, with the decoder utilizing interleaved time-wise and track-wise cross-attention layers -- meaning, that each output "sees" information both from past frames in its own track and from other tracks. Their system compares favorably to 3 standard baselines taken as-is from prior work or reproduced.
strength_weakness: strength_weakness: The main weaknesses of the paper are primarily in terms of presentation, rather than methodology or evaluation. My comments are mainly meant to help the authors improve their presentation. 

* Clarity of music concepts: Sometimes, the authors introduce or reference concepts which are not straightforward without domain knowledge. Examples of that are: * The orchestral function of x is introduced as a column sum of activation indicators over the different columns of X. Given that the columns of X represent MIDI pitches, this would mean that at each frame, the authors are summing the notes that are active. Yet, in p.3, l.115, they write that this indicator function "counts each note onset position as 1". Why onset only? This would mean that each note is counted once. But the equation states $x_t^k>0$, which appears to be evaluated for each frame (thus, not only at the onset of each note, but for its entire duration). Further, they state that "it [orchestral function] describes the rhythm and groove of each track". It is not at all clear to me why this orchestral function should describe the rhythm? It only appears to describe the amount of notes active at a given time. It's even less clear why it should describe the groove, given the subjective nature of "groove". It is okay that the authors attempt to give a layman's explanation to their method in this part, but it is also recommended that they better describe those concepts (and do so in a way that would be acceptable even to those that are not familiar with concepts from music theory).

 * There are other terms, such as "counterpoint", "phrase", etc. which will not be evident in an audience not familiar with them. While it is understandable that in a music generation paper, there will be a lot of domain language, it would be useful if the authors commented on their importance in their discussion of results. For example, the ILS metric measures the similarity within a music phrase vs the rest of the song. What does it mean that the authors' method does better than the baselines in this metric? That it produces more coherent phrases that are more easily distinguishable from the rest of the song? Adding a sentence to clarify the importance of each result (in, as far as this is possible, layman's terms) would improve the readability and outreach of the paper 

* Comparability to baselines: The authors compare to three established baselines. However, with the exception of PopMAG, which the authors reproduce on part of the data they used to train their own model, the other two baselines are not strictly comparable. AMT requires ground-truth accompaniment (arguably a limitation of that method compared to the authors'), and GETMusic was trained on a different data (and additionally handles different instrumental tracks). While this is not a no-go, and is common practice for generative methods in general, it should be mentioned in 5.2. 

* The description of *interleaving* layers in the autoencoder -- according to the authors themselves, a major novelty of the work -- does not feature as prominently in their text as it should (given its stated importance), and could be improved in terms of clarity. Specifically, it is not entirely clear what the "Track Encoder" does in Fig. 2. Is it a standard residual "transformer" (it is not clear from A.2 if a residual connection is there) layer that essentially combines information across tokens? It would be beneficial to describe this in 3.3 using simple equations like in 3.2. One important open question is why this track encoder, if it is indeed based on self-attention, is different than the decoder? Specifically, why does one integrate information in the time-axis and the other on the track-axis? It would be important to show the dimensions here. I guess you have: $(B, K, N, P)$, where B is the batch size, K the number of tracks, N the duration of each note, and P the number of notes (I guess N&P are eventually downsampled from their original dimensions). So the inter-track encoder would operate on tensors $(B\cdot N, K, P)$ and then the intra-track decoder would operate on tensors $(B\cdot K, N, P)$? All this of course would be done in autoregressive fashion, so it would go from $\{1...N\}$. It would be nice to show this using the notation of the authors. * The VQ-VAE is described (A.1) and portrayed (Fig. 1) as having a convolutional encoder with a stride of 4 (thus a downsampling of 4) and a fully-connected decoder. It would be useful to show how this fully-connected decoder mitigates the downsampling of the encoder to have an output size equal to the input. Minor comments: + p.7, l.240: Best substitute "denoising process" with denoising diffusion probabilistic model (if this is what this meant), as the term denoising can be ambiguous
questions: questions: There is only one question, regarding the interleaving of layers, but since it is also a weakness, I have included it there instead.
limitations: limitations: The authors have adequately discussed their limitations.
paper_summary: paper_summary: The paper suggests creating multi-instrument accompaniment by using the piano reduction and the instrument note density (referred to as the 'Orchestration Function') as bootstrap representations. By effectively applying VAE and an autoregressive sequence generation framework, the paper demonstrates the high potential and effectiveness of the proposed approach for learning musically structured accompaniment.
The paper also proposed a 'layer interleaving architecture' that processes the orchestration codec by alternating between the time axis and the track axis.
The paper compares with previous works with objective and subjective evaluation and shows its validity.
strength_weakness: strength_weakness: The proposed methodology utilizes piano reduction as a condition, so the quality of the final orchestra accompaniment is expected to vary depending on the quality of the piano reduction. Upon listening to the provided examples, it is evident that notes not present in the piano reduction were added, confirming that the proposed model's role extends beyond merely rearranging the notes of the piano reduction. However, there needs to be a discussion on the impact of using piano reduction on the model's high evaluation compared to other models. Additionally, while it was mentioned in L168 that another module was used, a more detailed explanation of how the piano reduction was generated is necessary.
questions: questions: Q1. From my understanding, the 'orchestration function' in Eq. 1 eliminates pitch information. Thus, I expect the priors (${s_t}$) to provide note-density-like bare information. What do you expect to be encoded in $s$? Also, as a follow-up question, what kinds of correlation are expected along with $s_t$, and why does it make sense to train a sequential prediction model for $s$? I'm not sure about the expected role of the prior model.
limitations: limitations: I generally agree with the authors about the limitation presented in Appendix E.
paper_summary: paper_summary: The paper presents a style transfer-based music accompaniment generation system. It starts with the lead melody and fleshes out the accompaniment tracks based on various high-level information, such as instrument type and rhythmic structure. The main goal of the proposed model is to be able to generate coherent structure in long music sequences, while they are also cohesive. Model architecture adopts VQVAE baselines to learn latent representations while the sequence learning part is done via transformer models applied to tracks and time dimensions orthogonally. The objective and subjective test scores look promising.
strength_weakness: strength_weakness: - The description of the proposed model is not organized well, so it is difficult to understand. 

- Some high-level choices the authors made are not well justified, such as discrete embeddings vs. continuous, Gaussian noise vs. other noise, etc.
questions: questions: - In Fig. 1, the autoencoder takes the piano reduction information as its first input. I imagine it would correspond to the lead sheet. Does it mean that during the training time the output of the autoencoder is the full sheet with all tracks? Then this isn't an autoencoder, right? The precise training procedures are not detailed enough in this manuscript. 

- If the input PN is the full music during training, how does the model generalize to the test-time scenario when the input has to be only the lead sheet?

- The manuscript seems to be based on the concept of 1 segment = 8 beats. Is this correct? Any rationale behind this choice?

- Gaussian noise is used to regularize the model. But if it's to deal with the domain shift between piano reduction representations, wouldn't there be a different type of noise that's more suitable, i.e. more discrete random variables?

- It appears that the "s" embeddings contain important information about the track to be generated. Since it's learned in an unsupervised way, it's a little hard to imagine how these embeddings are more interpretable than other existing conditioning vectors as the authors claim.
limitations: limitations: - Basically it's not clear how Fig. 1 and Fig. 2 are combined to form the pipeline described in Fig 3. Colored figures, shades, and hashes show the authors' effort in describing this method, but it's not entirely clear to me. 

- It appears that the system takes the user input to designate which instrumental track to generate based on the instrument embedding. It's a nice feature, but it also means that, for evaluation, somebody has to come up with a nice orchestration (i.e., which instruments go well given the lead melody). Was this critical selection of instruments done by the authors to generate the test sequences? Unless I missed this part, it has to be clearly mentioned, as it can affect the performance of the model (in comparison to other methods that don't have such a concept).
paper_summary: paper_summary: This paper introduces a novel music AI system that creates multi-track accompaniments from a lead sheet by leveraging disentangled style factors to enhance context coherence, creativity, and computational efficiency. The proposed two-stage process begins with generating a piano arrangement using piano texture styles, followed by orchestrating multiple tracks by incorporating orchestral function styles. The system employs vector quantization and a multi-stream Transformer for modeling orchestration styles, significantly improving arrangement quality and providing flexible control over various music genres and compositional levels.
strength_weakness: strength_weakness: 1. The introduction lacks clarity, with key terms and relationships not sufficiently defined before introducing mathematical notations and formulations. This obscures the motivation and makes it difficult to understand, especially as the abstract outlines challenges related to coherency, creativity, and efficiency that are not clearly addressed in the introduction.

2. The contributions of the paper, as discussed between lines 46-60 and 85-94, appear to be incremental rather than significant, primarily building upon work from AccoMontage.

3. There is a consistent omission of motivation when new concepts are introduced. For instance, the motivation behind formulating the multi-track accompaniment arrangement task is not made clear in the introduction after discussing style representations. Similarly, the reasons behind specific mechanisms in sections 3 and 4 are not elaborated upon.
questions: questions: 1. Why does concatenating style factors potentially contradict the structure of existing conditions as mentioned on line 25?

2. The paper discusses conflicts when combining style factors in sequence generation tasks; could these be resolved using different methods like diffusion models?

3. How do the initial discussions on introducing style into long-term generation relate to the paper’s main theme of multi-track accompaniment arrangement?

4. What prompted the formulation of the problem between lines 32-36 in paragraph 3, and how is time t determined?

5. Is there a distinction between 'piano reduction' and 'piano track'? The terminology varies between italics and normal text; does this signify a difference?

6. What is the rationale behind using VQ-VAE and VAE to encode orchestral functions and piano reductions in section 3.2?

7. What motivations underlie the construction of the Multi-Track Orchestral Function Prior in section 3.3?

8. Can you explain what inputs the proposed model receives and provide comparisons to highlight its unique features, as seen in Figure 4?

9. How does the model handle both ABC notation and Midi, and is there a conversion of ABC notation into the representation discussed in section 3.1?

10.Why is the computational efficiency of this model less optimal compared to [20]?

11. How are the 'style' factors controlled and tested in the experiments?
limitations: limitations: Limitations are addressed.

Comments: comment: Thank you for your detailed and thoughtful response to my review. Your clarifications, particularly regarding the significance of your contribution and the motivations for the methodologies employed, provide a deeper understanding of your work. I look forward to seeing these enhanced explanations incorporated into the revised manuscript, as they will undoubtedly strengthen its clarity and justification.