Title: Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling
Venue: NeurIPS 2024
Accepted: yes
Keywords: symbolic music generation

style transfer

accompaniment arrangement
Abstract: In the realm of music AI, arranging rich and structured multi-track accompaniments from a simple lead sheet presents significant challenges. Such challenges include maintaining track cohesion, ensuring long-term coherence, and optimizing computational efficiency. In this paper, we introduce a novel system that leverages prior modelling over disentangled style factors to address these challenges. Our method presents a two-stage process: initially, a piano arrangement is derived from the lead sheet by retrieving piano texture styles; subsequently, a multi-track orchestration is generated by infusing orchestral function styles into the piano arrangement. Our key design is the use of vector quantization and a unique multi-stream Transformer to model the long-term flow of the orchestration style, which enables flexible, controllable, and structured music generation. Experiments show that by factorizing the arrangement task into interpretable sub-stages, our approach enhances generative capacity while improving efficiency. Additionally, our system supports a variety of music genres and provides style control at different composition hierarchies. We further show that our system achieves superior coherence, structure, and overall arrangement quality compared to existing baselines.

Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling

Abstract

In the realm of music AI, arranging rich and structured multi-track accompaniments from a simple lead sheet presents significant challenges. Such challenges include maintaining track cohesion, ensuring long-term coherence, and optimizing computational efficiency. In this paper, we introduce a novel system that leverages prior modelling over disentangled style factors to address these challenges. Our method presents a two-stage process: initially, a piano arrangement is derived from the lead sheet by retrieving piano texture styles; subsequently, a multi-track orchestration is generated by infusing orchestral function styles into the piano arrangement. Our key design is the use of vector quantization and a unique multistream Transformer to model the long-term flow of the orchestration style, which enables flexible, controllable, and structured music generation. Experiments show that by factorizing the arrangement task into interpretable sub-stages, our approach enhances generative capacity while improving efficiency. Additionally, our system supports a variety of music genres and provides style control at different composition hierarchies. We further show that our system achieves superior coherence, structure, and overall arrangement quality compared to existing baselines.

1 Introduction

Representation learning techniques have enabled new possibilities for controllable generative modelling. By learning implicit style representations, which are often hard to explicitly label ( e.g. , timbre of music audio [21], texture of music composition [39], and artistic style in paintings [20]), new music and artworks can be created via style transfer and latent space sampling. These learned style factors can also serve as external controls for downstream generative models, including Transformers [18, 36] and diffusion models [42]. However, applying style factors to long-term sequence generation remains a challenging task. Existing approaches rely on style templates specified manually or by heuristic rules [36, 42, 51], which are impractical for long-term generation. Moreover, when structural constraints are imposed, misaligned style factors can result in incoherent outputs.

To address these challenges, we aim to develop a novel sequence generation framework leveraging a global style planner, or prior , which models the conditional distribution of style factors given the model input's content factors . Both style and content factors are sequences of compact, structurally aligned latent codes over a disentangled representation space. By infusing the style back to the content, we can recover the observational target with globally coherent style patterns.

In this paper, we study style prior modelling through the task of multi-track accompaniment arrangement , a typical scenario for long-term conditional sequence generation. We assume the input of a

piano accompaniment score, which typically carries a verse-chorus structure. Our target is to generate corresponding multi-track arrangements featuring band orchestration. We start by disentangling a band score at time t into piano reduction c t (content factor) and orchestral function s k t (style factors for individual tracks k = 1 , 2 , · · · , K ). On top of this, we model the prior of finding appropriate functions to orchestrate a given piano score , or formally p ( s 1: K 1: T | c 1: T ) . To model dependencies in both time ( T ) and track ( K ) directions, we develop a multi-stream Transformer with interleaved time-wise and track-wise layers. The track-wise layer allows for flexible control over the choice of instruments and the number of tracks, while the time-wise layer ensures structural alignment through cross-attention to the piano reduction . Decoding the inferred s 1: K 1: T with c 1: T , we can address accompaniment arrangement in a flexible multi-track form with extended whole-song structure.

Experiments show that our method outperforms existing sequential token prediction approaches and provides better multi-track cohesion, structural coherence, and computational efficiency. Additionally, compared to existing designs of multi-stream language models, our model handles flexible stream combinations more effectively with enhanced generative capacity.

To summarize, our contributions in this paper are three-folded:

We propose style prior modelling, a hierarchical generative methodology addressing both long-term structure (via style prior at high level) and fine-grained condition/control (via representation disentanglement at low level). Our approach moves beyond the limitation of manual specification of style factors, providing a flexible, efficient, and self-supervised solution for long-term sequence prediction and generation tasks.

We propose a novel layer interleaving architecture for multi-stream language modelling. In our case, it models parallel music tracks with a flexible track number, controllable instruments, and manageable computation. To our knowledge, it is the first multi-stream language model with tractable generalization to flexible stream combinations.

Integrating our previous study on piano texture style transfer [39, 50], we present a complete music automation system arranging an input lead sheet (a basic music form with melody and chord only) via piano accompaniment to multi-track arrangement. The entire system is interpretable at two composition hierarchies: 1) piano texture and 2) orchestral function , and demonstrates state-of-the-art arrangement performance for varied genres of music. 1

2 Related Works

In this section, we overview three topics related to our study. Section 2.1 reviews existing studies on representation disentanglement. Section 2.2 summarizes prior modelling methods in music generation. Section 2.3 reviews the current progress with the task of accompaniment arrangement.

2.1 Content-Style Disentanglement via Representation Learning

Representation disentanglement is a popular technique in deep generative modelling [3, 16, 48, 49]. In the music domain, this approach has proven valuable by learning compositional factors related to music style and content . By manipulating these factors through interpolation [32], swapping [39], and prior sampling [46], it provides a self-supervised and controllable pathway for various music automation tasks. Recent works leverage disentangled style factors as control signals for long-term music generation [36, 42]. However, these approaches typically treat style representations as fixed condition sequences during training, requiring manual specification or additional algorithms for control during inference. In contrast, we model the prior of the style to apply conditional on the given music content, which is a more generalized and flexible approach.

2.2 Music Generation with Latent Prior

In sequence generation tasks ( e.g. , music and audio), learning a prior sampler over a compact, latent representation space is often more efficient and effective. Jukebox [7] models the latent codes encoded by VQ-VAEs [34] as music priors, which can further reconstruct minutes of music audio. More recently, MusicLM [2] and MusicGen [4] learn multi-modal priors for generating music from text

prompts. While prior modelling facilitates long-term generation, the latent codes in these works are not interpretable, thus lacking a precise control by music content-based signals ( e.g. , music structure). Such controls are essential for conditional generation tasks, including accompaniment arrangement. In this paper, we model a style prior conditional on the disentangled music content, which allows for structured long-term music generation, enhancing both interpretability and controllability.

2.3 Accompaniment Arrangement

Accompaniment arrangement aims to compose the accompaniment part given a lead sheet, which is a difficult conditional generation task involving structural constraints. Existing methods mainly train a conditional language model based on sequential note-level tokenization [14, 15, 30, 33], which often suffer from slow inference speed, truncated structural context, and/or simplified instrumentation. Recent attempts with diffusion models show higher sample quality with faster inference [23, 26, 27], but still consider limited instruments or tracks. AccoMontage [47, 50] maintains a whole-song structure by manipulating high-level composition factors, but is limited to piano arrangement alone. Our paper presents a two-stage approach: from lead sheet to piano accompaniment, and from piano to multi-track, both leveraging prior modelling of high-level style factors. This approach offers modularity [11] and enables high-quality whole-song and multi-track accompaniment arrangement.

3 Method

We develop a model that takes a piano reduction as input and outputs an orchestrated multi-track arrangement. Using an autoencoder, we disentangle a multi-track music score into its piano reduction (content factor) and orchestral function (style factor). We then design a prior model to infer orchestral functions given the piano reduction . The autoencoder operates at segment level, while the prior model works on the whole song. The entire model can operate as an orchestrator module in a complete arrangement system. In this section, we introduce our data representation in Section 3.1, autoencoder framework in Section 3.2, and prior model design in Section 3.3.

3.1 Data Representation

We summarize our data representations in Table 1. Let x be a K -track arrangement score. We split it into T segments and represent x k t -each segment track - as a matrix of shape P × N . Here P = 128 represents 128 MIDI pitches and N is the time dimension of a segment. This matrix representation aligns with the modified piano roll in [39], where each non-zero entry ( p, n ) > 0 indicates a note onset and its value indicates the note duration. In this paper, we primarily focus on music pieces in 4/4 time signature with 1/4-beat resolution. Duration values range from 1 (for sixteenth notes) to 32 (for double whole notes). We consider 1 segment = 8 beats (2 bars) and derive N = 32 , which is a proper scale for learning music content/style representations [37, 39-41, 46].

The piano reduction of x is notated as pn[ x ] . It is approximated by downmixing all K tracks into a single-track mixture similar to [8]. When concurring notes are found across tracks, we keep the one with the largest duration ( i.e. , track-wise maximum). Segment-wise, pn[ x ] t is also a P × N matrix. It preserves the overall music content while discarding the multi-track form.

The orchestral function of x is notated as fn[ x ] . It describes the rhythm and grooving patterns [45] of each segment track, which serves as the 'skeleton' of a multi-track form. Formally,

where indicator function 1 {·} counts each note onset position as 1 ; colsum( · ) sums up the pitch dimension, deriving a 1 × N time-series feature; max _ sum = 14 is for normalization. The orchestral

pn[

𝐱𝐱

]

𝑡𝑡

Conv1d

FC

⋯

⋯

FC

⋯

⋯

(128x16)

�

𝐱𝐱

𝒕𝒕

𝟏𝟏

�

𝐱𝐱

𝒕𝒕

𝟐𝟐

⋯

�

𝐱𝐱

𝒕𝒕

𝑲𝑲

𝐜𝐜

𝑡𝑡

𝐳𝐳

𝑡𝑡

1

𝐳𝐳

𝑡𝑡

2

𝐳𝐳

𝑡𝑡

𝐾𝐾

Track Axis

𝐬𝐬

1

𝐬𝐬

2

𝐬𝐬

𝐾𝐾

⋯

⋯

fn[

𝐱𝐱

]

𝑡𝑡

𝑘𝑘

�

fn[

𝐱𝐱

]

𝑡𝑡

𝑘𝑘

Sep

(Track Separator )

Track Decoder

Piano Encoder

Fn. Enc.

VQ Codebook

Enc

p

Enc

f

Enc

f

Enc

f

Dec

f

Dec

f

Dec

f

Dec

t

Dec

t

Dec

t

Fn. Dec.

Function Encoder

Function Decoder

𝑡𝑡

𝑡𝑡

𝑡𝑡

function fn[ x ] essentially describes the form, or layout, of multi-track music x . It indicates the rhythmic intensity of parallel tracks and informs where to put more notes and where to keep silent.

3.2 Autoencoder

Our autoencoder consists of two components as shown in Figure 1. A VQ-VAE submodule (right of Figure 1) encodes orchestral function fn[ x ] k t . A VAE module (left of Figure 1) encodes piano reduction pn[ x ] t and reconstructs individual tracks x 1: K t leveraging the cues from fn[ x ] 1: K t . During training, both inputs pn[ x ] and fn[ x ] are deterministic transforms from the output x and the entire model is self-supervised. We see similar techniques for representation disentanglement in [39, 41, 46].

The VQ-VAE consists of Function Encoder Enc f and Decoder Dec f . Encoder Enc f contains a 1-D convolutional layer followed by a vector quantization block. Our intuition for applying a VQ-V AE is that orchestral function commonly consists of rhythm patterns (such as syncopation, arpeggio, etc. ) that can naturally be categorized as discrete variables. In our case, each segment track is encoded into 8 discrete embeddings on a 1-beat scale, indicating the flow of orchestration style. Formally,

where s k τ is the latent orchestral function code for the k -th track at the τ -th beat. We encode fn[ x ] k t at a finer 1-beat scale (instead of segment) to preserve fine-grained rhythmic details. The new scale is re-indexed by τ = 1 , 2 , · · · , 8 T . We collectively denote each 8-code grouping as s k t for conciseness.

The VAE consists of Piano Encoder Enc p , Track Separator Sep , and Track Decoder Dec t . Encoder Enc p learns content representation c t from piano reduction pn[ x ] t . Here c t is a continuous representation (without vector quantization) that captures more nuanced music content. Decoder Dec t reconstructs individual tracks x k t from track representation z k t . Notably, z 1: K t are recovered from c t using the orchestral function cues from s 1: K t . Formally,

where Track Separator Sep is a Transformer encoder. In this process, each s k t queries c t to recover the corresponding track ( k ), while they also attend to each other to maintain the dependency among parallel tracks. Learnable instrument embeddings [51] are added to each track based on its instrument class. We provide details of the autoencoder architecture in Appendix A.1.

3.3 Style Prior Modelling

The VQ-VAE in Section 3.2 derives latent codes s 1: K 1:8 T for orchestral function as a multi-stream time series. Here k = 1 , 2 , · · · K is the stream (track) index and τ = 1 , 2 , · · · , 8 T is the time (beat) index. The purpose of style prior modelling is to infer orchestral function given piano reduction so that the former can be leveraged to orchestrate the latter into multi-track music. We design our prior model as shown in Figure 2. It is an encoder-decoder framework that models parallel tracks/streams of orchestral function codes conditional on the piano reduction .

The decoder module (right of Figure 2) has alternate layers of Track Encoder and Auto-Regressive Decoder. Track Encoder is a standard Transformer encoder layer [35] and it aggregates inter-track information along the track axis. Auto-Regressive Decoder is a Transformer decoder layer (with self-attention and cross-attention) and it predicts next-step orchestral function codes on the time axis.

2

Context Encoder

𝐜𝐜1

:

𝑇𝑇

1

⋯

⋯

⋯

𝑠𝑠

𝜏𝜏

𝐾𝐾

1

1

𝑠𝑠1

1

𝑠𝑠1

𝐾𝐾

𝑠𝑠2

𝐾𝐾

𝑠𝑠3

𝐾𝐾

𝑠𝑠

𝜏𝜏+1

𝐾𝐾

Time Axis

+

+

+

+

Cross-Attention

⋯

𝑠𝑠

𝜏𝜏+1

1

Auto-Regressive Decoder

Sinusoidal Positional Encoding

Relative Positional Embedding

Music Timing Condition

Instrument Embedding

𝑠𝑠

𝜏𝜏

𝑠𝑠2

𝑠𝑠1

sos

By orthogonally stacking two types of layers, we can model track-wise and time-wise dependencies simultaneously with a manageable computational cost. Compared to sequential tokenization methods in previous studies [9, 30, 36], our method brings down the complexity from O ( N 2 T 2 ) to O (max( N,T ) 2 ) . Moreover, we support a flexible multi-track form ( N being variable) with a diverse instrumentation option. We add instrument embedding [51] and relative positional embedding [15] to the track axis, where 34 instrument classes [25] are supported. We add music timing condition [7] to the time axis, which encodes the positions in a training excerpt as fractions of the complete song, helping the model capture the overall structure of a song.

The encoder module (left of Figure 2) of our prior model is a standard Transformer encoder, which takes piano reduction c 1: T as global context. It is connected to the decoder module via cross-attention and maintains the global phrase structure. During training, both c 1: T and s 1: K 1:8 T are derived from the same multi-track piece and the entire model is self-supervised. Let p θ be the distribution of orchestral function codes fitted by our prior model θ , the training objective is the mean of negative log-likelihood of next-step code prediction:

We provide more implementation details of the prior model in Appendix A.2. We note that there is a potential domain shift from our approximated piano reduction to real piano arrangements. To prevent overfitting, we use a Gaussian noise ϵ to blur c 1: T while preserving its high-level structure . During training, ϵ is combined with c 1: T using a weighted summation with noise weight γ ranging from 0 to 1. It encourages a partial unconditional generation capability. At inference time, γ is a parameter that can balance creativity with faithfulness. An experiment on γ is covered in Appendix C.

4 Whole-Song Multi-Track Accompaniment Arrangement

We finalize a complete music automation system by applying style prior modelling at two cascaded stages. As shown in Figure 3, our autoencoder and orchestral function prior operate on Stage 2 for piano to multi-track arrangement. On Stage 1, we adopt our previous study, a piano texture prior [50]

Piano Texture Prior

Orchestral Function Prior

Lead Sheet

A Complete Multi-Track

Accompaniment Arrangement System

Piano Texture

Style

Orchestral Function

Style

External Control

Multi-Track Arrangement

Stage 1

Stage 2

Counterpoint Relation between Tracks

Metrical Division between String and Brass

Harmonic/Melodic Division between Two Guitars

Long-Term Phrase Coherence over Extended Context

41

13

Chorus

1

Bb

F/A

Gm

Eb

Bb

Eb

F/A

Bb/D

A.G. 1

A.G. 2

E.P. 1

Vlns.

Br.

E.B.

Chorus3

Bb

F/A

Gm

E

B

Eb

F

F/A

Bb/D

A.G.1

A.G.2

Pno

Vlns.

Br.

E.B.

on top of chord/texture representation learning [39], for lead sheet to piano arrangement. Given a lead sheet, the first stage generates a piano accompaniment, establishing the rough whole-song structure. Our system then orchestrates the piano accompaniment into a complete multi-track arrangement with band instrumentation. This two-stage approach mirrors musicians' creative workflow [1] and allows for control at both composition levels. In particular, we provide three control options:

Texture Selection: To filter piano textures on Stage 1 by metadata and statistical features.

Instrumentation: To customize the track number and choice of instruments on Stage 2.

Orchestral Prompt: To prompt the orchestration process with an orchestral function template.

We showcase an arrangement example by the complete system in Figure 4. The system input is a lead sheet shown by the Mel staff. The final output is the accompaniment in the rest staves. Notably, the lead sheet consists of 60 bars in a structure of i4A8B8B8x4A8B8B8O4 (using notations by [5]). Here, i4 , x4 , and O4 each denote a 4-bar intro, interlude, and outro. A8 and B8 represent an 8-bar verse and chorus, respectively. Figure 4 shows the arrangement result for the first and third choruses, spanning from bar 13 to 41. We leverage control option 2 to customize the instrumentation as celesta, acoustic guitars (2), electric pianos (2), acoustic piano, violin, brass, and electric bass in a total of K = 9 tracks. The complete arrangement score is included in Appendix E.

In Figure 4, we observe some multi-track arrangement patterns that are common in practice. Purple blocks highlight a counterpoint relation between guitar track A . G . 1 and electric piano track E . P . 1 .

Green blocks show two guitar tracks with complementary orchestral functions: one melodic ( A . G . 1 ) and the other harmonic ( A . G . 2 ). Yellow blocks illustrate the metrical division between the string ( Vlns . ) and the brass ( Br . ) sections, with strings on the downbeat and brass on the offbeat. These patterns demonstrate a natural and cohesive multi-track arrangement by our system. Moreover, we see consistent accompaniment patterns echoing in both chorus parts that span over 30 bars (shown by red dotted boxes), while the latter adds a piano arpeggio track ( Pno . ) to enhance the musical flow. This demonstrates structured whole-song arrangement over extended music contexts.

5 Experiment

In this section, we evaluate the performance of our multi-track accompaniment arrangement system. Given that existing methods primarily focus on lead sheet to multi-track arrangement, we ensure a fair comparison by using the two-stage approach discussed in Section 4. In Section 5.1, we present the datasets used and the training details of our model. In Section 5.2, we describe the baseline models used for comparison. Our evaluation is divided into two parts: objective evaluation, detailed in Section 5.3, and subjective evaluation, covered in Section 5.4. For the single-stage piano to multi-track (Stage 2) and lead sheet to piano (Stage 1) arrangement tasks, we perform additional comparisons with various ablation architectures in Section 5.5 and 5.6, respectively.

5.1 Datasets and Training Details

We use two datasets to train the autoencoder and the style prior, respectively. The autoencoder is trained with Slahk2100 [25], which contains 2K curated multi-track pieces with 34 instrument classes in a balanced distribution. We discard the drum track and clip each piece into 2-bar segments with 1-bar hop size. We use the official training split and augment training samples by transposing to all 12 keys. The autoencoder comprises 19M learnable parameters and is trained with batch size 64 for 30 epochs on an RTX A5000 GPU with 24GB memory. We use Adam optimizer [19] with a learning rate from 1e-3 exponentially decayed to 1e-5. We use exponential moving average (EMA) [29] and random restart [7] to update the codebook with commitment ratio β = 0 . 25 .

We use Lakh MIDI Dataset (LMD) [28] to train the prior model. It contains 170k music pieces and is a benchmark dataset for training music generative models. We collect 2/4 and 4/4 pieces (110k after processing) and randomly split LMD at song level into training (95%) and validation (5%) sets. We further clip each piece into 32-bar training excerpts ( i.e. , T = 16 at maximum) with a 4-bar hop size. Our prior model has 30M parameters and is trained with batch size 16 for 10 epochs (600K iterations) on two RTX A5000 GPUs. We apply AdamW optimizer [22] with a learning rate of 1e-4, scheduled by a 1k-step linear warm-up followed by a single cycle of cosine decay to a final rate of 1e-6.

For model inference and testing, we consider two additional datasets: Nottingham [10] and WikiMT [44]. Both datasets contain lead sheets (in ABC notation) that are not seen during training or validation. Moreover, they cover diverse music genres including folk, pop, and jazz. When arranging a piece, we leverage control option 2 to set up the instrumentation. Without loss of generality, this control choice is randomly sampled from Slakh2100 validation/test sets. To arrange music longer than the prior model's context length (32 bars), we use windowed sampling [7], where we move ahead our context window by 4 bars and continue sampling based on the previous 28 bars. We apply nucleus sampling [13] with top probability p = 0 . 05 and temperature t = 6 .

5.2 Baseline Models

We compare our system with three existing methods: PopMAG [30], Anticipatory Music Transformer (AMT) [33], and GETMusic [23]. PopMAG and GETMusic generate multi-track accompaniments from an input lead sheet based on a Transformer and a diffusion model, respectively. AMT is Transformer-based and it continues the accompaniment part from an input melody with starting accompaniment prompt. We provide detailed configurations in the following.

PopMAG is an encoder-decoder architecture based on Transformer-XL [6]. It represents multi-track music by sequential note-level tokenization and is fully supervised. The encoder takes a lead sheet as input and the decoder generates multi-track accompaniment auto-regressively. Since the model is not open source, We reproduce it on LMD with lead sheets extracted by [24] (melody) and [17] (chord).

Anticipatory Music Transformer (AMT) is a decoder-only Transformer architecture with note-level tokenization. It introduces an 'anticipation' method, where conditional tokens (melody and starting prompt) and generative tokens (accompaniment continuation) are interleaved to train a conditional generative model. Since our testing dataset does not provide ground-truth accompaniment, the starting prompt is given by the generation result (first 2 bars) of our system. We use the official implementation of the AMT model, 2 which is also trained on LMD.

GETMusic represents multi-track music as an image-like matrix resembling score arrangement, based on which a denoising diffusion probabilistic model is trained with a mask reconstruction objective. Given a lead sheet, it supports generating 5 accompaniment tracks using piano, guitar, string, bass, and drum or their subsets. In our experiment, we generate all 5 accompaniment tracks. We use the official implementation of the GETMusic model, 3 which is trained on internal data.

5.3 Objective Evaluation on Multi-Track Arrangement

We introduce four metrics to evaluate multi-track arrangement performance: chord accuracy [23, 30], degree of arrangement ( DOA ), structure awareness [42], and inference latency [30]. Among them, chord accuracy measures the multi-track harmony that reflects the fitness of the accompaniment to the lead sheet; DOA measures inter-track tonal diversity that reflects the creativity of the instrumentation. Both metrics demonstrate music cohesion at local scales. On the other hand, structure awareness measures phrase-level content similarity that reflects long-term structural coherence of the whole song. Finally, we use inference latency (in second/bar) to evaluate computational efficiency of each method. The detailed computation of each metric is provided in Appendix B. In Table 2, we compute ground-truth DOA using 1000 random pieces from LMD. We compute ground-truth structure awareness using 857 pieces in 4/4 from POP909 Dataset [38].

We randomly sample 50 pieces in 4/4 time signature from Nottingham and WikiMT respectively (100 pieces in total) to conduct experiment. The length of each piece ranges from 16 to 32 bars. We run our method and baseline models at each piece in 3 independent rounds, deriving 300 sets of multi-track arrangement samples. In Table 2, we report the evaluation results with mean value, standard error of mean ( sem ), and statistical significance computed by Wilcoxon signed rank test [43]. We find significant differences between our method and all baselines (p-value p < 0.05/6, using Bonferroni correction). In particular, our method outperforms in chord accuracy , structure awareness , and DOA , indicating the capability of arranging harmonious, structured, and creative accompaniments. The diffusion baseline outperforms in inference latency as it applies only 100 diffusion steps. Our method's efficiency is on par with it, while being 10 times faster than vanilla note-level auto-regression.

5.4 Subjective Evaluation on Multi-Track Arrangement

Wealso conduct a double-blind online survey to test music quality. Our survey consists of 5 evaluation sets, each containing an input lead sheet followed by 4 arrangement samples by our method and each baseline. Each sample is 24-32 bars long and is synthesized to audio at 90 BPM (~1 minute per sample). Both the set order and the sample order in each set are randomized. We request participants to listen to 2 sets and evaluate the musical quality based on a 5-point Likert scale from 1 to 5.



  









































The evaluation is based on 5 criteria: 1) Harmony and Texture Coherency , 2) Long-Term Structure , 3) Naturalness , 4) Creativity , and 5) Overall Musicality .

A total of 23 participants (8 female and 15 male) with diverse musical backgrounds have completed our survey, with an average completion time of 22 minutes. The mean ratings and standard errors, computed by within-subject (repeated-measures) ANOVA [31], are presented in Figure 5. Significant differences are observed across all criteria (p-value p < 0.05). Notably, our method outperforms all baselines on each criterion, aligning with the results from the objective evaluation.

5.5 Ablation Study on Style Prior Architecture

We now validate our design with the prior model by exclusively evaluating on the piano to multi-track arrangement task. Our prior model is based on a unique layer interleaving design, which enables multi-stream time series modelling with explicit stream-wise attention. We compare it with two other multi-stream architectures: 1) Parallel : summing up parallel code embeddings for joint language modelling [18], and 2) Delay : leveraging a 1-step delay code interleaving to catch implicit streamwise dependency [4]. Both Parallel and Delay are trained under the same setup as our model. We additionally introduce 3) Random : a naive prior based on random template retrieval. The templates are sampled every 2 bars with shared instrumentation from the validation/test sets of Slakh2100.

We introduce two metrics to evaluate piano to multi-track arrangement: faithfulness and degree of arrangement ( DOA ). Faithfulness measures if the generated arrangement faithfully reflects the original content from the piano. It computes the similarity between i) the input piano, and ii) the piano reduction of the generated multi-track arrangement. In our case, we compute cosine similarity over two features: a statistical ( stats. ) pitch class histogram [45] and a latent texture representation [39], which emphasize tonal and rhythmic similarity, respectively. DOA measures the creativity as defined in Section 5.3. We also report the NLL loss for our model, Parallel , and Delay .

We conduct experiments using the test set of POP909 [39], which consists of 88 piano arrangement pieces. In our experiment, we use the first section of each piece, which contains 2 to 4 complete phrases totally spanning 24 to 32 bars. We use control option 3 to prompt our model, Parallel , and Delay with the same 2-bar orchestral function template (sampled from Slakh2100) and see how it is developed. We report mean value, standard error of mean ( sem ), and statistical significance in Table 3 and find significant differences in both faithfulness and DOA . We also conduct a subjective evaluation in the same setup as Section 5.4, with the results presented in Figure 6. Here we consider an additional criterion, Instrumentation , which reflects the well-formedness of the multi-

track arrangement. Significant differences are observed across all criteria (p-value p < 0.05). Overall, Parallel and Delay both fall short in performance because they assume a preset stream combination, while in our setting, both track numbers and choices of instruments are flexible. By explicitly modelling stream-wise attention, our layer interleaving design fits well to that generalized scenario.

5.6 Ablation Study on Piano Arrangement

Now we validate our choice for the lead sheet to piano arrangement module on the first stage of our two-stage system. Our choice is a piano texture prior as covered in Section 4. We conduct an ablation study by replacing it with the Whole-Song-Gen model [42], which, to our knowledge, is the only existing alternative that can handle a whole-song structure. The ablation study is conducted in the same setup as Section 5.3. In Table 4, we report chord accuracy , structure awareness , and DOA regarding the final multi-track arrangement results. We further compare our piano texture prior with Whole-Song-Gen exclusively on the piano accompaniment arrangement task. In Table 5, we report chord accuracy and structure awareness regarding piano arrangement for both models. Significant differences (p-value p < 0.05) are found in all metrics based on Wilcoxon signed rank test.

By comparing Table 4 and Table 5, we can see that a higher-quality piano arrangement generally encourages a more musical and creative final multi-track arrangement result. Specifically, the piano arrangement on Stage 1 lays the groundwork for (at least) chord progression and phrase structure for Stage 2, both of which are important for capturing the long-term structure in whole-song multi-track arrangement. Moreover, we see that our piano texture prior outperforms existing alternatives and guarantees a decent piano quality, thus being the best choice for our system.

6 Conclusion

To sum up, we contribute a music automation system for multi-track accompaniment arrangement. The main novelty lies in our proposed style prior modelling , a generic methodology for structured sequence generation with fine-grained control. By modelling the prior of disentangled style factors given content, we build a cascaded arrangement process: from lead sheet to piano texture style, and then from piano to orchestral function style. Our system first generates a piano accompaniment from a lead sheet, establishing the rough whole-song structure. It then orchestrates the piano accompaniment into a complete multi-track arrangement with band instrumentation. Extensive experiments show that our system generates structured, creative, and natural multi-track arrangements with state-of-the-art quality. At a higher level, we elaborate our methodology as interpretable modular representation learning , which leverages finely disentangled and manipulable music representations to tackle complex tasks with a compositional hierarchy. We hope our research brings new perspectives to broader domains of music creation, sequence data modelling, and representation learning.

References

Samuel Adler and Peter Hesterman. The study of orchestration , volume 2. WW Norton New York, NY, 1989.

Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325 , 2023.

Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A. Efros. Everybody dance now. In 2019 IEEE/CVF International Conference on Computer Vision , pages 5932-5941, 2019.

Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. In Advances in neural information processing systems , volume 36, 2023.

Shuqi Dai, Huan Zhang, and Roger B Dannenberg. Automatic analysis and influence of hierarchical structure on melody, rhythm and harmony in popular music. arXiv preprint arXiv:2010.07518 , 2020.

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Conference of the Association for Computational Linguistics , pages 2978-2988, 2019.

Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341 , 2020.

Hao-Wen Dong, Chris Donahue, Taylor Berg-Kirkpatrick, and Julian J. McAuley. Towards automatic instrumentation by learning to separate parts in symbolic multitrack music. In Proceedings of the 22nd International Society for Music Information Retrieval Conference , pages 159-166, 2021.

Hao-Wen Dong, Ke Chen, Shlomo Dubnov, Julian McAuley, and Taylor Berg-Kirkpatrick. Multitrack music transformer. In International Conference on Acoustics, Speech and Signal Processing , pages 1-5. IEEE, 2023.

Eric Foxley. Nottingham database. [EB/OL], 2011. https://ifdo.ca/~seymour/ nottingham/nottingham.html Accessed May 17, 2023.

Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse H. Engel, and Douglas Eck. Enabling factorized piano music modeling and generation with the MAESTRO dataset. In 7th International Conference on Learning Representations , 2019.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations , 2020.

Wen-Yi Hsiao, Jen-Yu Liu, Yin-Cheng Yeh, and Yi-Hsuan Yang. Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 178-186, 2021.

Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. In 7th International Conference on Learning Representations , 2019.

Rongjie Huang, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech. In Advances in neural information processing systems , volume 35, 2022.

Junyan Jiang, Ke Chen, Wei Li, and Gus Xia. Large-vocabulary chord transcription via chord structure decomposition. In Proceedings of the 20th International Society for Music Information Retrieval Conference , pages 644-651, 2019.

Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. Text-free prosody-aware generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , pages 8666-8681, 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Dmytro Kotovenko, Artsiom Sanakoyeu, Sabine Lang, and Björn Ommer. Content and style disentanglement for artistic style transfer. In 2019 IEEE/CVF International Conference on Computer Vision , pages 4421-4430. IEEE, 2019.

Liwei Lin, Gus Xia, Qiuqiang Kong, and Junyan Jiang. A unified model for zero-shot music source separation, transcription and synthesis. In Proceedings of the 22nd International Society for Music Information Retrieval Conference , pages 381-388, 2021.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 6th International Conference on Learning Representations , 2018.

Ang Lv, Xu Tan, Peiling Lu, Wei Ye, Shikun Zhang, Jiang Bian, and Rui Yan. Getmusic: Generating any music tracks with a unified representation and diffusion framework. arXiv preprint arXiv:2305.10841 , 2023.

Xichu Ma, Xiao Liu, Bowen Zhang, and Ye Wang. Robust melody track identification in symbolic music. In Proceedings of the 23rd International Society for Music Information Retrieval Conference , pages 842-849, 2022.

Ethan Manilow, Gordon Wichern, Prem Seetharaman, and Jonathan Le Roux. Cutting music source separation some slakh: A dataset to study the impact of training data quality and quantity. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics , pages 45-49, 2019.

Lejun Min, Junyan Jiang, Gus Xia, and Jingwei Zhao. Polyffusion: A diffusion model for polyphonic score generation with internal and external controls. In Proceedings of the 24th International Society for Music Information Retrieval Conference , pages 231-238, 2023.

Matthias Plasser, Silvan Peter, and Gerhard Widmer. Discrete diffusion probabilistic models for symbolic music generation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence , pages 5842-5850, 2023.

Colin Raffel. Learning-Based Methods for Comparing Sequences, with Applications to Audioto-MIDI Alignment and Matching . PhD thesis, Columbia University, USA, 2016.

Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in neural information processing systems , volume 32, 2019.

Yi Ren, Jinzheng He, Xu Tan, Tao Qin, Zhou Zhao, and Tie-Yan Liu. Popmag: Pop music accompaniment generation. In Proceedings of the 28th ACM International Conference on Multimedia , pages 1198-1206, 2020.

Henry Scheffe. The analysis of variance , volume 72. John Wiley & Sons, 1999.

Hao Hao Tan and Dorien Herremans. Music fadernets: Controllable music generation based on high-level features via low-level feature modelling. In Proceedings of the 21st International Society for Music Information Retrieval Conference , pages 109-116, 2020.

John Thickstun, David Leo Wright Hall, Chris Donahue, and Percy Liang. Anticipatory music transformer. Transactions on Machine Learning Research , 2024.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in neural information processing systems , volume 30, 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , volume 30, 2017.

Dimitri von Rütte, Luca Biggio, Yannic Kilcher, and Thomas Hoffman. Figaro: Generating symbolic music with fine-grained artistic control. In 11th International Conference on Learning Representations , 2023.

Ziyu Wang and Gus Xia. Musebert: Pre-training music representation for music understanding and controllable generation. In Proceedings of the 22nd International Society for Music Information Retrieval Conference , pages 722-729, 2021.

Ziyu Wang, Ke Chen, Junyan Jiang, Yiyi Zhang, Maoran Xu, Shuqi Dai, and Gus Xia. POP909: A pop-song dataset for music arrangement generation. In Proceedings of the 21st International Society for Music Information Retrieval Conference , pages 38-45, 2020.

Ziyu Wang, Dingsu Wang, Yixiao Zhang, and Gus Xia. Learning interpretable representation for controllable polyphonic music generation. In Proceedings of the 21st International Society for Music Information Retrieval Conference , pages 662-669, 2020.

Ziyu Wang, Yiyi Zhang, Yixiao Zhang, Junyan Jiang, Ruihan Yang, Gus Xia, and Junbo Zhao. PIANOTREE VAE: Structured representation learning for polyphonic music. In Proceedings of the 21st International Society for Music Information Retrieval Conference , pages 368-375, 2020.

Ziyu Wang, Dejing Xu, Gus Xia, and Ying Shan. Audio-to-symbolic arrangement via crossmodal music representation learning. In International Conference on Acoustics, Speech and Signal Processing , pages 181-185. IEEE, 2022.

Ziyu Wang, Lejun Min, and Gus Xia. Whole-song hierarchical generation of symbolic music using cascaded diffusion models. In 12th International Conference on Learning Representations , 2024.

Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics: Methodology and distribution , pages 196-202. Springer, 1992.

Shangda Wu, Dingyao Yu, Xu Tan, and Maosong Sun. Clamp: Contrastive language-music pre-training for cross-modal symbolic music information retrieval. In Proceedings of the 24th International Society for Music Information Retrieval Conference , pages 157-165, 2023.

Shih-Lun Wu and Yi-Hsuan Yang. The jazz transformer on the front line: Exploring the shortcomings of ai-composed music through quantitative measures. In Proceedings of the 21st International Society for Music Information Retrieval Conference , pages 142-149, 2020.

Ruihan Yang, Dingsu Wang, Ziyu Wang, Tianyao Chen, Junyan Jiang, and Gus Xia. Deep music analogy via latent representation disentanglement. In Proceedings of the 20th International Society for Music Information Retrieval Conference , pages 596-603, 2019.

Li Yi, Haochen Hu, Jingwei Zhao, and Gus Xia. Accomontage2: A complete harmonization and accompaniment arrangement system. In Proceedings of the 23rd International Society for Music Information Retrieval Conference , pages 248-255, 2022.

Wenjie Yin, Hang Yin, Kim Baraka, Danica Kragic, and Mårten Björkman. Dance style transfer with cross-modal transformer. In IEEE/CVF Winter Conference on Applications of Computer Vision , pages 5047-5056, 2023.

Siyang Yuan, Pengyu Cheng, Ruiyi Zhang, Weituo Hao, Zhe Gan, and Lawrence Carin. Improving zero-shot voice style transfer via disentangled representation learning. In 9th International Conference on Learning Representations , 2021.

Jingwei Zhao and Gus Xia. Accomontage: Accompaniment arrangement via phrase selection and style transfer. In Proceedings of the 22nd International Society for Music Information Retrieval Conference , pages 833-840, 2021.

Jingwei Zhao, Gus Xia, and Ye Wang. Q&A: Query-based representation learning for multitrack symbolic music re-arrangement. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence , pages 5878-5886, 2023.

A Implementation Details

A.1 Autoencoder

The autoencoder consists of a VQ-VAE submodule and an overarching VAE. The encoder of the VQVAE consists of a 1-D convolutional layer of kernel size 4, stride 4, and 16 output channels, followed by a vector quantization block with codebook size 128. The decoder takes the concatenated latent codes and leverages two fully-connected layers (shape 128 × 256 and 256 × 32 ) for reconstruction. In the overarching VAE, Piano Encoder and Track Decoder are adapted from PianoTree VAE [40]. The encoder first applies a pitch-wise bi-directional GRU to summarize concurrent notes at time step n and then applies a time-wise GRU to encode the full representation. The decoder mirrors the encoder structure with time- and pitch-wise uni-directional GRUs to reconstruct individual tracks. We use hidden size 256 in a single layer for pitch GRUs and 512 for time GRUs. The Track Separator is a 2-layer Transformer encoder with 8 attention heads, 0.1 dropout ratio, and GELU activation [12]. The hidden dimensions of self-attention d model and feed-forward layers d ff are 512 and 1024, respectively.

The autoencoder is trained with joint reconstruction loss for orchestral function (MSE) and individual tracks (cross entropy). The VQ-VAE is additionally regularized with latent loss and commitment loss with commitment ratio β = 0 . 25 . The VAE is regularized with KL loss over all continuous factors ( c t and z 1: K t ) based on KL annealing [41] with a ratio exponentially increasing from 0 to 0.5.

A.2 Prior Model

The prior model consists of a 12-layer Context Encoder and a 12-layer Auto-Regressive Decoder. The latter is interleaved with another 12 track-wise Track Encoder layers. For each layer, we apply 8 attention heads, 0.1 dropout ratio, and GELU activation. We apply layer normalization before self-attention ( i.e. , norm first). The hidden dimensions of self-attention d model and feed-forward layers d ff are 256 and 1024, respectively. We apply relative positional embedding [15] to Track Encoder so that two tracks initialized with identical instruments can still generate different content.

Our prior model is trained on the latent codes c 1: T and s 1: K 1:8 T inferred by a well-trained autoencoder on LMD. For discrete code s , we take the codebook indices and learn a new embedding.

B Objective Evaluation Metrics

B.1 Degree of Arrangement

In multi-track arrangement, parallel tracks typically play a unique role to each other in the overall arrangement. We are interested in capturing the diversity and creativity inherent in these roles.

To achieve this, we consider the pitch class histogram [45] as a probability distribution P . Let P t,k be the distribution of the t -th bar in track k , and P pn t be that of the t -th bar in the piano reduction . Recall that in this paper we approximate the piano reduction of a multi-track piece by downmixing all tracks. Both P t,k and P pn t are 12-D vectors, describing tonality of individual tracks and the overall arrangement, respectively. We compute the KL divergence of each track to the piano reduction :

where T is the total number of bars.

Interpreting d k in terms of KL divergence, we see it as the 'excess surprise' from the overall arrangement ( piano reduction ) when track k is played in isolation. A large d k indicates that track k possesses a unique quality, such as a bass track playing the root or a counter-melody track focusing on tensions. Conversely, a small d k suggests that track k serves as a foundational element in the arrangement, such as string padding that establishes the harmonic foundation.

If all d k values are small, it implies homogeneity across tracks and thus a low degree of arrangement. Conversely, if all d k values are high, it suggests a composition dominated by counterpoints, a scenario less common in pop music. A well-orchestrated piece typically exhibits a diverse range of d k values, encompassing both foundational and unique decorative tracks. We thus define the degree of

arrangement DOA as the standard deviation of d k for k = 1 , 2 , · · · , K across all tracks:

where d is the mean. K is the total number of tracks. To establish a reference point, we calculate the ground-truth DOA = 0 . 333 based on 1000 random pieces from the LMD dataset with at least 5 tracks. Within this context, a higher DOA signifies a more creative arrangement.

B.2 Strcture Awareness

We introduce Inter-phrase Latent Similarity ( ILS ) from [42] to measure the structural awareness of long-term arrangement. ILS calculates the content similarity among same-type phrases ( e.g. , chorus) versus the whole song. It leverages pre-trained disentangled V AEs that encode music notes into latent representations and then compare cosine similarities in the latent space. In our case, we compute ILS over the piano reduction of a generated arrangement since it contains the overall content. We apply the texture VAE [39] and obtain a latent texture representation c txt t for every 2-bar segment. For odd-numbered phrases, we repeat its final bar and pad it to the end of the phrase. Suppose there are M different types of phrases in one piece and let I m be the set of segment indices in the typem phrase, ILS is defined as the ratio between same-type phrase similarity and global average similarity:

̸

̸

where | · | is the cardinality of a set. T is the number of 2-bar segments. When applying ILS , we use [5] to automatically label the phrase structure of a piece. To establish a reference point, we calculate the ground-truth ILS = 1 . 980 based on the POP909 dataset (with phrase annotation by human). Within this context, a higher ILS signifies saliency with long-term phrase-level structure.

B.3 Chord Accuracy

We introduce chord accuracy from [30] to measure if the chords of the generated arrangement match the conditional chord sequence in the lead sheet. It reflects the harmonicity of the generated music and is defined as follows:

where N chord is the number of chords in a piece; C i is the i -th chord in the (ground-truth) lead sheet; and ˆ C i is the aligned chord in the generated arrangement.

The original formulation in [30] considers chord accuracy for individual tracks. Given our system's capability to accommodate a variable combination of tracks, we opt for a broader evaluation for the overall arrangement. In our case, we extract the chord sequence of a generated arrangement with [17] and compare it in root and quality with ground-truth at 1-beat granularity, which is more rigorous.

B.4 Orchestration Faithfulness

We measure the faithfulness of orchestration by the similarity between i) the input piano and ii) the piano reduction of the generated multi-track arrangement. Let e in t and e pn t be vector features derived from the t -th segment of the input and the reduction, respectively. Orchestration faithfulness OF is defined as follows:

where cos( · , · ) is cosine similarity. T is the number of segments.

In our work, we select two options for vector feature e . One is a statistical pitch class histogram [45], which is a 12-D vector describing pitch class distribution. The other is a latent 256-D texture representation learned by a pre-trained V AE [39]. Both features are general descriptors of the musical content with respective focus on tonal harmony and rhythmic grooves.

C Experiment on Noise Weight γ

Continuing from Section 3.3, we compare different γ values and see their impact on the model performance. When applying our model to piano to multi-track arrangement, γ balances the force of a noisy factor added to the piano, which encourages a partial unconditional generation capability. The experimental settings are the same as Section 5.5. We evaluate the results based on faithfulness and DOA . In Table 6, we report mean value, standard error of mean ( sem ), and statistical significance computed by Wilcoxon signed rank test. By varying the γ value, we observe a controllable balance between faithfulness and creativity. Specifically, a larger γ encourages creativity (higher DOA ) at the cost of faithfulness . If not mentioned otherwise, we use γ = 0 . 25 for experiments in this paper.

D Online Survey Specifics

We distribute our survey via SurveyMonkey. 4 Our survey consists of 5 sample sets for both the lead sheet to multi-track and the piano to multi-track arrangement tasks (10 sets in total). Each sample is 24-32 bars long and is synthesized to audio at 90 BPM using BandLab 5 with the default soundfont. Each participant listens to 2 sets (in random order) and the mean time spent is 22 minutes. Figure 7 shows the sample pages of our survey with instructions to the participants.

08

New Evaluation on Full-Band Accom

entArrangement&Orchestration

Arangement Set 2

We are going to arrange a lead sheetinto full-band music. A lead sheet* is a simple musical fom

with melody and chords only. The lead sheet as follows (model input) has 32 bars in an AABB

structure:

Now, we will present four arrangement samples (model outputs). Please evaluate the music quality

of each sample based on the following criteria:

Coherency: The fitness of the arrangement to the melody in harmony and texture.

• Naturalness: How likely a human arranger produces the music.

• Structure: The dynamic flow with music development in the long run.

• Muslcality: The overall musical quality.

. Creativity: How creative the music is.

*9.How would you rate the following arrangement?

Naturalness

Coherency

very low

O

low

neutral

high

very high

Structure

Musicalty

Creativty

O

*10.How would you rate the following arrangement?

203

(a) Lead sheet to multi-track arrangement.

田☆

New Evaluation on Full-Band Accompaniment Arrangement & Orchestration

Orchestration Set 4

We are going toarrange apiece of pian solo intful-bandmusicThe piano solo (model input is

Now, we will present four arrangement samples (model outputs) starting with the same 2-bar

prompt. Please evaluate the music quality of each sample based on the following criteria:

Instrumentation: The dversity,hamonicity and well-fomedness of fl-band arrangement.

Structure: The dynamic flow with music development in the long run.

Musicality: The overallmusical qualty.

• Creativity: How creative the music is.

*5.Howwould yourate the following arrangement?

very low

heutral

very high

Structure

Musicalty

Creativty

O

*6.Howwould yourate the following arrangement?

(b) Piano to multi-track arrangement.

E Example on Structured Arrangement

We demonstrate an example of accompaniment arrangement by our proposed system. The input lead sheet is a complete pop song shown in Section E.1. Our system first arranges a piano accompaniment for the whole song, which is shown in Section E.2. The piano score is then orchestrated into a multi-track arrangement with customized instrumentation, which is shown in Section E.3.

E.1 Lead Sheet

We use our system to arrange for Can You Feel the Love Tonight , a pop song by Elton John. As shown in Figure 8, the entire song is 60 bars long and it presents a structure of i4A8B8B8x4A8B8B8O4 , where i , x , O , A , and B each refer to intro, interlude, outro, verse, and chorus.

25

33

5

9

29

21

17

13



























 

















 





  











 

















  











 





  









 









 





















 



 



  











 

 

  







 







 



 







 











  













 























































































































































=	65

B8

B8

O4

x4

i4

A8

E♭

Cm

Gm

B♭/D

B♭/D

B♭/D

Cm7

B♭

B♭

Cm/E♭

E♭

F7sus

E♭

F/A

B♭/D

B♭/F

B♭

E♭

B♭/D

E♭/B♭

Cm/E♭

B♭

F

C/E

E♭

E♭

E♭

B♭/D

B♭

B♭/D

B♭/D

F/A

C7/E

Gm

F/A

F

F

E♭

Gm

C7/E

Gm

F

Cm

B♭/D

E♭

F/A

B♭/D

E♭

E♭/G

E♭

E♭

B♭/D

A♭

Gm

Cm/E♭

B♭

Cm7

F7sus

E♭

E♭/B♭

Gm

F/A

B♭

Cm7

B♭/F

E♭

E♭

B♭/D

E♭/G

E♭

B♭/D

F/A

Cm7

B♭

B♭

B♭

F/A

E♭

B♭/D

E♭

B♭/D

1.

2.































 















 























E.2 Piano Arrangement

The piano arrangement result is shown from Figure 9 to Figure 10. It roughly establishes a whole-song structure and lays the groundwork for band orchestration at the next stage. Demo audio for the piano score is available at https://zhaojw1998.github.io/structured-arrangement/ .





25

17

13

9

21

5

Melody

Pno.

Pno.

Pno.

Pno.

Pno.

Pno.

Piano







































































































































































































































































































































































































































































  





























































 













































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































=	65

B8

i4

A8

B8

E♭

Cm/E♭

E♭

B♭/D

B♭

B♭/D

Gm

Gm

E♭

E♭/G

E♭

B♭

B♭/D

Cm7

B♭

F/A

B♭/D

E♭

C7/E

B♭/D

B♭/D

E♭

B♭/D

E♭

Cm/E♭

F/A

Cm7

B♭

F/A

Gm

B♭

E♭

F

F

B♭

E♭

E♭

E♭

F/A

Gm

B♭/D

C/E

F

E♭

B♭/D

C7/E

B♭/D

Cm7

B♭

B♭/D

E♭

B♭/F

F

Cm

E♭

F7sus

A♭

E♭/B♭

F/A

Gm

E♭







































































































29

45

33

37

53

49

41

57

Pno.

Pno.

Pno.

Pno.

Pno.

Pno.

Pno.

Pno.







































































































































































































































































































































































 















































































































































 









































 



 



 





 



























































 







 



 





 











































 

























 















































 

















 





























































 

  





















 

 























 













































  











 











 























































 

















 







































































 

















































































 















 



























































































 

















 









 



 





























































































































































































































































































































































































































































































































































A8

B8

O4

B8

x4

Cm

Gm

E♭

B♭

F

B♭/D

Gm

B♭

F

B♭/D

C/E

B♭/D

Gm

E♭

Cm7

Gm

E♭

E♭

B♭/D

B♭/D

Gm

F/A

Cm/E♭

B♭

F

F/A

E♭/B♭

B♭/D

E♭

B♭/D

E♭

F7sus

B♭

B♭/D

Cm/E♭

B♭/F

E♭/G

F/A

E♭

B♭/D

E♭

A♭

Cm

B♭

E♭

E♭

F/A

E♭

B♭

F

C7/E

Cm7

B♭/D

F/A

B♭/D

Cm7

E♭

E♭

E♭

E♭

E♭/B♭

B♭/D

C7/E

E♭

B♭

F7sus

Gm

B♭

Cm/E♭

B♭/D

E♭































































E.3 Multi-Track Arrangement

The multi-track arrangement is shown from Figure 11 to Figure 15. We customize the instrumentation as celesta, acoustic guitars (2), electric pianos (2), acoustic piano, violin, brass, and electric bass in a total of K = 9 tracks. We can see that the structure of the accompaniment follows the lead sheet. Demo audio is available at https://zhaojw1998.github.io/structured-arrangement/ . More detailed analysis on this arrangement demo is covered in Section 4.

Melody

Celesta

Acoustic	Guitar	1

Acoustic	Guitar	2

Piano

Electric	Piano	1

Electric	Piano	2

Brass

Electric	Bass









 







 



 

























 





  





























































































































 























































































































































































































































































































=	65

i4

E♭/G

F/A

F/A

B♭

B♭

Cm7

B♭/D

B♭/D

B♭/F

E♭



























5















A8

































E♭





























































B♭/D































 

















B♭/D



























B♭/D

















































E♭























































E♭



































































Cm7



















































 





F/A



 





















































A.G.	2

E.P.	1

E.P.	2

Vlns.

Br.

E.B.



































17

13

9

A.G.	1

A.G.	1

A.G.	1

A.G.	2

A.G.	2

A.G.	2

Pno.

E.P.	1

E.P.	1

E.P.	1

E.P.	2

Vlns.

Vlns.

Vlns.

Br.

Br.

Br.

E.B.

E.B.

E.B.



















































































































































































































 



































































































 















































































































 

































 













 













 





































 















































































































































 





























































 









































































































































































































































































































































































































































































B8

Gm

F

E♭

Gm

B♭/D

A♭

C/E

E♭

C7/E

B♭

Cm/E♭

B♭/D

F

B♭/D

E♭

Cm7

E♭

B♭

E♭

F/A

E♭

Gm

F

E♭

B♭/D







































































33

21

25

29

A.G.	1

A.G.	1

A.G.	1

A.G.	2

A.G.	2

A.G.	2

Pno.

Pno.

E.P.	2

Vlns.

Vlns.

Br.

Br.

E.B.

E.B.

E.B.

E.B.







































































































































































 







 







 





































































































































































 























 







































































































 

































  



 

 







































































  

























 

































 























































































































 

























































































































































































































































































































































































































































































































































































x4

A8

B8

B♭/D

B♭/D

E♭

F/A

E♭

Cm

B♭/D

E♭

E♭

E♭/B♭

B♭

Gm

E♭

F/A

B♭

B♭/F

B♭

B♭

Cm/E♭

Cm7

B♭/D

E♭

B♭/D

F/A

F/A

B♭

F7sus

B♭/D

B♭/D

Cm7

E♭

E♭

F

E♭/G

C7/E

Gm

















































37

41

45

Cel.

A.G.	1

A.G.	1

A.G.	2

A.G.	2

Pno.

Pno.

Pno.

Vlns.

Vlns.

Br.

Br.

E.B.

E.B.

E.B.



























































































































 





 















































 













 









 







 







 









 















































































































 



 

























 











 



 

















 





































































 



 







 











 



























 





  



 





 







 













 



 

  





 

  



























  





 



 









 





  





















































































































































































































































































































































































































































































































































B8

E♭

Cm7

Cm/E♭

B♭/D

B♭/D

A♭

F/A

E♭

Gm

B♭/D

B♭

C7/E

C/E

F

E♭

F

E♭

B♭/D

E♭

E♭

Gm

B♭

Gm

E♭

F































49

57

53

A.G.	1

A.G.	1

A.G.	2

A.G.	2

A.G.	2

Pno.

Pno.

E.P.	1

E.P.	2

Vlns.

Vlns.

Vlns.

Br.

Br.

E.B.

E.B.

E.B.







































































































































































 











 

































 

  



























































































  















 















































 



















 





 











 

















 



 





















 















































  











 





































































 



































 







































 

























































































 





































































































































































































































































































































































































































































O4

B8

Cm/E♭

E♭

F7sus

E♭

E♭/B♭

E♭

Cm/E♭

Gm

E♭

Gm

F/A

B♭

B♭

E♭

C7/E

F

Cm

B♭

Gm

E♭

B♭

B♭/D

Cm

E♭/B♭

B♭/D

F7sus

B♭/D

B♭/D































F Limitation

We propose a two-stage system for whole-song, multi-track accompaniment arrangement. In the context of this paper, we acknowledge that our current system exclusively supports tonal tracks in quadruple meters while disregarding triple meters, triplet notes, and drums. However, we perceive this as a technical limitation rather than a scientific challenge. We also acknowledge that our current system primarily emphasizes the composition level, thereby omitting the modelling of MIDI velocity, dynamic timing, and MIDI control messages. Consequently, the generated results do not encompass performance MIDI and may lack expressive qualities. Nevertheless, we believe that our composition-centric work serves as a solid and vital foundation for further advancements in those specific areas, thus facilitating the development of enhanced techniques and features. As a pioneering work, our system is the foremost accomplishment in solving whole-song multi-track accompaniment arrangement, characterized by flexible controllability on track number and choice of instruments.

G Broader Impacts

Our multi-track accompaniment arrangement system, which incorporates style to generate accompaniment, is designed to enhance originality and creativity. It serves as a platform for human-AI co-creation, where the user provides content-based material (in our case, lead sheet) that remains fundamentally original, while the AI agent infuses style, enriches the form, and enhances creativity . Our system therefore empowers musicians to explore new musical ideas and expand their creative boundaries. This approach also allows for rapid mock-up with different styles and arrangements, fostering an environment where innovation and artistic expression can thrive.

However, we acknowledge the need to address potential risks. The accessibility of our system may inadvertently lead to excessive reliance on automation, potentially impeding the development of fundamental skills among musicians. Additionally, widespread adoption of the system may contribute to the homogenization of music, threatening the distinctiveness and individuality that are crucial to artistic expression. We recognize that our datasets predominantly feature contemporary Western music, which introduces a cultural bias that could limit the diversity of generated compositions.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our claims in the abstract and introduction accurately reflect our paper's contributions and scope with support from experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Appendix F discusses the limitations of our work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We introduce our experimental settings thoroughly in Section 5. We also provide details of model architectures necessary for reproduction in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All datasets used in this work are open-source. We release our code and more resources at https://github.com/zhaojw1998/Structured-Arrangement-Code .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Training and testing details are covered in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For objective evaluation, we report mean, standard error of mean, and statistical significance computed by Wilcoxon signed rank test [43], which does not assume normality. For subjective evaluation, we report mean, standard error, and statistical significance computed by within-subject ANOVA [31], where normality of errors is verified.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details of computational resources are covered in Section 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed and conformed in every respect with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines .

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss both positive and negative impacts of our work in Appendix G.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: All datasets used in this paper are open-source datasets with minimum risk of misuse, and so are the models trained on them.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use two datasets for model training: Lakh MIDI [28] and Slakh2100 [25], both licensed under CC BY 4.0. We use three datasets for model testing: WikiMT [44] and POP909 [38], which are under the MIT License, and Nottingham [10], for which no license information is available. We use the official code of two baseline models: AMT [33], licensed under Apache-2.0, and GETMusic [23], which is under the MIT License.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We release code, checkpoint, and notebook tutorial of this work under the MIT License at https://github.com/zhaojw1998/Structured-Arrangement-Code .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: We provide crowdsourcing details (including screenshots) in Appendix D.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: The human study in our experiment is based on online crowdsourcing, which bears minimum risk. Participants are informed that participation in our study is entirely voluntary and that they may choose to stop participating at any time without any negative consequences. No personally identifying information is collected in the human study.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel music AI system that creates multi-track accompaniments from a lead sheet by leveraging disentangled style factors to enhance context coherence, creativity, and computational efficiency. The proposed two-stage process begins with generating a piano arrangement using piano texture styles, followed by orchestrating multiple tracks by incorporating orchestral function styles. The system employs vector quantization and a multi-stream Transformer for modeling orchestration styles, significantly improving arrangement quality and providing flexible control over various music genres and compositional levels.\n\nSTRENGTHS:\n1. The paper addresses the significant task of Multi-Track Accompaniment Arrangement, making notable improvements in the field.\n\n2. The authors have diligently constructed a comprehensive framework in both the experimental design and methodology, contributing valuable insights and data to the field.\n\nWEAKNESSES:\n1. The introduction lacks clarity, with key terms and relationships not sufficiently defined before introducing mathematical notations and formulations. This obscures the motivation and makes it difficult to understand, especially as the abstract outlines challenges related to coherency, creativity, and efficiency that are not clearly addressed in the introduction.\n\n2. The contributions of the paper, as discussed between lines 46-60 and 85-94, appear to be incremental rather than significant, primarily building upon work from AccoMontage.\n\n3. There is a consistent omission of motivation when new concepts are introduced. For instance, the motivation behind formulating the multi-track accompaniment arrangement task is not made clear in the introduction after discussing style representations. Similarly, the reasons behind specific mechanisms in sections 3 and 4 are not elaborated upon.\n\nQUESTIONS:\n1. Why does concatenating style factors potentially contradict the structure of existing conditions as mentioned on line 25?\n\n2. The paper discusses conflicts when combining style factors in sequence generation tasks; could these be resolved using different methods like diffusion models?\n\n3. How do the initial discussions on introducing style into long-term generation relate to the paper’s main theme of multi-track accompaniment arrangement?\n\n4. What prompted the formulation of the problem between lines 32-36 in paragraph 3, and how is time t determined?\n\n5. Is there a distinction between 'piano reduction' and 'piano track'? The terminology varies between italics and normal text; does this signify a difference?\n\n6. What is the rationale behind using VQ-VAE and VAE to encode orchestral functions and piano reductions in section 3.2?\n\n7. What motivations underlie the construction of the Multi-Track Orchestral Function Prior in section 3.3?\n\n8. Can you explain what inputs the proposed model receives and provide comparisons to highlight its unique features, as seen in Figure 4?\n\n9. How does the model handle both ABC notation and Midi, and is there a conversion of ABC notation into the representation discussed in section 3.1?\n\n10.Why is the computational efficiency of this model less optimal compared to [20]?\n\n11. How are the 'style' factors controlled and tested in the experiments?\n\nLIMITATIONS:\nLimitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a style transfer-based music accompaniment generation system. It starts with the lead melody and fleshes out the accompaniment tracks based on various high-level information, such as instrument type and rhythmic structure. The main goal of the proposed model is to be able to generate coherent structure in long music sequences, while they are also cohesive. Model architecture adopts VQVAE baselines to learn latent representations while the sequence learning part is done via transformer models applied to tracks and time dimensions orthogonally. The objective and subjective test scores look promising.\n\nSTRENGTHS:\n- The paper presents reasonable intermediate representations of symbolic music, piano reduction and orchestral function, which are used to learn discrete embeddings by the VQVAE model. \n\n- The proposed model is evaluated by both objective and subjective manners, where it outperformed the other methods.\n\nWEAKNESSES:\n- The description of the proposed model is not organized well, so it is difficult to understand. \n\n- Some high-level choices the authors made are not well justified, such as discrete embeddings vs. continuous, Gaussian noise vs. other noise, etc.\n\nQUESTIONS:\n- In Fig. 1, the autoencoder takes the piano reduction information as its first input. I imagine it would correspond to the lead sheet. Does it mean that during the training time the output of the autoencoder is the full sheet with all tracks? Then this isn't an autoencoder, right? The precise training procedures are not detailed enough in this manuscript. \n\n- If the input PN is the full music during training, how does the model generalize to the test-time scenario when the input has to be only the lead sheet?\n\n- The manuscript seems to be based on the concept of 1 segment = 8 beats. Is this correct? Any rationale behind this choice?\n\n- Gaussian noise is used to regularize the model. But if it's to deal with the domain shift between piano reduction representations, wouldn't there be a different type of noise that's more suitable, i.e. more discrete random variables?\n\n- It appears that the \"s\" embeddings contain important information about the track to be generated. Since it's learned in an unsupervised way, it's a little hard to imagine how these embeddings are more interpretable than other existing conditioning vectors as the authors claim.\n\nLIMITATIONS:\n- Basically it's not clear how Fig. 1 and Fig. 2 are combined to form the pipeline described in Fig 3. Colored figures, shades, and hashes show the authors' effort in describing this method, but it's not entirely clear to me. \n\n- It appears that the system takes the user input to designate which instrumental track to generate based on the instrument embedding. It's a nice feature, but it also means that, for evaluation, somebody has to come up with a nice orchestration (i.e., which instruments go well given the lead melody). Was this critical selection of instruments done by the authors to generate the test sequences? Unless I missed this part, it has to be clearly mentioned, as it can affect the performance of the model (in comparison to other methods that don't have such a concept).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper suggests creating multi-instrument accompaniment by using the piano reduction and the instrument note density (referred to as the 'Orchestration Function') as bootstrap representations. By effectively applying VAE and an autoregressive sequence generation framework, the paper demonstrates the high potential and effectiveness of the proposed approach for learning musically structured accompaniment.\nThe paper also proposed a 'layer interleaving architecture' that processes the orchestration codec by alternating between the time axis and the track axis.\nThe paper compares with previous works with objective and subjective evaluation and shows its validity.\n\nSTRENGTHS:\nThe strength of this paper, I believe, lies in the factorization of accompaniment generation by leveraging the fact that piano reduction and 'orchestration function' can be freely produced as middle-level features from multi-track MIDI. By separating each process, as the authors claim, it becomes applicable to scenarios where users can exert control. Additionally, by dividing the modeling of each accompaniment's prior and the modeling of detailed notes, the efficiency of learning for each model is expected to have increased.\n\nWEAKNESSES:\nThe proposed methodology utilizes piano reduction as a condition, so the quality of the final orchestra accompaniment is expected to vary depending on the quality of the piano reduction. Upon listening to the provided examples, it is evident that notes not present in the piano reduction were added, confirming that the proposed model's role extends beyond merely rearranging the notes of the piano reduction. However, there needs to be a discussion on the impact of using piano reduction on the model's high evaluation compared to other models. Additionally, while it was mentioned in L168 that another module was used, a more detailed explanation of how the piano reduction was generated is necessary.\n\nQUESTIONS:\nQ1. From my understanding, the 'orchestration function' in Eq. 1 eliminates pitch information. Thus, I expect the priors (${s_t}$) to provide note-density-like bare information. What do you expect to be encoded in $s$? Also, as a follow-up question, what kinds of correlation are expected along with $s_t$, and why does it make sense to train a sequential prediction model for $s$? I'm not sure about the expected role of the prior model.\n\nLIMITATIONS:\nI generally agree with the authors about the limitation presented in Appendix E.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a new model for generating multi-track accompaniment given a lead sheet. While being strictly a two-stage method (first generate a piano arrangement from the lead sheet and *then* generate the accompaniment), the authors' contribution is focused on the 2nd stage, and they rely on existing modules for the 1st stage. Their proposed method features two key components: **first**, a VQ-VAE submodule whose aim is to learn (quantized) representations of \"orchestral functions\" (orchestral functions being 1-D time-series representations of a track, with each element representing the sum of notes active in each segment, more on that in weaknesses), and, **second**, a transformer-based \"mixing\" VAE which combines the representations of the VQ-VAE (i.e., the representations of the orchestral functions) with learnt representations of \"piano reductions\" of each song (derived by averaging all individual tracks) to generate latent representations which are used to synthesize the full song. A key novelty of their model is the \"mixing\" part of the VAE, with the decoder utilizing interleaved time-wise and track-wise cross-attention layers -- meaning, that each output \"sees\" information both from past frames in its own track and from other tracks. Their system compares favorably to 3 standard baselines taken as-is from prior work or reproduced.\n\nSTRENGTHS:\nOverall, barring some presentation issues discussed below, this is a well-written paper that features state-of-the-art results and enough novelty to make it relevant for both the broader generative AI community (given the difficulty of generating long-form, complex music which the authors tackle with hierarchical modelling) and the more niche music-generation community. This is why I am leaning positive towards accepting the paper. I will mention some more of its strengths in detail: + The authors target a challenging task which is to generate a multi-track arrangement given a lead sheet. Even though they \"only\" focus on the 2nd stage of that pipeline, namely the generation of the multi-track arrangement given a piano arrangement, it is still a valid contribution. + The idea to disentangle the piano reduction and orchestral function and combine them in the way they do is interesting and novel. This also provides a nice element of controllability. + The fact that the model can be trained in \"self-supervised\" fashion (i.e. without manual annotations) is commendable. + The authors have done a thorough job of documenting experimental parameters (data, GPU runtime, etc.) and submit code to reproduce their experiments. + The authors have accounted for different sources of variability and provided multiple \"knobs\" to twist (and counteract domain mismatches) through the clever use of multiple \"positional\" embeddings (Fig. 2).\n\nWEAKNESSES:\nThe main weaknesses of the paper are primarily in terms of presentation, rather than methodology or evaluation. My comments are mainly meant to help the authors improve their presentation. \n\n* Clarity of music concepts: Sometimes, the authors introduce or reference concepts which are not straightforward without domain knowledge. Examples of that are: * The orchestral function of x is introduced as a column sum of activation indicators over the different columns of X. Given that the columns of X represent MIDI pitches, this would mean that at each frame, the authors are summing the notes that are active. Yet, in p.3, l.115, they write that this indicator function \"counts each note onset position as 1\". Why onset only? This would mean that each note is counted once. But the equation states $x_t^k>0$, which appears to be evaluated for each frame (thus, not only at the onset of each note, but for its entire duration). Further, they state that \"it [orchestral function] describes the rhythm and groove of each track\". It is not at all clear to me why this orchestral function should describe the rhythm? It only appears to describe the amount of notes active at a given time. It's even less clear why it should describe the groove, given the subjective nature of \"groove\". It is okay that the authors attempt to give a layman's explanation to their method in this part, but it is also recommended that they better describe those concepts (and do so in a way that would be acceptable even to those that are not familiar with concepts from music theory).\n\n * There are other terms, such as \"counterpoint\", \"phrase\", etc. which will not be evident in an audience not familiar with them. While it is understandable that in a music generation paper, there will be a lot of domain language, it would be useful if the authors commented on their importance in their discussion of results. For example, the ILS metric measures the similarity within a music phrase vs the rest of the song. What does it mean that the authors' method does better than the baselines in this metric? That it produces more coherent phrases that are more easily distinguishable from the rest of the song? Adding a sentence to clarify the importance of each result (in, as far as this is possible, layman's terms) would improve the readability and outreach of the paper \n\n* Comparability to baselines: The authors compare to three established baselines. However, with the exception of PopMAG, which the authors reproduce on part of the data they used to train their own model, the other two baselines are not strictly comparable. AMT requires ground-truth accompaniment (arguably a limitation of that method compared to the authors'), and GETMusic was trained on a different data (and additionally handles different instrumental tracks). While this is not a no-go, and is common practice for generative methods in general, it should be mentioned in 5.2. \n\n* The description of *interleaving* layers in the autoencoder -- according to the authors themselves, a major novelty of the work -- does not feature as prominently in their text as it should (given its stated importance), and could be improved in terms of clarity. Specifically, it is not entirely clear what the \"Track Encoder\" does in Fig. 2. Is it a standard residual \"transformer\" (it is not clear from A.2 if a residual connection is there) layer that essentially combines information across tokens? It would be beneficial to describe this in 3.3 using simple equations like in 3.2. One important open question is why this track encoder, if it is indeed based on self-attention, is different than the decoder? Specifically, why does one integrate information in the time-axis and the other on the track-axis? It would be important to show the dimensions here. I guess you have: $(B, K, N, P)$, where B is the batch size, K the number of tracks, N the duration of each note, and P the number of notes (I guess N&P are eventually downsampled from their original dimensions). So the inter-track encoder would operate on tensors $(B\\cdot N, K, P)$ and then the intra-track decoder would operate on tensors $(B\\cdot K, N, P)$? All this of course would be done in autoregressive fashion, so it would go from $\\{1...N\\}$. It would be nice to show this using the notation of the authors. * The VQ-VAE is described (A.1) and portrayed (Fig. 1) as having a convolutional encoder with a stride of 4 (thus a downsampling of 4) and a fully-connected decoder. It would be useful to show how this fully-connected decoder mitigates the downsampling of the encoder to have an output size equal to the input. Minor comments: + p.7, l.240: Best substitute \"denoising process\" with denoising diffusion probabilistic model (if this is what this meant), as the term denoising can be ambiguous\n\nQUESTIONS:\nThere is only one question, regarding the interleaving of layers, but since it is also a weakness, I have included it there instead.\n\nLIMITATIONS:\nThe authors have adequately discussed their limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-16"}