Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling
Authors: Jingwei Zhao
Gus Xia
Ziyu Wang
Ye Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: symbolic music generation
style transfer
accompaniment arrangement
Primary area: generative_models
TL;DR: A cascaded style prior modelling approach to whole-song multi-track accompaniment arrangement

Abstract: In the realm of music AI, arranging rich and structured multi-track accompaniments from a simple lead sheet presents significant challenges. Such challenges include maintaining track cohesion, ensuring long-term coherence, and optimizing computational efficiency. In this paper, we introduce a novel system that leverages prior modelling over disentangled style factors to address these challenges. Our method presents a two-stage process: initially, a piano arrangement is derived from the lead sheet by retrieving piano texture styles; subsequently, a multi-track orchestration is generated by infusing orchestral function styles into the piano arrangement. Our key design is the use of vector quantization and a unique multi-stream Transformer to model the long-term flow of the orchestration style, which enables flexible, controllable, and structured music generation. Experiments show that by factorizing the arrangement task into interpretable sub-stages, our approach enhances generative capacity while improving efficiency. Additionally, our system supports a variety of music genres and provides style control at different composition hierarchies. We further show that our system achieves superior coherence, structure, and overall arrangement quality compared to existing baselines.

Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling
Jingwei Zhao! 1.3
Gus Xia
Ziyu Wang
Ye Wang2.1.
Institute of Data Science, NUS Integratevv Sciences and Engineering
"School of Computing, NUS Programme, NUS Graduate School 'Computer Science Department, NYU Shanghai gus. xia.mbzuabzaii.. .c.aai.a.......................e ae wangye@comp nus. edu.
Machine learning Department, MBZUAI jzhao@u.nus nus. edu ziyu werr.wanneny..o. edu
Abstract
In the realm of music AI, arranging rich and structured multi-track accompani- ments from simple lead sheet presents significant challenges. Such challenges include maintaining track cohesion, ensuring long-term coherence, and optimizing computational efficiency. In this paper, introduce novel system that leverages prior modelling over disentangled style factors to address these challenges. Our method presents two-stage process: initially, a piano arrangement is derived from the lead sheet by retrieving piano texture styles; subsequently, a multi-trace orchestration is generated by infusing orchestral function styles into the piano arrangement. Our key design is the use of vector quantization and unique multi- stream Transformer model the long-term flow of the orchestration style, which enables flexible. controllable, and structured music generation. Experiments show that by factorizing the arrangement task into interpretable sub-stages, our approach enhances generative capacity while improving efficiency. Additionally, our system supports variety of music genres and provides style control different compo- sition hierarchies. We further show that our system achieves superior coherence, structure, and overall arrangement quality compared to existing baselines.
Introduction
Representation learning techniques have enabled new possibilities for controllable generative mod- elling. By learning implicit style representations, which are often hard to explicitly label (e.g.. timbre of music audio [21], texture of music composition [39], and artistic style paintings [20]), new music and artworks can be created via style transfer and latent space sampling. These learned style factors can also serve as external controls for downstream generative models, including Trans- formers [18, 36] and diffusion models [42]. However, applying style factors to long-term sequence generation remains a challenging task. Existing approaches rely on style templates specified manually or by heuristic rules [36, 42, 51], which are impractical for long-term generation. Moreover, when structural constraints are imposed, misaligned style factors can result in incoherent outputs.
To address these challenges, we aim to develop a novel sequence generation framework leveraging global style planner, or prior, which models the conditional distribution of style factors given the model input's content factors. Both style and content factors are sequences of compact, structurally aligned latent codes over disentangled representation space. By infusing the style back to the content, we can recover the observational target with globally coherent style patterns.
In this paper, we study style prior modelling through the task of malti-track accompaniment arrange- ment, a typical scenario for long-term conditional sequence generation. We assume the input of a
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

piano accompaniment score, which typically carries a verse-chorus structure. Our target is to generate corresponding multi-track arrangements featuring band orchestration. We start by disentangling a band score at time L into piano reduction C₁ (content factor) and orchestral function st (style factors for individual tracks k = =1,2,... K). On top of this, we model the prior of finding appropriate functions to orchestrate given piano score, or formally p(s (1:T)- To model dependencies in both time (2') and track (K) directions, we develop a multi-stream Transformer with interleaved time-wise and track-wise layers. The track-wise layer allows for flexible control over the choice of instruments and the number of tracks, while the time-wise layer ensures structural alignment through cross-attention to the piano reduction. Decoding the inferred s with C1:7, we can address accompaniment arrangement in a flexible multi-track form with extended whole-song structure.
Experiments show that our method outperforms existing sequential token prediction approaches and provides better multi-track cohesion, structural coherence, and computational efficiency. Additionally, compared to existing designs of multi-stream language models, our model handles flexible stream combinations more effectively with enhanced generative capacity.
To summarize, our contributions in this paper are three-folded:
We propose style prior modelling, hierarchical generative methodology addressing both long-term structure (via style prior at high level) and fineggraieed condition/control (via representation disentanglement at low level). Our approach moves beyond the limitation of manual specification of style factors, providing flexible, efficient, and self-supervised solution for long-term sequence prediction and generation tasks.
We propose a novel layer interleaving architecture for multi-stream language modelling. In our case, it models parallel music tracks with a flexible track number, controllable instruments, and manageable computation. To our knowledge, it is the first multi-stream language model with tractable generalization to flexible stream combinations.
Integrating our previous study on piano fexture style transfer [39,50]] we present a complete music automation system arranging an input lead sheet (a basic music form with melody and chord only) via piano accompaniment to multi-track arrangement. The entire system is interpretable at two composition hierarchies: 1) piano texture and 2) orchestral function, and demonstrates state-of-the-ar arrangement performance for varied genres of music.'
2 Related Works
In this section, we overview three topics related to our study. Section 2.1 reviews existing studies on representation disentanglement. Section 2.2 summarizes prior modelling methods in music generation. Section 2.3 reviews the current progress with the task of accompaniment arrangement.
2.1 Content-Style Disentanglement via Representation Learning
Representation disentanglement a popular technique in deep generative modelling [3, 16, 48, 49]. In the music domain, this approach has proven valuable by learning compositional factors related to music style and content. By manipulating these factors through interpolation [32], swapping [39], and prior sampling [46], it provides a self-supervised and controllable pathway for various music automation tasks. Recent works leverage disentangled style factors as control signals for long-term music generation [36, 42]. However, these approaches typically treat style representations as fixed condition sequences during training, requiring manual specification or additional algorithms for control during inference. In contrast, model the prior of the style to apply conditional the given music content, which is more generalized and flexible approach.
2.2 Music Generation with Latent Prior
In sequence generation tasks (e.g., music and audio), learning a prior sampler over compact, latent representation space is often more efficient and effective. Jukebox [7] models the latent codes encoded by VQ-VAEs [34] as music priors, which can further reconstruct minutes of music audio. More recently, MusicL. [2] and MusicGen [4] learn multi-modal priors for generating music from text
'Demo and more resources: : /tttas//2hhojv119998 http:////.......a.ooooooutttaat.
io/structured /ttp:/tuucuueddaaaraaeeeettt

---

Table Summary of the data representations applied in this paper. We use notation [a..b] to denote the integer interval {i
S S b, D E ž) including both endpoints.
Multi-Track Arrangement
Piano Reduction
Orchestral Function
Data Representation
X 32 XXX32222
pn x] E [0..32] x32x128 fu|x] E [0. 1
Latent Dimension
z E RTxXxx50
E R
8 [0..127]6TxK
prompts. While prior modelling facilitates long-tern seng-term generation, the latent codes in these works are not interpretable, thus lacking precise control by music content-based signals (e.g., music structure). Such controls are essential for conditional generation tasks, including accompaniment arrangement. In this paper, we model a style prior conditional on the disentangled music content, which allows for structured long-term music generation, enhancing both interpretability and controllability.
2.3 Accompaniment Arrangement
Accompaniment arrangement aims to compose the accompaniment part given a lead sheet, which is a difficult conditional generation task involving structural constraints. Existing methods mainly train conditional language model based on sequential note-level tokenization [14, 15, 30, 33), which often suffer from slow inference speed, truncated structural context, and/or simplified instrumentation. Recent attempts with diffusion models show higher sample quality with faster inference [23, 26, 27], but still consider limited instruments or tracks. AccoMontage [47, 50] maintains whole-song structure by manipulating high-level composition factors, but is limited to piano arrangement alone. Our paper presents two-stage approach: from lead sheet to piano accompaniment, and from piano to multi-track, both leveraging prior modelling of high-level style factors. This approach offers modularity [11] and enables high-quality whole-song and multi-track accompaniment arrangement.
3 Method
We develop a model that takes a piano reduction as input and outputs an orchestrated multi-trace arrangement. Using an autoencoder, we disentangle multi-track music score into its piano reduction (content factor) and orchestral function (style factor). We then design a prior model to infer orchestral functions given the piano reduction. The autoencoder operates at segment level, while the prior model works on the whole song. The entire model can operate as an orchestrator module in a complete arrangement system. In this section, we introduce our data representation in Section 3.1, autoencoder framework in Section 3.2, and prior model design in Section 3.3.
3.1 Data Representation
We summarize data representations in Table 1. Let x be K-track arrangement score. We split it into T segments and represent x - each segment track- - as a matrix of shape P x N. Here P 128 represents 128 MIDI pitches and N is the time dimension of segment. This matrix representation aligns with the modified piano roll in [39]. where each non-zero entry (p, n) > 0 indicates note onset and its value indicates the note duration. In this paper, on music pieces in 4/4 time signature with 1/4-beat resolution. Duration values range from (for sixteenth notes) to 32 (for double whole notes). We consider I segment = 8 beats (2 bars) and derive N = 32, which is a proper scale for learning music content/style representations [37, 39-41, 46).
primarily focus
The piano reduction of x is notated as pn[x]. It is approximated by downmixing all K tracks into single-track mixture similar to [8]. When concurring notes are found across tracks, we keep the one with the largest duration (i.e., track-wise maximum). Segment-wise pn[x]; is also a P x N matrix. preserves the overall music content while discarding the multi-track form.
The orchestral function of x is notated as fi[xx] It describes the rhythm and grooving patterns [45] of each segment track, which serves as the "skeleton" of a multi-track form. Formally,
fn [x]] = (10 (aarsso)mmmosseeeee mmx_sum,
(1)
where indicator function 143 counts each note onset position as 1 colsum(-) sums up the pitch dimension, deriving a 1 x N time-series feature, max_sum 14 is for normalization. The orchestral
3

---

Track Decoder
z
Trick Axis
Doe'
Dee
Dee
VQ Codeebook
(12x116)
Sep (Track Separalot)
Fr. Fric.
Fa. Dec.
In [x]]
pn[X]
Eac!
Decf
Enc
Deef
Piano
Encoder
sf
Function Encoder
Function Decoder
Figure I: The autoencoder architecture. It learns content representation c; from piano reduction, style representations s!:k from orchestral function, and leverages both to reconstruct individual tracks.
function fn(x essentially describes the form, or layout, of multi-track music x. It indicates the rhythmic intensity of parallel tracks and informs where to put more notes and where to keep silent.
3.2 Autoencoder
Our autoencoder consists of components as shown in Figure A VQ-VAE submodule (right of Figure 1) encodes orchestral function fn [x]] A VAE module (left of Figure 1) encodes piano reduction pn[x] and reconstructs individual tracks x x! leveraging the cues from fn[x] During training, both inputs pu[x] and fu [x]] are deterministic transforms from the output x and the entire model is self-supervised. We see similar techniques for representation disentanglement [39, 41, 46].
The VQ- VAE consists of Function Encoder Enc' and Decoder Dec Encoder Enc contains 1-D convolutional layer followed by a vector quantization block. Our intuition for applying a VQ- is that orchestral function commonly consists of rhythm patterns (such syncopation, arpeggio, etc.) that can naturally be categorized as discrete variables. In our case, each segment track encoded into 8 discrete embeddings 1-beat scale, indicating the flow of orchestration style. Formally,
11t 1577==8--7 Enc' (fu[x]/), k = 1,2, . K,
(2)
where s is the latent orchestral function code for the k-th track at the 7-th beat. We encode fn[x]] a finer 1-beat scale (instead of segment) to preserve fine-grained rhythmic details. The new scale is re-indexed by 1, ST. We collectively denote each 8-code grouping st for conciseness.
The VAE consists of Piano Encoder EncP, Track Separator Sep, and Track Decoder Dec Encoder EncP learns content representation c, from piano reduction pn[x]x- Here c, a continuous repre- sentation (without vector quantization) that captures more nuanced music content. Decoder Dec reconstructs individual tracks x from track representation z Notably, z!! are recovered from ca using the orchestral function cues from s, Formally,
K c.),
where Track Separator Sep Transformer encoder. In this process, each s queries C, recover the corresponding track (k), while they also attend to each other to maintain the dependency among parallel tracks. Learnable instrument embeddings [51] are added to each track based on its instrument class. We provide details of the autoencoder architecture in Appendix A.I.
3.3 Style Prior Modelling
The VQ-VAE in Section 3.2 derives latent codes 1:K $188 for orchestral function as a multi-stream time series. Here l 1, K is the stream (track) index and 1,2, ST is the time (beat) index. The purpose style prior modelling is to infer orchestral function given piano reduction so that the former can be leveraged to orchestrate the latter into multi-track music. We design our prior model as shown in Figure? 2. It is an encoder-decoder framework that models parallel tracks/streams of orchestral function codes conditional on the piano reduction.
The decoder module (right of Figure 2) has alternate layers of Track Encoder and ReggeReeeeeevve Decoder. Track Encoder is a standard Transformer encoder layer [35] and it aggregates inter-track information along the track axis. Auto- Regresegeee Decoder is a Transformer decoder layer (with self-attention and cross- atteenttt)t and it predicts next-step orchestral function codes the time axis.

---

r+1
Cross-Allentin
Context Encoder
st+
C.T
Auto-Regressive Decoder
Track Encoder
Sinusuidal Positional Encoding
Relative Pusitional Emboolding
Track Asis
Music Timing Condition
Embeelding
Gaussian Nake
Time Axis
505
$
Figure 2: The prior model architecture. The overall architecture is an encoder-decoder Transformer, while the decoder module interleaved with orthogonal time-wise and track-wise layers.
By orthogonally stacking two types of layers, we can model track-wise and time-wise dependen- cies simultancously with manageable computational cost. Compared to sequential tokenization methods in previous studies [9, 30, 36], our method brings down the complexity from O(N2T²) to (mmax(NV 2). Moreover, we support a flexible multi-track form (N being variable) with a diverse instrumentation option. We add instrument embedding [51] and relative positional embedding [15] to the track axis, where 34 instrument classes [25] are supported. We add music timing condition [7] to the time axis, which encodes the positions in a training excerpt as fractions of the complete song, helping the model capture the overall structure
song.
The encoder module (left of Figure 2) of our prior model is standard Transformer encoder, which takes piano reduction C1.7 as global context. is connected to the decoder module via cross-attention and maintains the global phrase structure. During training, both C1.7 and $1887 are derived from the same multi-track piece and the entire model self-supervised. Let pe be the distribution of orchestral function codes fitted by our prior model 0, the training objective is the mean of negative log-likedt oog-likelhhod of next-step code prediction:
slKK
K
1 K
((0)
log po
1 CIT)).
L:K
provide more implementation details the prior model in Appendix A.2. We note that there is potential domain shift from our approximated piano reduction to real piano arrangements. To prevent overfitting, we use a Gaussian noise to blur C1.7 while preserving its high-level structure. During training, e is combined with C1.7' using a weighted summation with noise weight y ranging from O to 1. It encourages a partial unconditional generation capability. At inference time, is a parameter that can balance creativity with faithfulness. An experiment y is covered in Appendix C.
4 Whole-Song Multi-Track Accompaniment Arrangement
We finalize a complete music automation system by applying style prior modelling at two cascaded stages. As shown in Figure our autoencoder and orchestral function prior operate Stage 2 for piano to multi-track arrangement. On Stage we adopt our previous study, a piano texture prior [50]
Stage
Lead Sheet
Piano Texture Prior
External Control
Stage 2
Piano Arrangement
Orchestrul Function Prior
Piano Texture Style
Orchestral Fanction Style
A Complete Multi-Track
Multi-Track Arrangement
Accompaniment Arrangement System
Figure 3: A complete accompaniment arrangement system based on cascaded prior modelling. The first stage models piano texture style given lead sheet while the second stage models orchestral function style given piano. Besides modularity, the system offers control on both composition levels.
5

---

Chorus
56
FiA
Gn
PiA
B>
BvD
A..
Counterpoint Relation between Tracks
Vins.
8c.
0.1
Long-Term Coherence over Extended
Chorus
BvD
Harmonic/ Melodic between
Vias.
Metrical Division between String and Bruss
Figure 4: Arrangement for Can You Feel the Love Tonight, pop song in total of 60 bars. We show two chorus parts from bar 13 to 41. We use red dotted boxes to show coherence in long-term structure. We coloured blocks to show naturalness and cohesion in multi-track arrangement.
on top chord/texture representation learning [39], for lead sheet to piano arrangement. Given lead sheet, the first stage generates piano accompaniment, establishing the rough whole-song structure. Our system then orchestrates the piano accompaniment into complete multi-track arrangement with band instrumentation. This two-stage approach mirrors musicians' creative workflow [1] and allows for control at both composition levels. In particular, we provide three control options:
1. Texture Selection: To filter piano textures on Stage by metadata and statistical features.
2. Instrumentation: To customize the track number and choice of instruments on Stage 2.
3. Orchestral Prompt: To prompt the orchestration process with orchestral function template.
We showcase an arrangement example by the complete system in Figure 4. The system imput is a lead sheet shown by the Mel staff. The final output is the accompaniment in the rest staves. Notably, the lead sheet consists of 60 bars structure of 14A888B6x4A888900 (using notations by [5]). Here, i4, x4, and 04 each denote 4-bar intro, interlude, and outro. A8 and B8 represent an 8-bar verse and chorus, respectively. Figure 4 shows the arrangement result for the first and third choruses, spanning from bar 13 41. We leverage control option 2 to customize the instrumentation celesta, acoustic guitars (2), electric pianos (2), acoustic piano, violin, brass, and electric bass total of K tracks. The complete arrangement score included in Appendix
In Figure 4, we observe some multi-track arrangement patterns that are common in practice. Purple blocks highlight counterpoint relation between guitar track A.G.1 and electric piano track E.P.1.

---

Green blocks show two guitar tracks with complementary orchestral functions: one melodic (A.G.1) and the other harmonic (A.G.2). Yellow blocks illustrate the metrical division between the string (Vlns.) and the brass (Br.) sections, with strings on the downbeat and brass on the offbeat. These patterns demonstrate a natural and cohesive multi-track arrangement by our system. Moreover, we see consistent accompaniment patterns echoing in both chorus parts that span over 30 bars (shown by red dotted boxes), while the latter adds a piano arpeggio track (Pno.) to enhance the musical flow. This demonstrates structured whole-song arrangement over extended music contexts.
5 Experiment
In this section, we evaluate the performance of our multi-track accompaniment arrangement system. Given that existing methods primarily focus on lead sheet to multi-track arrangement, we ensure a fair comparison by using the two-stage approach discussed in Section 4. In Section 5.1, present the datasets used and the training details of our model. In Section 5.2, we describe the baseline models used for comparison. Our evaluation is divided into two parts: objective evaluation, detailed in Section 5.3, and subjective evaluation, covered in Section 5.4. For the single-stage piano to multi-track (Stage 2) and lead sheet to piano (Stage 1) arrangement tasks, we perform additional comparisons with various ablation architectures in Section 5.5 and 5.6, respectively.
5.1 Datasets and Training Details
We use two datasets to train the autoencoder and the style prior, respectively. The autoencoder is trained with Slahk2100 [25]. which contains 2K curated multi-track pieces with 34 instrument classes in a balanced distribution. We discard the drum track and clip each piece into 2-bar segments with I-bar hop size. We use the official training split and augment training samples by transposing to all 12 keys. The autoencoder comprises 19M learnable parameters and is trained with batch size 64 for 30 epochs on an RTX A5000 GPU with 24GB memory. We use Adam optimizer [19] with a learning rate from 1e-3 exponentially decayed to le-5. We use exponential moving average (EMA) [29] and random restart [7] to update the codebook with commitment ratio 3 0.25.
We use Lakh MIDI Dataset (LMD) [28] to train the prior model. It contains 170k music pieces and is a benchmark dataset for training music generative models. We collect 2/4 and 4/4 pieces (1 10k after processing) and randomly split LMD at song level into training (95%) and validation (5%) sets. We further clip each piece into 32-bar training excerpts (i.e., T 16 at maximum) with b-r hop size. Our prior model has 30M parameters and trained with batch size 16 for 10 epochs (600K iterations) on two RTX A5000 GPUs We apply AdamW optimizer [22] with learning rate of le-4, scheduled by a Ik-step linear warm-up followed by single cycle of cosine decay to a final rate of 1e-6.
For model inference and testing, we consider two additional datasets: Nottingham [10] and WikiMT [44]. Both datasets contain lead sheets (in ABC notation) that are not seen during training or validation. Moreover, they cover diverse music genres including folk, pop, and jazz. When arranging piece, we leverage control option 2 set up the instrumentation. Without loss of generality, this control choice is randomly sampled from Slakh2100 validation/test sets. To arrange music longer than the prior model's 's context length (32 bars), we use windowed sampling [7], where we move ahead our context window by 4 bars and continue sampling based on the previous 28 bars. We apply nucleus sampling [13] with top probability p = 0.05 and temperature t= 66
5.2 Baseline Models
We compare our system with three existing methods: PopMAG [30], Anticipatory Music Transformer (AMT) [33], and GETMusic [23]. PopMAG and GETMusic generate multi-track accompaniments from input lead sheet based on Transformer and diffusion model, respectively. AMT is Transformer- based and it continues the accompaniment part from an input melody with starting accompaniment prompt. We provide detailed configurations in the following.
PopMAG is an encoder-decoder architecture based on Transformer-Xi [6]. It represents multi-track music by sequential note-level tokenization and is fully supervised. The encoder takes a lead sheet as input and the decoder generates multi-track accompaniment ruto-regressively. Since the model not open source, We reproduce it on LMD with lead sheets extracted by [24] (melody) and [17] (chord).
7

---

Table 2: Objective evaluation results for lead sheet to multi-track arrangement (Section 5.3). All
entries are of the form mean + sem', where is letter. Different letters within a column indicate significant differences (p < 0.05/6) based on Wilcoxon signed rank test with Bonferroni correction.
Model
Chord Acc.
DOA
Structure t
Latency +
Ours
0.567 0.014"
0.300 + 0.004"
1.520 + 0.030"
0,461 + 0.005
AMT [33]
0.446 + 0.013
0.294 + 0.006
1.094 + 0.009
6.320 + 0.2124
GETMusic [23]
0.423 = 0.012
0.225 0.007
1.243 1 0.017
0.450 = 0.002
PopMAG [30]
0.470 + 0.013
0.270 + 0.007
1.086 + 0.008
0.638 + 0.013
Ground-Truie
0.333 + 0.009
1.980 + 0.019
Anticipatory Music Transformer (AMT) is a decoder-only Transformer architecture with note-level tokenization. It introduces an "anticipation" method, where conditional tokens (melody and starting prompt) and generative tokens (accompaniment continuation) are interleaved to train a conditional generative model. Since our testing dataset does not provide ground-truth accompaniment, the starting prompt is given by the generation result (first 2 bars) of our system. We use the official implementation of the AMT model, which is also trained on LMD.
GETMusic represents multi-track music as an image-like matrix resembling score arrangement, based on which denoising diffusion probabilistic model is trained with mask reconstruction objective. Given a lead sheet, it supports generating 5 accompaniment tracks using piano, guitar, string, bass, and drum or their subsets. In our experiment, we generate all 5 accompaniment tracks. We use the official implementation of the GETMusic model,' which is trained on internal data.
5.3 Objective Evaluation on Multi-Track Arrangement
We introduce four metrics to evaluate multi-track arrangement performance: chord accuracy [23, 30], degree of arrangement (DOA), structure awareness [42], and inference latency [30]. Among them, chord accuracy measures the multi-track harmony that reflects the fitness of the accompaniment the lead sheet; DOA measures inter-track tonal diversity that reflects the creativity of the instrumentation. Both metrics demonstrate music cohesion local scales. On the other hand, structure awareness measures phrase-level content similarity that reflects long-term structural coherence of the whole song. Finally, we use inference latency (in second/bar) each method. The detailed computation of each metric is provided in Appendix B. In Table 2. we compute ground-trute truth DOA using 1000 random pieces from LMD. We compute ground-truth structure awareness using 857 pieces in 4/4 from POP909 Dataset [38].
evaluate computational efficiency of
We randomly sample 50 pieces in 4/4 time signature from Nottingham and WikiMT respectively (100 pieces in total) to conduct experiment. The length of each piece ranges from 16 to 32 bars. We run our method and baseline models at each piece in independent rounds, deriving 300 sets of multi-track arrangement samples. In Table 2, we report the evaluation results with mean value, standard error of mean (sem), and statistical significance computed by Wilcoxon signed rank test [43]. We find significant differences between our method and all baselines (p-value p < 0.05/6, using Bonferroni correction). In particular, our method outperforms in chord accuracy, structure awareness, and DOA, indicating the capability of arranging harmonious, structured, and creative accompaniments. The diffusion baseline outperforms in inference latency as applies only 100 diffusion steps. Our method's efficiency is on par with it, while being 10 times faster than vanilla note-level auto-regression.
5.4 Subjective Evaluation on Multi-Track Arrangement
We also conduct double-blind online survey to test music quality. Our survey consists of 5 evaluation sets, each containing an input lead sheet followed by 4 arrangement samples by our method and each baseline. Each sample is 24-32 bars long and is synthesized to audio 90 BPM (-1 minute per sample). Both the set order and the sample order in each set are randomized. We request participants to listen 2 sets and evaluate the musical quality based on a 5-point Likert scale from I to 5.
8

---

Table Objective evaluation results for piano to multi-traae track arrangement (Section 5.5). All entries are of the form mean + sem", where is a letter. Different letters within a column indicate significant differences (p < 0.05/6) based on Wilcoxon signed rank test with Bonferroni correction.
Prior
Faithfulness (stats.)
Faithfulness (latent)
DOA
NLL +
Ours
0.945 0.001
0.215 + 0.005
0.308 0.005
0.411 0.004
Parallel
0.937 + 0.002
0.153 + 0.003
0.233 + 0.005
0.960 + 0.010
Delay
0.915 + 0.004
0.133 + 0.003
0.207 + 0.005
1.024 + 0.006
Random
0.913 + 0.003
0.113 + 0.003d
0.262 + 0.005
Ours
AMT
GETMusic
PopMAG
Ours
Parallel
Delay
Random
4.0
4.0
3.0
3.0
2.0
2.0-
1.0
1.0
Coherency Structure Naturalnees NaturalneesCrraiivvii Musicality
Instrumentation Structure
Creativity
Musicality
Figure 5: Subjective evaluation results on lead sheet to multi-track arrangement (Section 5.4).
Figure 6: Subjective evaluation results on piano to multi-track arrangement (Section 5.5).
The evaluation based on criteria: 1) Harmony and Texture Coherency, 2) Long- -Tem Structure, 3) Naturalness, 4) Creativity, and 5) Overall Musicality.
A total of 23 participants (8 female and 15 male) with diverse musical backgrounds have completed our survey, with an average completion time of 22 minutes. The mean ratings and standard errors, computed by within-subject (repeated-measuree) ANOVA [31], are presented in Figure 5. Significant differences are observed across all criteria (p-value p < 0.05). Notably, our method outperforms all baselines on each criterion, aligning with the results from the objective evaluation.
5.5 Ablation Study on Style Prior Architecture
We now validate our design with the prior model by exclusively evaluating on the piano to multi-track arrangement task. Our prior model based on unique layer interleaving design, which enables multi-stream time series modelling with explicit stream-wise attention. We compare it with two other multi-stream architectures: 1) Purallel: summing up parallel code embeddings for joint language modelling [18], and 2) Delay: leveraging 1-step delay code interleaving catch implicit stream- wise dependency [4]. Both Parallel and Delay are trained under the same setup as our model. We additionally introduce 3) Random; naive prior based on random template retrieval. The templates are sampled every? 2 bars with shared instrumentation from the validation/test sets of Slakh2100.
We introduce two metrics to evaluate piano to multi-track arrangement: faithfulness and degree of arrangement (DOA). Faithfulness measures if the generated arrangement faithfully reflects the original content from the piano. It computes the similarity between i) the input piano, and ii) the piano reduction of the generated multi-track arrangement. our case, we compute cosine similarity over two features: a statistical (stafs.) pitch class histogram [45] and a lateni texture representation [39], which emphasize tonal and thyth similarity, respectively. DOA measures the creativity as defined in Section 5.3. We also report the NLL loss for model, Parallel, and Delay
We conduct experiments using the test set of POP909 [39], which consists of 88 piano arrangement pieces. In our experiment, we use the first section of each piece, which contains 2 to 4 complete phrases totally spanning 24 to 32 bars. We use control option
prompt our model, Parallel, and Delay with the same 2-bar orchestral function template (sampled from Slakh2 100) and see how it developed. We report mean value, standard error of mean (sem), and statistical significance in Table 3 and find significant differences in both faithfulness and DOA. We also conduct a subjective evaluation the same setup as Section 5.4, with the results presented Figure 6. Here we consider an additional criterion, Instrumentation, which reflects the well fell-formedsess of the multi-

---

Table Ablation study on alternative lead sheet to piano arrangement (i... Stage 1) modules. Here we investigate the impact of Stage I on the entire two-stage system. Evaluation results are based on the final multi-track arrangement using respective Stage modules.
Two-Stage System
Chord Acc. t
Structure
DOA
Ours (Stage + Stage 2) Whole-Song- Gen [42] + Stage 2
0.567 + 0.014 0.509 + 0.015
1.520 = 0.030
0.300 + 0.004
1.121 = 0.006
0.277 + 0.006
Table 5: Evaluation results for lead sheet piano arrangement exclusively Stage 1.
Piano Arrangement Module
Chord Acc. 1
Structure t
Ours (Piano Texture Prior)
0.540 + 0.016 0.430 + 0.020
1.983 = 0.147
Whole-Song-Gee [42]
1.153 + 0.180
track arrangement. Significant differences are observed across all criteria (p-value p 0.05), Overall, Parallel and Delay both fall short in performance because they assume preset stream combination, while in our setting, both track numbers and choices of instruments are flexible. By explicitly modelling stream-wise attention, our layer interleaving design fits well to that generalized scenario.
5.6 Ablation Study on Piano Arrangement
Now we validate our choice for the lead sheet to piano arrangement module on the first stage of our two-stage system. Our choice is piano texture prior as covered in Section 4. We conduct an ablation study by replacing it with the Whole-Song Son---onnGeeee -en model [42], which, to our knowledge, is the only existing alternative that can handle whole-song structure. The ablation study is conducted in the same setup as Scction 5.3. In Table 4, we report chord accuracy, structure awareness, and DOA regarding the final multi-track arrangement results. We further compare our piano texture prior with Sorole-SongGGerr exclusively on the piano accompaniment arrangement task. In Table 5, we report chord accuracy and structure awareness regarding piano arrangement for both models. Significant differences (p-value < 0.05) are found in all metrics based on Wilcoxon signed rank test.
By comparing Table 4 and Table 5, we can see that higher-quality piano arrangement generally encourages more musical and creative final multi-track arrangement result. Specifically, the piano arrangement on Stage 1 lays the groundwork for (at least) chord progression and phrase structure for Stage 2, both of which are important for capturing the long-term structure in whole-song multi-track arrangement. Morcover,
see that our piano texture prior outperforms existing alternatives and guarantees a decent piano quality, thus being the best choice for our system.
Conclusion
To sum up, contribute a music automation system for multi-track accompaniment arrangement. The main novelty lies in our proposed style prior modelling. a generic methodology for structured sequence generation with fine-grained control. By modelling the prior of disentangled style factors given content, we build a cascaded arrangement process: from lead sheet to piano texture style, and then from piano orchestral function functioon style. Our system first generates piano accompaniment from
lead sheet, establishing the rough whole-song structure. then orchestrates the piano accompaniment into a complete multi-track arrangement with band instrumentation. Extensive experiments show that our system generates structured, creative, and natural multi-track arrangements with state-of-the-art quality. At higher level, we elaborate our methodology as interpretable modular representation learning, which leverages finely disentangled and manipulable music representations to tackle complex tasks with a compositional hierarchy. We hope our research brings new perspectives to broader domains of music creation, sequence data modelling, and representation learning.
References
[1] Samuel Adler and Peter Hesterman. The study of orchestration, volume 2. WW Norton New York, NY, 1989.
10

---

[2] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi. et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.1 1325, 2023.
[3] Caroline Chan, Shiry Ginosar. Tinghui Zhou, and Alexei Efros. Everybody dance now, In 2019 IEEF/CVF International Conference on Computer Vision, pages 5932-5941, 2019.
[4] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. In Advances in neural information processing systems, volume 36, 2023.
[5] Shuqi Dai, Huan Zhang, and Roger B Dannenberg. Automatic analysis and influence of hierarchical structure on melody, rhythm and harmony in popular music. arXiv preprint 1r00::2000.075/8 2020.
[6] Zihang Dai, Zhilin Yang. Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhut- dinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 2978-2988.
2019.
[7] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00344, 2020.
[8] Hao-Wen Dong, Chris Donahue, Taylor Berg- Kink-aiika, and Julian McAuley. Towards automatic instrumentation by learning to separate parts in symbolic multitrack music In Proceedings of the 22nd International Society for Music Information Retrieval Conference, pages 159-166, 2021.
[9] Hao- Wen Dong. Ke Chen, Shlomo Dubnov, Julian McAuley, and Taylor Multitrack music transformer. In International Conference on Acousties, Speech and Signal Processing, pages 1-5. IEEE, 2023.
[10] Eric Foxley. Nottingham database. [EB/OL], 2011. nottingghmmnottiiggann htm. Accessed May 17, 2023.
https://ifdo.. cat.sddoor////syouurr
[11] Curtis Hawthome, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse H. Engel, and Douglas Eck. Enabling factorized piano music modeling and generation with the MAESTRO dataset. In 7th International Conference Learning Representations, 2019.
[12] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 160i 08415, 2016.
[13] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, 2020.
[14] Wen-Yi Hsiao, Jen-Ye Yu Liu, Yin-Cheng Yeh, and Yi-Hsuan Yang. Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 178-186, 2021.
[15] Cheng-Zhi -hhn Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. 7th International Conference on Learning Representations, 2019.
[16] Rongjie Huang, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech. In Advances in neural information processing systems, volume 35, 2022.
[17] Junyan Jiang, Ke Chen, Wei Li, and Gus Xia. Large- chord transcription via chord structure decomposition. In Proceedings of the 20th International Society for Music Information Retrieval Conference, pages 644-651, 2019.

---

[18] Eugene Kharitonov, Ann Lee, Adam Polyak. Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. Text-free prosody-aware generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 8666-8681,22 2022.
[19] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[20] Dmytro Kotovenko, Artsiom Sanakoyeu, Sabine Lang, and Bjöm Ommer. Content and style disentanglement for artistic style transfer. In 2019 IEEE/CVF International Conference on Computer Vision, pages 4421 4421-4430. IEEE, 2019.
[21] Liwei Lin, Gus Xia, Qiuqiang Kong, and Junyan Jiang. A unified model for zero-shot music source separation, transcription and synthesis. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, pages 38 3818388, 2021.
[22] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 6th International Conference on Learning Representations, 2018.
[23] Ang Lv, Xu Tan, Peiling Lu, Wei Ye, Shikun Zhang, Jiang Bian, and Rui Yan. Getmusic: Generating any music tracks with a unified representation and diffusion framework. arXiv preprint arXiv:2305. 1084, 2023.
[24] Xichu Ma, Xiao Liu, Bowen Zhang, and Ye Wang. Robust melody track identification in symbolic music. In Proceedings of the 23rd International Society for Music Information Retrieval Conference, pages 342-849, 2022.
[25] Ethan Manilow, Gordon Wichern, Prem Seetharaman, and Jonathan Le Roux. Cutting music source separation some slakh: A dataset to study the impact of training data quality and quantity. In 2019 IEEE Workshop Applications of Signal Processing Audio and Acoustics, pages 45-49, 2019.
[26] Lejun Min, Junyan Jiang, Gus Xia, and Jingwei Zhao. Polyffusion: diffusion model for polyphonic score generation with internal and external controls. In Proceedings of the 24th International Society for Music Information Retrieval Conference, pages 231-238, 2023.
[27] Matthias Plasser, Silvan Peter, and Gerhard Widmer. Discrete diffusion probabilistic models for symbolic music generation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. pages 5842-5850, 2023.
[28] Colin Raffel. Learning-Bate Based Methods for Comparing Sequences, with Applications to Audio- to-MIDI Alignment and Matching. PhD thesis, Columbia University, USA, 2016.
[29] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with 2q-vae-2. In Advances in neural information processing systems, volume 32, 2019.
[30] Yi Ren, Jinzheng He, Xu Tan, Tao Qin, Zhou Zhao, and Tie-Yan Liu. Popmag: Pop music accompaniment generation. In Proceedings of the 28th ACM International Conference on 2020.
Multimedia, pages
[31] Henry Scheffe. The analysis of variance, volume 72. John Wiley & Sons, 1999.
[32] Hao Hao Tan and Dorien Herremans. Music fadernets: Controllable music generation based on high-level features via low-level feature modelling. In Proceedings of the 2 International Society for Music Information Retrieval Conference, pages 109-116, 2020.
[33] John Thickstun, David Leo Wright Hall, Chris Donahue, and Percy Liang. Anticipatory music transformer. Transactions on Machine Learning Research, 2024.
[34] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in neural information processing systems, volume 30, 2017.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, volume 30, 2017.
12

---

[36] Dimitri von Rütte, Luca Biggio. Yannic Kilcher, and Thomas Hoffman. Figaro: Generating symbolic music with fine-gree seee-grried artistic control. In I 1tth International Conference on Learning Representations, 2023.
[37] Ziyu Wang and Gus Xia. Musebert: Pre-training music representation for music understanding and controllable generation. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, pages 722-729, 2021.
[38] Ziyu Wang, Ke Chen, Junyan Jiang, Yiyi Zhang, Maoran Xu, Shuqi Dai, and Gus Xia. POP909: A pop-song dataset for music arrangement generation. In Proceedings of the 21st International Society for Music Information Retrieval Conference, pages 38-45, 2020.
[39] Ziyu Wang, Dingsu Wang, Yixiao Zhang, and Gus Xia. Learning interpretable representation for controllable polyphonic music generation. In Proceedings of the 2 International Society for Music Information Retrieval Conference, pages 662-669, 2020.
[40] Ziyu Wang. Yiyi Zhang. Yixiao Zhang, Junyan Jiang. Ruihan Yang. Gus Xia, and Junbo Zhao. PIANOTREE VAE: Structured representation learning for polyphonic music. In Proceedings of the 21st International Society for Music Information Retrieval Conference, pages 368-375, 2020.
[41] Ziyu Wang, Dejing Xu, Gus Xia, and Ying Shan. Audio-to-symbolic arrangement via cross- modal music representation learning. In International Conference on Acoustics, Speech and Signal Processing, pages 181-185. IEEE, 2022.
[42] Ziyu Wang, Lejun Min, and Gus Xia. Whole-song hierarchical generation of symbolic music using cascaded diffusion models. In 12th International Conference on Learning Representations, 2024.
[43] Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics: Methodology and distribution, pages 196-202. Springer, 1992.
[44] Shangda Wu, Dingyao Yu, Xu Tan, and Maosong Sun. Clamp: Contrastive language- music pre-training for cross-modal symbolic music information retrieval. In Proceedings of the 24th International Society for Music Information Retrieval Conference, pages 157-165, 2023.
[45] Shih-Lun Wu and Yi-Hsuan Yang. The jazz transformer on the front line: Exploring the shortcomings of ai-composed music through quantitative measures. In Proceedings of the s International Society for Music Information Retrieval Conference, pages 142-149, 2020.
[46] Ruihan Yang. Dingsu Wang. Ziyu Wang, Tianyao Chen, Junyan Jiang, and Gus Xia. Deep music analogy via latent representation disentanglement. In Proceedings of the 20th International Society for Music Information Retrieval Conference, pages 996-603, 2019.
[47] Li Yi, Haochen Hu, Jingwei Zhao, and Gus Xia. Accomontage2: A complete harmonization and accompaniment arrangement system. In Proceedings of the 23rd International Society for Music Information Retrieval Conference, pages 248-255, 2022.
[48] Wenjie Yin, Hang Yin, Kim Baraka, Danica Kragic, and Märten Björkman. Dance style transfer with cross-modal transformer. In IEEE/CVE Winter Conference Applications of Computer Vision, pages 5047-5056, 2023.
[49] Siyang Yuan, Pengyu Cheng, Ruiyi Zhang, Weituo Hao, Zhe Gan, and Lawrence Carin. Improv- ing zero-shot voice style transfer via disentangled representation learning. In 9th International Conference on Learning Representations, 2021.
[50] Jingwei Zhao and Gus Xia. Accomontage: Accompaniment arrangement via phrase selection and style transfer. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, pages 833-840, 2021.
[51] Jingwei Zhao, Gus Xia, and Ye Wang. Q&A: Query-based representation learning for multi- track symbolic music re- re-arrangemen. In Proceedings of the Thirty- International Joint Conference Artificial Intelligence, pages 5878-5886 2023.
13

---

A Implementation Details
A.1 Autoencoder
The autoencoder consists of a VQ-VAE submodule and an overarching VAE. The encoder of the VQ- VAE consists of a I-D convolutional layer of kernel size 4, stride 4, and output channels, followed by vector quantization block with codebook size 128. The decoder takes the concatenated latent codes and leverages two fully-connected layers (shape 128 x 256 and 256 x 32) for reconstruction. In the overarching VAE, Piano Encoder and Track Decoder are adapted from Piano Tree VAE [40]. The encoder first applies a pitch-wise bi-directional GRU to summarize concurrent notes at time step n and then applies time-wise GRU to encode the full representation. The decoder mirrors the encoder structure with time- and pitch-wise uni miidirrctional GRUs to reconstruct individual tracks. We use hidden size 256 in single layer for pitch GRUs and 512 for time GRUs. The Track Separator is 2-layer Transformer encoder with attention heads, 0.1 dropout ratio, and GELU activation [12]. The hidden dimensions of self-attention dmodel and feed-forward layers dg are 512 and 1024, respectively.
The autoencoder is trained with joint reconstruction loss for orchestral function (MSE) and individual tracks (cross entropy). The VQ-VAE is additionally regularized with latent loss and commitment loss with commitment ratio 3 = 0.25. The VAE is regularized with KL loss over all continuous factors (c, and z. : based on KL annealing [41] with ratio exponentially increasing from 0 to 0.5.
A.2 Prior Model
The prior model consists of a 12-layer Context Encoder and a 12-layer Auto- Regressgeee Decoder. The latter is interleaved with another 12 track-wise Track Encoder layers. For each layer, apply 8 attention heads, 0.1 dropout ratio, and GELU activation. We apply layer normalization before self-attention (i.e., norm first). The hidden dimensions of self-attention dmoddl and feed-forward layers du are 256 and 1024, respectively. We apply relative positional embedding [15] to Track Encoder so that two tracks initialized with identical instruments can still generate different content.
Our prior model is trained on the latent codes C1.7 and gl:K inferred by a well-trained autoencoder on LMD. For discrete code s, we take the codebook indices and learn embedding.
B Objective Evaluation Metrics
B.1 Degree of Arrangement
In multi-track arrangement, parallel tracks typically play a unique role to each other in the overall arrangement. We are interested in capturing the diversity and creativity inherent in these roles.
To achieve this, we consider the pitch class histogram [45] as a probability distribution P. Let Pt,k be the distribution of the t-th bar in track k, and pmn be that of the t-th bar in the piano reduction. Recall that in this paper we approximate the piano reduction of a multi-track piece by downmixing all tracks. Both Pr, and ppd are 12-D vectors, describing tonality of individual tracks and the overall arrangement, respectively. We compute the KI. divergence of each track to the piano reduction:
T
KL P!
(5)
T
where 7 is the total number of bars.
Interpreting de in terms of KL divergence, see it the "excess surprise" from the overall arrangement (piano reduction) when track & is played in isolation. A large de indicates that track possesses unique quality, such as bass track playing the root counter-melody track focusing on tensions. Conversely, a small de suggests that track & serves as a foundational element in the arrangement, such as string padding that establishes the harmonic foundation.
If all de values are small, it implies homogeneity across tracks and thus a low degree of arrangement. Conversely, if all dy values are high, it suggests composition dominated by counterpoints, scenario less common pop music. A well-orchestrated piece typically exhibits diverse range of d, values, encompassing both foundational and unique decorative tracks. We thus define the degree of
14

---

arrangement DOA as the standard deviation of d2 for l = 1,2,- ............ K across all tracks:
Σ == (dk 2)²
DOA
(6)
K
where d is the mean. K is the total number of tracks. To establish reference point, we calculate the ground-truth DOA = 0.333 based on 1000 random pieces from the LMD dataset with at least tracks. Within this context, higher DOA signifies creative arrangement.
B.2 Strcture Awareness
We introduce Inter- aher-p Latent Similarity (ILS) from [42] to measure the structural awareness of long-term arrangement. ILS calculates the content similarity among same-type phrases (e.g., chorus) versus the whole song. It leverages pre-trained disentangled VAEs that encode music notes into latent representations and then compare cosine similarities in the latent space. In our case, we compute ILS over the piano reduction of generated arrangement since contains the overall content. We apply the texture VAE [39] and obtain a latent texture representation ct for every 2-bar segment. For odd numbeeede phrases, we repeat its final bar and pad it to the end of the phrase. Suppose there M different types of phrases in piece and let Lm be the set of segment indices in the type-m phrase, ILS is defined as the ratio between same-type phrase similarity and global average similarity:
(EME) Eiprer cos( c .C) c.xt)) ()/(E M
M
I'm
(mm))
ILS
(7)
EiSTAJST cos(ce .. //T2 7))
where | is the cardinality of set. T is the number of 2-bar segments. When applying ILS, we use [5] to automatically label the phrase structure of a piece. To establish a reference point, we calculate the ground-truth ILS 1.980 based the POP909 dataset (with phrase annotation by human). Within this context, a higher ILS signifies saliency with long-term phrase-level structure.
B.3 Chord Accuracy
We introduce chord accuracy from [30] to measure if the chords of the generated arrangement match the conditional chord sequence in the lead sheet. reflects the harmonicity of the generated music and is defined as follows:
Nehond
CA
E (c)=
Nehord
where Nehora is the number of chords in piece; C₁ is the i-th chord in the (ground-truth) lead sheet; and Ö, is the aligned chord in the generated arrangement.
The original formulation in [30] considers chord accuracy for individual tracks. Given our system's capability to accommodate variable combination of tracks, we opt for a broader evaluation for the overall arrangement. In our ease, we extract the chord sequence of generated arrangement with [17] and compare it in root and quality with ground-truth at 1-beat granularity, which is more rigorous.
B.4 Orchestration Faithfulness
We measure the faithfulness of orchestration by the similarity between i) the input piano and ii) the piano reduction of the generated multi-track arrangement. Let e!" and e,"t be vector features derived from the t-th segment of the input and the reduction, respectively. Orchestration faithfulness OF is defined as follows:
1 T Σ cos(e," ef"), OF
(9)
where cos(..)) is cosine similarity. T is the number of segments.
In our work, we select two options for vector feature e. One a statistical pitch class histogram [45], which is 12-D vector describing pitch class distribution. The other is latent 256-D texture representation learned by a pre-trained VAE [39]. Both features are general descriptors of the musical content with respective focus tonal harmony and rhythmic grooves.
15

---

C Experiment on Noise Weight Y
Continuing from Section 3.3, we compare different Y values and see their impact on the model performance. When applying our model to piano to mulli--trck arrangement, 7 balances the force a noisy factor added to the piano, which encourages a partial unconditional generation capability, The experimental settings are the same Section 5.5. We evaluate the results based on faithfulness and DOA. In Table 6, we report mean value, standard error mean (sem), and statistical significance computed by Wilcoxon signed rank test. By varying the value, observe controllable balance between faithfulness and creativity. Specifically, a larger ? encourages creativity (higher DOA) at the cost of faithfulness. If not mentioned otherwise,
0.25 for experiments in this paper.
Table 6: Objective evaluation results on the impact of noise weight in Appendix C.
Noise Weight y
Faithfulness (stats.) t Faithfulness (latent) f
DOA t
7
0.946 + 0.001
0.228 0.005*
0.300 0.005
7 0.25
0.945 + 0.001
0.215 + 0.005*
0.308 = 0. 0.005 he:
0,5
0.944 = 0.001
0.187 + 0.004
0.320 = 0. 0.006
= 1
0.936 + 0.002
0.127 = 0.003
0.325 + 0.007
D Online Survey Specifies
We distribute our survey via SurveyMonkey.' Our survey consists of 5 sample sets for both the lead sheet to multi-track and the piano to multi-track arrangement tasks (10 sets in total). Each sample is 24-32 bars long and is synthesized audio 90 BPM using BandLab' with the default soundfont. Each participant listens to sets (in random order) and the mean time spent is 22 minutes. Figure 7 shows the sample pages of our survey with instructions the participants.
Ascrppatie Arsaqueeat deternake
Ban saage Dr
ale gaigga araga skeetin dchaan manernaic "sad skeple
pringho apore wnk pire pare le
votherionns ettmll o0
will fur amangemett oes reode cututt variing with sume nato anerarter dientty moiidyy wabl mpement Marcoallly rveel mosiafe
prean' fur assagenaa sanp suquin
desccus The orant Boy musk dedioomen ihh bri
moad beiiwiing erangpeees
Craaielly Hour wnler mesiiss
st New world ene beleving imeegmment?
(a) Lead sheet to multi-track arrangement.
(b) Piano to multi-track arrangement.
Figure 7: Screenshots of survey pages and instructions of our online survey.
'https: h//w:: www.waa.dbbb... com/
16

---

E Example on Structured Arrangement
We demonstrate an example of accompaniment arrangement by our proposed system. The input lead sheet is complete pop song shown in Section E. Our system first arranges a piano accompaniment for the whole song, which is shown in Section E.2. The piano score then orchestrated into multi-track arrangement with customized instrumentation, which is shown in Section E.3.
E.1
Lead Sheet
We use our system to arrange for Can You Feel the Love Tonight, a pop song by Elton John. As shown in Figure 8, the entire song is 60 bars long and it presents structure of 1448B9BBx4A8B8B004 where i, O, A, and each refer to intro, interlude, outro, verse, and chorus.
65
F/A
B&GG
Bb/F
By/D
F/A
Cm7
Bb/D
A8
Eb
Bb/D
Eb
Bb/D
Eb
Bb/D
Cm7
F/A
Eb
Bb/D
Eb
Bb/D
Eb
Gm
Ab
F
B8
13
Bb
F/A
Gm
Eb
Bb
E>
C7/R
17
Eb
Bb/D
Gm
E
Cm7
Bb/D
Cm/Eb
C/E
B8|
21
1bb
F/A
Gm
Eb
Bb
E6b
C7/E
F
0
25
Eb
Bb/D
Gm
Cm
Bb/D
Cm/Eb F7sus
Eb/Bb Bb
x4
29
1..bb
F/A
Eb/G
Bb/F
Eb
Bb/D
F/A
Cm7 Bb/D
04
33
12.gb
Bb/D
m
Eb
Cm
Bb/D
Cm/Eb F7sus
Eb/Bb Bb
Figure 8: Lead sheet for pop song Can You Feel the Love Tonight.
17

---

E.2 Piano Arrangement
The piano arrangement result is shown from Figure 9 to Figure 10. It roughly establishes a whole-song structure and lays the groundwork for band orchestration the next stage. Demo audio for the piano score is available at https : /t/zzzaaaww999 .it.eb............. www.sstttuedrrrrnneeeeeeee
CeT Dhp
Plano
A8
10
364
Phn
1
9
B8
Phn
17
Pno.
B8
204
25
Figure 9: Piano arrangement score (page 1).
18

---

x4
300
1/9
1VD
FA
R10
300
1
A8
93
300
001
15
364
Pon
Pnn
B8
45
144
B8
Puo.
Pnn
04
Pnn
Figure 10: Piano arrangement score (page 2, last page).
19

---

E.3 Multi-Track Arrangement
The multi-track arrangement shown from Figure to Figure 13. We customize the instrumentation as celesta, acoustic guitars (2), electric pianos (2), acoustic piano, violin, brass, and electric bass in a to- tal of K = tracks. We can see that the structure the accompaniment follows the lead sheet. Demo audio is available at https: htt////haowww// tailed analysis on this arrangement demo is covered in Section
github. io/ Lo/structured. arrangement/ More de-
Maledy
Celesta
Acoustic Guitar
Acoustic Guitar 2
Piano
Electric Piano
Electric Piano
Violins
Brass
Electric Bass
A8
A.G.
E.P.
Vins
Br.
R.B.
Figure 11: Multi-track arrangement score (page 1).
20

---

160
N61
A.G.
Pno.
E.P.
Vlns.
B.B.
B8
A.G.
A.G.2
E.P.
Br.
A.G.
A.G.2
E.P.
Vlns.
E.B.
Figure 12: Multi-track arrangement score (page 2).
21

---

B8
FA
A.G.1
A.G. 2
Vins.
Br.
E.B.
0
16
000
Md
A.G.
A.G.2
Vins
Br.
29 X4
Bkm
A.G. 1
A.G.2 2
Poo.
E.P. 2
R.B.
A8
K0
M6
Poo.
E.B.
Figure 13: Multi-track arrangement score (page 3).
22

---

37
400
160
Na1
Cel.
Pno.
E.B.
B8
41
A.G.
A.G.
Рno.
Vina.
E.B.
160
161
A.G.
A.G. 2
Pno.
Vlns.
Br.
E.B.
Figure 14: Multi-track arrangement score (page 4).
23

---

B8
7W
36
A.G.
A.G.2
E.B.
A.G.
A.G.2
E.B. 2
57
A.G.
P
E.P.1
E.P.2
Tix Fiiyfo
Vina
E.D.
Figure 15: Multi-track arrangement score (page last page).
24

---

Limitation
We propose two-stage system for whole-song, multi-track accompaniment arrangement. In the context of this paper, we acknowledge that our current system exclusively supports tonal tracks in quadruple meters while disregarding triple meters, triplet notes, and drums. However, we perceive this as a technical limitation rather than scientific challenge. We also acknowledge that our current system primarily emphasizes the composition level, thereby omitting the modelling of MIDI velocity, dynamic timing, and MIDI control messages. Consequently, the generated results do not encompass performance MIDI and may lack expressive qualities. Nevertheless, we believe that our composition-centrii work serves solid and vital foundation for further advancements in those specific areas, thus facilitating the development of enhanced techniques and features. As pioncering work, system is the foremost accomplishment in solving whole-song multi-track accompaniment arrangement, characterized by flexible controllability on track number and choice instruments.
G Broader Impacts
Our multi-track accompaniment arrangement system, which incorporates style generate accom- paniment, is designed to enhance originality and creativity. It serves as a platform for human-Al AI co-creation, where the user provides content-based material (in our case, lead sheet) that remains fundamentally original, while the Al agent infuses style, enriches the form, and enhances creativity. Our system therefore empowers musicians to explore new musical ideas and expand their creative boundaries. This approach also allows for rapid mock-up with different styles and arrangements, fostering an environment where innovation and artistic expression can thrive.
However, we acknowledge the need to address potential risks. The accessibility of our system may inadvertently lead excessive reliance automation, potentially impeding the development of fundamental skills among musicians. Additionally, widespread adoption the system may contribute the homogenization of music, threatening the distinctiveness and individuality that crucial to artistic expression. We recognize that our datasets predominantly feature contemporary Western music, which introduces cultural bias that could limit the diversity of generated compositions.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims in the abstract and introduction accurately reflect paper's contributions and scope with support from experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Appendix F discusses the limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We introduce our experimental settings thoroughly in Section 5. We also provide details of model architectures necessary for reproduction in Appendix A.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification: All datasets used in this work are open- We release our code and more resources at httpp://gitae /ttp//ggtttt..../// wwtp///aaoo11///////rttaoooo Stw.uuuuettrrrrrreeeeeeoe aararaaarrrruuuuudrrraraaeeeeeee Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. "nips.cc/ppppp cccppblic/guu getggguiicccsssai wwteesummisiioPpooo ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Training and testing details are covered in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For objective evaluation, report mean, standard error of mean, and statistical significance computed by Wilcoxon signed rank test [43], which does not assume normal- ity. For subjective evaluation, we report mean, standard error, and statistical significance computed by within-subject ANOVA [31], where normality of errors verified.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Details of computational resources are covered in Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurlPS Code of Ethics attpp::/neuurji: http:///e/eerpp..ccppiieeeeee wttpccccp/ppttteeeeeeeeeee hwwblli//Ehiicchidellinneee
didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: The authors have reviewed and conformed in every respect with the NeurIPS
Code of Ethics https: htttp:://nneripptt httpp//c/cc///ttttssssieeee.eee.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss both positive and negative impacts of our work in Appendix G.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: All datasets used in this paper open-source datasets with minimum risk of misuse, and so are the models trained on them.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use two datasets for model training: Lakh MIDI [28] and Slakh2100 [25]. both licensed under CC BY 4.0. We use three datasets for model testing: WikiMT [44] and POP909 [38], which are under the MIT License, and Nottingham [10], for which no license information is available. We use the official code of two baseline models: AMT [33], licensed under Apache-2.0, and GETMusic [23], which is under the MIT License.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
30

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcom thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release code, checkpoint, and notebook tutorial of this work under the MIT Arr License at ht /tt//ggtttuu..../ httpa//......aaooog///////oooooa Arrangement- Code.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We provide crowdsourcing details (including screenshots) in Appendix D.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
The human study in our experiment is based on online crowdsourcing, which bears minimum risk. Participants are informed that participation in our study is entirely voluntary and that they may choose to stop participating at any time without any negative consequences. No personally identifying information collected in the human study.
Justification:
Guidelines:
31

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "fEdu61y2Fa", "reviewer": "Reviewer_iTts", "summary": "The authors introduce a new model for generating multi-track accompaniment given a lead sheet. While being strictly a two-stage method (first generate a piano arrangement from the lead sheet and *then* generate the accompaniment), the authors' contribution is focused on the 2nd stage, and they rely on existing modules for the 1st stage. Their proposed method features two key components: **first**, a VQ-VAE submodule whose aim is to learn (quantized) representations of \"orchestral functions\" (orchestral functions being 1-D time-series representations of a track, with each element representing the sum of notes active in each segment, more on that in weaknesses), and, **second**, a transformer-based \"mixing\" VAE which combines the representations of the VQ-VAE (i.e., the representations of the orchestral functions) with learnt representations of \"piano reductions\" of each song (derived by averaging all individual tracks) to generate latent representations which are used to synthesize the full song. A key novelty of their model is the \"mixing\" part of the VAE, with the decoder utilizing interleaved time-wise and track-wise cross-attention layers -- meaning, that each output \"sees\" information both from past frames in its own track and from other tracks. Their system compares favorably to 3 standard baselines taken as-is from prior work or reproduced.", "questions": "There is only one question, regarding the interleaving of layers, but since it is also a weakness, I have included it there instead.", "limitations": "The authors have adequately discussed their limitations.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Overall, barring some presentation issues discussed below, this is a well-written paper that features state-of-the-art results and enough novelty to make it relevant for both the broader generative AI community (given the difficulty of generating long-form, complex music which the authors tackle with hierarchical modelling) and the more niche music-generation community. This is why I am leaning positive towards accepting the paper. I will mention some more of its strengths in detail: + The authors target a challenging task which is to generate a multi-track arrangement given a lead sheet. Even though they \"only\" focus on the 2nd stage of that pipeline, namely the generation of the multi-track arrangement given a piano arrangement, it is still a valid contribution. + The idea to disentangle the piano reduction and orchestral function and combine them in the way they do is interesting and novel. This also provides a nice element of controllability. + The fact that the model can be trained in \"self-supervised\" fashion (i.e. without manual annotations) is commendable. + The authors have done a thorough job of documenting experimental parameters (data, GPU runtime, etc.) and submit code to reproduce their experiments. + The authors have accounted for different sources of variability and provided multiple \"knobs\" to twist (and counteract domain mismatches) through the clever use of multiple \"positional\" embeddings (Fig. 2).", "weaknesses": "The main weaknesses of the paper are primarily in terms of presentation, rather than methodology or evaluation. My comments are mainly meant to help the authors improve their presentation. \n\n* Clarity of music concepts: Sometimes, the authors introduce or reference concepts which are not straightforward without domain knowledge. Examples of that are: * The orchestral function of x is introduced as a column sum of activation indicators over the different columns of X. Given that the columns of X represent MIDI pitches, this would mean that at each frame, the authors are summing the notes that are active. Yet, in p.3, l.115, they write that this indicator function \"counts each note onset position as 1\". Why onset only? This would mean that each note is counted once. But the equation states $x_t^k>0$, which appears to be evaluated for each frame (thus, not only at the onset of each note, but for its entire duration). Further, they state that \"it [orchestral function] describes the rhythm and groove of each track\". It is not at all clear to me why this orchestral function should describe the rhythm? It only appears to describe the amount of notes active at a given time. It's even less clear why it should describe the groove, given the subjective nature of \"groove\". It is okay that the authors attempt to give a layman's explanation to their method in this part, but it is also recommended that they better describe those concepts (and do so in a way that would be acceptable even to those that are not familiar with concepts from music theory).\n\n * There are other terms, such as \"counterpoint\", \"phrase\", etc. which will not be evident in an audience not familiar with them. While it is understandable that in a music generation paper, there will be a lot of domain language, it would be useful if the authors commented on their importance in their discussion of results. For example, the ILS metric measures the similarity within a music phrase vs the rest of the song. What does it mean that the authors' method does better than the baselines in this metric? That it produces more coherent phrases that are more easily distinguishable from the rest of the song? Adding a sentence to clarify the importance of each result (in, as far as this is possible, layman's terms) would improve the readability and outreach of the paper \n\n* Comparability to baselines: The authors compare to three established baselines. However, with the exception of PopMAG, which the authors reproduce on part of the data they used to train their own model, the other two baselines are not strictly comparable. AMT requires ground-truth accompaniment (arguably a limitation of that method compared to the authors'), and GETMusic was trained on a different data (and additionally handles different instrumental tracks). While this is not a no-go, and is common practice for generative methods in general, it should be mentioned in 5.2. \n\n* The description of *interleaving* layers in the autoencoder -- according to the authors themselves, a major novelty of the work -- does not feature as prominently in their text as it should (given its stated importance), and could be improved in terms of clarity. Specifically, it is not entirely clear what the \"Track Encoder\" does in Fig. 2. Is it a standard residual \"transformer\" (it is not clear from A.2 if a residual connection is there) layer that essentially combines information across tokens? It would be beneficial to describe this in 3.3 using simple equations like in 3.2. One important open question is why this track encoder, if it is indeed based on self-attention, is different than the decoder? Specifically, why does one integrate information in the time-axis and the other on the track-axis? It would be important to show the dimensions here. I guess you have: $(B, K, N, P)$, where B is the batch size, K the number of tracks, N the duration of each note, and P the number of notes (I guess N&P are eventually downsampled from their original dimensions). So the inter-track encoder would operate on tensors $(B\\cdot N, K, P)$ and then the intra-track decoder would operate on tensors $(B\\cdot K, N, P)$? All this of course would be done in autoregressive fashion, so it would go from $\\{1...N\\}$. It would be nice to show this using the notation of the authors. * The VQ-VAE is described (A.1) and portrayed (Fig. 1) as having a convolutional encoder with a stride of 4 (thus a downsampling of 4) and a fully-connected decoder. It would be useful to show how this fully-connected decoder mitigates the downsampling of the encoder to have an output size equal to the input. Minor comments: + p.7, l.240: Best substitute \"denoising process\" with denoising diffusion probabilistic model (if this is what this meant), as the term denoising can be ambiguous", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive feedback! We value your thorough and insightful comments and will revise our manuscript based on these suggestions to improve the presentation. We also hope the following could address your existing concerns:\n\n**Weakness 1. Clarification of music concepts**\n\nA: Yes, we see the importance of clarifying music-related terms to make our paper more comprehensible. Certain terms (like 'groove') may have slightly varied definitions in music and AI research. We will use them in the research context and make the best effort to explain them well. We provide an explanation to the raised points as follows:\n\n(Count each note onset position as 1) - In this paper, we represent MIDI tracks using the *modified* pianoroll representation [1]. Instead of spreading out the full duration of a note, each non-zero $\\mathbf{x}_t^k =1, 2, \\cdots, 32$ denotes the duration of a note at timestep $t$ and track $k$. Hence the indicator function $\\mathbf{x}_t^k>0$ recovers the note onset positions.\n\n(*Orchestral function*, rhythm, and groove) - Our intuition with *orchestral function* is inspired by the \"grooving pattern\" introduced in [2], the essence of which is to use note onset densities (in our case, intensities) to describe rhythms. Groove admittedly has a subjective nature, but it is generally associated with the rhythm. In our paper, we mention \"groove\" in the same context as [2] and we will clarify this in our future revision.\n\n[1] Z. Wang, et al. Learning interpretable representation for controllable polyphonic music generation. ISMIR 2020.\n\n[2] S.-L. Wu and Y.-H. Yang. The jazz transformer on the front line: Exploring the shortcomings of ai-composed music through quantitative measures. ISMIR 2020.\n\n**Weakness 2. Clarification with other terms**\n\nA: Yes, we see the benefits of interpreting the essence of specific terms that may otherwise be too abstract to comprehend. Your interpretation of the ILS metric is well aligned with our intention. We will follow this example to elaborate on the other terms.\n\n**Weakness 3. Comparability to baseline**\n\nA: We will clarify our model’s comparability to each baseline in Section 5.2.\n\n**Weakness 4. (Q1.) Interleaving Layers**\n\nWe will improve the presentation of layer interleaving in our future revision. Regarding model dimensions, we have tensors $(B, K, T, D)$, each being batch size, track number, time frames (downsampled), and feature dimension. The inter-track encoder operates on tensor $(B \\times T, K, D)$, and the autoregressive decoder on $(B \\times K, T, D)$. We note that the inter-track encoder is not autoregressive, but a standard residual Transformer Encoder layer. This gives the user full control of initializing the number ($K$) and instruments of tracks.\n\nWe will supplement the missing details in the VQ-VAE framework and revise the ambiguous terms. Thank you again for your constructive advice on the paper presentation!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rk960UrVY7", "reviewer": "Reviewer_Y3LG", "summary": "The paper suggests creating multi-instrument accompaniment by using the piano reduction and the instrument note density (referred to as the 'Orchestration Function') as bootstrap representations. By effectively applying VAE and an autoregressive sequence generation framework, the paper demonstrates the high potential and effectiveness of the proposed approach for learning musically structured accompaniment.\nThe paper also proposed a 'layer interleaving architecture' that processes the orchestration codec by alternating between the time axis and the track axis.\nThe paper compares with previous works with objective and subjective evaluation and shows its validity.", "questions": "Q1. From my understanding, the 'orchestration function' in Eq. 1 eliminates pitch information. Thus, I expect the priors (${s_t}$) to provide note-density-like bare information. What do you expect to be encoded in $s$? Also, as a follow-up question, what kinds of correlation are expected along with $s_t$, and why does it make sense to train a sequential prediction model for $s$? I'm not sure about the expected role of the prior model.", "limitations": "I generally agree with the authors about the limitation presented in Appendix E.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The strength of this paper, I believe, lies in the factorization of accompaniment generation by leveraging the fact that piano reduction and 'orchestration function' can be freely produced as middle-level features from multi-track MIDI. By separating each process, as the authors claim, it becomes applicable to scenarios where users can exert control. Additionally, by dividing the modeling of each accompaniment's prior and the modeling of detailed notes, the efficiency of learning for each model is expected to have increased.", "weaknesses": "The proposed methodology utilizes piano reduction as a condition, so the quality of the final orchestra accompaniment is expected to vary depending on the quality of the piano reduction. Upon listening to the provided examples, it is evident that notes not present in the piano reduction were added, confirming that the proposed model's role extends beyond merely rearranging the notes of the piano reduction. However, there needs to be a discussion on the impact of using piano reduction on the model's high evaluation compared to other models. Additionally, while it was mentioned in L168 that another module was used, a more detailed explanation of how the piano reduction was generated is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and thoughtful feedback! We hope the following will address your concerns:\n\n**Weakness: Discussion on the impact of the piano reduction quality**\n\nA: We introduce *piano reduction* as an intermediate representation from the input lead sheet to the final orchestra arrangement. Intuitively, *piano reduction* is a hierarchical and more abstract planning of the final orchestra. The orchestrator at Stage 2 may add notes, but it still falls within the scope implied by the *piano reduction*.\n\nTo formally investigate the impact of *piano reduction*, we conduct an ablation study by replacing the original piano arranger with the *Whole-Song-Gen* model [1], which, to our knowledge, is the only existing alternative that can handle a whole-song structure. The ablation study is conducted in the same setting as Section 5.3. We report objective evaluation results regarding the final orchestra’s *chord accuracy*, *structure awareness*, and *degree of arrangement* (*DOA*) as follows:\n| | *Chord Acc*  $\\uparrow$ |*Structure*  $\\uparrow$ |*DOA* $\\uparrow$ |\n| -------- | ------- | ------- | ------- |\n| Ours\t\t\t| $0.567 \\pm 0.014^a$ | $1.520 \\pm 0.030^a$ | $0.300 \\pm 0.004^a$ |\n| Using *Whole-Song-Gen*| $0.509 \\pm 0.015^b$ | $1.121 \\pm 0.006^b$ | $0.277 \\pm 0.006^b$ |\n\n\nWe can observe that *Whole-Song-Gen* at Stage 1 generally deteriorates the quality of the final orchestra. To see why this happens, we further compare *Whole-Song-Gen* with our original piano arranger exclusively on the piano arrangement stage. We report objective evaluation results regarding the piano’s *chord accuracy* and *structure awareness* as follows:\n| | *Chord Acc*  $\\uparrow$ |*Structure*  $\\uparrow$|\n| -------- | ------- | ------- |\n| Original piano arranger| $0.540 ± 0.016^a$ | $1.983 ± 0.147^a$ | \n| *Whole-Song-Gen*| $0.430 ± 0.020^b$ | $1.153 ± 0.18^b$ |\n\nBy comparing the two tables, we can see that a higher-quality *piano reduction* generally encourages a more musical and creative final orchestra result. Particularly, *piano reduction* *lays the groundwork for (at least) chord progression and phrase structure*, both of which are important for capturing the long-term structure in whole-song arrangement.\n\nOverall, we confirm that the *piano reduction* is an abstract planning of the final orchestra result. Thus its quality is positively correlated to the orchestra quality. Meanwhile, we see that *our current piano arranger significantly outperforms existing alternatives and guarantees decent piano quality*, thus being the best choice for our model. Additionally, both the piano arranger at Stage 1 and the orchestrator at Stage 2 can be applied as independent modules to address respective subtasks.\n\n[1] Z. Wang, et al. Whole-song hierarchical generation of symbolic music using cascaded diffusion models. ICLR 2024.\n\n**Q1. What information is encoded in $\\mathbf{s}$**\n\nA: The $\\mathbf{s}$ embeddings are encoded from *orchestral function* $\\mathrm{fn}[\\mathbf{x}]$, which essentially describes the *form*, or *layout*, of orchestra sheet $\\mathbf{x}$. It contains rhythmic intensity information, telling the model where to put more notes and where to keep silent.\n\n**Q2. What kind of correlation to be expected along with $\\mathbf{s}\\_{1:T}$**\n\nA: $\\mathbf{s}\\_{1:T}$ encodes the sequential *form* of orchestra sheet. As $t$ goes from $1$ to $T$, *with the development of music*, we may see new rhythm patterns being introduced and new instrumental tracks being activated (e.g., in Figure 4 of the manuscript, the piano track is activated in the second half of the piece to mark increased atmosphere). More importantly, the lead sheet (and piano reduction) $\\mathbf{c}\\_{1:T}$ implies a verse-chorus structure of the whole song. Our prior model infers $\\mathbf{s}\\_{1:T}$ conditional on $\\mathbf{c}\\_{1:T}$, thus guaranteeing the structural alignment between $\\mathbf{s}\\_{1:T}$ and $\\mathbf{c}\\_{1:T}$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IpyjAUqhOc", "reviewer": "Reviewer_ANaR", "summary": "The paper presents a style transfer-based music accompaniment generation system. It starts with the lead melody and fleshes out the accompaniment tracks based on various high-level information, such as instrument type and rhythmic structure. The main goal of the proposed model is to be able to generate coherent structure in long music sequences, while they are also cohesive. Model architecture adopts VQVAE baselines to learn latent representations while the sequence learning part is done via transformer models applied to tracks and time dimensions orthogonally. The objective and subjective test scores look promising.", "questions": "- In Fig. 1, the autoencoder takes the piano reduction information as its first input. I imagine it would correspond to the lead sheet. Does it mean that during the training time the output of the autoencoder is the full sheet with all tracks? Then this isn't an autoencoder, right? The precise training procedures are not detailed enough in this manuscript. \n\n- If the input PN is the full music during training, how does the model generalize to the test-time scenario when the input has to be only the lead sheet?\n\n- The manuscript seems to be based on the concept of 1 segment = 8 beats. Is this correct? Any rationale behind this choice?\n\n- Gaussian noise is used to regularize the model. But if it's to deal with the domain shift between piano reduction representations, wouldn't there be a different type of noise that's more suitable, i.e. more discrete random variables?\n\n- It appears that the \"s\" embeddings contain important information about the track to be generated. Since it's learned in an unsupervised way, it's a little hard to imagine how these embeddings are more interpretable than other existing conditioning vectors as the authors claim.", "limitations": "- Basically it's not clear how Fig. 1 and Fig. 2 are combined to form the pipeline described in Fig 3. Colored figures, shades, and hashes show the authors' effort in describing this method, but it's not entirely clear to me. \n\n- It appears that the system takes the user input to designate which instrumental track to generate based on the instrument embedding. It's a nice feature, but it also means that, for evaluation, somebody has to come up with a nice orchestration (i.e., which instruments go well given the lead melody). Was this critical selection of instruments done by the authors to generate the test sequences? Unless I missed this part, it has to be clearly mentioned, as it can affect the performance of the model (in comparison to other methods that don't have such a concept).", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- The paper presents reasonable intermediate representations of symbolic music, piano reduction and orchestral function, which are used to learn discrete embeddings by the VQVAE model. \n\n- The proposed model is evaluated by both objective and subjective manners, where it outperformed the other methods.", "weaknesses": "- The description of the proposed model is not organized well, so it is difficult to understand. \n\n- Some high-level choices the authors made are not well justified, such as discrete embeddings vs. continuous, Gaussian noise vs. other noise, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive feedback! Please allow us to first respond to the points raised in **Limitations** and **Questions**. We hope this could address your concerns. We will then comprehensively incorporate these clarified ideas into our manuscript to provide sufficiently clear organization (**W1**) and well-justified motivation (**W2**).\n\n**L1. Elaboration on the model architecture**\n\nA: In this paper, we introduce a two-stage system to address the challenging task of multi-track accompaniment arrangement. Stage 1 arranges lead sheet into piano, and Stage 2 arranges piano into multi-track full sheet. Fig. 3 in the manuscript demonstrates the two-stage pipeline, which sequentially arranges piano/full sheets using respective style factors. The technical method of this paper mainly focuses on Stage 2, where we use the autoencoder (Fig. 1) to disentangle *piano reduction* $\\mathrm{pn}[\\mathbf{x}]$ (content factor) and *orchestral function* $\\mathrm{fn}[\\mathbf{x}]$ (style factor) from full sheet $\\mathbf{x}$. In Stage 2, $\\mathrm{pn}$ is given as condition and we propose the prior model (Fig. 2) to infer $\\mathrm{fn}$.\n\n**L2. User control on instrument designation**\n\nA: Our system does rely on user designation of instrumental tracks, and we see it as an intuitive and handy control. In practice, users can try out preset instrument ensembles (e.g., pop band with piano, guitars, and bass) without added burden. In our experiment, without loss of generality, this control choice is randomly sampled from Slakh2100 test set (mentioned in line 222). \n\n**Q1. Input/output of autoencoder and training detail**\n\nA: The autoencoder takes two inputs: the *piano reduction* $\\mathrm{pn}[\\mathbf{x}]$  and the *orchestral function* $\\mathrm{fn}[\\mathbf{x}]$. The output is the full sheet $\\mathbf{x}$. Note that both $\\mathrm{pn}[\\mathbf{x}]$ and $\\mathrm{fn}[\\mathbf{x}]$ are deterministic transforms from $\\mathbf{x}$. This is an inductive bias for content/style disentanglement. The ultimate input is $\\mathbf{x}$ and the training is based on a self-supervised reconstruction objective. We can see similar autoencoder designs in other disentanglement works [1] as well.\n\n**Q2. How the model generalizes to the test-time scenario**\n\nA: At test time, the autoencoder takes the piano arrangement from Stage 1 as its first input. \n\n**Q3. The rationale behind segment scale**\n\nA: Yes. We consider 1 segment = 8 beats. This is a proper scale (i.e., neither too short nor too long) to capture composition structures. Existing studies on music representation learning have also applied the 8-beat scale in their work [2,3]. \n\n**Q4. The rationale for Gaussian noise against discrete ones**\n\nA: We note that the *piano reduction* encoder learns *continuous* content representations instead of discrete ones (vector quantization is not applied here). Music content can be nuanced and thus better described with continuous variables. We hence see Gaussian noise as a natural choice for the domain shift regularization.\n\n**Q5. How the $\\mathbf{s}$ embeddings are interpretable**\n\nA: The $\\mathbf{s}$ embeddings are encoded from *orchestral function* $\\mathrm{fn}[\\mathbf{x}]$, which essentially describes the *form*, or *layout*, of multi-track music $\\mathbf{x}$. It contains rhythmic intensity information, telling the model where to put more notes and where to keep silent. When learning $\\mathbf{s}$ embedding from $\\mathrm{fn}[\\mathbf{x}]$, we apply vector quantization because common rhythmic patterns (like syncopation, arpeggio, etc.) can naturally be categorized as discrete variables. Moreover, VQ-VAE learns a discrete latent space that facilitates a prior model to mount on.\n\n[1] Z. Wang, et al. Audio-to-symbolic arrangement via cross-modal music representation learning. ICASSP 2022.\n\n[2] A. Robert, et al. A hierarchical latent vector model for learning long-term structure in music. ICML 2018\n\n[3] R. Yang, et al. Deep music analogy via latent representation disentanglement. ISMIR 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y4Eqe2uAmU", "reviewer": "Reviewer_AQ7F", "summary": "This paper introduces a novel music AI system that creates multi-track accompaniments from a lead sheet by leveraging disentangled style factors to enhance context coherence, creativity, and computational efficiency. The proposed two-stage process begins with generating a piano arrangement using piano texture styles, followed by orchestrating multiple tracks by incorporating orchestral function styles. The system employs vector quantization and a multi-stream Transformer for modeling orchestration styles, significantly improving arrangement quality and providing flexible control over various music genres and compositional levels.", "questions": "1. Why does concatenating style factors potentially contradict the structure of existing conditions as mentioned on line 25?\n\n2. The paper discusses conflicts when combining style factors in sequence generation tasks; could these be resolved using different methods like diffusion models?\n\n3. How do the initial discussions on introducing style into long-term generation relate to the paper’s main theme of multi-track accompaniment arrangement?\n\n4. What prompted the formulation of the problem between lines 32-36 in paragraph 3, and how is time t determined?\n\n5. Is there a distinction between 'piano reduction' and 'piano track'? The terminology varies between italics and normal text; does this signify a difference?\n\n6. What is the rationale behind using VQ-VAE and VAE to encode orchestral functions and piano reductions in section 3.2?\n\n7. What motivations underlie the construction of the Multi-Track Orchestral Function Prior in section 3.3?\n\n8. Can you explain what inputs the proposed model receives and provide comparisons to highlight its unique features, as seen in Figure 4?\n\n9. How does the model handle both ABC notation and Midi, and is there a conversion of ABC notation into the representation discussed in section 3.1?\n\n10.Why is the computational efficiency of this model less optimal compared to [20]?\n\n11. How are the 'style' factors controlled and tested in the experiments?", "limitations": "Limitations are addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses the significant task of Multi-Track Accompaniment Arrangement, making notable improvements in the field.\n\n2. The authors have diligently constructed a comprehensive framework in both the experimental design and methodology, contributing valuable insights and data to the field.", "weaknesses": "1. The introduction lacks clarity, with key terms and relationships not sufficiently defined before introducing mathematical notations and formulations. This obscures the motivation and makes it difficult to understand, especially as the abstract outlines challenges related to coherency, creativity, and efficiency that are not clearly addressed in the introduction.\n\n2. The contributions of the paper, as discussed between lines 46-60 and 85-94, appear to be incremental rather than significant, primarily building upon work from AccoMontage.\n\n3. There is a consistent omission of motivation when new concepts are introduced. For instance, the motivation behind formulating the multi-track accompaniment arrangement task is not made clear in the introduction after discussing style representations. Similarly, the reasons behind specific mechanisms in sections 3 and 4 are not elaborated upon.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive feedback! We acknowledge that some terms in our paper may not be sufficiently clear (**W1**). We also recognize our shortcomings in providing adequate motivation to justify our design choices (**W3**). Please allow us to first respond to the relevant points raised in **W2** and **Questions**. We hope this could address your concerns. We will then comprehensively incorporate these clarified ideas into our manuscript to provide sufficiently clear introduction and well-justified motivation.\n\n**W2. Significance of Contribution**\n\nOur proposed *style prior modelling* is not built upon AccoMontage. It is an original methodology that effectively addresses long-term, structured conditional sequence generation. AccoMontage addresses piano arrangement only, while our method is more scalable to solve more challenging practical problems. Building upon *style prior modelling*, we introduce the first whole-song, multi-track accompaniment arrangement system, which supports variable music length ($T$) and controllable instrumental tracks ($K$).\n\n**Q1. Why  concatenating style factors may contradict the structure of existing conditions**\n\nA: The general problem that we study is *conditional sequence generation*. The input is a conditional sequence $\\mathbf{c}\\_{1:T}$ and the output is an observational sequence $\\mathbf{o}\\_{1:T}$. *We assume there underlies a style factor sequence $\\mathbf{s}\\_{1:T}$*, which can render/arrange the content of $\\mathbf{c}\\_{1:T}$ to realize $\\mathbf{o}\\_{1:T}$. By *style prior modelling* of $\\mathbf{s}\\_{1:T}$, we aim to improve interpretability, controllability, and performance. We note that the existing condition $\\mathbf{c}\\_{1:T}$ often implies a long-term structure. The style factor $\\mathbf{s}\\_{1:T}$, if not structurally aligned with $\\mathbf{c}\\_{1:T}$, will lead to incoherent results. \n\n**Q2. Can diffusion models address this?**\n\nA: Diffusion models and Transformers can be distinct implementations of the same methodology. The context dependency of diffusion models differs from that of Transformers and we will explore this in our future research.\n\n**Q3. Relation to accompaniment arrangement**\n\nA: Accompaniment arrangement is a typical task of *long-term* conditional sequence generation. The input is a lead sheet that implies a verse-chorus structure of the whole song. The output is the accompaniment, and the style factor regards the sequential *form* of the accompaniment. In this paper, we consider multi-track, whole-song arrangement, where it is challenging to maintain both track cohesion and structural coherence.\n\n**Q4. Prompt of the formulation in lines 32-36**\n\nA: Continuing from Q1 & Q3, our general idea is to model $\\mathbf{s}$ conditional on $\\mathbf{c}$. We thus come to the formulation of $p(\\mathbf{s}\\_{1:T}^{1:K} | \\mathbf{c}\\_{1:T})$. Superscript $k=1, 2, \\cdots K$ denotes track indices of multi-track arrangement. Subscript $t = 1, 2, \\cdots, T$ denotes component segments of the whole song. We consider each segment as a 2-bar music snippet, which is a proper scale for learning content/style factor representations.\n\n**Q5. *Piano reduction* vs 'piano track'**\n\nA: We use *piano reduction* to denote the overall content from a multi-track piece. We will unify all instances of *piano reduction* into italic text. On the other hand, a ‘piano track’ simply refers to a general track played by piano. \n\n**Q6. Rationale for VQ-VAE and VAE**\n\nA: We use VAE to learn the nuanced content from the *piano reduction* because existing works have shown VAE’s effectiveness for music representation learning. We further choose VQ-VAE for the *orchestral function* because common patterns of orchestral tracks (like syncopation, arpeggio, etc.) can naturally be categorized as discrete variables. Moreover, VQ-VAE learns a discrete latent space that facilitates a prior model to mount on.\n\n**Q7. Motivations for Multi-Track Orchestral Function Prior**\n\nA: By constructing Multi-Track Orchestral Function Prior, we *recover the underlying style factor sequence* that can further render/arrange the input content into multi-track accompaniment. This design is more interpretable and controllable while also adhering to music composition practice. Experiments also demonstrate superior performance against non-prior baselines.\n\n**Q8. Model inputs in Figure 4**\n\nA: In Figure 4, the model receives two inputs: a lead sheet shown by the “Mel” staff, and a set of instruments (user control) shown by the rest staff labels. The output is the accompaniment in the rest staves. The length of lead sheet determines $T$ and the number of instruments determines $K$.\n\n**Q9. ABC notation**\n\nA: ABC is essentially a score notation for lead sheet, which can be converted into MIDI and the representation in Section 3.1.\n\n**Q10. Computational Efficiency of GETMusic**\n\nA: GETMusic is a diffusion model with only 100 diffusion steps, thus achieving notable computational efficiency. \n\n**Q11. How the style factors are controlled and tested**\n\nA: The control over style factors is covered in lines 173-176. Firstly, user can customize the instruments to steer the Multi-Track Orchestral Function Prior. Moreover, a starting prompt can optionally be provided. In our experiment, these control choices are randomly sampled from Slakh2100 test set. The effectiveness of $\\mathbf{s}\\_{1:T}$ is manifested from the quality of the resulting $\\mathbf{o}_{1:T}$. Experiments show that *our method achieves the top chord, structure, and DOA scores for long-term arrangement*, which demonstrates the effectiveness of style factors and the superiority of *style prior modelling*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary**: The paper introduces a new system for generating multi-track accompaniments from a lead sheet using a style prior modelling approach. The method involves a two-stage process: generating a piano arrangement and then orchestrating multiple tracks. The system leverages vector quantization and a multi-stream Transformer to model orchestration styles, enhancing generative capacity and providing flexible control.

**Strengths**: The paper presents a unique method for disentangling piano reduction and orchestral function, providing a novel element of controllability. The proposed model is technically robust and has a well-executed evaluation. The authors have documented experimental parameters thoroughly and provided code for reproducibility. The method addresses a challenging task in music AI, making significant contributions to the broader generative AI community and the niche music-generation community.

**Weaknesses**: The reviewers noted that the paper could benefit from clearer explanations of concepts and better organization of the model description (e.g., what are interleaving layers). Some baselines used for comparison are not strictly comparable, which should be clarified in the paper. Some high-level design choices, such as discrete embeddings and Gaussian noise, need better justification.

**Recommendation**: The reviewers unanimously agree that the paper should be accepted. As such, it is recommended for acceptance conditioned on minor revisions being implemented to address the presentation issues and provide clearer justifications for high-level design choices. The unique approach and state-of-the-art performance make it a valuable contribution to the field of music AI.
Decision: Accept (poster)