FIGARO: Controllable Music Generation using Learned and Expert Features
Authors: Dimitri von Rütte
Luca Biggio
Yannic Kilcher
Thomas Hofmann
Venue: ICLR 2023 poster (2023)
Keywords: symbolic music
style transfer
music generation
controllable generation
human-interpretability
self-supervised learning

Abstract: Recent symbolic music generative models have achieved significant improvements in the quality of the generated samples. Nevertheless, it remains hard for users to control the output in such a way that it matches their expectation. To address this limitation, high-level, human-interpretable conditioning is essential. In this work, we release FIGARO, a Transformer-based conditional model trained to generate symbolic music based on a sequence of high-level control codes. To this end, we propose description-to-sequence learning, which consists of automatically extracting fine-grained, human-interpretable features (the description) and training a sequence-to-sequence model to reconstruct the original sequence given only the description as input. FIGARO achieves state-of-the-art performance in multi-track symbolic music generation both in terms of style transfer and sample quality. We show that performance can be further improved by combining human-interpretable with learned features. Our extensive experimental evaluation shows that FIGARO is able to generate samples that closely adhere to the content of the input descriptions, even when they deviate significantly from the training distribution.

Published as a conference paper at ICLR 2023
FIGARO: CONTROLLABLE MUSIC GENERATION USING EXPERT AND LEARNED FEATURES
Dimitri von Rütte, Luca Biggio; Yannic Kilcher, Thomas Hofmann Department of Computer Science, ETH Zürich dimitri. von inf.ethz.
ABSTRACT
Recent symbolic music generative models have achieved significant improvements in the quality of the generated samples. Nevertheless, it remains hard for users to control the output in such a way that matches their expectation. To address this limitation, high-level, human-interpretable conditioning is essential. In this work, we release FIGARO, a Transformer-base bassed conditional model trained to generate symbolic music based sequence of high-level control codes. To this end, we propose pnnerription-o-seqeeee extracting tin-ggrined, human- interpretable features (the description) and training a sequence-to -oueenece----eueenee model to reconstruct the original sequence given only the description input. FIGARO achieves state-of-the-ard performance in multi-track symbolic music generation both in terms of style transfer and sample quality. We show that performance can be further improved by combining human-intennn interpetatte with learned features. Our extensive experimental evaluation shows that FIGARO is able to generate samples that closely adhere to the content of the input descriptions, even when they deviate significantly from the training distribution.
learning, which consists automatically
INTRODUCTION
Music a fascinating subject that surrounds us constantly, being a source of inspiration and canvas for imagination many. To some, creating music is topic worthy of dedicating one's 's life to, which is a testament to the artistry and mastery involved. While composition is an intricate form of art that requires deep understanding of the human experience and domain knowledge, the idea of devising a systematic or algorithmic approach to music creation has been around for centuries (Nierhaus) 2009). With the advent of deep learning, automatic music generation has witnessed renewed interest Hernandez-Olivan & Beltran (021). Especially the Transformer architecture (Vaswani al.) 2017), which has seen applications many Machine Learning domains (Brown) et al.] 2020, Dosovitskiy et al., 2021] Lample & Charton 2019 Biggio et 2021, has proven to be a powerful tool for musical sequence modelling. Initial breakthroughs by Huang et al. (2018) and Payne (2119) applied language modelling techniques to symbolic music to achieve state-of-the-art music generation. Though these models were capable of some form of conditional generation (e.g. melody or genre conditioning). other conditioning mechanisms and different types of control have
since been proposed (Ens & Pasquier. 2020 Choi et al. 2020 Wu & Yang 2021).
As deep generative models are improving and producing more and more realistic samples, it remains an area of active research how humans can interact with these models and steer them to generate a desirable result. Recent efforts in text-to-image generation (Ramesh al. 2022 Saharia al. 2022 have shown the potential in usability and artistic applications of human- cuuman-interpretablle controllable generative models. Whereas text-based conditioning has yielded human- numan-interpretable control for image generation, the same conditioning mechanisms are not easily applicable to music generation. We aim extend this kind of control other domains, in this case music generation. As scale has proven to be key for achieving capable models, we cannot rely on scarce annotated data and instead propose self-supervised objective, which we call pessrrpttion-to-sequeen inspiration from recent text-to- to-tto- approaches, but instead of a natural language description of the target, we automatically extract a sequence of high-level features (the description). These can
learning. We take
"qqual contribution

---

Published as a conference paper at ICLR 2023
At inforence time:
User Input
Learned Description
Latent Description Space
per bar
Rato-Regressive Decoder
VQ-VAE Encoder
Bar-level
of
Global, fine grained control description space
VO-VAE loss
sea2se0
Expert Description
Humen-interpretable Description Space
Bidirectional Encoder
Auto-Regressive Decoder
Symbolic Extraction
Sequence rec.
desc-to-seq loss
Figure I: Overview of FIGARO. Dashed lines indicate components that are only used during training of the learned description.
either be hand-crafted using domain knowledge or learned. The description then serves as the input to conditional model to reconstruct the original sequence. To this end, we define a description function which extracts said features from a given piece of music. The choice of description function determines the characteristics of the resulting model and serves an inductive bias, allowing to emphasize desirable properties such as human- intrannttteerrttbblittyy or fine- ane-grained control over instruments and chord progression. Note that the general nature of the proposed framework allows for applications to other domains despite our focus on symbolic music.
Our main contribution is FIGARO (Fine-grained music Generation via Attention-based. RObust control), a model trained on the proposed description-to-seqquee separate description functions: 1) The hand-crafted expert description, which provides global context sequence and 2) the learned description, where
objective by combining two
in the form of a high-level, human-interprettal use representation learning to extract salient features from the source sequence. The learned description is intended to amend the expert description with the latter might be incomplete, albeit at the cost of human- interrrettttttbilityy... The model trained on conditional generation, mapping descriptions to music. An illustrated overview of the model is given Figure [ At inference time, users may interact with the model space. We provide a simple interface in the form of an online demo of our model We also release the source code and model weights for anyone to download and use freely Our secondary contribution is REMI+, extension the REMI input representation (Huang & Yang 2020) which opens the way to multi-trace multi-time-sipe signatree music.
information in places where
caman-interpretabll description
We evaluate FIGARO on its ability adhere to the prescribed condition by comparing it to state-of- the-art methods for controllable symbolic music generation (Choi et al.) 22000; Wu & Yang 202T). We demonstrate empirically that technique outperforms the state-of-the-ann in controllable generation and sample quality. To evaluate sample quality, we employ subjective evaluation in the form of a listening study. We further demonstrate that FIGARO is robust with respect to distributional shifts in description space and performs well on constructed samples outside the training distribution, indicating that the proposed objective effective learning generalized concepts about the data.
CONTROLLABLE SYMBOLIC MUSIC GENERATION
In the context of generative modelling, controllability an important issue, as such models only become useful if the user is able to steer the generation process in a desired direction. This has recently been observed for text-to-image models and intend to take closer look at controllable music generation. We identify two different levels of controllability: global and fine-grained control. Global conditioning, where the generation is guided by a constant set of attributes that do not change during the generation process, is the most prevalent form of control. Examples of global control include prompt- based conditioning (Payne 2019 or conditional decoding of latent representations
'Online demonstration FIGARO is available on Google Colab (urpps//tinyyurl.co///eez2)) We recom-
mend selecting GPU environment for improved inference speed. "Code and model weights are available through GitHub (ftpss:/githubb.cco//vvveeeeegg
2

---

Published as a conference paper at ICLR 2023
Method
Input Rep.
Multi- Multi-Sig.
Global Ctrl. Fine-Grained Ctrl.
MIDI-VAL Brunner (2018)
Pianoroll
MuseNet Payne
2019
MIDI-like
MMM (Ens Pasquier 2020
MIDI-like
Choi ct al (2220)
MIDI-like
MuseMorphoose (Wu& & Yang. 2021
REMI
FIGARO (ours)
REMI+
Table I:
Generative capability and controllability comparison between different methods proposed in the literature. We compare our method to other state-of-the-art symbolic music generation methods on modelling capability (can the model generate controllability (can the generation be controlled on a global/fine-gran level?).
-ime--ignatture music?) and
(Brunner al. 2018). Fine-grained control is achieved when the generation process can be guided any point in time, f.e. if the control attributes can be arbitrarily varied over time. Consider controlling what instruments are playing in the generated sequence an example of global control in contrast to controlling what instruments are playing at any point in time as an example of fine-grained control. Note that fine-grained control also implies global control, global control can be achieved by fixing the control attributes to some constant value. Fine-graine control is therefore strictly more powerful property and seemingly harder to obtain, as highlighted in Table [
2.1 RELATED WORK
The capabilities of symbolic music generative models have been steadily improving with many recent contributions (Huang et al. 2018 Payne 2019 Huang & Yang 2020 Hsiao et al. 2021 Wu et al. 2022p. This line of work focuses on improving the quality of generated samples but does not contribute substantially toward controllability. An exception to that is MuseNet (Payne (209)) which allows some control through prompt-base based conditioning with control tokens. Even still, prompt-based control is very limited, as control tokens are "forgotten" by the model once the generation advances beyond the initial context size.
Another line of work focuses on finding ways of controlling the generation process. VAE-based methods often achieve global control through latent conditioning vectors, enabling genre transfer (Brunner et al.) 2018) capabilities or control over arousal (Tan & Herremans (020). Transformer auto- encoders have been used for melody conditioning (Huang et al.J | [2018] Choi et al. 2020) and encoding of musical style Choi et al. (2020). [Ens & Pasquier (2020) present MMM which is capable of bar-level and track-level symbolic music inpainting. Finally, Wu & Yang (02T)) introduce MuseMorphose for fine-grained attribute conditioning and latent space style editing. All of these approaches have some limitations that are highlighted in Table Recent work tackling the problem of controllable music to global control.
style/Choi
generation is often either limited single-track, fungle-time-sinattte
Fine-grained control has been a topic of interest in the recent literature (Choi et al. 2020 Wu & Yang. 2021 Di et al. [02T Ferreira & Whitehead (021) and is an essential property when considering user-directed applications. In essence, fine-grained control is necessary to allow control over salient features in the generation, as saliency in music least partly lies in how it changes over time. In addition, salient features may be impossible or prohibitively expensive to quantify directly (Choi et al. 2020 Ferreira & Whitehead, [222I), emphasizing the need for un- or self-supervised ine-grained control. Our proposed method provides both global and fine-grained control through description- eescrittiontto-squuemne to-sequence learning.
DESCRIPTION--T-SSEUUEE
LEARNING
pnaeription-to-sequeee learning is self-supervised objective based the reconstruction of sequence given automatically extracted features of that same sequence (the description) as input. Conceptually, an information bottleneck is applied by mapping a sample the description space and the model is trained to learn a probabilistic map from description space to sample space.
While it is possible to control the style (via artist tags) and instruments of the generated sequence, this information is "forgotten" by the model due context scrolling once the generation advances beyond the initial prompt.
*Fine-grained control is limited to changing the note density through bar-level inpainting. 5 "Fine-graa FFineggrained control is limited to optionally prescribing the melody of the generated sequence.
3

---

Published as a conference paper at ICLR 2023
<bos>>
Bar_I TimeSignature_4/4 NoteDensity_3 .3 McanPitch_14 MeanVelocity_19 MeanDuration.33
Instrument Instrument. Instrument EPiano Sertumment-SlaBaaa
E: maj F#:min7
Bar.2
TimeSignature .4/4 NoteDensity.3 3 McanPiteh..15 MeanVelocity_Je MeanDuration.27
Instrument.Drumi Instrument Instrument. Instrument.Guitar
Chard.C#:mi Chord. E: maj Chord.A: maj7
Bar_3
<cos>
Figure 2: Example of an expert description. The description contains information about time signature, note density, pitch, velocity and duration as well which instruments and chords played throughout each bar.
Objective. Let Veeq denote sequence vocabulary and Veeee description vocabulary. Let x Vaeg an input sequence and Veeg - desc be partition-wise feature extraction function that extracts descriptive features given a partition I1,...,, Un of x such that x concat(E1....... We then define the description of x to be:
Destription-to-Sequene
(oucat(((
For simplicity, we write = F(x) in the following. The description-to-sequeene objective is then given by the cross-entropy reconstruction loss of x given F(x):
( Ex-x Exxxl (x(
Here X' is the true distribution of sequences and ø denotes the parameters of a sequence-to-sequuee model. An illustrated overview of this objective is given in F Note that level of fine-grained control guaranteed by the way we partition prior feature extraction, the length of partitions puts an upper bound on the receptive field of the description function.
3.1 EXPERT DESCRIPTION
We aim for human- mamnan-itterrretblee fine-grained conditioning mechanism by using domain knowledge to automatically extract high-level features that are easy to understand for human experts and relevant from a compositional stand- point. We identify time signature, chords and in- struments as such features and also consider note density, mean pitch, mean velocity and mean du- ration as auxiliary features quantifying musical style (Choi et al., 2020). All of these features are computed bar-per-bar and quantized accord- ing special-purposs. vocabulary. Finally, all resulting tokens are concatenated into a sequence, Fexpert (x), of which an example is given in Figure 2 and quantization can be found in Appendix
Source sequence
Deseiition
High lioi dose.
Sequence Kidel
Neconstruded
Figure 3: Schematic overview of description-to- sequence learning the context of music.
which yields the expert description function More details the feature extraction algorithm
3.2 LEARNED DESCRIPTION
While the expert description serves as a human-interrin cumaanniteerpretable inductive bias, is only able to capture fairly low-fidelity features by design and suffers from non-injectivity (i.e. there can be many sequences that map to the same description). In an attempt to alleviate some of these limitations, representation learning to extract features that maximize the information content about the underlying sequence. We choose the VQ-VAE framework as the basis for our learned description function it produces discrete codes that allow for learning a prior sequence model and has been shown to be effective for various tasks such music generation in particular (Dhariwal et al. 2020). The latent representations given by this model is used the output of the learned description function Flearne (x). This way, we add high-fidelity information and reduce non-injectivity. albeit at the cost of human interpretability.
We later show that combining both descriptions yields better performance than using either one on its
own (Section 6.2) and that the resulting model is fairly robust (Section 6.3
4

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTAL SETUP
REMI+ Input Representation. The Revamped MIDI (REM)) representation (Huang & Yang 2020 is a heat-based input representation, which has recently been shown to give better sample quality than traditional time-based representation. REMI represents notes with four consecutive tokens encoding note position, pitch, velocity and duration and also includes chord and tempo events.
In order to make it applicable to any general MIDI piece, we extend REMI by adding instrument and time- signatire information, yielding REMI+. We add tokens encoding the time signature of a given bar at the beginning of each bar and instrument tokens at the beginning of each note, thus making it suitable for multi-time-signan signature, multi-track modelling tasks. We also increase the temporal resolution note positions from 4 to 12 sub-beats per quarter note. This allows for precise quantization of triplets in addition to sixteenths, which was previously
possible. For exact details on quantization and tokenization, we refer to Appendix [ A limitation left unaddressed by REMI+ is the inability encode expressive performances, which requires frequent shifts in tempo or small time offsets. Time-based representations are still preferred for that setting.
In our experiments, we tokenize MIDI files using the REMI+ input representation and train FIGARO on the proposed objective using the combination of the proposed expert and latent description functions.
Description-toSSqqeence Model. We use the original Transformer auto-encoder (Vawwani et al. 2017) architecture with encoder and 6 decoder layers as the to-iueee--- backbone. Three objective (Equation using
separate models are trained using the proposed description-to-sequue either Fexpert(X] Flearned (x) or a combination of both descriptions. Combining descriptions is done by simply summing their respective learned token embeddings. The model architecture is kept deliberately simple, so as to be able to accurately assess the effectiveness of the various description functions. Further details and hyperparameters are given in Appendix
VQ-VAE Model. For our bar-level VQ-VAE model, we use the same Transformer auto-encoder backbone but with additional quantization bottleneck consisting of 16 latent codes with shared codebook of size 2048 between the encoder and decoder. The model is trained on reconstruction for individual bars. Note that training FIGARO two-stage process: In the first step, we train the VQ-VAE for feature extraction, which is later used to compute Fiearned The VQ-VAE model is frozen during training of the pecrrittiontt-sequenne model.
Training Setup. We use the LakhMIDI dataset (Raffel 2016) as training data in all of our experiments, which to the best of our knowledge is the largest publicly available symbolic music dataset. We use 80%-10% 166-10% 100% training-validation--e 2080 Ti GPUs. For evaluation, we generate samples conditioned on descriptions sampled from the test set, generating 32 bars for each sample. For training details, we refer to Appendix A
split. Each model is trained for 24 hours on 4 Nvidia GTX
5 PERFORMANCE EVALUATION
5.1 FLUENCY
We use perplexity (PPL.) as way to measure fluency and to compare the likelihood of different models in addition to task-specific metrics. The perplexity measures the likelihood of sequences while normalizing over the sequence length, which makes it better suited to comparing sequences of different lengths than the negative ligeliili.oodd
5.2 DESCRIPTION FIDELITY
We also quantitatively evaluate the fidelity of generated sequences to the given condition. Let denote a test sample and F(xx) its description. Then we generate x by sampling the model conditioned on F(x) and examine x and & for similarity. Metrics computed as empirical estimate over the test distribution. More details and exact formulas are given in Appendix [
Accuracy. We compute accuracy metrics for categorical values, namely for instruments, chords and time signature. Instruments and chord are multi-label features for which we compute the mean F, score. As there is only time signature per bar, we compute the standard accuracy score.
5

---

Published as a conference paper at ICLR 2023
Macro Overlapping Area. Previous work has used the overlapping area (OA) metric to quantify similarity between two musical sequences for a given feature (Choi et al. 2020 Wu & Yang 2021). However, we find that the standard OA metric fails to take the order of the sequences into account, as feature histograms are computed over the entire sequence. For example, a sequence has maximal overlapping area with itself in reverse order, even though the resulting sequences will sound completely different. To alleviate this limitation, we propose the macro overlapping area (MOA), which partition-wise computes the overlap in the distributions of given feature, taking sequential order into account. We use the MOA metric to compute similarity in pitch, velocity and duration between ground truth and reconstruction. The exact definition of our MOA metric given in Appendix D
Normalized Moot--MeanSquare Error. We compute the normalized RMSE (NRMSE) for bar-wise note density. This helps compare similarity accross different feature magnitudes.
Cosine Similarity. We also evaluate chroma and grooving similarity as a way to quantify similarity in sound and rhythm as proposed by Wu & Yang (202)) We compute bar-wise cosine similarity for the chroma vectors (Fujishimal 1999) and grooving vectors (Dixon et al. 2004
6 EXPERIMENTS
Recent work on controllable generation has largely focused on single-track symbolic music and to the best of our knowledge, there are no competitive baselines for multi-track music generation with fine-grained control. To make due, we train two state-of-the-art methods for single-track controllable generation (Choi et al. [2200] Wu & Yang (02T) on the REMI+ input representation, which allows us to apply them to a multi traci-ttack setting. We also train an unconditional baseline based on Music sanity check: It has no additional information about the target sequence and essentially performs "random guessing". i.e. it uniformly samples a sequence from the training distribution. If any conditional model performs worse than unconditional on similarity metrics, it is an indication for mode collapse (Dieng et al., 2018). Indeed, although the training of MuseMorphose (Wu & Yang 2021 initially appeared to be successful, quantitative analysis revealed that the model had suffered mode collapse, as evidenced by the small chord and instrument entropy in Table 2 and the worse-than-uneee -han-uncondiiional performance on some of the style transfer metries in Table 3a As this model was originally designed for modelling single-tres track piano music and the MIDI dataset constitutes a quite drastic domain shift, we instead focus on comparison to Choi et al. 2020). Details on baseline training are given in Appendix F
Transformer (Huang et al. 2018), which acts
6.1 HUMAN INPUT
The question of how well FIGARO can handle human-generate generated descriptions is both important and tricky to evaluate quantitatively. as both Fexpert and especially Flearned require time and domain knowledge to create (albeit substantially less than composing a piece of music from scratch). We therefore demonstrate the generative capabilities of our model in interactive demo based on Fexpert- We provide four hand-crafted examples, namely a simple four-bar piano progression, an 8-bar description containing uncommon instruments, a 9-bar description based on well-known Jazz standard and a longer 20-bar description demonstrating control over multiple features in parallel with fine-gre mne-grained instructions over instruments, chords, note density and mean pitch. These example are meant to show several attributes of FIGARO, namely its level of controllability and its ability to generate meaningful outputs even when input descriptions are numan-made and present features that rarely appear in the training set. The reader is invited to generate samples based on these descriptions or to come up with their own creations. Demonstrating the use of Fleacned on human input less straight- forward as in the absence of a prior model, latent codes need to be extracted from an existing piece of music. We approximate this and the previous setting by using pieces of music from the test set extract descriptions and perform quantitative evaluation style transfer and controllable generation in Section 6.2 and 6.3 respectively.
6.2 STYLE TRANSFER
In this experiment, we extract conditioning sequences by feeding samples drawn from the test set uniformly at random through neural bottlenecks in the case of MuseMorphose, Choi et al. (2020) and
'Online demonstration of FIGARO Google Colab (ttpps//ttnwwrr.coo///e22277)

---

Published as a conference paper at ICLR 2023
Flearned or symbolic bottleneck in the case of Fexpert. New samples are then generated based on the condition and closeness in style to the original sequence is evaluated using the metrics from Section 5 Note that many of these metries are directly present in F Fexport and FIGARO (expert) expectedly does well in that regard. The fact that the results obtained with the expert description adhere to the content of the description itself is a highly desirable feature for controllable music generation. However, it also performs well on chroma and grooving similarity, both of which are not directly present. This shows that the expert description can act as a strong inductive bias and helps guide the generation process even beyond what informa- tion is directly represented. By using this de- scription, the degree of control exerted by the user is enhanced and the results match expec- tations. We also observe that adding Iearned features further improves performance across the board, with FIGARO beating FIGARO (ex- pert) in every category. The success of this hy- brid approach means that we can preserve the interpretability and inductive bias, yet increase the fidelity of the generated music by exploiting black-box AI. Our expert description and hybrid models significantly outperform all baselines, and the learned description model outperforms Choi et al.] (2020) on most metries by a slight margin. The difference in performance is explained by the conditioning used for Choi et al. (2020), where the conditioning vector is temporally aggregated over the entire sample with any style progres- sion throughout the sequence being lost. We provide a non-cherrypicked collection of samples and encourage the reader to get impression of the quality and diversity of the music by listening to some of them on SoundCloud The full list of results is provided in Table 3a
Model
H Inst
A
Ground truth
(3.763)
(4.077)
Unconditional
-0.521
-0.328
Choi et al. (2020)
-0.116
-0.128
Wu & Yang (2021
-1.94
- -1.57
FIGARO (expert)
-0.011
-0.028
FIGARO (learned)
0.062
0.008
FIGARO
0.102
-0.06
Table 2: Difference in instrument entropy Histt and chord entropy Hehord between ground truth and modelled distributions. Model entropies are empirical estimates over samples from the style transfer task. Ground truth values are absolute, deltas are relative to ground truth.
6.3 CONTROLLABLE GENERATION
To evaluate the robustness of our model with respect to out-of-distribution conditions, we combine two sequences (1) and (2) (either splicing or mixing the descriptions) and generate samples conditioned on the resulting descriptions. This way, we create examples that have unusual transitions (in the case of splicing) or conflicting information (in the case of mixing) and are not part of the training distribution in general.
Description splicing. For this experiment, we take 16 bars of each sequence and concatenate them into novel description x b(8) ... b bl7 || ... || b We then input the description F(x) to the model and sample the output distribution to generate a "medley" of the input sequences. drop in performance
Importantly, none of the models are finetuned for this specific task. We in all evaluation metrics for every model compared to the style transfer task, which is expected due to distributional shifts in the data. However, FIGARO and FIGARO (expert) drop significantly less than FIGARO (learned) and Choi et al. (020)). showing that the expert description provides a strong inductive bias and robust with respect distribution shifts. We sometimes even observe FIGARO turning the hard cut-off points from bar-splicing into smooth transitions, which can be attributed to its capacity to pay attention to future bars and anticipate significant changes in style. The complete list of results is available in Table 3b We omit Wu & Yang (202I) from these experiments we do not expect competitive performance due to mode collapse.
Description mixing. Description mixing is done by combining Fexpert (x()) with Fiearned (x(2) and thus is a scenario that is only applicable to FIGARO, the only model that takes both description types as input. Note that mixing deseriptions in this way potentially provides conflicting information to the model as the two sequences might not agree with each other on certain features which might decrease sample quality or lead to a collapse of the model's performance in the worst case. Conflicting information also makes quantitative evaluation more difficult, which is why we focus qualitative evaluation. We find that mixing descriptions in this way is not detrimental to the sample quality and that the model relies on different descriptions for different attributes of the generated sample. In general terms, the model seems to rely on the expert description for attributes including instruments, harmonics and time signature (one might call these "low-fidelity" attributes) and on the learned
7

---

Published as a conference paper at ICLR 2023
Fluency
Accuracy C1
Fidelity
Model
PPL
I1
TS 1
ND
P1
V1
D
5e t
9g
Unconditional
1,988
.191
.048
.751
2.192
.563
.153
.312
.306
.510
Choi al. (2020)
2.049
.658
.184
.908
1.679
.646
.574
.484
.514
.688
Wu & Yang 2021
2.094
.179
.050
1.00
0.873
.492
.050
.207
.312
.529
FIGARO (exper))
1,913
.957
.561
.996
0.319
.759
.658
.514
.712
.637
FIGARO (learned)
1.973
.594
.195
.969
0.738
.701
.653
.546
.544
.697
FIGARO
1.705
.960
.593
.997
0.238
.827
.735
.748
.790
.853
(a) Perplexity and similarity metrics for style transfer LakhMIDI. Best values are highlighted.
Fluency
Accuracy
Fidelity
Model
PPL 4
If
Ct
TS t
ND
4
Pt
V
D
Se +
8g f
Choi et al.
2020)
2.213
441
.129
.808
1.407
.603
.396
.448
.437
.643
FIGARO (expert)
1,824
.944
.524
.992
0.384
.741
.559
.497
.705
.575
FIGARO (learned)
2.186
.381
.128
.829
0.831
.649
.424
.478
.446
.614
FIGARO
1.782
.977
.144
.988
0.335
.807
.702
.694
.748
.744
(b) Description splicing perplexity and similarity metries. Best values are highlighted.
Fluency
Accuracy
Fidelity
Model
PPL. +
If
C
TS f
ND 4
P
V
1
D
8c
8g
FIGARO (expert)
1.894
.955
.553
.996
0.360
.700
.646
.434
.710
.639
w/o instruments
1,980
.373
.568
1.00
0,424
.674
.586
.436
.687
.625
w/o chords
2.023
.855
.100
.955
0.564
.672
.603
.413
.294
.615
w/o meta info.
1.966
.908
.536
.795
0.878
.574
.205
.334
.636
.584
(c) Ablation study perplexity and similarity metrics. Best/worst values are hightighecd/emhhasize..
Table 3: We compare our models to the unconditional baseline based on Huang et al. (2018), (Choi et al. 2020) and MuseMorphose (Wu & Yang 202T) perplexity (PPL) and similarity metrics. Similarity metrics include instrument Fi-score (I), chord Fi-score (C) and time signature accuracy (TS) well as note density NRMSE (ND), pitch MOA (P), velocity MOA (V), duration MOA (D), chroma similarity 8, and grooving similarity 8g-
description for attributes including texture and rhythmic intensity ("high-fidelity" attributes). This is not surprising the expert description is designed make this kind of low-fidelity information readily available and the learned deseription intended to "fill in the gaps" the expert description. We invite the reader to examine how the models combines two pieces of music into one by listening to some of the samples on (SundCloodd We also provide evaluation metrics with respect to the two source samples in Appendix
6.4 ABLATION STUDY
To evaluate which parts of the expert description are essential, we group it into three components: instruments, chords and meta-information. Instruments and chords include all tokens with information about instruments and chords respectively while all other tokens (time signature, note density and mean pitch, velocity and duration) are classified as meta-tokens. We train separate models with one part of the description removed and compare the performance FIGARO (expert), which receives the full expert description as input.
As one would expect, removing each component reduces the performance significantly in the respective metrics, indicating that each component carries useful information not entirely inferable through the remaining components. Interestingly, our experiments show that removing any component slightly decreases the over-all performance to be affected. Removing instrument information, for example, increases the error for note density, pitch, velocity and duration, indicating that the instruments also carry mutual information about those features. This seems plausible considering the fact that different styles (or genres) music usually identifies set of instruments that is common for said style. Similar arguments can be made for the other two components. The full list of results is available in Table 3c
in metrics that we would not necessarily expect
6.5 SUBJECTIVE EVALUATION
Finally, we evaluate the subjective quality of generated samples through a listening study, comparing best model to the baselines. In this experiment we not interested in how closely the generated
8

---

Published as a conference paper at ICLR 2023
samples follows the prescribed condition, but instead try to quantify the perceived quality of samples by recording pairwise preferences. To this end, we have conducted a survey where participants were asked to indicate their preference between 20 second excerpts of two samples chosen uniformly at random In total, we gathered 7569 comparisons by 691 unique participants, whose expertise ranged everything between non-musicians and professional musicians, although no prior musical knowledge was required. In two types of questions, participants had to choose 1) between a real sample and generated sample or 2) between two generated samples. Generated samples are taken from the style transfer task. Question type 1) ranks the different methods on how good generated samples are compared to real, human- composed music. In this respect, FIGARO beats all other baseline with win rate of 39.3% compared to the next best model by CCoi et al.] (2020), which has win rate of 33. 2%. As evidenced by the fact that the unconditional baseline (Huang et al. 2018) and Choi et al. (2020) are very close in terms of win rates, sample quality and reconstruction accuracy are not necessarily correlated, highlighting the need for a dedicated evaluation of sample quality. Win rates of the differrntt models are displayed in Figure 4 with 90% confidence intervals obtained through normal approximation. While(Chor et al.
(2020) is able to adhere to the preseribed con- dition as shown in Section 6.2] the quality of generated samples is approximately on par with
Choi
the unconditional model. FIGARO on the other
hand is able to surpass the unconditional base-
Huangster
line in sample quality, while also providing con- trollable generation capabilities. Question type 2) is used to construct a pairwise ranking of mod- els by applying the Wilcoxon signed-rank test on the study results. In this ranking, our model beats each of the baselines with a p-value of < 10 7. The complete ranking and test results can be found in Appendix G
& Yrng
Winnlt Inth
Figure 4: Win rates of generated samples against real samples. We compare FIGARO, Huang et al. (2018), Choi et al. (2020) and Wu & Yang (2021). Real samples are from the test set.
7 DISCUSSION
Future Work. While our work represents step towards high-quality symbolic music generation with cuman-interpretable control, we recognize three main avenues for future work. Firstly, the proposed model is strictly speaking not generative model as creative input in the form of description is still required. Training a prior model to generate such descriptions unconditionally could allow for faster generation with less human input required while still retaining the ability influence the generation process in description space. Secondly, adding new description functions or extending the proposed ones could lead to further improved controllability and sample quality. Melody conditioning stands out as useful attribute still lacking control in FIGARO. Lastly, applying the description to-sequence objective to other domains such as natural language could potentially enable human-intery eettreetterreeeable control in a way that is not currently possible. For example, the long-range conditioning ability could be interesting in the setting of story generation.
We present the self-supervised text of symbolic music generation using expert and learned features. We train and release FIGARO, which combines two description functions in model that achieves human- -nttrr-ntterretablle fine-grained control on multi-track, multi-time- dime-sinnature music while beating state-of-the-as controllable genera- tion models both in terms of sample quality and style transfer. The proposed description functions complement each other: The expert description is a human-interpretable sequence that is easy to create and acts as a strong inductive bias to the model. The learned description is able mitigate non-injectivity present in the expert description and amend it with high-fidelity detailed information. In the case of conflicting deseriptions, the model still produces plausible samples, thus preserving even in the presence of learned features. We can combine both descriptions to achieve state-of -o-tte----h---rr symbolic music generation with iamaannnterpretable, fine-grained control. On a broader perspective, we hope that the proposed method is step toward facilitating artists in their creative process as well as enabling amateurs to express themselves by lowering the barrier of entry to music creation and making the process faster and easier overall.
Conclusion.
wttptiionnnt-suuunneeee objective and apply it in the con-
Samples used in the study are available for download: hamp::/hinyurllcoom//sss22z

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Automatic music generation may raise ethical concerns similar to those of large language models. The models may exhibit biases toward a certain style of music and may not represent the music of marginalized cultures accurately. In our case, the majority of training samples are western music, which is not necessarily representative of music at large. The model might also reproduce copyrighted material that is present in the training data and potentially generate samples that infringe on copyright law. Additionally, powerful music model capable of competing with human composers have the potential to create a big impact on the music industry. In this regard, we believe that our contribution is step toward human-Al collaboration rather than competition.
REFERENCES
Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascandolo. Neural symbolic regression that scales. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 936-945. PMLR, 18-24 Jul 2021. URL
mlr. wttps///1l/..ggiooo.2.....
html
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vons Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Gino Brunner, Andres Konrad, Yuyi Wang, and Roger Wattenhofer. MIDI VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer. In arXiv: http:::/000000000 [cs, eess, star), September 2018.
abtpg///////// 1tt.//1899.0 07600
arXiv: 809.07600.
Kristy Choi, Curtis Hawthorne, Ian Simon, Monica Dinculescu, and Jesse Engel. Encoding Musical Style with Transformer Autoencoders. In Proceedings of the 37th International Conference on Machine Learning, pp. 1899-1908. PMLR, November 2020. URL httlttts//pprceeddnng
mlr. wwtpp//wssee//eeeeeeeeeeeeeeeeee
html
ISSN: 2640-3498.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:/810.04805 [cs], May 2019, URL arXiv: 1810.04805.
http://arxiv. org/ abs/ 04805
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A Generative Model for Music. arXiv:2005.003: 00341 [cs, eess, stat], April 2020. URL
http: org/ abs / 2005 00341
arXiv: 2005.00341.
Shangzhe Di, Zeren Jiang, Si Liu, Zhaokai Wang, Leyan Zhu, Zexin He, Hongming Liu, and Shuicheng Yan. Video Background Music Generation with Controllable Music Transformer. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 2037-2045, October 2021. URL https: doi. org 11w.1....1111010000 01114//444488.337555999 3475195
Adji B. Dieng, Yoon Kim, Alexander M. Rush, and David M. Blei. Avoiding latent variable collapse with generative skip models, 2018. URL https::////// /ttps:://aaxivvi atwpvvivvvv.////.i.//eaaaa/..vtv wwt.:bs//aaaaaceooooo 180708100000 04863
Simon D Fabien Gouyon, and Gerhard Widmer. Towards Characterisation of Music via Rhythmic Patterns. ISMIR, 2004.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16 Words: Transformers for Image Recognition Scale. arXiv:2010. 19999 2010.11929.
[cs], June 2021. URL /00.1
arXiv:
Jeff Ens and Philippe Pasquier. MMM Exploring Conditional Multi-Track Music Generation with the Transformer. arXiv:2008, 06048 [cs], August 2020. URL htttp://axiiv. http///xix.oooo/a///// arXiv:
2008. 06048
10

---

Published as a conference paper at ICLR 2023
Lucas N. Ferreira and Jim Whitehead. Learning to Generate Music With Sentiment. wrXiv:2133001122 06122 arXiv: 2103.06125.
[cs, eess], March 2021. URL http:: /tt//aaxivi..
http://wwww...rraa///iaaa// /
Takuya Fujishima. Real-time chord recognition of musical sound: A system using common lisp music. Proc. ICMC, Oct. 1999, pp. 464 367, 1999.
Carlos Hermandez-Olivan and Jose R. Beltran. Music Composition with Deep Learning: A Re- view. ar0iv:2108. 12290 [cs, eess), September 2021.
12290 arXiv: 2108.12299
Wen- Yi Hsiao, Jen- Yu Liu, Yin-Cheng Yeh, and Yi-Hsuan Yang. Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs. arXiv:2101 .02402 [cs, eess]. January 2021.
org/ att///aa/21111 02402 arXiv: 2101.02402
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music Transformer. arXiv: 04281 [cs, eess, stat], December 2018. URL http::/aaxti..rrtttt
abs/1809. 04281
arXiv: 1809.04281.
Yu-Siang Huang and Yi-Hsuan Yang. Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions. arXiv:2002.002// [cs, eess, stat], August 2020. URL 00212
http:///axxxv.org//ee/
arXiv: 2002.00212.
Zhiheng Huang, Davis Liang Peng Xu, and Bing Xiang. Improve Transformer Models with Better Relative Position Embeddings. arXiv:2009, 33658 [cs]. September 2020. URL |RLhttp
www.iv.crg/ab////0///.. 2009, 13658 arXiv: 2009.13658.
Lukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Parmar, Samy Bengio, Jakob Uszkoreit, and Noam Shazeer. Fast Decoding Sequence Models using Discrete Latent Variables. arXiv:/803.0338 [cs], June 2018. URL
abs 08033 03382
arXiv: 1803.03382.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs], January 2017. URL httt:: wttp://www....r/////1.22.... 6980 arXiv: 1412.6980.
Guillaume Lample and François Charton. Deep learning for symbolic mathematics, 2019.
Gerhard Nierhaus. Historical Development of Algorithmic Procedures. In Algorithmic Composition: Paradigms of Automated Music Generation, pp- 7-66. Springer, Vienna, 2009. ISBN 978-3- 211-75540-2.
doi: 10. 00010077978-3-211-555------ URL https:
pittps://do:: http://////oo.../0..0//0..0
978 3- -
Aaron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learning. arXiv:1711.009) [cs]. May 2018. URL/att:://rxxv.oo wttpi//.rrgaaab//////aa/a
abs /1711, 00937
arXiv: 1711.00937.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. arXiv: aee:/800..377 [cs, star], January 2019. http:://x/// org arXiv: 1807.03748.
abs/ 03748
Christine Payne. MuseNet, April 2019. UtLhttts:///oeett wot.//o/////...... http://ewi..com/booo/mmmeeeeeeee
Colin Raffel. Learning-Bon Based Methods for Comparing Sequences, with Applications to Audio-to- MIDI Alignment and Matching. PhD thesis, Columbia University, 2016.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. URL URLhttps:: RRlltttts:///rxivv http::///axvvvvooeee /
06125
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistie text-to-image diffusion models with deep language understanding, 2022. URL bttpt::: arxiv. wttp/i...or//........
22055 1665.11887 11487
11

---

Published as a conference paper at ICLR 2023
Hao Hao Tan and Dorien Herremans. Music fademnes:: Controllable music generation based on high-level features via low-level feature modelling. 2020. doi: 10.4550AAKXY.200.14447.
Utttps::://axxi org/ abw/aaa...222....... 154744 14474
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, pp. 1998-6008. Curran Associates, Inc., 2017.
Guowei Wu, Shipei Liu, and Xiaoya Fan. The power of fragmentation: A hierarchical transformer model for structural segmentation in symbolic music generation, 2022. httph:ttt//aaa..
org/abs/2200 /205. 08579
Shih-Lun Wu and Yi-Hsuan Yang. MuseMorphose: Full-Song and Fine-Grained Music Style
Transfer with Just One Transformer VAE. arXiv:2105.04000 04090 [cs, eess), May 2021. htth abarg//ba///ee //arxiv. org/ abs 22100 04090
arXiv: 2105,04090.
12

---

Published as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
A.1 DESCRIPTION- TO-SEQUENCE MODEL.
We largely follow the original paper for both of our Transformer auto-encoders. Unless specified differently, hyperparameters are left unchanged. One exception is the reduced context size of 256 tokens for the sake of training time. In the same spirit, we also reduce the number of encoder layers to 4 but leave the number of decoder layers at 6. We use relative positional embeddings (Huang al. 2020 this has been shown be beneficial for symbolic music generation (Huang al. 2018). In addition to the positional embeddings, we also add learned bar embedding and learned beat-position embedding. This puts limit on the maximum number of bars (512 in our case). With a 4/4 time signature at 120 BPM this equates to 17 minutes of music, which we deem an acceptable limitation. In the case of using both descriptions, we simply add their token embeddings before model has 44.6 M trainable parameters.
passing it to the encoder. In total, the description-to-sequuee
In the presence the learned description Flearned, the discrete latent codes from the VQ-VAE feature extractor are embedded using the frozen codebook from the VQ-VAE instead of re-learning new embeddings. This done to ensure stable training and reduce the number of trainable parameters.
A.2 VQ- VAE MODEL
For our bar-level VQ-VAE model, we use the same Transformer auto-encoder backbone for the description-to-seqquene model. The final layer of the encoder is pooled as proposed by Devlin et al. 2019) and then fed through vector-quantizatie from a codebook of size 2048. We use a modified version of the sliced vector quantization scheme proposed by Kaiser et al. (2018) as our discretization bottleneck. Specifically, we decompose the latent representation z into 16 slices Z1...., Z16 and discretize each of them to a shared codebook with |C| 2048 using the k-means discretization technique from the original paper (Oord et al.) 2018). The latent vector is provided to the decoder through cross-attention. We use a linear layer before and after vector- quantt-quanizztioonn to project between model space and latent space. The VQ-VAE model has 43.7 M trainable parameters in total.
bottleneck, quantizing it to 16 latent codes
The model trained on bar-level reconstruction by minimizing the canonical B-VQ-VAE loss without the auxiliary codebook loss (Oord et al. 2019) given by
LvQ-VAE( = Ex-x log (x (x)) + 31|=(x) - (x)2,(x)||I
where sg(x) denotes the stop-gradient operator, ø are the model parameters and B 0.02. The codebook is updated using the EMA update step as proposed in the original paper. We employ random restarts (Dhariwal et al., 2020)) ensure optimal codebook usage. Bars with more tokens than the context size are truncated to fit.
A.3 TRAINING DETAILS
We train each model for 100k steps with batch size of 512 sequences. Models are optimized using the Adam optimizer (Kingma & Ba 2017) with 31 0.9, 82 = 0.999, e 10-6 and 0.01 weight decay. We use the swverse-square-ooo learning rate schedule with initial constant warmup at 10-4 given by 10 10-4 //mx((1 / ///N) where N 4000 is the number of warmup steps. For additional details and hyperparameters, we refer to the source code.
B REMI+ INPUT REPRESENTATION
We extend the original REMI representation (Huang Yang 2020) to make suitable for general multi-track, multi-signature symbolic music sequences. We make modifications that add time signature and instrument information, we determine a unique order of events and use quantization schemes that allow for accurate representation for a diverse set of music. An example of REMI+ sequence is given in Figure 5
Bar Tokens. To provide the model with additional context, we include the index of the current bar in each bar token. This, along with the bar embedding, should help the model retrieve the correct information from the description and help determine the end of piece generation time.
13

---

Published as a conference paper at ICLR 2023
Time signature. We add time signature token at the beginning of each bar, indicating the time signature of the bar which is to follow. We adapt the convention that time signature changes may only happen at the beginning of bar, which is commonly true in written music.
Instruments. We add instrument information as an additional token before each note event, indicating which instrument will play the following note.
Order of events. theory, the order notes within each bar can be arbitrary without compromising the validity of the sequence. But to make the modelling task easier, define unique and determin- istic order of events. Specifically, we sort events by (Bar, Position, EventTyppe, Instrument, Pitch) in ascending order (valid event types are (Chord, Tempo, Note)). This order is unique since given instrument can only ever play a single note with given pitch at a given time
quantization. The most significant deviation is the use of 12 note onset positions per quarter note instead of 4 as proposed in the original work. For example, there will be 48 unique note onset positions for the 4/4 time signature and 36 note onset positions for the 3/4 time signature. This allows both triplet and sixteenth notes to be quantized accurately, which is important when considering a diverse set of music.
Quantization. We largely follow Huang & Yang (2020)
Instruments and note pitches are not quantized as they are categorical variables with 128 possible values by the MIDI specification. Note velocity quantized to 32 intervals in [0, 128) and note duration is quantized to position intervals defined by the following mesh:
M = f2, U
{12 3i ie li (I.... 4)}
3)] {12+4i 31) U
624+6) / [])...,,))4
[48 12i iE 12).. U
[192 + 24i | (1,
This ensures single position accuracy up quarter notes, then 16th and triplet accuracy up to half notes and 8th note accuracy up to a full note. To limit vocabulary size, we switch to quarter note steps up full notes and half note steps up 16 full notes after that. Notes longer than full notes are truncated to this length. Finally, tempo change events are discretized to 32 intervals in [0, 240).
<bos>>
TimeSignature
Pos_0 Tempo 120
Pos.O Chord.C:min
Pos.O Instrument. Pitch_36 Vel.90 Dur_0
Pas_O Instrument_Ping
Pitch_64 Vel_85 Dur_4
Pos_4 Instrument_Piano Pitch_66 Vel_85 Dur_4
Bar_2 TimeSignature.3/4
Pas Tempo 120
<eos>
Figure 5: Example sequence represented in the REMI+ representation. At the beginning of each bar time signature, tempo and the current chord are noted, after which each note is represented through five subsequent tokens.
C EXPERT DESCRIPTION ALGORITHM
Pseudocode for generating the expert description is given in Algorithm I We quantize note density,
mean pitch and velocity to 32 linearly spaced intervals in [0, 12). [0, 128 and [0, 128) respectively. Mean duration is quantized to 32 logarithmically spaced intervals in [o, 128) positions (12 positions per quarter note). Chords are extracted with an adapted version of the Viterbi algorithm also used by Huang & Yang (200)).
This is assumption that could be violated in theory but does hold in practice.
14

---

Published as a conference paper at ICLR 2023
Algorithm 1 ExpertDescription
input musical sequence X output description
d <-<
bi...., bn < PARTIIINNLYTOBBRSS
for b, E (b1...., bn) do
f- {n n is note with onset in b;)
{ finst inst is being played during b:)
C + {chord chord is being played during b
q f duration of in quarter notes
ts < time signature at beginning of b;
nd <- [
I
mp < INT
Σ
EREN PITCH(n)
mv [N] NEN VELOCITY(1)
1 md < [N] EnEN DURATION(1)
Quantize nd,, mp, mv and md
d: - (i, s, nd, mp, mv, md) list(Z) || list(C)
d d || d
end for return d
D EVALUATION METRICS
D.I MACRO OVERLAPPING AREA
As used by previous work, the overlapping area (OA) metric does not consider the sequential order of the investigated feature, as feature histograms are computed over the entire sequence. For example, the overlapping area of x and reverse(x) would be maximal, even though the reversed sequence does not sound like the original sequence in general.
To alleviate this limitation, adapt the OA metric to also consider temporal order. Let and y denote two musical sequences and let b.x) and b5²²) denote the i-th bar of x and y respectively. We compute the overlap in feature distributions for each bar by fitting a Gaussian distribution to the feature under examination (e.g. note pitch) and compute the overlapping area between the two distributions. Let this overlap be given by overlap(b b())). Then the macro overlapping area (MOA) between x and y is given by
MOA(x,y) = N Σ overlap (6/06660))
D.2 NORMALIZED ROOT-MEAN SQUARE ERROR
In order to normalize for different feature magnitudes between different samples, compute the normalized RMSE (NRMSE) for bar-wise note density. Let x denote the ground truth, & denote the reconstruction and N denote the length of the sequences. Then the NRMSE is given by
N
RMSE (2,EE(,C0) 2) =
Σ
(G
)2)2
N
RMSE ( +)
NRMSE(2,
mean (z)
D.3 COSINE SIMILARITY
Let ((x) and v (y) denote the chroma vector (Fujishima 1999) or grooving vector (Dixon et al. 2004) for the i-th bar in and y respectively. We then average the cosine similarity over the entire sequence
15

---

Published as a conference paper at ICLR 2023
Accuracy
Fidelity
1
C1
TS
ND 4
P1
V
D 1
8g T
FIGARO
.600
.593
.997
.238
.827
.735
.748
.790
.853
Expert Similarity
.819
.351
.993
.453
.625
.354
.473
.595
.621
Learned Similarity
.288
.057
.753
.696
.699
.670
.623
.398
.741
Mean
.554
.004
.873
.55
.662
.512
.548
.497
.681
Best case
.819
.351
.993
.453
.699
.670
.623
.595
.741
Table 4: Description mixing performance on similarity metrics. "FIGARO" refers to the performance on unaltered descriptions. Expert/cerned similarity refers similarity of generated samples the sequence from which the expert/learned description was extracted from. Highlights indicate which of the two descriptions was more closely adhered in of the respective metric.
to get the chroma/grooving similarity:
(y) v
simy (x,y)
E BENCHMARK MODELS
E. HUANG AL. (2088)
We reimplement the model proposed by Huang al. 20188 8) as an unconditional baseline. We train the model on the REMI+ representation to allow for direct comparison to our method. We use the improved relative attention from Huang et al. 2020) eliminate any possible advantage arising from different attention mechanisms. We largely stick to the hyperparameters from our models, using 6 decoder layers with hidden size of 512 and filter size of 2048. Training and optimization hyperparameters are also the same as for our models.
E.2
CHOI ET AL. (2020)
We reimplement the model proposed by Choi et al. (2020) and train it the REMI+ representation to allow for direct comparison to our method. We again use the improved relative attention from Huang et al. (2020) and largely stick to the hyperparameters from the original paper, using 6 encoder and 6 decoder layers. Unlike the original work, we do not use data augmentation since the dataset is large enough and in order allow for fair comparison between the models. Due to GPU memory constraints we reduce the context size from 2048 to 1024 and use an accumulated batch size of 16, ensuring stable training. Training and optimization hyperparameters are the same as for our models.
E.3
WU & YANG (201
We train MuseMorphose the REMI+ representation to allow for direct comparison method. Adapting the released implementation for our experiments, we reduce the context size from 1280 to 512 tokens due GPU memory constraints but leave all other hyperparameters as they were proposed in the original paper. The model is trained until convergence (approx. 125k steps). We limit the training data subset of the entire dataset (20k samples) due technical limitations. This is still considerably more training data than what was used in the original paper (Ik samples) and should not affect performance significantly compared using the full dataset.
F DESCRIPTION MIXING
Quantitative evaluation for this task is not straight-forwart there are two different "ground truth" samples with potentially conflicting features and it is unclear, which one the model should adhere to for which features. It is clear though that the model follows one of the descriptions more closely for some features, which points to some degree of separation concerns between the two descriptions. Exact numbers provided in Table
16

---

Published as a conference paper at ICLR 2023
Model
Opponent
Winrate
p-value
1. Ground truth
0.669
2418
FIGARO
0.605
10-6
595
Huang et al. (2018)
0.663
< 10
605
Choi al. (2020)
0.647
10-12
586
Wu & Yang (2021)
0.756
10
632
2. FIGARO
0,581
2439
Huang et al.
2018
0.609
10-7
635
Choi al. (2020)
0.624
10-9
631
Wu & Yang
202
0.696
10 20
578
3. Huang et al. (0018)
0.463
2467
Choi al. 2020
0.543
0.0162
613
Wu & Yang (2021
0.581
10
614
4. Choi et al. 2020)
0.447
2490
Wu & Yang (2021)
0.589
10-5
660
5. Wu & Yang (021)
0.345
2484
Table 5: Ranking of the different methods by winrate according to the user study. The Wilcoxon signed-rank test used to calculate pairwise ranking p-values. "Ground truth" denotes sampling the test set.
G SUBJECTIVE EVALUATION
The listening study includes 7569 comparisons by 691 participants, each averaging 11 answers. In each comparison, participants were presented with two different samples and were asked to indicate, which the two they preferred. For the following ranking test, samples were chosen uniformly at random from two different generative models, the pair of which again was chosen at random. In this setup, we treat real samples as one possible model, for which we simply sample the test distribution. For the other models, generate samples as described for the style transfer experiment (Section 6.2) Screenshots from the study are presented in Figure (
We apply the Wilcoxon signed-rank test to the results in order to establish ranking of the different models. FIGARO beats each baseline except ground truth with a win rate of more than 60% and p-value of less than 10-7 Out of all methods, FIGARO also has the highest win rate against real samples with win rate of 39.3%, which is 6% advantage over the next best method. Rankings and corresponding p-values reported in Table
17

---

Published as a conference paper at ICLR 2023
Survey on Computer-Generated Music
Thank you for participating in our survey about computee--generated music. The goal of this survey is compare different algorithms (artificial intelligence) for generating music. The survey consists 15 questions total and takes about 10 minutes to complete. Each of the questions will require you to listen to compuren-generated samples of music (20 seconds) and indicate which of the two you prefer. As the samples are computer-gen- erated, the soundscape will not be as you're used to, but-by its computer-generated na- ture-synthetic.
You will not have to write reason for choosing one sample the other, you can sim- ply press button.
Choosing your Preference
Think of yourself as jury in composition contest where you have to decide which of the two pieces, generated by different algorithms, going to win.
Instructions
You are going to be presented with two samples: "Sample A and "Sample B". you prefer the first sample, click prefer A". If you prefer the second sample, click prefer B".
For best results, please this survey in quiet environment and using headphones.
(a) Instructions that were provided to the participants of the listening study.
Listen to these two samples:
Sample
Sample
0:00
0:20
0:00
0:20
Which sample do you prefer?
prefer A
prefer
(b) Screenshot of the survey questions that were provided to the study participants after reading the instructions.
Figure 6: Screenshots of the listening study. Participants were first presented with the instructions (a) before answering 15 questions of the type presented (b).
18


Reviews:
[{"review_id": "j5WtXw8Gs2", "reviewer": "Reviewer_GHu3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a system for training a multi-instrument symbolic music generation model conditioned on two types of features: human-interpretable attributes that can be extracted programmatically, and latent VQ-VAE encodings that are learned.  Both feature types can be used to exert fine-grained control over a piece of generated music, as demonstrated in several experiments.", "summary_of_the_review": "The paper seems correct and is a valid research contribution to the area of music generation.  I sincerely hope the authors (or someone else) writes another paper that explores the HCI questions relevant to the system presented in this paper.", "strength_and_weaknesses": "Strengths\n-----------\n1) The paper does exactly what it claims: proposes a system for controlling multi-instrument music generation with human-interpretable and learned latent features, and evaluates the technical aspects of the system.\n\n2) The samples on SoundCloud sound quite reasonable.\n\nWeaknesses\n--------------\n1) Far and away the biggest thing missing here, even though the authors may consider it out of scope for ICLR, is some kind of user study demonstrating that this method of controlling a music generation system is helpful to musicians or composers, expert or novice.  The experiments in the paper demonstrate that the system *obeys* the controls and produces plausible music, but not that it is usable.  And based on the baseline models, it isn't surprising that the model would obey the conditioning.  Much of the effectiveness of such a system seems like it would depend on exactly what attributes are used for expert conditioning, and this question isn't explored beyond measuring the impact of dropping out individual attributes, nor is there discussion of why these attributes were chosen.\n\n2) Two major weaknesses of the system that I suspect limit its real-world utility are its lack of support for a) infilling and b) missing controls.  Even though the conditioning signal is temporally fine-grained, a change to the conditioning at time T1 may have effects at time T2 distant from T1, which is likely not what users want.  And having to provide two types of conditioning values (one with multiple attributes) at all positions in a piece of music is likely to be very cumbersome for users.\n\n3) Minor issues:\n    * The paper mentions that an online demo is available; however no link is provided.\n    * \"GitHub\" is misspelled in the footnote on page 2.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "j1TsuBeL5eS", "reviewer": "Reviewer_g33b", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "FIGARO is a model trained on a seq2seq task, where the input is either a human-interpretable high-level description of the music or a learned sequence of VQ-VAE tokens. Output is a symbolic music composition in the proposed REMI+ format, an extension to REMI that adds multi-instrument and multi-time signature capabilities. The authors show via several experiments and metrics that the produced music reflects the inputs given to the model.", "summary_of_the_review": "FIGARO provides a way to condition symbolic music generation on several per-bar controls, and experiments show that these controls are successful. However, the use case is not completely clear.", "strength_and_weaknesses": "**Strengths**\n\n- The experiments and metrics are convincing that the model does utilize the inputs when constructing the output music.\n- The listening study indicates that the output music is of an overall high quality (or at least as high as the training dataset).\n- Code and weights are available for download.\n\n**Weaknesses**\n\nMy main concern is that the motivation for the model is somewhat unclear. The authors say that providing the expert description is easier than actually writing the music, but it seems to be only marginally easier. You still have to annotate every bar and include attributes that aren't necessarily musically intuitive, like note density and mean pitch. I could perhaps see constructing an UI that makes these attributes more intuitive. The authors do mention some of these issues in the Discussion section, but it would have been nice to see a more clear path toward direct usability.\n\nI also don't understand the motivation for the learned representation at all. It doesn't seem to be suitable for human input, so I guess another model would be necessary to translate between another representation and the learned tokens?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Pqueo_fQu_", "reviewer": "Reviewer_W9Fj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a music generation approach which aims to extend the type of conditioning variables typically used in this area of research. The authors describe this as so-called description-to-sequence learning. Inspired by recent progress in text-conditioned image generation, the authors propose to derive descriptions of music in terms of musical attributes which can be easily extracted from the data itself. These descriptions are termed as expert descriptions. In addition, to fill in the gaps in the expert descriptions per bar, they also propose to learn descriptions using a VQ-VAE. Combining the two: expert and learned descriptions, the authors learn a sequence to sequence model transforming said descriptions to musical sequences. The proposed method is compared against competing methods using both objective and subjective evaluations. In all cases, FIGARO outperforms the baseline approaches. The authors also carry out various ablation studies showing the benefit of different types of expert descriptions.", "summary_of_the_review": "My overall impression of this paper is that it presents a simple yet effective method for controllable music generation. My only issue is that it feels a little incomplete and does not extend existing literature to the extent I would have hoped. Despite that, the results sound convincing and the user study shows that this method outperforms some competing approaches. Thus I am rating it as a weak accept.", "strength_and_weaknesses": "Strengths:\n- The proposed method is elegant in combining domain knowledge of music to extract attributes along with a VQ-VAE to learn latent descriptors of measures which are fed to a transformer encoder-decoder architecture to generate musical sequences.\n- The approach extends the fine-grained control of previous approaches to more musical dimensions thus affording more control in the output.\n- The results, both qualitative and quantitative look and sound good.\n\nWeakness:\n- The contributions seem a bit incremental. While it's true that this has not been done before in symbolic music, the setup is pretty similar to Jukebox with the addition of some extra musical attributes as conditioning minus the prior to generate conditioning latents/descriptions. In fact, the absence of the prior feels like a big weakness, but this is left as future work by the authors.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4O3F9pHRXEU", "reviewer": "Reviewer_itPu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a transformer-based conditional model to generate multitrack symbolic music. The model utilizes a \"description to sequence\" learning scheme. The authors show that the model produces more desirable musical sequences when human-interpretable features are incorporated during training.\n\n", "summary_of_the_review": "The work advances the state-of-the-art. The usage of learned and expert descriptions is sound. The experiments are comprehensive, and the dataset is large and representative enough (for music information research). The authors also extend the existing data representation in existing datasets. Given these strengths, I would like to recommend the paper for acceptance.\n", "strength_and_weaknesses": "Strengths:\n\n- Novel training objective\n- The model can handle multi-track, multi-signature symbolic music\n- Comparative experiments between strong baselines in different settings with respect to numerous evaluation measures\n\nThere are a few cosmetic issues, which are explained below.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster