Simple and Controllable Music Generation
Authors: Jade Copet
Felix Kreuk
Itai Gat
Tal Remez
David Kant
Gabriel Synnaeve
Yossi Adi
Alexandre Défossez
Venue: NeurIPS 2023 poster (2023)
Keywords: Music generation
Generative AI
Transformer
Language Models
TL;DR: MusicGen is simple, efficient, and controllable music generation, that can be conditioned on text and melody.

Abstract: We tackle the task of conditional music generation. We introduce MusicGen, a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for cascading several models, e.g., hierarchically or upsampling. Following this approach, we demonstrate how MusicGen can generate high-quality samples, both mono and stereo, while being conditioned on textual description or melodic features, allowing better controls over the generated output. We conduct extensive empirical evaluation, considering both automatic and human studies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark. Through ablation studies, we shed light over the importance of each of the components comprising MusicGen. Music samples, code, and models are available at https://github.com/facebookresearch/audiocraft

Simple and Controllable Music Generation
Jade Copet
Felix Kreuk
Itai Gat
Tal Remez
David Kant
Gabriel Synnaeve 0
Yossi Adi
Alexandre Défossez 9
equal contributions, C core Meta AI
{jadecopet, felixkreuk, adiyoss>@meta,coo wwwee.ma...mmm
Abstract
We tackle the task conditional music generation. We introduce MUSICGEN, sin- gle Language Model (LM) that operates over several streams of compressed discrete music representation, i.c., tokens. Unlike prior work. MUSICGEN is comprised single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for cascading several models, e.g., hierarchically or up- sampling. Following this approach, demonstrate how MUSICGEN can generate high-quality samples, both mono and stereo, while being conditioned on textual description melodic features, allowing better controls over the generated output. We conduct extensive empirical evaluation, considering both automatic and human studies, showing the proposed approach is superior the evaluated baselines standard text-to-music benchmark. Through ablation studies, we shed light over the importance of each of the components comprising MUSICGEN. Music samples,
code, and models are available at
1 Introduction
Text-to-music is the task of generating musical pieces given text descriptions, e.g., "90s rock song with guitar riff". Generating music challenging task as it requires modeling long range sequences. Unlike speech, music requires the use of the full frequency spectrum [Müller, 2015]. That means sampling the signal at a higher rate, i.c., the standard sampling rates of music recordings are 44.1 kHz or 48 kHz vs. 16 kHz for speech. Moreover, music contains harmonies and melodies from different instruments, which create complex structures. Human listeners are highly sensitive to disharmony [Fedorenko et al., 2012, Norman-Haignan et al., 2019], hence generating music does not leave lot of room for making melodic errors. Lastly, the ability control the generation process in a diverse set of methods, e.g., key, instruments, melody, genre, etc. is essential for music creators.
Recent advances self-supervised audio representation learning [Balestriero et al., 2023], sequential modeling [Touvron et al., 2023], and audio synthesis [TTan et al., 2021] provide the conditions to develop such models. To make audio modeling more tractable, recent studies proposed representing audio signals as multiple streams of discrete tokens representing the same signal [Défossez et al., 2022]. This allows both high-quality audio generation and effective audio modeling. However, this comes at the cost of jointly modeling several parallel dependent streams.
Kharitonov et al. [2022], Kreuk al. [2022] proposed modeling nulti-streams of speech tokens in parallel following a delay approach, i.c., introduce offsets between the different streams. Agostinelli al. [2023] proposed representing musical segments using multiple sequences of discrete tokens different granularity and model them using a hierarchy of autoregressive models. In parallel, Donahue al. [2023] follows a similar approach but for the task of singing accompaniment generation. Recently, Wang et al. [2023] proposed tackling this problem in two stages: (i) modeling the first
Yossi Adi Affiliated with The Hebrew University of Jerusalem MetaAI.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Flatten ng Patzern
Para Parallel Pattern
Residial codebaoks
Residual codehooks
@
4
4
6
0
's
Max-]
Sequence steps
Sequence steps
Coarse First Pattern
Delay Pattern
Residial codebooks
Residhal codebooks
Q
0
3 f
%
SA
tu
5
5
Foll Sail Pass A... Stout Fin
41
Sad %
Sequence steps
Sequence steps
Figure 1: Codebook interleaving patterns presented in Section 2.2. Each time step t,, t2, by is composed of 4 quantized values (corresponding to .i,..., 1ay...,,44.. k4). When doing autoregressive modelling. we can flatten or interleave them in various ways, resulting in a new sequence with 4 parallel streams and steps $1 82, ....... 8mn The total number of sequence steps S depends the pattern and original number of steps T. 0 is a special token indicating empty positions in the pattern.
stream of tokens only; (ii) then, apply a post-network to jointly model the rest of the streams in non-autoregressive manner.
In this work, introduce MUSICGEN, simple and controllable music generation model, which able to generate high-quality music given textual description. We propose general framework for modeling multiple parallel streams of acoustic tokens, which serves generalization of previous studies (see Figure 1). We show how this framework allows to extend generation to stereo audio at no extra computational cost. To improve controllability of the generated samples, we additionally introduce unsupervised melody conditioning, which allows the model to generate music that matches given harmonic and melodic structure. We conduct an extensive evaluation of MUSICGEN and show the proposed method is superior to the evaluated baselines by large margin, with a subjective rating of 84.8 out of 100 for MUSICGEN against 80.5 for the best baseline. We additionally provide an ablation study which sheds light on the importance of each of the components on the overall model performance. Lastly, human evaluation suggests that MUSICGEN yields high quality samples which are better melodically aligned with given harmonic structure, while adhering to textual description.
Our contribution: (i) We introduce simple and efficient model to generate high quality music 32 kHz. We show that MUSICGEN can generate consistent music with a single-stage language model through an efficient codebook interleaving strategy. (ii) We present a single model perform both text and the provided melody and faithful to the text conditioning information. (iii) We provide extensive objective and human evaluations on the key design choices behind our method.
generation and demonstrate that the generated audio is coherent with
2 Method
MUSICGEN consists in an autoregressive transformer-based decoder [Vaswani et al., 2017]. condi- tioned on text or melody representation. The (language) model is over the quantized units from an EnCodec [Défossez et al., 2022] audio tokenizer, which provides high fidelity reconstruction from low frame rate discrete representation. Compression models such as [Défossez al., 2022, Zeghidour et al., 2021] employ Residual Vector Quantization (RVQ) which results in several parallel streams. Under this setting, each stream comprised of discrete tokens originating from different learned codebooks. Prior work, proposed several modeling strategies to handle this issue [Kharitonov et al., 2022, Agostinelli et al., 2023, Wang et al., 2023]. this work, we introduce a novel modeling framework, which generalizes to various codebook interleaving patterns, and explore several

---

variants, Through patterns, we can leverage the internal structure of the quantized audio tokens. Finally, MUSICGEN supports conditional generation based on either text or melody.
2.1 Audio tokenization
We use EnCodec [Défossez et al., 2022], a convolutional auto-encoder with a latent space quantized using Residual Vector Quantization (RVQ) [Zeghidour et al., 2021], and an adversarial reconstruction loss. Given a reference audio random variable X E TRd- with d the audio duration and f. the sample rate, EnCodec encodes it into a continuous tensor with frame rate fr & fs. This representation is then quantized into Q E {I..... M)&f-XK, with K being the number of codebooks used in RVQ and M being the codebook size. Notice, after quantization we are left with K' parallel discrete tokens sequences, each of length T = d- fr. representing the audio sample. In RVQ, each quantizer encodes the quantization error left by the previous quantizer, thus quantized values for different codebooks are general not independent, and the first codebook the most important one.
2.2 Codebook interleaving patterns (see Figure 1)
Exact flattened autoregressive decomposition. An autoregressive model requires a discrete random sequence U E {I,...,M, MMS with S the sequence length. By convention, we will take Un = 0, a deterministic special token indicating the beginning of the sequence. We can then model the distribution
vt 0, Pt (Ul-1,.... Un) A PU Uo}
(1)
Let us build second sequence of random variables U using the - density p. e.g. define recursively Uo = 0, and for all t > O,
0, P Ü ...,, Un] = pp
Then, we immediately have that U and ú follow the same distribution. This means that if we can fit perfect estimate of with deep learning model, then we can fit exactly the distribution of U.
As stated before, the main issue with the representation Q we obtained from the EnCodec model is that there are K codebooks for each time step. One solution would be to flatten out Q, thus taking S d- f. K, e.g. first predicting the first codebook of the first time step. then the second codebook of the first time step. etc. Then, using eq. (1) and eq. (2), we could theoretically fit an exact model of the distribution of Q. The downside however is the increased complexity, with part of the gain coming from the lowest sample rate f. being lost.
More than one possible flattening exists, and not all the P. functions need to be estimated through single model. For instance, MusicLM [Agostinelli et al., 2023] uses two models, one modeling the flattened first K/ K/2 codebooks, and second one the other K/2 flattened codebooks, conditioned on the decision of the first model. In that case, the number of autoregressive steps is still df.- K.
Inexact autoregressive decomposition. Another possibility is to consider autoregressive de- composition, where some codebooks are predicted in parallel. For instance, let us define another sequence with Vo 0 and for all f (I...., T), le E [1....,K] K}. Vú, = Qr, When dropping the codebook index kt e.g. V., we mean the concatenation
all the codebooks at time
Pt,k (V--1,.... Vo) A P [V., Vi-,...... Vo).
(3)
Let's define again recursively Vo 0 and for all t O,
Vt 0, Vk.
Ve
Pr,k (Ve-1..... vo).
(4)
Unlike in (2), we no longer have in the general case that V follows the same distribution as V. even assuming we have access to the exact distribution pt,k- In fact, we would only have proper generative model for I, (V.,),, ) are independent conditionally on Vi-1,..., Vo. As increases, the errors will compound and the two distributions can grow further apart. Such decomposition inexact, but allows to keep the original frame rate which can considerably speed up training and inference, especially for long sequences.
3

---

Arbitrary codebook interleaving patterns. In order to experiment with various such decompositions, and measure exactly the impact of using an inexact decomposition, we introduce codebook interleav- ing patterns. Let us consider f2 = {(t, k) {1..... d. fr), k E (I..... }}} be the set of all pairs of time steps and codebook indexes. A codebook pattern is sequence P = (Po: P1. P2. ..... Ps), with Po = 9, and for all 0 < I S, Pa C Ω2 such that P is partition of f2. We model Q by predicting in parallel all the positions in Ps. conditionally on all the positions in Po. Pl..... Pa-1- Pragmatically, we restrict ourselves to patterns where each codebook index appears at most once in any of the Pa.
We can now easily define a number of decompositions, for instance the "parallel" pattern given by
P, {ss,k k (1....:K)) K}).
(5)
It is also possible to introduce a "delay" between the codebooks, as in Kharitonov et al. [2022], e.g..
Ps = ((s + k) k E (1,.... K]. k 2 0).
Through empirical evaluations, we show the benefits and drawbacks of various codebook patterns, shedding light on the importance of exact modeling of the parallel codebook sequences.
2.3 Model conditioning
Text conditioning, Given a textual description matching the input audio X, we compute a condi- tioning tensor C E TR with D being the inner dimension used in the autoregressive model. Generally, there are three main approaches for representing text for conditional audio generation.
Kreuk et al. [2022] proposed using a pretrained text encoder, specifically T5 [Raffel al., 2020].
Chung et al. [2022] show that using instruct-based language models provide superior performance. Lastly, Agostinelli et al. [2023], Liu et al. [2023], Huang et al. [2023a]. Sheffer and Adi [2023] claimed that joint text-audio representation, such as CLAP [Wu* et al., 2023], provides better-quality generations. We experiment with all of the above, respectively: T5 encoder, FLAN-T5, and CLAP.
Melody conditioning. While text is the prominent approach in conditional generative models nowadays, a more natural approach for music is conditioning on melodic structure from another audio track or even whistling or humming. Such an approach also allows for an iterative refinement of the model's output. To support that, we experiment with controlling the melodic structure via jointly conditioning on the input's 's chromagram and text description. In preliminary experiments, we observed that conditioning on the raw chromagram often led reconstructing the original sample, resulting in overfitting. To reduce it, we introduce an information bottleneck by choosing the dominant fume-frequency bin in each time step. While a similar capability was shown in Agostinelli al. [2023], the authors used supervised proprietary data, which is tedious and costly to collect. In this work, we take an unsupervised approach, elimínating the requirement for supervised data.
2.4 Model architecture
Codebook projection and positional embedding, Given a codebook pattern, only some codebooks present at each pattern step P.. We retrieve from Q the values corresponding to the indices in P.. As noted in Section 2.2, each codebook is present at most once in Ps or not at all. If it is present, we use learned embedding table with N entries and dimension D to represent the associated value from Q. Otherwise, we use a special token indicating its absence. We sum the contribution from each codebook after this transformation. As Po 0, the first input is always the sum of all the special tokens. Finally, we sum a sinusoidal embedding to encode the current step (Vaswani et al., 2017].
Transformer decoder. The input is fed into transformer with L. layers and dimension D. Each layer consists of a causal self-attention block. We then use a cross-attention block that is fed with the conditioning signal C. When using melody conditioning, we instead provide the conditioning tensor C' as a prefix to the transformer input. The layer ends with a fully connected block consisting of a linear layer from D to 4- channels, a ReLU, and linear layer back to D channels. The attention and fully connected blocks are wrapped with a residual skip connection. Layer normalization [Ba et al., 2016] is applied to each block before being summed with the residual skip connection "prre-nomm").
Logits prediction. The output from the transformer decoder at pattern step P. is transformed into logits prediction for the values of Q taken at the indices given by Ps+1- Each codebook is present at most once in P-+1- If codebook is present, the logits prediction is obtained by applying a codebook specific linear layer from D channels to N.

---

3 Experimental setup
3.1 Models and hyperparameters
Audio tokenization model. We use non-causal five layers EnCodec model for 32 kHz monophonic audio with a stride of 640, resulting in frame rate of 50 Hz, and an initial hidden size of 64, doubling at each of the model's five layers. The embeddings are quantized with a RVQ with four quantizers, each with a codebook size of 2048. We follow Défossez et al. [2022] to train the model on one-second audio segments cropped at random in the audio sequence.
Transformer model. We train autoregressive transformer models at different sizes: 300M, 1.5B, 3.3B parameters. We use a memory efficient Flash attention [Dao al., 2022] from the xFormers package [Lefaudeux et al., 2022] to improve both speed and memory usage with long sequences. We study the impact of the size of the model in Section 4. We use the porammereee model for all of our ablations. We train on 30-second audio crops sampled at random from the full track. We train the models for 1M steps with the Adam optimizer [Loshchilov and Hutter, 2017], batch size of 192 examples, 31 0.9, 32 = 0.95, a decoupled weight decay of 0.1 and gradient clipping of 1.0. We further rely on D-Adaptation based automatic step-sizes [Defazio and Mishchenko, 2023] for the 300M model as it improves model convergence but showed no gain for the bigger models. We use a cosine learning rate schedule with a warmup of 4000 steps. Additionally, we use an exponential moving average with decay of 0.99. We train the 300M, 1.5B and 3.3B parameter models, using respectively 32, 64 and 96 GPUs, with mixed precision. More specifically, we use float as bfloat 16 was leading instabilities in our setup. Finally, for sampling, we employ top-k sampling [Fan et al., 2018] with keeping the top 250 tokens and a temperature of 1.0.
Text preprocessing. Kreuk et al. [2022] proposed a text normalization scheme, in which stop words are omitted and the remaining text lemmatized. We denote this method by text-normalization When considering musical datasets, additional annotations tags such as musical key, tempo, type of instruments, etc. are often available. We also experiment with concatenating such annotations to the text description. We denote this approach by condition-merging. Finally, we explored using word dropout as another text augmentation strategy. For the final models, we used condition condition-merging with probability of 0.25. Upon merging, we apply text description dropout with probability of 0.5. We use a word dropout with a probability of 0.3 on the resulting text. A full comparison of the different text preprocessing strategies can be found in Appendix A.2.
Codebook patterns and conditioning. We use the "delay" interleaving pattern from Section 2.2, This translates 30 seconds of audio into 1500 autoregressive steps. For text conditioning. we use the T5 [Raffel et al., 2020] text encoder, optionally with the addition of the melody conditioning presented in Section 2.3. We also experiment with FLAN-T5 [Chung et al., 2022], and CLAP [Wu* et al., 2023] and compare the performance of MUSICGEN using each of these text encoders in the Appendix A.2. For melody conditioning, and a hop size of 212. Using a large window prevents the model from recovering fine temporal details. We further quantize the chromagram by taking the argmax at each time step. We follow similar approach to Kreuk et al. [2022] and implement classifier freeee guidance when sampling from the model's is logits. Specifically, during training we drop the condition with probability of 0.2 and during inference we use a guidance scale of 3.0.
compute the chromagrams with a window size of 214
3.2 Datasets
Training datasets. We use 20K hours of licensed music to train MUSICGEN. Specifically, we rely on an internal dataset of 10K high-quality music tracks, and on the ShutterStock and Pond5 music data collections" with respectively 25K and 365K instrument-only music tracks. All datasets consist of full-length music sampled at 32 kHz with metadata composed of a textual description and information such as the genre, BPM, and tags. We downmix the audio to mono unless stated otherwise.
Evaluation datasets. For the main results and comparison with prior work, evaluate the proposed method on the MusicCaps benchmark [Agostinelli et al., 2023]. MusicCaps is composed of 5.5K samples (ten-second long) prepared by expert musicians and 1K subset balanced across genres. We report objective metrics on the unbalanced set, while we sample examples from the genre- balancd
www.shutttrssocckcommmuus and www.pond5.com
5

---

Table 1: Text-to-Muse generation. We compare objective and subjective metries for MUSICGEN against a number of baselines. We report both mean and C195 scores. The Mousai model is retrained on the same dataset, while for MusicIM we use the public API for human studies. We report the original FAD on MusicCaps for Music and MusicIM. "MUSICGEN w. random melody" refers MUSICGEN trained with chromagram and text. At evaluation time, we sample the chromagrams at random from held-out set.
MUSICCAPS Test Set
MODEL
Fgg +
K1. 4
CLAP 1
OVL. f
REL. t
Riffusion
14.8
2.06
0.19
79.31
74.20
Mousai
7.5
1.59
0.23
76. 1..5
77.35
MusicIM
4.0
80.51
82.35
Music
2.1
MUSICGEN w.o melody (300M)
3.1
1.28
0.31
78.43
MUSICGEN melody (L.5B)
3.4
1.23
0.32
80.74
83.70
MUSICGEN w.o melody (3.3B)
3.8
1.22
0.31
84.81 10.9
82.47 1.25
MUSICGEN w. random melody (1.5B)
5.0
1.31
0.28
81.30+1299
81.98
set for qualitative evaluations. For melody evaluation and the ablation studies, we use samples from an in-domain held out evaluation set of 528 music tracks, with no artist overlap with the training set.
3.3 Evaluation
Baselines. We compare MUSICGEN to two baselines for text-to-music generation: Riffusion [Fors- gren and Martiros] and Mousai [Schneider al., 2023]. We the open source Riffusion model to run inference For Mousai, we train a model using our dataset for a fair comparison, using the open source implemennation provided by the authors'. Additionally, when possible, we compare to MusicLM [Agostinelli et al., 2023] and Moise2Music [Huang et al., 2023b].
Evaluation metrics. We evaluate the proposed method using objective and subjective metrics. For the objective methods, we use three metrics: the Fréchet Audio Distance (FAD), the Kullback-Leiber Divergence (KL) and the CLAP score (CLAP). We report the FAD [Kilgour al., 2018] using the official implementation in Tensorflow with the VGGish model 5. A low FAD score indicates the state-of-the-art audio classifier
generated audio is plausible. Following Kreuk al. [2022], trained for classification on AudioSet [Kutini et al., 2021] to compute the KI--divergence over the probabilities of the labels between the original and the generated music. The generated music is expected to share similar concepts with the reference music when the KI. is low. Last, the CLAP score [Wu* et al., 2023, Huang et al., 2023a] is computed between the track description and the generated audio to quantify audio-text alignment, using the official pretrained CLAP model
For the human studies, follow the same setup as in Kreuk al. [2022]. We ask human raters to evaluate two aspects of the audio samples (i) overall quality (OVL), and (ii) relevance to the text input (REL). For the overall quality test, raters were asked to rate the perceptual quality of the provided samples a range of to 100. For the text relevance test, raters were asked to rate the match between audio and text platform. We evaluate randomly sampled files, where each sample was evaluated by at least raters. We the CrowdMOS package" to filter noisy annotations and outliers. We remove annotators who did not listen to the full recordings, annotators who rate the reference recordings less than 85, and the rest of the recommended recipes from CrowdMOS [Ribeiro al., 2011]. For fairness, all samples are normalized at - 14dB LUFS [ITU-R, 2017].
scale of 1 to 100. Raters were recruited using the Amazon Mechanical Turk
Using riffusion-modd model-vi from github.com cithuuu.com/ifftsso//iffisssoooo app (on May 10, 2023) 'Implementation from
(March 2023) audio distance
httpss//githubb.omm/AAAee AI/LAP http://www.cowwdddossoo/wooooooo

---

Table 2: We report cosine similarity between reference and generated melody (SIM.) and subjective metrics including alignment with the melody (MEL.). All results are reported for MUSICGEN 1.5B.
In Domain Test Set
TRAIN CONDITION
TEST CONDITION
SIM.
MEL. f
OVI.
REL..
Text Text+Chroma
Text
0.10
64.441.0.3
82.18
81.54
Text
0.10
61 20.89
81.65=1.13
82.50
Text+Chroma
Text+Chroma
0.66
72.87
83.94
1.028
4 Results
We start by presenting results of the proposed method on the task of text-to-music generation and compare MUSICGEN to prior work in the field. Next. we evaluate the ability of the proposed method to generate music conditioned on melodic features. We further show how to simply extend our codebook patterns for stereo audio generation. We conclude with ablation study. Music samples, code, and models are available at guthub..om//aebbooteeercccaaaate
4.1 Comparison with the baselines
Table 1 presents the comparison of the proposed method against Mousai, Riffusion, MusicLM, and Noise2Music. As there no official implementation of Noise2Music, nor pre-trained model, we report only FAD on MusicCaps, which is reported in the Noise2Music manuscript. Similarly, MusicLM implementation is not public. We use the MusicLM public demo for our subjective tests while reporting the FAD as reported by the authors. While the original MusicLM model is trained on data with vocals, the model behind the API is instrument-onle onnl.. For the human study, we restrict ourselves to 40 instrument-only samples from MusicCaps. To prevent leakage in MUSICGEN trained with chromagram, we sample chromagrams at random from a held-out set during test time.
Results suggest that MUSICGEN performs better than the evaluated baselines as evaluated by human listeners, both in terms of audio quality and adherence to the provided text description. 2uisc2Music performs the best in terms of FAD on MusicCaps, followed by MUSICGEN trained with text condi- tioning. Interestingly, adding a melody conditioning degrades the objective metrics, however, it does not significantly affect human ratings, while still being superior to the evaluated baselines.
We notice that for the worst rated model, the FAD is correlated with the overall subjective rating, but it is not the case for the best rated models. We noticed that a large number of samples in MusicCaps [Agostinelli et al., 2023] contains a description stating that the recording is "noisy". It is possible that due to those noisy samples, improvements in the quality of the generated audio might deteriorate the FAD MusicCaps once certain quality threshold is achieved.
4.2 Melody evaluation
We evaluate MUSICGEN, conditioned jointly on textual and melodic representations, using objective and subjective metric on the held out evaluation set. For the objective evaluation, we introduce a new metric: chroma posine-similarity. which measures the average cosine-similarity between frames corresponding to the same time steps, taken from the quantized chroma of the reference and the generated samples. We evaluate using 1000 randomly sampled files from a held-out set. To better evaluate the relation between the conditioned melody to the generated music, we introduce another human study. To that end, we present human raters with a reference musical piece, followed by a set of generated pieces. For each generated sample, the listeners are asked to rate how well the melody of the generated piece matches that of the reference on a scale of to 100. We use 40 samples of 10 seconds at random from the held-out set. Results are reported in Table 2. Results suggest that MUSICGEN trained with chromagram conditioning successfully generates music that follows a given melody. Thus, allowing for better control over the generated output. Interestingly, MUSICGEN is robust to dropping the chroma at inference time with both OVL. and REI. staying roughly the same.
http::/bbbgggggeeecchooooyyainnn airgll--i--est--iteeeee
7

---

Stereo
Delay ay Pattern
Stereo Partial Delay Pattern
425%
Maliciee 2.0
Rosihal Rondhal coolbooke
Rothhll Bochhal Codelodke
1.0%
L55%
Sagaroor ange
Sepuanco doyu
(a) Visualizing codebook patterns for stereo models with two pos- sible interleaving of the left (blue) and right (red) codebooks. For a given codebook index, the "stereo delay" pattern uses different delays for the left and right channels, while the "stereo partial delay" predicts the two channels in parallel.
(b) Memorization results for the first codebook tokens on 5-second audio generations, considering exact (solid line) and 80% partial (dashed line) matches when prompted with extracts of varying duration from the train set.
Figure 2: Stereo codebooks (left) and memorization results (right)
Table 3: Stereophonic Text-to-Music generation. EnCodec processes separately the left and right channels, giving us 8 codebooks instead of 4. We experiment with two codebook patterns, depicted in Figure 2a. We also measure one of the stereo model after being downmixed to mono. We use a 1.5B MUSICGEN model conditioned only on text.
MUSICCAPS Test Set
CB. PATTERN
STEREO?
OVL. t
REL.. 1
Mono Delay
80.61 1..95
Stereo Partial Delay
X*
1..4
79.3911.16
Stereo Partial Delay
/
86.73
80.41+1.15
Stereo Delay
85.51
78.32 11.21
downmixed mono after generation.
4.3 Fine-tuning for stereophonic generation
We experiment with for extending generation to stereophonic data. We use the same EnCodec tokenizer, which we applied independently to the left and right channels, providing 2 K = codebooks per frame. Starting from pre-trained monophonic MUSICGEN model, we fine tune it for 200K batches the same dataset with stereo audio. We reuse the "delay" pattern, with two possible variations: (i) "stereo delay" introduces a delay between the left and right channels for the same codebook level, while (ii) "stereo partial delay" apply the same delay to the left and right channel codebooks for given level, as depicted in Figure 2a. Note that using this simple strategy, we can generate stereo audio at no extra computational cost. We provide in Table 3 the subjective evaluations for those models. We notice that when downmixing the stereo output mono, we are almost equivalent in perceived quality to a mono model. Stereo audio was overall rated higher than the mono counterpart, and the "stereo partial delay" benefits from small boost both in overall quality and text relevance compared with the "stereo delay" pattern.
4.4 Ablation
This section provides an ablation study for the different codebook patterns, together with results for model scales and memorization study. Additionally, present results for different text augmentation strategies, text encoders, and audio tokenization models in Appendix A.2. All ablations are performed using 1K samples of 30 seconds, randomly sampled from the held-out evaluation set.
The effect of the codebook interleaving patterns. We evaluate various codebook patterns using the framework from Section 2.2, with K = 4, given by the audio tokenization model. Table reports results with the "delay" pattern. The "partial delay" consists delaying by the same amount the codebooks 2, 3, and 4. The "parallel" pattern predicts all the codebooks from the same time step in parallel. The "coarse first" pattern first predicts codebook for all steps, then predicts in parallel codebooks 2, 3, and 4. Thus, this pattern has twice the steps compared to other patterns. "Partial
8

---

Table Codebook pattern.. We compare different codebook interleaving patterns on 30-seconds, audio sequences. The "flattening" pattern achieves the best scores. The "delay" and "partial flattening" patterns achieve similar scores, while "parallel" obtains worse scores.
In Domain Test Set
CONFIGURATION
Nb. steps
FADagR
KL +
CLAP
OVL.
REL.
Delay
1500
0.96
0.52
0.35
79.69=1.46
79.67=1A1
Partial delay
1500
1.51
0.54
0.32
79.13=1.55
79.67=146
Parallel
1500
2.58
0.62
0.27
72.21 1:249
80.30
Partial flattening
3000
1.32
0.54
0.34
78.56+1.86
79.18+1449
Coarse first
3000
1.98
0.56
0.30
1..42
1..55
Flattening
6000
0.86
0.51
0.37
19.71
82.03
Table 5: Model scale. We compare 3 scales for our method, and evaluate it on an internal test set to limit the impact of the out of domain prediction issues we observed with MusicCaps. In terms of objective metrics we observe continuous improvement of the metrics, although subjective quality stop improving at 1.5B parameters. A 3.3B model however seems better fit the text prompt.
In Domain Test Set
Dim.
Heads
Depth
Param.
PPI.].
FA
Kt. J
CLAP
OVL. f
REL.
1024
16
24
300M
56.1
0.96
0.52
0.35
78.3214
82.5 11.6
1536
24
48
1.5B
48.4
0.86
0.50
0.35
81.9
82.9+15
2048
32
48
3.3B
46.1
0.82
0.50
0.36
79.2
83.5
flattening" is similar, but instead of sampling first codebook for all steps, it interleaves them with the parallel sampling codebooks 2, 3, and 4. Finally, the "flattening" pattern consists in flattening all the codebooks, similar to MusicI.M [Agostinellii et al., 2023]. We report objective and subjective evaluations in Table 4. While flattening improves generation, it comes at high computational cost and similar performance can be reached for a fraction of this cost using simple delay approach.
The effect of model size. In Table 5 report results for different model sizes, namely 300M, 1.5B, and 3.3B parameter models. As expected, scaling the model size results in better scores, however this comes at the expense of longer training and inference time. Regarding subjective evaluations, the overall quality is optimal at 1.5B, but a larger model can better understand the text prompt.
Memorization experiment. Following [Agostinelli al., 2023], we analyze the memorization abilities of MUSICGEN. We only consider the first stream of codebooks from MUSICGEN as contains the coarser-grain information. We randomly select N 20, 000 examples from training set and for each one we feed the model with a prompt of EnCodec codebooks corresponding to the original audio and the conditioning information. We generate a continuation of 250 audio tokens (5 second-long audio) using greedy decoding. We report exact matches as the fraction of examples for which the generated audio tokens and source audio tokens are identical over the whole sequence. In addition, we report partial matches as the fraction of the training examples for which the generated and source sequences have least 80% of the audio tokens matching. We present the memorization results for the different model sizes when varying the length of the audio prompt in Figure 2b.
Related work
Audio representation. In recent years, the prominent approach is to represent the music signals in compressed representation, discrete or continuous, and apply a generative model on top of it. Lakhotia et al. [2021] proposed quantizing speech representation using k-means to construct speech language models. Recently, Défossez et al. [2022], Zeghidour et al. [2021] proposed to apply VQ-VAE directly on the raw waveform using residual vector quantization. Later, several studies used such representation for aoxt-oo-audio generation. Next, we discuss the recent research in audio generation.
Music generation. Music generation has been long studied under various setups. Dong et al. [2018] proposed a GAN-based approach for symbolic music generation. Bassan et [2022] proposed an unsupervised segmentation for symbolic music which can be later used for generation. Yeart et al.

---

[2017] proposed modeling polyphonic music modeling using recurrent neural networks. Ji et al. [2020] conducted a comprehensive survey therein for deep learning methods for music generation.
Dhariwal et al. [2020] proposed representing music samples in multiple streams of discrete represen- tations using hierarchical VQ-VAE. Next, sparse transformers applied over the sequences to generate music. Gan et al. [2020] proposed generating music for a given video, while predicting its midi notes. Recently, Agostinelli al. [2023] proposed representing music using multiple streams of "semantic tokens" and "acoustic tokens". Then, they applied a cascade of transformer decoders conditioned on a textual-music joint representation [Huang et al., 2022]. Donahue et al. [2023] followed a similar modeling approach, but for the task of singing-to-acee ac-oammcmammmenntt generation.
An alternative approach is using diffusion models. Schneider al. [2023], Huang al. [2023b]. Maina [2023], Forsgren and Martiros proposed using a latent diffusion model for the task of text-to-music. Schneider et al. [2023] proposed using diffusion models for both audio encoder-decoder and latent generation. Huang et al. [2023b] proposed a cascade of diffusion model to generate audio and gradually increase its sampling rate. Forsgren and Martiros proposed fine-tuning Stable Diffusion Rombach et al. [2022] using spectrograms to generate five-second segments, then, using image-to-image mapping and latent interpolation generate long sequences.
Audio generation. Several studies were proposed for text-to-audio (environmental sounds) genera- tion. Yang et al. [2022] proposed representing audio spectrograms using a VQ VAE, then applying discrete diffusion model conditioned on textual CLIP embeddings for the generation part [Radford al., 2021]. Kreuk et al. [2022] proposed applying transformer language model over discrete audio representation, obtained by quantizing directly time-domain signals using EnCodec [Défossez et al., 2022]. Sheffer and Adi [2023] followed similar approach Kreuk al. [2022] for image-to-audee generation. Huang et al. [2023a], Liu et al. [2023] proposed using latent diffusion models for the task of tex--to-audio, while extending various other tasks such as inpainting, image-to-audio, etc.
Discussion
We introduced MUSICGEN, a state-of-the-art single stage controllable music generation model that can be conditioned on text and melody. We demonstrated that simple codebook interleaving strategies can be used to achieve high quality generation, even in stereo, while reducing the number of autore- gressive time steps compared to the flattening approach. We provided comprehensive study of the impact of model sizes, conditioning methods, and text pre-processing techniques. We also introduced a simple chromagram- based conditioning for controlling the melody of the generated audio.
Limitations Our simple generation method does not allow us to have fine-grained control over adherence of the generation to the conditioning, we rely mostly on CF guidance. Also, while is relatively straightforward to do data augmentation for text conditioning, conditioning on audio warrants further research on data augmentation, types and amount of guidance.
Broader impact. Large scale generative models present ethical challenges. We first ensured that all the data trained on covered by legal agreements with the right holders, in particular through an agreement with ShutterStock. A second aspect is the potential lack of diversity in the dataset used, which contains larger proportion of western-style music. However, the simplification we operate in this work, e.g., using a single stage language model and a reduced number of auto-regressive steps, can help broaden the applications to new datasets. Generative models can represent an unfair competition for artists, which is an open problem. Open research can ensure that all actors have equal access to these models. Through the development of more advanced controls, such as the melody conditioning we introduced, we hope that such models can become useful both to music amateurs and professionals.
believe
Acknowledgements.
The authors would like to thank Mary Williamson, Rashel Moritz and Joelle Pineau for supporting this project, thank Justin Luk, Prash Jain, Sidd Srinivasan, Rod Duenes, and Philip Woods for the dataset, and thank the xformers team: Daniel Haziza, Francisco Massa, and Michael Ramamonjisoa for the technical discussions.
Authors note. This paper was submitted in the wake of the tragic terror attack perpetrated by Hamas on October 7, 2023, which has left the Israeli nation profoundly devastated. The assault greatly
10

---

impacted us at personal level, thereby significantly impacting the course of this research work. This paper was finalized while we grieve and mourn our friends and family, under great stress, with scientific considerations being the last thing on our minds. It may contain subtle errors.
In memory of the countless lives shattered by Hamas actions.
References
Meinard Müller.
Fundamentals of music processing: Audio, analysis, algorithms, applications,
volume 5. Springer, 2015.
Evelina Fedorenko, Josh H McDermott, Sam Norman Hainnnre,, and Nancy Kanwisher. Sensitivity to musical structure in the human brain. Journal of neurophysiology, 108(12):3289-3000 2012.
Sam V Norman- Norman-Haigneer, Nancy Kanwisher, Josh H McDermott, and Bevil R Conway. Divergence in the functional organization of human and macaque auditory cortex revealed by finii responses to harmonic tones. Nature neuroscience, 22(7):1057-100 2019.
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, al. A cookbook of self- supervised learning, arXiv preprint Prxiiv2.304.122100 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Xu Tan, Tao Qin, Frank Soong, and Tie- Yan Liu. A survey on neural speech synthesis. arXiv preprint Sttv:2200.115611 2021.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint vrXiv:2210.134:33 2022.
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, et al. Text-free prosody- aware generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8666-8681, 2022.
Felix Kreuk, Gabriel Synnaeve. Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniy Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint wwXiiv2209.15355 2022.
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023.
Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli, Mauro Verzetti, Ian Simon, Olivier Pietquin, Neil Zeghidour, al. Singsong: Generating musical 2023.
accompaniments from singing. arXiv preprint
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv: 2301.0211, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, neurips. neurips.cc/paper
Inc., 2017. URL https: Sto0g2/5B4eeee9111bbbCccccCBBBe Paper .ddf.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. JEFE//CM Transactions on Audio, Speech, and Language Processing, 2021.

---

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (1 (1)::48855555) 2020.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction instruction-inetuned language models. arXiv preprint arXiv:22 10. 11166, 2022.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.125000 2023.
Rongjie Huang. Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. arXiv preprint arXiv:2301.12661 2023a.
Roy Sheffer and Yossi Adi. I hear your true colors: Image guided audio generation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEFF, 2023.
Yusong Wu", Ke Chen Tianyu Zhang*, Yuchen Hui", Taylor King-Kirppatrick, and Shlomo Dubnov. Large-scale contrastive sanguage-audin pretraining with feature fusion and keyword-to-captin augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1007.0600.064550 2016.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with I0- awarnesssss. In Advances in Neural Information Processing Systems, 2022.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library. attps://githe: com/ facebookrresearchh htccch//fooooe srmerrs, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05101, 2017.
Aaron Defazio and Konstantin Mishchenko. Leamningrrate-fre learning by d-adaptation. arXiv preprint arXiv:2301.07733, 2023.
Angela Fan, Mike Lewis. and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv:1805.0m 04833, 2018.
S Forsgren and H Martiros. Riffusion-stable diffusion for real-time music generation. 2022. URL https:/riffussinn com/about.
Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf. Mo)," usai: Text-to-music generation with long-context latent diffusion. arXiv preprint 2xXi::230..1157, 2023.
Qingqing Huang, Daniel S Park, Tao Wang, Timo Denk, Andy Ly. Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint or 2rXiv:2302.03317, 2023b.
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fri, 'echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint 1rXiv:1812.08466, 2018.
Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer, Efficient training of audio transformers with patchout. arXiv preprint 2rX:v:2110.05069, 2021.
Flávio Ribeiro, Dinei Floréncio, Cha Zhang, and Michael Seltzer. Crowdmos: An approach for crowdsourcing mean opinion score studies. In 201 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 2416-2419. IEEE, 2011.
12

---

ITU-R. Algorithms to measure audio programme loudness and true-peak, audio level, 2017.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, AbdeIrahman Mohamed, et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336-1354, 2021.
Hao- Wen Dong, Wen-Y Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang. Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Shahaf Bassan, Yossi Adi, and Jeffrey S Rosenschein. Unsupervised symbolic music segmentation using ensemble temporal prediction errors. arXiv preprint arXiv:2207.00760, 2022.
Adrien Ycart, Emmanouil Benetos, et study on Istm networks for polyphonic music sequence modelling. ISMIR, 2017.
Shulei Ji, Jing Luo, and Xinyu Yang. A comprehensive survey deep music generation: Multi-level representations, algorithms, evaluations, and future directions. arXiv preprint arXiv:2011.0680 1.08001, 2020.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00331, 2020.
Chuang Gan, Deng Huang, Peihao Chen, Joshua B Tenenbaum, and Antonio Torralba. Foley music: Learning generate music from videos. In Computer Vision-ECCo ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XI 16, pages 758-775 Springer, 2020.
Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel PW Ellis. Mulan: A joint embedding of music audio and natural language. arXiv preprint arXiv:2208. 12415, 2022.
Kinyugo Maina. Msani:: High fidelity music synthesis shoestring budget. arXiv preprint 0rXiv:2301.06440 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
Dongchao Yang. Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. arXiv preprint 000:22077,9983, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages PMLR, 2021.
Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis Bach. Music source separation in the waveform domain. arXiv preprint arXiv:1911.13255 2019.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. arXiv preprint arXiv:2306..66: 2023.
13

---

A
Appendix
Pa r p iiall
Flatten ing Pattern
Part
Delay Pattern
Residual cadebooks
Residual codebooks
5
Sequence steps
Sequence steps
Figure A.1: Visualizing partial flattening and partial delays codebook patterns applied on sequence with parallel streams of quantized values (corresponding to ...,.., ka) and N time steps (f1..... k,). "Partial flattening" separates the first codebook in dedicated steps and interleaves them with the parallel sampling of codebooks 2, 3, and 4, leading the number interleaved sequences steps M to be twice the number of original steps N. The "partial delay" pattern consists in delaying by the same amount the codebooks 2, 3, and in our case we use delay of 1. The total number steps of the interleave sequences is N (excluding the last step for simplicity).
A.1 Experimental details
Codebook interleaving patterns. Figure A.1 provides visual description of the additional codebook patterns introduced for the ablation in Section 4, namely "partial flattening" and "partial delay" patterns. The intuition behind such patterns is driven by the fact that the first codebook from RVQ is the most important one and predicting the rest of the codebooks in parallel would allow to limit the introduced flattening or delay while maintaining good modeling performance.
Melody conditioning. In this work, we provide an unsupervised approach for melody conditioning through conditioning the chromagram representation. As shown in Figure A.2, chromagram based conditioning successfully preserves the melodic structure when generating novel music samples. In preliminary experiments, noticed that the chromagram tends to be dominated by the lower frequency instruments, mainly by the drums and bass. To mitigate that, we used Demucs [Défossez al., 2019] to first decompose the reference track into four components: drums, bass, vocals, and other. Next, we omit the drums and bass to recover the melodic structure of the residual waveform. Finally, extract the quantized chromagram create the conditioning that is later fed the model.
Distribution genres. We provide in Figure A.3 a histogram of the top 50 musical genres present in the dataset. We notice clear dominance of the Dance/EDM genre, which in our experience is also one of the genres that is best generated by MUSICGEN. While we tried to explore a number of resampling scheme to boost the importance of other genres, observed that oversampling less represented genres would often lead to a worse model overall.
A.2 Additional experimental results
We provide further ablation studies on the core components of MUSICGEN, namely the text encoder used for text conditioning described in Section 2.3, text augmentation strategies presented in Sec- tion 3.1, and the used audio tokenization model. We report results on the MusicCaps dataset to better understand out-of-domain generalization abilities of the different approaches. Finally, we share additional experimental results on optimization methods.
The effect of text encoder. We investigate the impact of the text encoder, comparing three different encoders: T5 [Raffel et al., 2020] 9, Flan-T5 [Chung et al., 2022] and CLAP (Wu* et al., 2023] II with a quantization bottleneck. For the CLAP-based encoder, similarly to Agostinelli et al. [2023] we rely on the music embeddings during training and the text embeddings inference time and
10 "nttps::/hugginggaace cttetaaccc..ctttteteeeeeeeeeeeee co/t.-basst
ort.a e5_esc__
90. 14.
14

---

Table A.1: Text encoder results. We report results for T5, Flan-T5, and CLAP as text encoders. We observe similar results for T5 and Flan- T5an-T5 on all the objective metrics. Note that T5 is the text encoder used for the main MUSICGEN models. CLAP encoder performs consistently worse on all the metrics but CLAP score. All comparisons are done with 300M MUSICGEN model using text conditioning only.
MUSICCAPS Test Set
MODEL
FADvgg +
KL
CLAP scr
OVL.
REL.
T5
3.12
1.29
0.31
85.04+1.23
87.33+:19
Flan-T5
3.36
1.30
0.32
85.54
85.00+1.6
CLAP
4.16
1.36
0.35
82.13+1.29
83.56+1554
CLAP (no normalization)
4.14
1.38
0.35
84.87
85.0611772
CLAP (no quantization)
5.07
1.37
0.37
84.13
84.67
Table A.2: Text augmentations strategics results. We report objective metrics using only the original text description (no augmentation) and for different text augmentation strategies: using condition merging to augment the text description with metadata, using next-normalization (text-norm.) and applying word dropout on the resulting text. We use 300M MUSICGEN models trained for 500K steps. Condition merging improves the result over training only over the original text description. Other augmentations perform worst all metrics. We use the Condition Merging with Word dropout, showing the best text relevance, in our main models.
MUSICCAPS Test Sct
CONFIGURATION
FADugg vgg 4
KL L
CLAPser
OVL. f
REL. f
No augmentation Condition Merging (CM)
3.68
1.28
0.31
83.40+1.44
81.16=1.20
3.28
1.26
0.31
82.60+1.41
84.45 11.16
CM + Text-norm. (TN)
3.78
1.30
0.29
80.57+2.14
82,40=1.09
CM+ Word dropout (WD)
3.31
1.31
0.30
82.52+155
85.27=0097
CM + TN + WD
3.41
1.39
0.30
81.18+1191
84.32 11.59
we train RVQ layer on top of the extracted embeddings. More specifically, we first normalize the embeddings and use RVQ with 12 quantizers, each with codebook size of 1024. Quantizing the CLAP embeddings leads to homogeneous representation with the discrete tokens further reducing the gap between the audio encoding used train time and text encoding at test time. We report results for the different text encoders in Table A. Both T5 and Flan-T5 perform similarly in terms of objective metrics, the overall quality being slighly better for T5. The CLAP-based model however suffers worse objective and subjective metrics, with the exception of the CLAP score which rely on the same underlying audio and text encoder.
The effect of text augmentations. We examine the impact of text augmentation strategies for the proposed method. In particular, we study the use of condition merging (i.e. concatenating additional metadata to the text description), text normalization (text-norm.) and word dropout. We report objective metrics for the different augmentation strategies Table A.2. We observe a gain in FAD and KL when leveraging the additional metadata with condition merging. However, neither text normalization or word dropout improves the results terms of objective metrics.
The effect of the audio tokenizer. We experiment with replacing EnCodec with Descript Audio different
Codec (DAC) [Kumar al., 2023] similar audio compression models, that training set, a similar adversarial loss enhanced with multiband discriminators, but performs quantization in a lower dimension space to improve codebook usage. DAC compresses audio at 44.1 kHz with 9 codebooks, and a framerate of 86 Hz. We trained a small (300M parameters) and a medium (1.5B parameters) MUSICGEN model using both DAC and EnCodec as an audio tokenizer. on vocal-free version of our dataset. The results provided in Table A.3 show a worse FAD and KL on our in domain test set. On MusicCaps, the FAD improved by using DAC, although the KL is worse, as well as the subjective evaluations. The EnCodec model used in this work was specifically
12 Using the public implementation github.co//esccrpppo//ssccc..uud
15

---

MUSICGEN + DAC small Table A.3: We test replacing EnCodec with DAC [Kumar al., 2023] using their implementation. DAC is 44.1 kHz model with codebooks and frame rate of 86 Hz. Those models are trained on a vocal-free version of our dataset, hence the objective metrics will not match those reported in the other tables. We report objective metrics both our in domain test set and MusicCaps [Agostinelli al., 2023), and subjective metrics only on MusicCaps.
In Domain Test Set
MUSICCAPS Test Sct
MODEL
FADvgg
KL
FADvga
|
KL
OVL.
3,45
0.58
4.46
1.35
83.32+:0995
MUSICGEN + DAC medium
2.42
0.57
4.32
1.30
84.46++0.97
MUSICGEN + EnCodec small
0.67
0.54
5.26
1.27
84.69+0..0
MUSICGEN + EnCodec medium
0.49
0.52
5.05
1.23
86.09+0.88
designed operate lower frame rate (50 Hz) than DAC (86 Hz), thus reducing by 40% the inference runtime for producing the same audio. Note finally that DAC was trained on different dataset than EnCodec, and further experiments would be required to understand exactly what influences the fitmess of such compression models to be used with auto-regressive language models.
Effect of D-Adaptation.
D-Adaptation is novel automated way selecting the overall learning rate the Adam optimizer, ic. its parameter, dynamically throughout the training. introduced by Defazio and Mishchenko [2023]. We observed improved convergence for the 300M parameter model, although for larger models, e.g. 1.5B and 3.3B, observed the automated rule led to deteriorated performance, both on the train and validation set. Further investigation is required to better understand the effect of D-Adaptation, and whether it can scale to the largest model. The convergence for both methods can be observed on the train and validation set in Figure A.4.
16

---

Ground-truth
Generated chroma
Generated w/o. chroma
10
10
Chromi
Chroma
Chroma
Time
Time
Time
10
10
10
Chroma
Chroma
Chromo
5
20
40
60
20
40
60
20
40
60
Time
Time
Time
10
10
Chrome
Chroma
Chroma
20
40
60
40
60
20
40
60
Time
Time
Time
Figure A.2: Visualization of quantized chromagram bins over time from reference melody (left), generated music conditioned on chroma and text (middle) and generated music with text-only conditioning (right). Each row is generated using different chroma condition, all rows share the same text condition: "90s rock song with electric guitar and heavy drums". We observe strong adherence to the input melody for the music samples generated with chroma conditioning while rendering novel styles guided by the input text.
140000
80000
60000
40000
Figure A.3: Histogram of the top 50 musical genres in the training dataset.
Train Perplexity
Valid Perplexity
DAdam, 300M
60
Adam, 300M
40
DAdam, I.5B
Adam, 1.5B
500
35
0
200
400
200
400
Epoch
Epoch
Figure A.4: Comparison of Adam and Adam with D-Adaptation [Defazio and Mishchenko, 2023]. While D-Adaptation provided consistent gains for the 300M parameters model, observed worse convergence both on the train (left) and validation (right) set for the 1.5B parameters model.
17


Reviews:
[{"review_id": "hNJNAYaMVF", "reviewer": "Reviewer_A118", "summary": "This paper introduces MUSICGEN, an approach for generating music conditioned on either text or melody representation. MUSICGEN consists of a single-stage transformer language model (LM) augmented with efficient token interleaving patterns, eliminating the requirement of employing multiple cascaded models. The authors extensively evaluate the proposed approach through a combination of automatic evaluations and human studies, demonstrating its superiority over the evaluated baselines on a widely-used text-to-music benchmark.", "questions": "Despite having some concerns regarding the motivation and novelty, I am impressed by the thoroughness of the experiments. If the authors can further improve the writing quality, I am willing to give a higher score.", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The experimental setup and results presented in the paper are robust and persuasive, providing strong evidence for the proposed approach.\n2. The paper includes a detailed and comprehensive discussion of the patterns related to the multi-codebook technique, enriching the understanding of this aspect of the research.\n3. The demos in supplementary materials are impressive.", "weaknesses": "1. The novelty of the proposed approach is somewhat limited, and the motivation for the research is not clearly articulated. Additionally, certain design choices in the methodology lack explanation, necessitating further elaboration and clarification.\n2. Some notations in the formulas are confusing, such as the usage of $\\widetilde{U}$ in Equation (2) and $\\hat{p}$ in Line 85. These notations should be introduced and defined more explicitly to avoid confusion.\n3. Attention to writing details is required. For instance, when referencing equations in the main text, only the number is provided without explicitly mentioning “Equation” (e.g., Line 89, Line 101). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**_Regarding novelty and motivation:_** The novelty and motivation of the proposed method (MusicGen) are designing a simple and efficient auto-regressive model to perform text-to-music generation. Unlike prior work, which consists of a cascade of models using either super-resolution or semantic tokens, we present a single-stage language model through an efficient codebook interleaving strategy. Additionally, we present a single model to perform both text and melody-conditioned generation and demonstrate that the generated audio is coherent with the provided melody and faithful to the text-conditioning information. \n\nFollowing the reviewer's concern, we additionally include an extension of MusicGen to stereophonic music generation, making it the first model to generate stereophonic music conditioned on textual descriptions. We encode separately the left and right channels using the same EnCodec model and use a specifically designed codebook interleaving pattern depicted in Figure 1 of the rebuttal document. OVL. and REL. scores can be seen below (and inside the attached PDF file), samples were shared with the AC through an external link as per the NeurIPS guidelines. Due to all of the above, we believe our work is novel and the contributions are important and would be valuable to the community. \n\n\n### Stereo Model Human Study. \n\n\n| Cb. Pattern          | Stereo? | OVL.           | REL.           |\n|----------------------|---------|----------------|----------------|\n| mono delay           | ✗       | 85.0 ± 1.6     | **80.6** ± 1.2 |\n| stereo partial delay | ✗*      | 84.5 ± 1.8     | 79.4 ± 1.2     |\n| stereo partial delay | ✓       | **86.7** ± 1.1 | **80.4** ± 1.1 |\n| stereo delay         | ✓       | 85.5 ± 1.2     | 78.3 ± 1.2     |\n\n\n*: downmixed to mono\n\n**_Regarding notations and writing quality_:** We thank the reviewer for their feedback and suggestions. We will fix all writing issues raised by the reviewer, including Equation references and method notations. \nWhen introducing $\\tilde{U}$, we clarified as “Let us build a second sequence of random variables $\\tilde{U}$ using the auto-regressive density $p$, e.g. we define recursively $\\tilde{U}_0 = 0$ and for all t > 0, …\nWe introduce $\\hat{p}$ as an estimate of $p$ using a deep learning model. We clarified “This means that if we can fit a perfect estimate $\\hat{p}$ of $p$ with a deep learning model, then we can fit exactly the distribution of $U$.”"}, {"review_id": "71rydiA0tm", "reviewer": "Reviewer_VPXA", "summary": "The paper proposes a single-stage music generation model that can input text or melody. The proposed approach uses tokens from pre-trained neural audio codec tokens with multiple residual vector quantizers and investigates efficient language modeling to reduce the length of autoregressive steps. Experiments on large-scale datasets show better generative results than some baselines.", "questions": "1. How does the number of codebooks affect the modeling? The original Encodec has 8 codebooks, and SoundStream has 12 codebooks. The 'flatten' idea becomes much more computationally expensive, but how could you ensure it would degrade the performance when the number of codebooks increases? Similarly, the codebook size also could change the difficulty of the optimization.\n\n2. the authors tried various text encoders in the main paper, but the generated samples are not described. In particular, I am interested in comparing CLAP text embedding and T5.", "limitations": "As I mentioned in the weakness, data memorization could be an issue and I am looking forward to the author's response.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. It is challenging to achieve strong performance and efficiency simultaneously, but this paper shows these could be achieved elegantly.\n\n2. The samples in supplementary have high audio quality and text-music coherence. I could perceive the difference in comparison to other approaches.\n\n3. The paper is generally written well. The motivation and demonstrations are clear.\n\n4. I am glad that the code and models will be available for future comparison and reproduction.", "weaknesses": "1. While the generative samples are impressive, I could see the technical novelty in this paper is limited. The audio tokens or EnCodec models are widely used in various applications. The 'parallel' or 'delay' codebook interleaving patterns are proposed in previous papers. The text2music or melody2music tasks have been established. I was having a difficult time capturing the core contribution of this paper. From my perspective, the main contribution of this paper would be to perform extensive experiments, analyze the results and make comparisons to demonstrate the critical components for efficient modeling. From this aspect, additional experiments (discussed in the questions) might be needed to convince me fully.\n\n2. Nowadays, we know data is a core part of reaching successful training, and obviously, the collected licensed music data probably won't be released. It is not a unique situation, but reproduction and future comparison will become an issue, and I wish authors could discuss this aspect. Besides, a genre distribution of the collected data might be helpful.\n\n3. Evaluation of generated music is another tricky thing. I understand that FAD, KL-div, CLAP consistency, and subjective evaluations are almost the standard evaluation metrics, but none consider whether the musical structure makes sense. I am wondering when the evaluation gap between symbolic (midi) based and audio-based methods could be consistent.\n\n4. In the MusicLM paper, there is a specific section about data memorization, which could be problematic, especially for licensed data. I want authors to discuss how such an issue could be avoided/improved as the technical components become mature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**_Regarding the contribution of the proposed method_:** The novelty and contribution of our work are designing a simple and efficient auto-regressive model to perform text-to-music generation. Unlike prior work, which consists of a cascade of models using either super-resolution or semantic tokens, we present a single-stage language model through an efficient codebook interleaving strategy. With the codebook-interleaving patterns, we provide a framework that generalizes prior work in the field. Additionally, we present a single model to perform both text and melody-conditioned generation and demonstrate that the generated audio is coherent with the provided melody and faithful to the text-conditioning information. \n\nFollowing the reviewer's concern, we additionally include an extension of MusicGen to stereophonic music generation, making it the first model to generate stereophonic music conditioned on textual descriptions. OVL. and REL. scores can be seen in the table above (in response to reviewer A118) (and inside the attached rebuttal PDF file, Table 1), samples were shared with the AC through an external link as per the NeurIPS guidelines. Due to all of the above, we believe our work is novel and the contributions are important and would be valuable to the community. \n\n**_Regarding the used dataset_:** our dataset consists of 20K hours of licensed music, comprising 10k high-quality music data, ShutterStock, and Pond5 music data collections, which are available for licensing (lines 202-206). As per the reviewer's request, we provide a genre distribution of the training set in the attached rebuttal PDF file (Figure 3). Regarding comparison to MusicGen, we will open-source both training code and pre-trained models, so future research could: (1) directly compare the MusicGen models, and (ii) build new and improved music generation models on top of MusicGen. \n\n**_Regarding evaluation function_:** We agree that evaluating musical structure in the generated music is interesting and important for music generation. However, this is far from trivial as the generated output is audio samples and not interpretable discrete representations (like midi). Developing such metrics is an important future research direction, we hope the community (including us) will pursue it. Nevertheless, we believe MusicGen is an important step in the right direction of developing high-quality and controllable music generation models.\n\n**_Regarding memorization metrics_:** As per the reviewer’s request, we follow Agostinelli et al. [2023], and analyze the training data memorization abilities of MusicGen. We consider the first stream of codebooks from MusicGen as it contains the coarser grain and most important regarding the generated audio. We randomly select $N=20,000$ examples from our training set and for each one, we feed the model with a prompt of EnCodec codebooks corresponding to the original audio and the conditioning information. We generate a continuation of 250 audio tokens (corresponding to 5 seconds) using greedy decoding. We report exact matches as the fraction of examples for which the generated audio tokens and source audio tokens are identical over the whole sequence. In addition, we report partial matches with the fraction of examples for which the generated and source sequences have at least $80\\%$ of the audio tokens matching. The Figure can be seen in the attached rebuttal PDF file (Figure 2). As can be seen, the exact matches and partial matches are almost zero under all evaluated settings.\n\n**_Regarding the number of codebooks (question 1)_:** During early experiments, we compared several numbers of codebooks and codebook sizes. We noticed that larger codebooks (e.g. 2048) would improve quality but not increase it beyond that point (we tested 4096 and 8192). Early experiments indicated that using more codebooks (e.g. 8) was detrimental to MusicGen models with 1.5B parameters. However, as depicted in Table 1 in the Rebuttal Document, we show that we can extend to 8 codebooks to stereo data, using 4 codebooks for the left channel, and 4 codebooks for the right channel. In that case, we noticed no regression compared with the 4 codebooks model (e.g. downmixing the output to mono to compare to the previously trained mono models), while the model was clearly able to model the stereo image of the generated music extracted, as measured by subjective evaluations. \n\n**_Regarding samples for different text encoders (question 2)_:** As per the reviewer’s request, we shared samples for various text encoders. Specifically, we share samples for T5, Flan-T5, and CLAP using 12 codebooks (previously just CLAP, as done in MusicLM), CLAP using 24 codebooks, and CLAP without quantization. Interestingly, we observe that the quantization of the CLAP embedding as done in MusicLM is detrimental to the final score of the model. Samples were shared with the AC through an external link as per the NeurIPS guidelines. We additionally provide results for the different setups in the table below: \n\n| Text Encoder                | FAD      | KL       | Clap sc. | OVL.           | REL.           |\n|-----------------------------|----------|----------|----------|----------------|----------------|\n| T5 (default)                | **3.12** | **1.29** | 0.31     | 84.9 ± 1.8     | **82.5** ± 1.3 |\n| FLAN-T5                     | 3.36     | 1.30     | 0.32     | **86.3** ± 1.8 | 80.8 ± 1.9     |\n| CLAP (RVQ n_q=12 bins=1024) | 4.23     | 1.53     | 0.32     | 79.8 ± 1.8     | 77.3 ± 1.5     |\n| CLAP (RVQ n_q=24 bins=1024) | 4.18     | 1.47     | 0.32     | 82.0 ± 1.4     | 76.7 ± 1.3     |\n| CLAP (no quantizer)         | 5.13     | 1.53     | **0.34** | 84.5 ± 1.2     | 80.2 ± 1.      |"}, {"review_id": "HYJhHNuR4q", "reviewer": "Reviewer_7q8g", "summary": "MusicGen is an auto-regressive architecture for music audio generation conditioned on textual descriptions and an optional melody. The key proposal is a generic formulation of audio codebook interleaving strategy, which enables parallel code streams to be processed with a simple single-stage Transformer decoder while balancing efficiency with inter-stream dependency. Through extensive evaluation, MusicGen demonstrates a superior performance in acoustic coherency and semantic faithfulness compared to previous models. Sufficient ablation studies also validate its design choices for text pre-processing, melody conditioning, code interleaving pattern, and model scale.", "questions": "* There seems a notation clash on the variable $N$. On line 74, $N$ is defined as codebook size, but is later used to represent code sequence length in line 98, line 145, and the caption of Figure 1. \n\n* Line 95: Should the autoregressive steps of MusicLM be $df_r\\cdot K$, rather than $df_s\\cdot K$?\n\n* Line 177: What is the purpose of applying EMA to the Transformer model? Shouldn’t it be part of the audio tokenization model for updating the codebook?\n", "limitations": "In my view, the main potential limitation of this work lies in long-term music structure and fine-grained control, which has been detailed in the Weaknesses section. The latter has also been discussed in the paper. However, I understand that this paper focuses on text semantic control (with already compelling results) and those areas are currently out of the scope.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* **Unsupervised melody conditioning**: The paper uses quantized chromagrams to capture the most salient melodic features to conduct melody conditioning. To suppress interference with low-frequency instruments, it further leverages a pretrained source separation model to detach those bands before extracting chromagram. Such an unsupervised scheme is efficient, sound, and working reasonably well.\n\n* **Ablation study**: Extensive empirical studies are conducted to validate the key design choices, which are beneficial for the reference of follow-up research.\n\n* **Compelling performance**: Demos demonstrate a compelling acoustic and semantic quality. It is good to know that code and models will be open, for which a good impact of this work can be expected.\n", "weaknesses": "* **Long-term music structure**: The paper demonstrates several long generation examples created using windowed sampling, showcasing the ability of MusicGen to produce remarkably authentic grooves and looping patterns. However, one noticeable weakness lies in the relative absence of a distinct phrasal or sectional structure in the long run, which may lead to a lack of sense of musical development. To tackle such problem may require more musically insightful inductive bias in the framework design.\n\n* **Fine-grained control**: The text-conditional architecture of MusicGen focuses on global semantic control, which essentially guarantees a globally consistent “picture” throughout the generated piece. On the other hand, musicians sometimes intentionally want to create variations and inconsistency to make the piece more impressive and contagious. Achieving such purpose would require a more fine-grained controlling scheme, which is not currently supported by MusicGen.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**_Regarding long-range musical structure_:** We agree that evaluating musical structure in the generated music is interesting and important for music generation. However, it is far from trivial as the generated output is audio samples and not interpretable discrete representations (like midi). Developing such metrics is an important future research direction, we hope the community (including us) will pursue it. Nevertheless, we believe MusicGen is an important step in the right direction of developing high-quality music generation models.\n\n**_Regarding fine-grained control_:** We agree fine-grained control can benefit the music industry and specifically music creators. However, we believe global control is also important and can benefit creators who are not expert musicians. Nevertheless, we believe MusicGen is an important step in the right direction toward developing controllable music generation. \n\n**_Regarding notations_:** We fixed this notation. We renamed the $N$ which corresponds to the time steps into $T$ in line 98 and Figure 1.\n\n**_Regarding line 95, MusicLM_**: Thanks for the correction! We fixed it for the final manuscript. \n\n**_Regarding EMA_:** During preliminary experiments, we notice using an exponential moving average during model training smoothes out the validation loss. Such a technique was also used in prior work with transformers, see [1]. \n\n[1] Touvron, Hugo, et al. \"Training data-efficient image transformers & distillation through attention.\" International conference on machine learning. PMLR, 2021."}, {"review_id": "MjIhREL66A", "reviewer": "Reviewer_gnNB", "summary": "This paper introduces MusicGen, an innovative single-stage transformer language model for conditional music generation, using either text or melody as a conditioning factor. The authors focus on exploring different approaches of codebook interleaving patterns and propose an efficient approach of Delay Pattern. Experiments were performed with state-of-the-art conditional music generation models, and multiple ablation studies aids the efficacy of the proposed approach. The authors have further contributed to the community by releasing demo samples and inference code to the public.", "questions": "Please see the concerns raised at **Weakness** section. + theoretical explanation of the decoder interpreting *Delay Pattern* mentioned at **Strengths** section.", "limitations": "The paper comprehensively outlines its limitations and discussions in the manuscript, which align with the reviewer's.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is presented clearly, which greatly aids comprehension of the presented methods and results.\n- The proposed *Delay Pattern* is a novel approach that shows strengths in computational time. It’s surprising to see the decoder interpret the residual codebooks of the Delay Pattern, even when the delayed codebooks are summed and used as input. A more in-depth theoretical explanation of this would be valuable.\n- The empirical results are impressive, where the evaluation against state-of-the-art text to music generation systems provides solid evidence for the model's performance.\n- The authors' decision to open-source the code and model weights could significantly impact both academia and industry. I’m already seeing some online applications utilizing MusicGen's API.", "weaknesses": "- The reproduced *Valle-E pattern* among the *codebook interleaving patterns* seems inconsistent with the original manuscript [A].\nIn the Section **Introduction**, the authors correctly mention the codebook generation steps of *Valle-E* as “(i) modeling the first stream of tokens only; (ii) then, apply a post-network to jointly model the rest of the streams in a non-autoregressive manner.”\nYet, the authors reproduced the *Valle-E* pattern as the step (ii) being decoded in an autoregressive manner. The correct pattern of *Valle-E* would be to use two different decoders and perform step (ii) in a nonautoregressive manner (given the autoregressively generated tokens from step (i)), resulting in sequence steps of $n+3$.\n    - Thus, this discrepancy should be addressed either by 1. correcting the *Valle-E* pattern methodology or 2. simply renaming the “Valle-E Pattern” in the manuscript. #1 option would be to rectify *Figure 1, Section 2.2, Equation (6), Table 3 (Nb. steps)*, and updating the results of *Table 3*.\n\n- Some aspects of the paper lack details and justifications of method choice.\n    1. According to the ablation study results of *text augmentation* Table A.2, the best performing method is applying only the *condition merging*. Since the authors are using *Condition merging + Text-norm. + Word dropout* as the final text augmentation strategy, they should provide a justification for this.\n    2. The final choice of text encoder is unclear. Although the authors have stated that they performed an ablation study using all three different text encoders (T5, Flan-T5, and CLAP), the reviewer couldn’t find an indication which model was selected as the final text encoder. Furthermore, the results from the ablation study Table A.1 does not align with any of the scores from Table 1 & 2. If the authors used different configurations for Table A.1, they should mention it in the manuscript.\n\n- typo at line 196: “hope size” → “hop size”\n\n[A] Wang, Chengyi, et al. \"Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers.\" *arXiv preprint arXiv:2301.02111* (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**_Regarding the VALL-E codebook pattern_:** Indeed, we incorrectly describe the “VALL-E” codebook patterns as corresponding to VALL-E. We renamed it to “flat 1st codebook then parallel”.\n\n**_Regarding text augmentations_:** During model development, we experimented with condition merging, word dropout, and text normalization. For the final models, we use condition merging + word dropout as it gives the best overall performance. We agree the description in the text is confusing and we will clarify that for the final manuscript. To better support our design choice, we provide here a human study (OVL. and REL.) in the table below. \n\n### Text augmentations Human study\n\n| Configuration                                 | FAD      | KL   | CLAP sc. | OVL.           | REL.           |\n|-----------------------------------------------|----------|------|----------|----------------|----------------|\n| No augmentation                               | 3.68     | 1.28 | **0.31** | **83.4** ± 1.4 | 81.2 ± 1.3     |\n| Condition merging                             | **3.28** | 1.26 | **0.31** | 82.6 ± 1.4     | 84.5 ± 1.2     |\n| Condition merging + Text-norm.                | 3.78     | 1.30 | 0.29     | 80.6 ± 2.1     | 82.4 ± 1.1     |\n| Condition merging + word dropout              | 3.31     | 1.31 | 0.30     | 82.5 ± 1.6     | **85.3** ± 1.0 |\n| Condition merging + Text-norm. + Word dropout | 3.41     | 1.39 | 0.30     | 81.2 ± 1.9     | 84.3 ± 1.6     |\n\nResults suggest condition merging + word dropout provides comparable generation quality but better text relevance scores. \n\n**_Regarding the used text encoder_:** We use the T5 text encoder as noted in line 192. Considering all metrics, the T5 model gets the best overall performance (Table A.1). The configuration used in Table A.1 is indeed different. For Table A.1, as this is an ablation study, we use a 300M parameters model (see lines 170-171). We now indicate clearly when an Ablation is done with a 300M parameters model, and in Table A.1 we further repeat that T5 is the encoder used for the main models in the caption of Table A.1.\n\n**_Regarding typos_:** Thanks! We will fix the typo."}]

Author rebuttal:
We would like to thank the reviewers for their detailed reviews and valuable feedback. We are happy the reviewers found our experimental setup and results robust and persuasive. We are also glad the reviewers found our method to have both strong performance and efficient modeling. We address each of the reviewers' questions and concerns in a separate comment below. 

Meta-review: This paper proposes MusicGen, a single-stage Transformer language model to perform text-to-music generation, with efficient token interleaving patterns. All reviewers are positive about this paper. Although some reviewers have challenges with the technical novelty and motivations, the authors provide a good explanation, and let the reviewer support that this work is ''an important step in the right direction''. 
The open-source code and checkpoint also resolve the concerns about the unavailability of the training data to some extent. Overall, it is a good paper and I recommend an acceptance.
Decision: Accept (poster)