SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jade Copet\* Felix Kreuk\* Itai Gat Tal Remez David Kant Gabriel Synnaeve Yossi Adi\* Alexandre Defossez
$\spadesuit$  equal contributions,  $\diamond$  : core team
{jadecopet, felixkreuk, adiyoss}@meta.com
We tackle the task of conditional music generation. We introduce MUSICGEN, a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MUSICGEN is comprised of a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for cascading several models, e.g., hierarchically or upsampling. Following this approach, we demonstrate how MUSICGEN can generate high-quality samples, both mono and stereo, while being conditioned on textual description or melodic features, allowing better controls over the generated output. We conduct extensive empirical evaluation, considering both automatic and human studies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark. Through ablation studies, we shed light over the importance of each of the components comprising MUSICGEN. Music samples, code, and models are available at github.com/facebookresearch/audiocraft.
Text-to-music is the task of generating musical pieces given text descriptions, e.g., "90s rock song with a guitar riff". Generating music is a challenging task as it requires modeling long range sequences. Unlike speech, music requires the use of the full frequency spectrum [Müller, 2015]. That means sampling the signal at a higher rate, i.e., the standard sampling rates of music recordings are 44.1 kHz or 48 kHz vs. 16 kHz for speech. Moreover, music contains harmonies and melodies from different instruments, which create complex structures. Human listeners are highly sensitive to disharmony [Fedorenko et al., 2012, Norman-Haignere et al., 2019], hence generating music does not leave a lot of room for making melodic errors. Lastly, the ability to control the generation process in a diverse set of methods, e.g., key, instruments, melody, genre, etc. is essential for music creators.
Recent advances in self-supervised audio representation learning [Balestriero et al., 2023], sequential modeling [Touvron et al., 2023], and audio synthesis [Tan et al., 2021] provide the conditions to develop such models. To make audio modeling more tractable, recent studies proposed representing audio signals as multiple streams of discrete tokens representing the same signal [Défossez et al., 2022]. This allows both high-quality audio generation and effective audio modeling. However, this comes at the cost of jointly modeling several parallel dependent streams.
Kharitonov et al. [2022], Kreuk et al. [2022] proposed modeling multi-streams of speech tokens in parallel following a delay approach, i.e., introduce offsets between the different streams. Agostinelli et al. [2023] proposed representing musical segments using multiple sequences of discrete tokens at different granularity and model them using a hierarchy of autoregressive models. In parallel, Donahue et al. [2023] follows a similar approach but for the task of singing to accompaniment generation. Recently, Wang et al. [2023] proposed tackling this problem in two stages: (i) modeling the first
* Yossi Adi is Affiliated with both The Hebrew University of Jerusalem & MetaAI.
Figure 1: Codebook interleaving patterns presented in Section 2.2. Each time step  $t_1, t_2, \ldots, t_n$  is composed of 4 quantized values (corresponding to  $k_1, \ldots, k_4$ ). When doing autoregressive modelling, we can flatten or interleave them in various ways, resulting in a new sequence with 4 parallel streams and steps  $s_1, s_2, \ldots, s_m$ . The total number of sequence steps  $S$  depends on the pattern and original number of steps  $T$ . 0 is a special token indicating empty positions in the pattern.
stream of tokens only; (ii) then, apply a post-network to jointly model the rest of the streams in a non-autoregressive manner.
In this work, we introduce MUSICGEN, a simple and controllable music generation model, which is able to generate high-quality music given textual description. We propose a general framework for modeling multiple parallel streams of acoustic tokens, which serves as a generalization of previous studies (see Figure 1). We show how this framework allows to extend generation to stereo audio at no extra computational cost. To improve controllability of the generated samples, we additionally introduce unsupervised melody conditioning, which allows the model to generate music that matches a given harmonic and melodic structure. We conduct an extensive evaluation of MUSICGEN and show the proposed method is superior to the evaluated baselines by a large margin, with a subjective rating of 84.8 out of 100 for MUSICGEN against 80.5 for the best baseline. We additionally provide an ablation study which sheds light on the importance of each of the components on the overall model performance. Lastly, human evaluation suggests that MUSICGEN yields high quality samples which are better melodically aligned with a given harmonic structure, while adhering to a textual description.
Our contribution: (i) We introduce a simple and efficient model to generate high quality music at  $32\mathrm{kHz}$ . We show that MUSICGEN can generate consistent music with a single-stage language model through an efficient codebook interleaving strategy. (ii) We present a single model to perform both text and melody-conditioned generation and demonstrate that the generated audio is coherent with the provided melody and faithful to the text conditioning information. (iii) We provide extensive objective and human evaluations on the key design choices behind our method.
MUSICGEN consists in an autoregressive transformer-based decoder [Vaswani et al., 2017], conditioned on a text or melody representation. The (language) model is over the quantized units from an EnCodec [Défossez et al., 2022] audio tokenizer, which provides high fidelity reconstruction from a low frame rate discrete representation. Compression models such as [Défossez et al., 2022, Zeghidour et al., 2021] employ Residual Vector Quantization (RVQ) which results in several parallel streams. Under this setting, each stream is comprised of discrete tokens originating from different learned codebooks. Prior work, proposed several modeling strategies to handle this issue [Kharitonov et al., 2022, Agostinelli et al., 2023, Wang et al., 2023]. In this work, we introduce a novel modeling framework, which generalizes to various codebook interleaving patterns, and we explore several
variants. Through patterns, we can leverage the internal structure of the quantized audio tokens. Finally, MUSICGEN supports conditional generation based on either text or melody.
2.1 Audio tokenization
We use EnCodec [Défossez et al., 2022], a convolutional auto-encoder with a latent space quantized using Residual Vector Quantization (RVQ) [Zeghidour et al., 2021], and an adversarial reconstruction loss. Given a reference audio random variable  $X \in \mathbb{R}^{d \cdot f_s}$  with  $d$  the audio duration and  $f_s$  the sample rate, EnCodec encodes it into a continuous tensor with a frame rate  $f_r \ll f_s$ . This representation is then quantized into  $Q \in \{1, \dots, M\}^{d \cdot f_r \times K}$ , with  $K$  being the number of codebooks used in RVQ and  $M$  being the codebook size. Notice, after quantization we are left with  $K$  parallel discrete tokens sequences, each of length  $T = d \cdot f_r$ , representing the audio sample. In RVQ, each quantizer encodes the quantization error left by the previous quantizer, thus quantized values for different codebooks are in general not independent, and the first codebook is the most important one.
2.2 Codebook interleaving patterns (see Figure 1)
Exact flattened autoregressive decomposition. An autoregressive model requires a discrete random sequence  $U \in \{1, \dots, M\}^S$  with  $S$  the sequence length. By convention, we will take  $U_0 = 0$ , a deterministic special token indicating the beginning of the sequence. We can then model the distribution
$$
\forall t > 0, p _ {t} \left(U _ {t - 1}, \dots , U _ {0}\right) \triangleq \mathbb {P} \left[ U _ {t} \mid U _ {t - 1}, \dots , U _ {0} \right]. \tag {1}
$$
Let us build a second sequence of random variables  $\tilde{U}$  using the auto-regressive density  $p$ , e.g. we define recursively  $\tilde{U}_0 = 0$ , and for all  $t > 0$
$$
\forall t > 0, \mathbb {P} \left[ \tilde {U} _ {t} | \tilde {U} _ {t - 1} \dots , \tilde {U} _ {0} \right] = p _ {t} (\tilde {U} _ {t - 1}, \dots , \tilde {U} _ {0}). \tag {2}
$$
Then, we immediately have that  $U$  and  $\tilde{U}$  follow the same distribution. This means that if we can fit a perfect estimate  $\hat{p}$  of  $p$  with a deep learning model, then we can fit exactly the distribution of  $U$ .
As stated before, the main issue with the representation  $Q$  we obtained from the EnCodec model is that there are  $K$  codebooks for each time step. One solution would be to flatten out  $Q$ , thus taking  $S = d \cdot f_r \cdot K$ , e.g. first predicting the first codebook of the first time step, then the second codebook of the first time step, etc. Then, using eq. (1) and eq. (2), we could theoretically fit an exact model of the distribution of  $Q$ . The downside however is the increased complexity, with part of the gain coming from the lowest sample rate  $f_r$  being lost.
More than one possible flattening exists, and not all the  $\hat{p}_t$  functions need to be estimated through a single model. For instance, MusicLM [Agostinelli et al., 2023] uses two models, one modeling the flattened first  $K/2$  codebooks, and a second one the other  $K/2$  flattened codebooks, conditioned on the decision of the first model. In that case, the number of autoregressive steps is still  $df_r \cdot K$ .
Inexact autoregressive decomposition. Another possibility is to consider an autoregressive decomposition, where some codebooks are predicted in parallel. For instance, let us define another sequence with  $V_{0} = 0$  and for all  $t \in \{1, \dots, T\}$ ,  $k \in \{1, \dots, K\}$ ,  $V_{t,k} = Q_{t,k}$ . When dropping the codebook index  $k$ , e.g.  $V_{t}$ , we mean the concatenation of all the codebooks at time  $t$ .
$$
p _ {t, k} \left(V _ {t - 1}, \dots , V _ {0}\right) \triangleq \mathbb {P} \left[ V _ {t, k} \mid V _ {t - 1}, \dots , V _ {0} \right]. \tag {3}
$$
Let's define again recursively  $\tilde{V}_0 = 0$  and for all  $t > 0$ ,
$$
\forall t > 0, \forall k, \mathbb {P} \left[ \tilde {V} _ {t, k} \right] = p _ {t, k} \left(\tilde {V} _ {t - 1}, \dots , \tilde {V} _ {0}\right). \tag {4}
$$
Unlike in (2), we no longer have in the general case that  $\tilde{V}$  follows the same distribution as  $V$ , even assuming we have access to the exact distribution  $p_{t,k}$ . In fact, we would only have a proper generative model if for all  $t$ ,  $(V_{t,k})_k$  are independent conditionally on  $V_{t-1}, \ldots, V_0$ . As  $t$  increases, the errors will compound and the two distributions can grow further apart. Such a decomposition is inexact, but allows to keep the original frame rate which can considerably speed up training and inference, especially for long sequences.
Arbitrary codebook interleaving patterns. In order to experiment with various such decompositions, and measure exactly the impact of using an inexact decomposition, we introduce codebook interleaving patterns. Let us consider  $\Omega = \{(t,k):\{1,\ldots ,d\cdot f_r\} ,k\in \{1,\ldots ,K\} \}$  be the set of all pairs of time steps and codebook indexes. A codebook pattern is a sequence  $P = (P_{0},P_{1},P_{2},\dots,P_{S})$  with  $P_0 = \emptyset$ , and for all  $0 < s\leq S$ ,  $P_{s}\subset \Omega$ , such that  $P$  is partition of  $\Omega$ . We model  $Q$  by predicting in parallel all the positions in  $P_{s}$ , conditionally on all the positions in  $P_0,P_1,\dots,P_{s - 1}$ . Pragmatically, we restrict ourselves to patterns where each codebook index appears at most once in any of the  $P_{s}$ .
We can now easily define a number of decompositions, for instance the "parallel" pattern given by
$$
P _ {s} = \{(s, k): k \in \{1, \dots , K \} \}. \tag {5}
$$
It is also possible to introduce a "delay" between the codebooks, as in Kharitonov et al. [2022], e.g.,
$$
P _ {s} = \{(s - k + 1, k): k \in \{1, \dots , K \}, s - k \geq 0 \}. \tag {6}
$$
Through empirical evaluations, we show the benefits and drawbacks of various codebook patterns, shedding light on the importance of exact modeling of the parallel codebook sequences.
2.3 Model conditioning
Text conditioning. Given a textual description matching the input audio  $X$ , we compute a conditioning tensor  $C \in \mathbb{R}^{T_C \times D}$  with  $D$  being the inner dimension used in the autoregressive model. Generally, there are three main approaches for representing text for conditional audio generation. Kreuk et al. [2022] proposed using a pretrained text encoder, specifically T5 [Raffel et al., 2020]. Chung et al. [2022] show that using instruct-based language models provide superior performance. Lastly, Agostinelli et al. [2023], Liu et al. [2023], Huang et al. [2023a], Sheffer and Adi [2023] claimed that joint text-audio representation, such as CLAP [Wu* et al., 2023], provides better-quality generations. We experiment with all of the above, respectively: T5 encoder, FLAN-T5, and CLAP.
Melody conditioning. While text is the prominent approach in conditional generative models nowadays, a more natural approach for music is conditioning on a melodic structure from another audio track or even whistling or humming. Such an approach also allows for an iterative refinement of the model's output. To support that, we experiment with controlling the melodic structure via jointly conditioning on the input's chromagram and text description. In preliminary experiments, we observed that conditioning on the raw chromagram often led to reconstructing the original sample, resulting in overfitting. To reduce it, we introduce an information bottleneck by choosing the dominant time-frequency bin in each time step. While a similar capability was shown in Agostinelli et al. [2023], the authors used supervised proprietary data, which is tedious and costly to collect. In this work, we take an unsupervised approach, eliminating the requirement for supervised data.
Codebook projection and positional embedding. Given a codebook pattern, only some codebooks are present at each pattern step  $P_{s}$ . We retrieve from  $Q$  the values corresponding to the indices in  $P_{s}$ . As noted in Section 2.2, each codebook is present at most once in  $P_{s}$  or not at all. If it is present, we use a learned embedding table with  $N$  entries and dimension  $D$  to represent the associated value from  $Q$ . Otherwise, we use a special token indicating its absence. We sum the contribution from each codebook after this transformation. As  $P_{0} = \emptyset$ , the first input is always the sum of all the special tokens. Finally, we sum a sinusoidal embedding to encode the current step  $s$  [Vaswani et al., 2017].
Transformer decoder. The input is fed into a transformer with  $L$  layers and a dimension  $D$ . Each layer consists of a causal self-attention block. We then use a cross-attention block that is fed with the conditioning signal  $C$ . When using melody conditioning, we instead provide the conditioning tensor  $C$  as a prefix to the transformer input. The layer ends with a fully connected block consisting of a linear layer from  $D$  to  $4 \cdot D$  channels, a ReLU, and a linear layer back to  $D$  channels. The attention and fully connected blocks are wrapped with a residual skip connection. Layer normalization [Ba et al., 2016] is applied to each block before being summed with the residual skip connection ("pre-norm").
Logits prediction. The output from the transformer decoder at pattern step  $P_{s}$  is transformed into logits prediction for the values of  $Q$  taken at the indices given by  $P_{s + 1}$ . Each codebook is present at most once in  $P_{s + 1}$ . If a codebook is present, the logits prediction is obtained by applying a codebook specific linear layer from  $D$  channels to  $N$ .
3.1 Models and hyperparameters
Audio tokenization model. We use a non-causal five layers EnCodec model for  $32\mathrm{kHz}$  monophonic audio with a stride of 640, resulting in a frame rate of  $50\mathrm{Hz}$ , and an initial hidden size of 64, doubling at each of the model's five layers. The embeddings are quantized with a RVQ with four quantizers, each with a codebook size of 2048. We follow Defossez et al. [2022] to train the model on one-second audio segments cropped at random in the audio sequence.
Transformer model. We train autoregressive transformer models at different sizes: 300M, 1.5B, 3.3B parameters. We use a memory efficient Flash attention [Dao et al., 2022] from the xFormers package [Lefaudeauux et al., 2022] to improve both speed and memory usage with long sequences. We study the impact of the size of the model in Section 4. We use the 300M-parameter model for all of our ablations. We train on 30-second audio crops sampled at random from the full track. We train the models for 1M steps with the AdamW optimizer [Loshchilov and Hutter, 2017], a batch size of 192 examples,  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.95$ , a decoupled weight decay of 0.1 and gradient clipping of 1.0. We further rely on D-Adaptation based automatic step-sizes [Defazio and Mishchenko, 2023] for the 300M model as it improves model convergence but showed no gain for the bigger models. We use a cosine learning rate schedule with a warmup of 4000 steps. Additionally, we use an exponential moving average with a decay of 0.99. We train the 300M, 1.5B and 3.3B parameter models, using respectively 32, 64 and 96 GPUs, with mixed precision. More specifically, we use float16 as bfloat16 was leading to instabilities in our setup. Finally, for sampling, we employ top-k sampling [Fan et al., 2018] with keeping the top 250 tokens and a temperature of 1.0.
Text preprocessing. Kreuk et al. [2022] proposed a text normalization scheme, in which stop words are omitted and the remaining text is lemmatized. We denote this method by text-normalization. When considering musical datasets, additional annotations tags such as musical key, tempo, type of instruments, etc. are often available. We also experiment with concatenating such annotations to the text description. We denote this approach by condition-merging. Finally, we explored using word dropout as another text augmentation strategy. For the final models, we used condition-merging with a probability of 0.25. Upon merging, we apply a text description dropout with a probability of 0.5. We use a word dropout with a probability of 0.3 on the resulting text. A full comparison of the different text preprocessing strategies can be found in Appendix A.2.
Codebook patterns and conditioning. We use the "delay" interleaving pattern from Section 2.2, This translates 30 seconds of audio into 1500 autoregressive steps. For text conditioning, we use the T5 [Raffel et al., 2020] text encoder, optionally with the addition of the melody conditioning presented in Section 2.3. We also experiment with FLAN-T5 [Chung et al., 2022], and CLAP [Wu* et al., 2023] and compare the performance of MUSICGEN using each of these text encoders in the Appendix A.2. For melody conditioning, we compute the chromatograms with a window size of  $2^{14}$  and a hop size of  $2^{12}$ . Using a large window prevents the model from recovering fine temporal details. We further quantize the chromatogram by taking the argmax at each time step. We follow a similar approach to Kreuk et al. [2022] and implement classifier-free guidance when sampling from the model's logits. Specifically, during training we drop the condition with a probability of 0.2 and during inference we use a guidance scale of 3.0.
Training datasets. We use 20K hours of licensed music to train MUSICGEN. Specifically, we rely on an internal dataset of 10K high-quality music tracks, and on the ShutterStock and Pond5 music data collections² with respectively 25K and 365K instrument-only music tracks. All datasets consist of full-length music sampled at 32 kHz with metadata composed of a textual description and information such as the genre, BPM, and tags. We downmix the audio to mono unless stated otherwise.
Evaluation datasets. For the main results and comparison with prior work, we evaluate the proposed method on the MusicCaps benchmark [Agostinelli et al., 2023]. MusicCaps is composed of  $5.5\mathrm{K}$  samples (ten-second long) prepared by expert musicians and a 1K subset balanced across genres. We report objective metrics on the unbalanced set, while we sample examples from the genre-balanced
2www.Shutterstock.com/music and www.pond5.com
Table 1: Text-to-Music generation. We compare objective and subjective metrics for MUSICGEN against a number of baselines. We report both mean and CI95 scores. The Mousai model is retrained on the same dataset, while for MusicLM we use the public API for human studies. We report the original FAD on MusicCaps for Noise2Music and MusicLM. "MUSICGEN w. random melody" refers to MUSICGEN trained with chromatogram and text. At evaluation time, we sample the chromatograms at random from a held-out set.
MODEL	MUSICCAPS Test Set
FADvgg ↓	KL ↓	CLAPscr ↑	OVL. ↑	REL. ↑
Riffusion	14.8	2.06	0.19	79.31±1.37	74.20±2.17
Mousai	7.5	1.59	0.23	76.11±1.56	77.35±1.72
MusicLM	4.0	-	-	80.51±1.07	82.35±1.36
Noise2Music	2.1	-	-	-	-
MUSICGEN w.o melody (300M)	3.1	1.28	0.31	78.43±1.30	81.11±1.31
MUSICGEN w.o melody (1.5B)	3.4	1.23	0.32	80.74±1.17	83.70±1.21
MUSICGEN w.o melody (3.3B)	3.8	1.22	0.31	84.81±0.95	82.47±1.25
MUSICGEN w. random melody (1.5B)	5.0	1.31	0.28	81.30±1.29	81.98±1.79
set for qualitative evaluations. For melody evaluation and the ablation studies, we use samples from an in-domain held out evaluation set of 528 music tracks, with no artist overlap with the training set.
Baselines. We compare MUSICGEN to two baselines for text-to-music generation: Diffusion [Forsgren and Martiros] and Mousai [Schneider et al., 2023]. We use the open source Diffusion model to run inference<sup>3</sup>. For Mousai, we train a model using our dataset for a fair comparison, using the open source implementation provided by the authors<sup>4</sup>. Additionally, when possible, we compare to MusicLM [Agostinelli et al., 2023] and Noise2Music [Huang et al., 2023b].
Evaluation metrics. We evaluate the proposed method using objective and subjective metrics. For the objective methods, we use three metrics: the Fréchet Audio Distance (FAD), the Kullback-Leibler Divergence (KL) and the CLAP score (CLAP). We report the FAD [Kilgour et al., 2018] using the official implementation in Tensorflow with the VGGish model  $^{5}$ . A low FAD score indicates the generated audio is plausible. Following Kreuk et al. [2022], we use a state-of-the-art audio classifier trained for classification on AudioSet [Koutini et al., 2021] to compute the KL-divergence over the probabilities of the labels between the original and the generated music. The generated music is expected to share similar concepts with the reference music when the KL is low. Last, the CLAP score [Wu* et al., 2023, Huang et al., 2023a] is computed between the track description and the generated audio to quantify audio-text alignment, using the official pretrained CLAP model  $^{6}$ .