SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the human studies, we follow the same setup as in Kreuk et al. [2022]. We ask human raters to evaluate two aspects of the audio samples (i) overall quality (OVL), and (ii) relevance to the text input (REL). For the overall quality test, raters were asked to rate the perceptual quality of the provided samples in a range of 1 to 100. For the text relevance test, raters were asked to rate the match between audio and text on a scale of 1 to 100. Raters were recruited using the Amazon Mechanical Turk platform. We evaluate randomly sampled files, where each sample was evaluated by at least 5 raters. We use the CrowdMOS package<sup>7</sup> to filter noisy annotations and outliers. We remove annotators who did not listen to the full recordings, annotators who rate the reference recordings less than 85, and the rest of the recommended recipes from CrowdMOS [Ribeiro et al., 2011]. For fairness, all samples are normalized at  $-14\mathrm{dB}$  LUFS [ITU-R, 2017].
3Using riffusion-model-v1 from github.com/riffusion/riffusion-app (on May 10, 2023)
4Implementation from github.com/archinetai.audio-diffusion-pytorch (March 2023)
$^{5}$  github.com/google-research/google-research/tree/master/frechet.audio_distance
$^{6}$ https://github.com/LAION-AI/CLAP
<sup>7</sup>http://wwwCrowdmos.org/download/
Table 2: We report cosine similarity between reference and generated melody (SIM.) and subjective metrics including alignment with the melody (MEL.). All results are reported for MUSICGEN 1.5B.
Train CONDITION	TEST CONDITION	In Domain Test Set
SIM.↑	MEL.↑	OVL.↑	REL.↑
Text	Text	0.10	64.44±0.83	82.18±1.21	81.54±1.22
Text+Chroma	Text	0.10	61.89±0.96	81.65±1.13	82.50±0.98
Text+Chroma	Text+Chroma	0.66	72.87±0.93	83.94±1.99	80.28±1.06
We start by presenting results of the proposed method on the task of text-to-music generation and compare MUSICGEN to prior work in the field. Next, we evaluate the ability of the proposed method to generate music conditioned on melodic features. We further show how to simply extend our codebook patterns for stereo audio generation. We conclude with an ablation study. Music samples, code, and models are available at github.com/facebookresearch/audiocraft.
4.1 Comparison with the baselines
Table 1 presents the comparison of the proposed method against Mousai, Diffusion, MusicLM, and Noise2Music. As there is no official implementation of Noise2Music, nor pre-trained model, we report only FAD on MusicCaps, which is reported in the Noise2Music manuscript. Similarly, MusicLM implementation is not public. We use the MusicLM public demo for our subjective tests while reporting the FAD as reported by the authors. While the original MusicLM model is trained on data with vocals, the model behind the API is instrument-only. For the human study, we restrict ourselves to 40 instrument-only samples from MusicCaps. To prevent leakage in MUSICGEN trained with chromatogram, we sample chromatograms at random from a held-out set during test time.
Results suggest that MUSICGEN performs better than the evaluated baselines as evaluated by human listeners, both in terms of audio quality and adherence to the provided text description. Noise2Music performs the best in terms of FAD on MusicCaps, followed by MUSICGEN trained with text conditioning. Interestingly, adding a melody conditioning degrades the objective metrics, however, it does not significantly affect human ratings, while still being superior to the evaluated baselines.
We notice that for the worst rated model, the FAD is correlated with the overall subjective rating, but it is not the case for the best rated models. We noticed that a large number of samples in MusicCaps [Agostinelli et al., 2023] contains a description stating that the recording is "noisy". It is possible that due to those noisy samples, improvements in the quality of the generated audio might deteriorate the FAD on MusicCaps once a certain quality threshold is achieved.
4.2 Melody evaluation
We evaluate MUSICGEN, conditioned jointly on textual and melodic representations, using objective and subjective metric on the held out evaluation set. For the objective evaluation, we introduce a new metric: chroma cosine-similarity, which measures the average cosine-similarity between frames corresponding to the same time steps, taken from the quantized chroma of the reference and the generated samples. We evaluate using 1000 randomly sampled files from a held-out set. To better evaluate the relation between the conditioned melody to the generated music, we introduce another human study. To that end, we present human raters with a reference musical piece, followed by a set of generated pieces. For each generated sample, the listeners are asked to rate how well the melody of the generated piece matches that of the reference on a scale of 1 to 100. We use 40 samples of 10 seconds at random from the held-out set. Results are reported in Table 2. Results suggest that MUSICGEN trained with chromatogram conditioning successfully generates music that follows a given melody. Thus, allowing for better control over the generated output. Interestingly, MUSICGEN is robust to dropping the chroma at inference time with both OVL. and REL. staying roughly the same.
<sup>8</sup>https://blog.google/technology/ai/musiclm-google-ai-test-kitchen/
Figure 2: Stereo codebooks (left) and memorization results (right)
Table 3: Stereophonic Text-to-Music generation. EnCodec processes separately the left and right channels, giving us 8 codebooks instead of 4. We experiment with two codebook patterns, depicted in Figure 2a. We also measure one of the stereo model after being downmixed to mono. We use a 1.5B MUSICGEN model conditioned only on text.
CB. PATTERN	STEREO?	MUSICCAPS Test Set
OVL. ↑	REL. ↑
Mono Delay	✘	84.95±1.60	80.61±1.22
Stereo Partial Delay	✘*	84.49±1.80	79.39±1.16
Stereo Partial Delay	✓	86.73±1.06	80.41±1.15
Stereo Delay	✓	85.51±1.21	78.32 ±1.21
*: downmixed to mono after generation.
4.3 Fine-tuning for stereophonic generation
We experiment with for extending generation to stereophonic data. We use the same EnCodec tokenizer, which we applied independently to the left and right channels, providing  $2 \cdot K = 8$  codebooks per frame. Starting from a pre-trained monophonic MUSICGEN model, we fine tune it for 200K batches the same dataset with stereo audio. We reuse the "delay" pattern, with two possible variations: (i) "stereo delay" introduces a delay between the left and right channels for the same codebook level, while (ii) "stereo partial delay" apply the same delay to the left and right channel codebooks for a given level, as depicted in Figure 2a. Note that using this simple strategy, we can generate stereo audio at no extra computational cost. We provide in Table 3 the subjective evaluations for those models. We notice that when downmixing the stereo output to mono, we are almost equivalent in perceived quality to a mono model. Stereo audio was overall rated higher than the mono counterpart, and the "stereo partial delay" benefits from a small boost both in overall quality and text relevance compared with the "stereo delay" pattern.
This section provides an ablation study for the different codebook patterns, together with results for model scales and a memorization study. Additionally, we present results for different text augmentation strategies, text encoders, and audio tokenization models in Appendix A.2. All ablations are performed using 1K samples of 30 seconds, randomly sampled from the held-out evaluation set.
The effect of the codebook interleaving patterns. We evaluate various codebook patterns using the framework from Section 2.2, with  $K = 4$ , given by the audio tokenization model. Table 1 reports results with the "delay" pattern. The "partial delay" consists in delaying by the same amount the codebooks 2, 3, and 4. The "parallel" pattern predicts all the codebooks from the same time step in parallel. The "coarse first" pattern first predicts codebook 1 for all steps, then predicts in parallel codebooks 2, 3, and 4. Thus, this pattern has twice the steps compared to other patterns. "Partial
Table 4: Codebook patterns. We compare different codebook interleaving patterns on 30-seconds, audio sequences. The "flattening" pattern achieves the best scores. The "delay" and "partial flattening" patterns achieve similar scores, while "parallel" obtains worse scores.
CONFIGURATION	Nb. steps	In Domain Test Set
FADvgg ↓	KL ↓	CLAPscr ↑	OVL. ↑	REL. ↑
Delay	1500	0.96	0.52	0.35	79.69±1.46	79.67±1.41
Partial delay	1500	1.51	0.54	0.32	79.13±1.56	79.67±1.46
Parallel	1500	2.58	0.62	0.27	72.21±2.49	80.30±1.43
Partial flattening	3000	1.32	0.54	0.34	78.56±1.86	79.18±1.49
Coarse first	3000	1.98	0.56	0.30	74.42±2.28	76.55±1.67
Flattening	6000	0.86	0.51	0.37	79.71±1.58	82.03±1.1
Table 5: Model scale. We compare 3 scales for our method, and evaluate it on an internal test set to limit the impact of the out of domain prediction issues we observed with MusicCaps. In terms of objective metrics we observe a continuous improvement of the metrics, although subjective quality stop improving at 1.5B parameters. A 3.3B model however seems to better fit the text prompt.
Dim.	Heads	Depth	#Param.	In Domain Test Set
PPL↓	FADvgg ↓	KL ↓	CLAPscr ↑	OVL. ↑	REL. ↑
1024	16	24	300M	56.1	0.96	0.52	0.35	78.3±1.4	82.5 ±1.6
1536	24	48	1.5B	48.4	0.86	0.50	0.35	81.9±1.4	82.9±1.5
2048	32	48	3.3B	46.1	0.82	0.50	0.36	79.2±1.3	83.5±1.3
flattening" is similar, but instead of sampling first codebook 1 for all steps, it interleaves them with the parallel sampling of codebooks 2, 3, and 4. Finally, the "flattening" pattern consists in flattening all the codebooks, similar to MusicLM [Agostinelli et al., 2023]. We report objective and subjective evaluations in Table 4. While flattening improves generation, it comes at a high computational cost and similar performance can be reached for a fraction of this cost using a simple delay approach.
The effect of model size. In Table 5 we report results for different model sizes, namely 300M, 1.5B, and 3.3B parameter models. As expected, scaling the model size results in better scores, however this comes at the expense of longer training and inference time. Regarding subjective evaluations, the overall quality is optimal at 1.5B, but a larger model can better understand the text prompt.
Memorization experiment. Following [Agostinelli et al., 2023], we analyze the memorization abilities of MUSICGEN. We only consider the first stream of codebooks from MUSICGEN as it contains the coarser-grain information. We randomly select  $N = 20,000$  examples from our training set and for each one we feed the model with a prompt of EnCodec codebooks corresponding to the original audio and the conditioning information. We generate a continuation of 250 audio tokens (5 second-long audio) using greedy decoding. We report exact matches as the fraction of examples for which the generated audio tokens and source audio tokens are identical over the whole sequence. In addition, we report partial matches as the fraction of the training examples for which the generated and source sequences have at least  $80\%$  of the audio tokens matching. We present the memorization results for the different model sizes when varying the length of the audio prompt in Figure 2b.
Audio representation. In recent years, the prominent approach is to represent the music signals in a compressed representation, discrete or continuous, and apply a generative model on top of it. Lakhotia et al. [2021] proposed quantizing speech representation using k-means to construct speech language models. Recently, Defossez et al. [2022], Zeghidour et al. [2021] proposed to apply a VQ-VAE directly on the raw waveform using residual vector quantization. Later, several studies used such representation for text-to-audio generation. Next, we discuss the recent research in audio generation.
Music generation. Music generation has been long studied under various setups. Dong et al. [2018] proposed a GAN-based approach for symbolic music generation. Bassan et al. [2022] proposed an unsupervised segmentation for symbolic music which can be later used for generation. Ycart et al.
[2017] proposed modeling polyphonic music modeling using recurrent neural networks. Ji et al. [2020] conducted a comprehensive survey therein for deep learning methods for music generation.
Dhariwal et al. [2020] proposed representing music samples in multiple streams of discrete representations using a hierarchical VQ-VAE. Next, two sparse transformers applied over the sequences to generate music. Gan et al. [2020] proposed generating music for a given video, while predicting its midi notes. Recently, Agostinelli et al. [2023] proposed representing music using multiple streams of "semantic tokens" and "acoustic tokens". Then, they applied a cascade of transformer decoders conditioned on a textual-music joint representation [Huang et al., 2022]. Donahue et al. [2023] followed a similar modeling approach, but for the task of singing-to-accompanyment generation.
An alternative approach is using diffusion models. Schneider et al. [2023], Huang et al. [2023b], Maina [2023], Forsgren and Martiros proposed using a latent diffusion model for the task of text-to-music. Schneider et al. [2023] proposed using diffusion models for both audio encoder-decoder and latent generation. Huang et al. [2023b] proposed a cascade of diffusion model to generate audio and gradually increase its sampling rate. Forsgren and Martiros proposed fine-tuning Stable Diffusion Rombach et al. [2022] using spectrograms to generate five-second segments, then, using image-to-image mapping and latent interpolation to generate long sequences.
Audio generation. Several studies were proposed for text-to-audio (environmental sounds) generation. Yang et al. [2022] proposed representing audio spectrograms using a VQ-VAE, then applying a discrete diffusion model conditioned on textual CLIP embeddings for the generation part [Radford et al., 2021]. Kreuk et al. [2022] proposed applying a transformer language model over discrete audio representation, obtained by quantizing directly time-domain signals using EnCodec [Défossez et al., 2022]. Sheffer and Adi [2023] followed a similar approach to Kreuk et al. [2022] for image-to-audio generation. Huang et al. [2023a], Liu et al. [2023] proposed using latent diffusion models for the task of text-to-audio, while extending it to various other tasks such as inpainting, image-to-audio, etc.
We introduced MUSICGEN, a state-of-the-art single stage controllable music generation model that can be conditioned on text and melody. We demonstrated that simple codebook interleaving strategies can be used to achieve high quality generation, even in stereo, while reducing the number of autoregressive time steps compared to the flattening approach. We provided a comprehensive study of the impact of model sizes, conditioning methods, and text pre-processing techniques. We also introduced a simple chromatogram-based conditioning for controlling the melody of the generated audio.
**Limitations** Our simple generation method does not allow us to have fine-grained control over adherence of the generation to the conditioning, we rely mostly on CF guidance. Also, while it is relatively straightforward to do data augmentation for text conditioning, conditioning on audio warrants further research on data augmentation, types and amount of guidance.
Broader impact. Large scale generative models present ethical challenges. We first ensured that all the data we trained on was covered by legal agreements with the right holders, in particular through an agreement with ShutterStock. A second aspect is the potential lack of diversity in the dataset we used, which contains a larger proportion of western-style music. However, we believe the simplification we operate in this work, e.g., using a single stage language model and a reduced number of auto-regressive steps, can help broaden the applications to new datasets. Generative models can represent an unfair competition for artists, which is an open problem. Open research can ensure that all actors have equal access to these models. Through the development of more advanced controls, such as the melody conditioning we introduced, we hope that such models can become useful both to music amateurs and professionals.
The authors would like to thank Mary Williamson, Rashel Moritz and Joelle Pineau for supporting this project, thank Justin Luk, Prash Jain, Sidd Srinivasan, Rod Duenes, and Philip Woods for the dataset, and thank the xformers team: Daniel Haziza, Francisco Massa, and Michael Ramamonjisoa for the technical discussions.
Authors note. This paper was submitted in the wake of the tragic terror attack perpetrated by Hamas on October 7, 2023, which has left the Israeli nation profoundly devastated. The assault greatly
impacted us at a personal level, thereby significantly impacting the course of this research work. This paper was finalized while we grieve and mourn our friends and family, under great stress, with scientific considerations being the last thing on our minds. It may contain subtle errors.
In memory of the countless lives shattered by Hamas actions.
ITU-R. Algorithms to measure audio programme loudness and true-peak audio level, 2017.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336-1354, 2021.
Hao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang. Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Shahaf Bassan, Yossi Adi, and Jeffrey S Rosenschein. Unsupervised symbolic music segmentation using ensemble temporal prediction errors. arXiv preprint arXiv:2207.00760, 2022.
Adrien Ycart, Emmanouil Benetos, et al. A study on lstm networks for polyphonic music sequence modelling. ISMIR, 2017.
Shulei Ji, Jing Luo, and Xinyu Yang. A comprehensive survey on deep music generation: Multi-level representations, algorithms, evaluations, and future directions. arXiv preprint arXiv:2011.06801, 2020.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020.
Chuang Gan, Deng Huang, Peihao Chen, Joshua B Tenenbaum, and Antonio Torralba. Foley music: Learning to generate music from videos. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 758-775. Springer, 2020.
Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel PW Ellis. Mulan: A joint embedding of music audio and natural language. arXiv preprint arXiv:2208.12415, 2022.
Kinyugo Maina. Msanii: High fidelity music synthesis on a shoestring budget. arXiv preprint arXiv:2301.06468, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. arXiv preprint arXiv:2207.09983, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis Bach. Music source separation in the waveform domain. arXiv preprint arXiv:1911.13254, 2019.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved rvqgan. arXiv preprint arXiv:2306.06546, 2023.
Figure A.1: Visualizing partial flattening and partial delays codebook patterns applied on a sequence with 4 parallel streams of quantized values (corresponding to  $k_{1}, \ldots, k_{4}$ ) and  $N$  time steps  $(t_{1}, \ldots, t_{n})$ . "Partial flattening" separates the first codebook in dedicated steps and interleaves them with the parallel sampling of codebooks 2, 3, and 4, leading the number of interleaved sequences steps  $M$  to be twice the number of original steps  $N$ . The "partial delay" pattern consists in delaying by the same amount the codebooks 2, 3, and 4, in our case we use a delay of 1. The total number of steps of the interleave sequences is  $N$  (excluding the last step for simplicity).
A.1 Experimental details
Codebook interleaving patterns. Figure A.1 provides a visual description of the additional codebook patterns introduced for the ablation in Section 4, namely "partial flattening" and "partial delay" patterns. The intuition behind such patterns is driven by the fact that the first codebook from RVQ is the most important one and predicting the rest of the codebooks in parallel would allow to limit the introduced flattening or delay while maintaining good modeling performance.
Melody conditioning. In this work, we provide an unsupervised approach for melody conditioning through conditioning on the chromatogram representation. As shown in Figure A.2, chromatogram-based conditioning successfully preserves the melodic structure when generating novel music samples. In preliminary experiments, we noticed that the chromatogram tends to be dominated by the lower frequency instruments, mainly by the drums and bass. To mitigate that, we used Demucs [Défossez et al., 2019] to first decompose the reference track into four components: drums, bass, vocals, and other. Next, we omit the drums and bass to recover the melodic structure of the residual waveform. Finally, we extract the quantized chromatogram to create the conditioning that is later fed to the model.
Distribution of genres. We provide in Figure A.3 a histogram of the top 50 musical genres present in the dataset. We notice a clear dominance of the Dance/EDM genre, which in our experience is also one of the genres that is best generated by MUSICGEN. While we tried to explore a number of resampling schemes to boost the importance of other genres, we observed that oversampling less represented genres would often lead to a worse model overall.
A.2 Additional experimental results
We provide further ablation studies on the core components of MUSICGEN, namely the text encoder used for text conditioning described in Section 2.3, text augmentation strategies presented in Section 3.1, and the used audio tokenization model. We report results on the MusicCaps dataset to better understand out-of-domain generalization abilities of the different approaches. Finally, we share additional experimental results on optimization methods.
The effect of text encoder. We investigate the impact of the text encoder, comparing three different encoders: T5 [Raffel et al., 2020]  $^{9}$ , Flan-T5 [Chung et al., 2022]  $^{10}$  and CLAP [Wu* et al., 2023]  $^{11}$  with a quantization bottleneck. For the CLAP-based encoder, similarly to Agostinelli et al. [2023] we rely on the music embeddings during training and the text embeddings at inference time and
$^{9}$ https://huggingface.co/t5-base  
 $^{10}$ https://huggingface.co/google/flan-t5-base  
 $^{11}$ https://huggingface.co/lukewys/laion_clap/blob/main/music_audioset_epoch_15_esc_90.14.pt