SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table A.1: Text encoder results. We report results for T5, Flan-T5, and CLAP as text encoders. We observe similar results for T5 and Flan-T5 on all the objective metrics. Note that T5 is the text encoder used for the main MUSICGEN models. CLAP encoder performs consistently worse on all the metrics but CLAP score. All comparisons are done with a 300M MUSICGEN model using text conditioning only.
MODEL	MUSICCAPS Test Set
FADvgg ↓	KL ↓	CLAPscr ↑	OVL. ↑	REL. ↑
T5	3.12	1.29	0.31	85.04±1.23	87.33±1.9
Flan-T5	3.36	1.30	0.32	85.54±1.01	85.00±1.63
CLAP	4.16	1.36	0.35	82.13±1.29	83.56±1.54
CLAP (no normalization)	4.14	1.38	0.35	84.87±1.25	85.06±1.72
CLAP (no quantization)	5.07	1.37	0.37	84.13±1.02	84.67±1.42
Table A.2: Text augmentations strategies results. We report objective metrics using only the original text description (no augmentation) and for different text augmentation strategies: using condition merging to augment the text description with metadata, using text-normalization (text-norm.) and applying word dropout on the resulting text. We use 300M MUSICGEN models trained for 500K steps. Condition merging improves the result over training only over the original text description. Other augmentations perform worst on all metrics. We use the Condition Merging with Word dropout, showing the best text relevance, in our main models.
CONFIGURATION	MUSICCAPS Test Set
FADvgg ↓	KL ↓	CLAPscr	OVL. ↑	REL. ↑
No augmentation	3.68	1.28	0.31	83.40±1.44	81.16±1.29
Condition Merging (CM)	3.28	1.26	0.31	82.60±1.41	84.45 ±1.16
CM + Text-norm. (TN)	3.78	1.30	0.29	80.57±2.14	82.40±1.09
CM+ Word dropout (WD)	3.31	1.31	0.30	82.52±1.55	85.27±0.97
CM + TN + WD	3.41	1.39	0.30	81.18±1.91	84.32 ±1.59
we train a RVQ layer on top of the extracted embeddings. More specifically, we first normalize the embeddings and use RVQ with 12 quantizers, each with a codebook size of 1024. Quantizing the CLAP embeddings leads to a homogeneous representation with the discrete tokens further reducing the gap between the audio encoding used at train time and text encoding at test time. We report results for the different text encoders in Table A.1. Both T5 and Flan-T5 perform similarly in terms of objective metrics, the overall quality being slightly better for T5. The CLAP-based model however suffers worse objective and subjective metrics, with the exception of the CLAP score which rely on the same underlying audio and text encoder.
The effect of text augmentations. We examine the impact of text augmentation strategies for the proposed method. In particular, we study the use of condition merging (i.e. concatenating additional metadata to the text description), text normalization (text-norm.) and word dropout. We report objective metrics for the different augmentation strategies in Table A.2. We observe a gain in FAD and KL when leveraging the additional metadata with condition merging. However, neither text normalization or word dropout improves the results in terms of objective metrics.
The effect of the audio tokenizer. We experiment with replacing EnCodec with Descript Audio Codec (DAC) [Kumar et al., 2023] $^{12}$ , a similar audio compression models, that uses a different training set, a similar adversarial loss enhanced with multiband discriminators, but performs quantization in a lower dimension space to improve codebook usage. DAC compresses audio at  $44.1\mathrm{kHz}$  with 9 codebooks, and a framerate of  $86\mathrm{Hz}$ . We trained a small (300M parameters) and a medium (1.5B parameters) MUSICGEN model using both DAC and EnCodec as an audio tokenizer, on a vocal-free version of our dataset. The results provided in Table A.3 show a worse FAD and KL on our in domain test set. On MusicCaps, the FAD is improved by using DAC, although the KL is worse, as well as the subjective evaluations. The EnCodec model used in this work was specifically
<sup>12</sup>Using the public implementation github.com/descriptinc/descript-audio-codec.
Table A.3: We test replacing EnCodec with DAC [Kumar et al., 2023] using their implementation. DAC is a  $44.1\mathrm{kHz}$  model with 9 codebooks and a frame rate of  $86\mathrm{Hz}$ . Those models are trained on a vocal-free version of our dataset, hence the objective metrics will not match those reported in the other tables. We report objective metrics both on our in domain test set and MusicCaps [Agostinelli et al., 2023], and subjective metrics only on MusicCaps.
MODEL	In Domain Test Set	MUSICCAPS Test Set
FADvgg ↓	KL ↓	FADvgg ↓	KL ↓	OVL. ↑
MUSICGEN + DAC small	3.45	0.58	4.46	1.35	83.32±0.95
MUSICGEN + DAC medium	2.42	0.57	4.32	1.30	84.46±0.97
MUSICGEN + EnCodec small	0.67	0.54	5.26	1.27	84.69±0.90
MUSICGEN + EnCodec medium	0.49	0.52	5.05	1.23	86.09±0.88
designed to operate at a lower frame rate (50 Hz) than DAC (86 Hz), thus reducing by  $40\%$  the inference runtime for producing the same audio. Note finally that DAC was trained on a different dataset than EnCodec, and further experiments would be required to understand exactly what influences the fitness of such compression models to be used with auto-regressive language models.
Effect of D-Adaptation.
D-Adaptation is a novel automated way of selecting the overall learning rate of the Adam optimizer, i.e. its  $\alpha$  parameter, dynamically throughout the training, introduced by Defazio and Mishchenko [2023]. We observed improved convergence for the 300M parameter model, although for larger models, e.g. 1.5B and 3.3B, we observed the automated rule led to deteriorated performance, both on the train and validation set. Further investigation is required to better understand the effect of D-Adaptation, and whether it can scale to the largest model. The convergence for both methods can be observed on the train and validation set in Figure A.4.
Figure A.2: Visualization of quantized chromatogram bins over time from reference melody (left), generated music conditioned on chroma and text (middle) and generated music with text-only conditioning (right). Each row is generated using a different chroma condition, all rows share the same text condition: "90s rock song with electric guitar and heavy drums". We observe strong adherence to the input melody for the music samples generated with chroma conditioning while rendering novel styles guided by the input text.
Figure A.3: Histogram of the top 50 musical genres in the training dataset.
Figure A.4: Comparison of Adam and Adam with D-Adaptation [Defazio and Mishchenko, 2023]. While D-Adaptation provided consistent gains for the 300M parameters model, we observed worse convergence both on the train (left) and validation (right) set for the 1.5B parameters model.