SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where, for all  $i$ ,  $\mathbf{MLP}^{(i)}(\mathbf{x}) \coloneqq \mathrm{RMSNorm}(\mathbf{W}_2^{(i)}\mathrm{GELU}(\mathbf{x}\mathbf{W}_1^{(i)} + \mathbf{b}_1^{(i)}) + \mathbf{b}_2^{(i)})$  projects an arbitrary input  $\mathbf{x} \in \mathbb{R}^{D_{\mathrm{in}}}$  to  $\mathbb{R}^{D_{\mathrm{hid}}}$  using RMSNorm [65] and GELU activation [66] with learnable  $\mathbf{W}_1^{(i)} \in \mathbb{R}^{D_{\mathrm{in}} \times D_{\mathrm{hid}}}$ ,  $\mathbf{W}_2^{(i)} \in \mathbb{R}^{D_{\mathrm{hid}} \times D_{\mathrm{hid}}}$ ,  $\mathbf{b}_1^{(i)}, \mathbf{b}_2^{(i)} \in \mathbb{R}^{D_{\mathrm{hid}}}$ , and  $D_{\mathrm{hid}}$  is hidden dimension.
Encoding Semantic Tokens The remaining conditions are the discrete tokens representing semantic information  $\mathbf{u}_1, \ldots, \mathbf{u}_{T_{\mathrm{ST}}}$ . Following the typical approach for embedding natural languages [8], we directly use a lookup table of vectors to map any token  $\mathbf{u}_t \in \{1, \ldots, V_{\mathrm{ST}}\}$  into a real-valued vector  $E(\mathbf{u}_t) \in \mathbb{R}^{D_{\mathrm{hid}}}$ , where  $V_{\mathrm{ST}}$  denotes the vocabulary size of the semantic tokens, i.e., the number of clusters in k-means for Wav2Vec2-Conformer. By stacking the vectors along the time axis and applying another MLP block, we obtain  $\mathbf{E}_{\mathrm{ST}} := \mathbf{MLP}^{(2)}([E(\mathbf{u}_1), \ldots, E(\mathbf{u}_{T_{\mathrm{ST}}})]) \in \mathbb{R}^{T_{\mathrm{ST}} \times D_{\mathrm{hid}}}$ .
Conditional on the computed embeddings  $\mathbf{E}_{\delta}$  and  $\mathbf{E}_{\mathrm{ST}}$ , we next show how the network input, i.e.,  $\mathbf{z}_{\delta_t}$  for the case of having same noise scale  $\delta_t$  for all chunks and  $\bar{\mathbf{z}}_{\delta_1} \oplus \dots \oplus \bar{\mathbf{z}}_{\delta_M}$  for the case of having different noise scales, is processed in DPD for velocity prediction. For the simplicity of notation, we use  $\mathbf{z}_{\delta_t}$  to denote the network input here and below.  $\mathbf{z}_{\delta_t}$  is first linearly transformed and added up with the angle embedding of the same shape:  $\mathbf{H} := \mathrm{RMSNorm}\left(\mathbf{z}_{\delta_t}\mathbf{W}_{\mathrm{in}} + \mathbf{E}_{\delta}\right)$ , where  $\mathbf{W}_{\mathrm{in}} \in \mathbb{R}^{D \times D_{\mathrm{hid}}}$  is learnable. Then, a crucial segmentation operation is applied for dual-path modeling.
Segmentation As illustrated in Figure [3], the segmentation module divides a 2-D input into  $S$  half-overlapping segments each of length  $K$ , represented by a 3-D tensor  $\mathbb{H} := [\mathbf{0}, \mathbf{H}_1, \dots, \mathbf{H}_S, \mathbf{0}] \in \mathbb{R}^{S \times K \times D_{\mathrm{hid}}}$ , where  $\mathbf{H}_s := \mathbf{H}\left[\frac{(s-1)K}{2} : \frac{(s-1)K}{2} + K,:\right]$ , and  $\mathbb{H}$  is zero-padded such that we have
Figure 3: Diagram for visually understanding the segmentation operation for dual-path modeling
$S = \left\lceil \frac{2L}{K}\right\rceil +1$ . By choosing a segment size  $K\approx \sqrt{L}$ , the costs of sequence processing become sub-linear  $(\mathcal{O}(\sqrt{L}))$  as opposed to  $(\mathcal{O}(L))$ . This greatly reduces the difficulty of learning a very long sequence and permits MeLoDy to use higher-frequency latents for better audio quality. In this work, 250Hz latent sequences was used. In comparison, MusicLM [5] was built upon 50Hz codec.
Dual-Path Blocks After the segmentation, we obtain a 3-D tensor input. As shown in Figure 2, the tensor is subsequently passed to  $N$  dual-path blocks, where each block contains two processing stages corresponding to coarse-path (i.e., inter-segment) and fine-path (i.e., intra-segment) processing, respectively. Similar to the observations in [40, 41], we find it superior to use an attention-based network for coarse-path processing and to use a bi-directional RNN for fine-path processing. The goal of fine acoustic modeling is to better reconstruct the fine details from the roughly determined audio structure [3]. At a finer scope, only the nearby elements matter and contain the most information needed for refinement, as supported by the modeling perspectives in neural vocoding [33, 35]. Specifically, we employ the Roformer network [67] for coarse-path processing, where we use a self-attention layer followed by a cross-attention layer to be conditional on  $\mathbf{E}_{\mathrm{ST}}$  with rotary positional embeddings. On the other hand, we use a stack of 2-layer simple recurrent units (SRUs) [68] for fine-path processing. A feature-wise linear modulation (FiLM) [69] layer is applied to the output of SRUs to assist the denoising with the angle embedding  $\mathbf{E}_{\delta}$  and the pooled  $\mathbf{E}_{\mathrm{ST}}$ . The details of inner mechanism in each dual-path block is presented in Appendix B.
4.2.4 Music Generation and Continuation
Suppose we have a well-trained multi-chunk velocity model  $\hat{\mathbf{v}}_{\theta}$ , we begin with a  $L$ -length latent generation, where  $L$  is the length of latents we used in training. According to Appendix A, the DDIM sampling algorithm [26] can be re-formulated by applying the trigonometric identities:
$$
\mathbf {z} _ {\delta_ {t} - \omega_ {t}} = \cos (\omega_ {t}) \mathbf {z} _ {\delta_ {t}} - \sin (\omega_ {t}) \hat {\mathbf {v}} _ {\theta} \left(\mathbf {z} _ {\delta_ {t}}; \mathbf {c}\right), \tag {12}
$$
which, by running from  $t = T$  to  $t = 1$  using the defined  $\omega_{t}$  in Eq. (4), we can generate a sample of  $\mathbf{z}_0$  of length  $L$ . To continue generation, we append a new chunk composed of random noises to the generated  $\mathbf{z}_0$  and drop the first chunk in  $\mathbf{z}_0$ . Recall that the inputs to  $\hat{\mathbf{v}}_{\theta}$  are the  $M$  concatenated noisy latents of different noise scales. The continuation of generation is feasible since the conditions (i.e., the semantic tokens and the angle vector) defined in DPD have an autoregressive nature at inference time. On one hand, the semantic tokens are generated by the semantic LM in an autoregressive manner, therefore we can continue the generation of semantic tokens for the new chunk. On the other hand, since the multi-chunk model  $\hat{\mathbf{v}}_{\theta}$  is trained to tackle chunks of different noise scales with respect to the angle vector, we can simply ignore the generated audio (on the first  $M - 1$  chunks) by zeroing the respective values and setting ones for the newly appended chunk, i.e.,  $\delta_{\mathrm{new}} := [0]_{r=1}^{L - \lceil L / M \rceil} \oplus [\delta_t]_{r=1}^{\lceil L / M \rceil}$ . Then, the newly appended noise chunk can be transformed to meaningful music audio after  $\lceil T / M \rceil$  step of DDIM sampling. For more details of generation, we present the corresponding algorithms in Appendix C. Besides music continuation, based on MuLan, MeLoDy also supports music prompts to generate music of a similar style, as shown in Figure [1]. Examples of music continuation, and music prompting are shown on our demo page.
Table 2: The speed and the quality of our proposed MeLoDy on a CPU (Intel Xeon Platinum 8260 CPU @ 2.40GHz) or a GPU (NVIDIA Tesla V100) using different numbers of sampling steps.
Steps (T)	Speed on CPU (↑)	Speed on GPU (↑)	FAD (↓)	MCC (↑)
(MusicCaps)	-	-	-	0.43
5	1472Hz (0.06×)	181.1kHz (7.5×)	7.23	0.49
10	893Hz (0.04×)	104.8kHz (4.4×)	5.93	0.52
20	498Hz (0.02×)	56.9kHz (2.4×)	5.41	0.53
Data Preparation As shown in Table [1]. MeLoDy was trained on 257k hours of music data (6.4M 24kHz audios), which were filtered with [27] to focus on non-vocal music. Additionally, inspired by the text augmentation in [6], we enriched the tag-based texts to generate music captions by asking ChatGPT [70]. This music description pool is used for the training of our 195.3M MuLan, where we randomly paired each audio with either the generated caption or its respective tags. In this way, we robustly improve the model's capability of handling free-form text.
Semantic LM For semantic modeling, we trained a 429.5M LLaMA [71] with 24 layers, 8 heads, and 2048 hidden dimensions, which has a comparable number of parameters to that of the MusicLM [5]. For conditioning, we set up the MuLan RVQ using 12 1024-sized codebooks, resulting in 12 prefixing tokens. The training targets were 10s semantic tokens, which are obtained from discretizing the  $25\mathrm{Hz}$  embeddings from a 199.5M Wav2Vec2-Conformer with 1024-center k-means.
Dual-Path Diffusion For the DPD model, we set the hidden dimension to  $D_{\mathrm{hid}} = 768$ , and block number to  $N = 8$ , resulting in 296.6M parameters. For the input chunking strategy, we divide the 10s training inputs in a fixed length of  $L = 2500$  into  $M = 4$  parts. For segmentation, we used a segment size of  $K = 64$  (i.e., each segment is 256ms long), leading to  $S = 80$  segments. In addition, we applied the classifier-free guidance (CFG) [72] to DPD to improve the correspondence between samples and conditions. During training, the cross-attention to semantic tokens is randomly replaced by self-attention with a probability of 0.1. For sampling, the unconditional prediction  $\mathbf{v}_{\mathrm{uncond}}$  and the conditional prediction  $\mathbf{v}_{\mathrm{cond}}$  are linearly combined:  $\rho \mathbf{v}_{\mathrm{cond}} + (1 - \rho)\mathbf{v}_{\mathrm{uncond}}$  with a scale of  $\rho = 2.5$ .
Audio VAE-GAN For audio VAE-GAN, we used a hop size of 96, resulting in  $250\mathrm{Hz}$  latent sequences for encoding  $24\mathrm{kHz}$  music audio. The latent dimension  $D = 16$ , thus we have a total compression rate of  $6\times$ . The hidden channels used in the encoder were 256, whereas that used in the decoder were 768. The audio VAE-GAN in total contains 100.1M parameters.
5.2 Performance Analysis
Objective Metrics We use the VGGish-based [73] Frechet audio distance (FAD) [74] between the generated audios and the reference audios from MusicCaps [5] as a rough measure of generation fidelity. To measure text correlation, we use the MuLan cycle consistency (MCC) [5], which calculates the cosine similarity between text and audio embeddings using a pre-trained MuLan.
Inference Speed We first evaluate the sampling efficiency of our proposed MeLoDy. As DPD permits using different numbers of sampling steps depending on our needs, we report its generation speed in Table 2. Surprisingly, MeLoDy steadily achieved a higher MCC score than that of the reference set, even taking only 5 sampling steps. This means that (i) the MuLan model determined that our generated samples were more correlated to MusicCaps captions than reference audios, and (ii) the proposed DPD is capable of consistently completing the MuLan cycle at significantly lower costs than the nested LMs in [5].
2Note that MeLoDy was mainly trained with non-vocal music data, its sample distribution could not fit the reference one as well as in [5, 6], since about  $76\%$  audios in MusicCaps contain either vocals or speech.
3Since our MuLan model was trained with a different dataset, our MCC results cannot be compared to [5, 6].
Table 3: The comparison of MeLoDy with the SOTA text-to-music generation models. NFE is the number of function evaluations [58] for generating  $T$ -second audio  $^5$ Musicality, Quality, and Text Corr. are the winning proportions in terms of musicality, quality, and text correlation, respectively.
Model	NFE (↓)	Musicality (↑)	Quality (↑)	Text Corr. (↑)
MLM	N2M	MLM	N2M	MLM	N2M
MusicLM [5]	(25 + 200 + 400)T	0.541	-	0.465	-	0.548	-
Noise2Music [6]	1000 + 800 + 800	-	0.555	-	0.436	-	0.572
MeLoDy (20 steps)	25T + 20	0.459	0.445	0.535	0.564	0.452	0.428
Comparisons with SOTA models We evaluate the performance of MeLoDy by comparing it to MusicLM [5] and Noise2Music [6], which both were trained large-scale music datasets and demonstrated SOTA results for a wide range of text prompts. To conduct fair comparisons, we used the same text prompts in their demos (70 samples from MusicLM; 41 samples from Noise2Music) and asked seven music producers to select the best out of a pair of samples or voting for a tie (both win) in terms of musicality, audio quality, and text correlation. In total, we conducted 777 comparisons and collected 1,554 ratings. We detail the evaluation protocol in Appendix F. Table 3 shows the comparison results, where each category of ratings is separated into two columns, representing the comparison against MusicLM (MLM) or Noise2Music (N2M), respectively. Finally, MeLoDy consistently achieved comparable performances (all winning proportions fall into [0.4, 0.6]) in musicality and text correlation to MusicLM and Noise2Music. Regarding audio quality, MeLoDy outperformed MusicLM  $(p < 0.05)$  and Noise2Music  $(p < 0.01)$ , where the  $p$ -values were calculated using the Wilcoxon signed-rank test. We note that, to sample 10s and 30s music, MeLoDy only takes  $4.32\%$  and  $0.41\%$  NFEs of MusicLM, and  $10.4\%$  and  $29.6\%$  NFEs of Noise2Music, respectively.
Diversity Analysis Diffusion models are distinguished for its high diversity [25]. We conduct an additional experiment to study the diversity and validity of MeLoDy's generation given the same text prompt of open description, e.g., feelings or scenarios. The sampled results were shown on our demo page, in which we obtained samples with diverse combinations of instruments and textures.
Ablation Studies We also study the ablation on two aspects of the proposed method. In Appendix D, we compared the uniform angle schedule in [23] and the linear one proposed in DPD using the MCC metric and case-by-case qualitative analysis. It turns out that our proposed schedule tends to induce fewer acoustic issues when taking a small number of sampling steps. In Appendix E, we showed that the proposed dual-path architecture outperformed other architectures [23, 31] used for LDMs in terms of the signal-to-noise ratio (SNR) improvements using a subset of the training data.
**Limitation** We acknowledge the limitations of our proposed MeLoDy. To prevent from having any disruption caused by unnaturally sound vocals, our training data was prepared to mostly contain non-vocal music only, which may limit the range of effective prompts for MeLoDy. Besides, the training corpus we used was unbalanced and slightly biased towards pop and classical music. Lastly, as we trained the LM and DPD on 10s segments, the dynamics of a long generation may be limited.
Broader Impact We believe our work has a huge potential to grow into a music creation tool for music producers, content creators, or even normal users to seamlessly express their creative pursuits with a low entry barrier. MeLoDy also facilitates an interactive creation process, as in Midjourney [24], to take human feedback into account. For a more precise tune of MeLoDy on a musical style, the LoRA technique [75] can be potentially applied to MeLoDy, as in Stable Diffusion [49].
4All samples for evaluation are available at https://Efficient-MeLoDy.github.io/. Note that our samples were not cherry-picked, whereas the samples we compared were cherry-picked [6], constituting very strong baselines.
We use  $+$  to separate the counts for the iterative modules, i.e., LM or DPM. Suppose the cost of each module is comparable, then the time steps taken by LM and the diffusion steps taken by DPM can be fairly compared.
[69] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[70] OpenAI. Chatgpt. URL https://chat.openai.com/, 2023.
[71] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[72] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
[73] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In 2017 iiem international conference on acoustics, speech and signal processing (icassp), pages 131-135. IEEE, 2017.
[74] Kevin Kilgour, Maurizio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms. In *INTERSPEECH*, pages 2350–2354, 2019.
[75] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.