SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Max W. Y. Lam, Qiao Tian, Tang Li, Zongyu Yin, Siyuan Feng, Ming Tu, Yuliang Ji, Rui Xia, Mingbo Ma, Xuchen Song, Jitong Chen, Yuping Wang, Yuxuan Wang Speech, Audio & Music Intelligence (SAMI), ByteDance
Recent progress in music generation has been remarkably advanced by the state-of-the-art MusicLM, which comprises a hierarchy of three LMs, respectively, for semantic, coarse acoustic, and fine acoustic modelings. Yet, sampling with the MusicLM requires processing through these LMs one by one to obtain the fine-grained acoustic tokens, making it computationally expensive and prohibitive for a real-time generation. Efficient music generation with a quality on par with MusicLM remains a significant challenge. In this paper, we present MeLoDy (M for music; L for LM; D for diffusion), an LM-guided diffusion model that generates music audios of state-of-the-art quality meanwhile reducing  $95.7\%$  to  $99.6\%$  forward passes in MusicLM, respectively, for sampling 10s to 30s music. MeLoDy inherits the highest-level LM from MusicLM for semantic modeling, and applies a novel dual-path diffusion (DPD) model and an audio VAE-GAN to efficiently decode the conditioning semantic tokens into waveform. DPD is proposed to simultaneously model the coarse and fine acoustics by incorporating the semantic information into segments of latents effectively via cross-attention at each denoising step. Our experimental results suggest the superiority of MeLoDy, not only in its practical advantages on sampling speed and infinitely controllable generation, but also in its state-of-the-art musicality, audio quality, and text correlation.
Our samples are available at https://Efficient-MeLoDy.github.io/
Music is an art composed of harmony, melody, and rhythm that permeates every aspect of human life. With the blossoming of deep generative models [1-3], music generation has drawn much attention in recent years [4-6]. As a prominent class of generative models, language models (LMs) [7,8] showed extraordinary modeling capability in modeling complex relationships across long-term contexts [9-11]. In light of this, AudioLM [3] and many follow-up works [5,12-14] successfully applied LMs to audio synthesis. Concurrent to the LM-based approaches, diffusion probabilistic models (DPMs) [1,15,16], as another competitive class of generative models [2,17], have also demonstrated exceptional abilities in synthesizing speech [18-20], sounds [21,22] and music [6,23].
However, generating music from free-form text remains challenging as the permissible music descriptions can be very diverse and relate to any of the genres, instruments, tempo, scenarios, or even some subjective feelings. Conventional text-to-music generation models are listed in Table[1] where both MusicLM [5] and Noise2Music [6] were trained on large-scale music datasets and demonstrated the state-of-the-art (SOTA) generative performances with high fidelity and adherence to various aspects of text prompts. Yet, the success of these two methods comes with large computational costs, which would be a serious impediment to their practicalities. In comparison, Mousai [23] building upon DPMs made efficient samplings of high-quality music possible. Nevertheless, the number of their demonstrated cases was comparatively small and showed limited in-sample dynamics. Aiming for a feasible music creation tool, a high efficiency of the generative model is essential since it facilitates interactive creation with human feedback being taken into account as in [24].
Table 1: A comparison of MeLoDy with conventional text-to-music generation models in the literature. We use AC to denote whether audio continuation is supported, FR to denote whether the sampling is faster than real-time on a V100 GPU, VT to denote whether the model has been tested and demonstrated using various types of text prompts including instruments, genres, and long-form rich descriptions, and MP to denote whether the evaluation was done by music producers.
Model	Prompts	Training Data	AC	FR	VT	MP
Moussi [23]	Text	2.5k hours of music	✓	✓	✗	✗
MusicLM [5]	Text, Melody	280k hours of music	✓	✗	✓	✗
Noise2Music [6]	Text	340k hours of music	✗	✗	✓	✗
MeLoDy (Ours)	Text, Audio	257k hours of music	✓	✓	✓	✓
While LMs and DPMs both showed promising results, we believe the relevant question is not whether one should be preferred over another but whether we can leverage both approaches with respect to their individual advantages, e.g., [25]. After analyzing the success of MusicLM, we leverage the highest-level LM in MusicLM, termed as semantic LM, to model the semantic structure of music, determining the overall arrangement of melody, rhythm, dynamics, timbre, and tempo. Conditional on this semantic LM, we exploit the non-autoregressive nature of DPMs to model the acoustics efficiently and effectively with the help of a successful sampling acceleration technique [26]. All in all, in this paper, we introduce several novelties that constitute our main contributions:
1. We present MeLoDy (M for music; L for LM; D for diffusion), an LM-guided diffusion model that generates music of competitive quality while reducing  $95.7\%$  and  $99.6\%$  iterations of MusicLM to sample 10s and 30s music, being faster than real-time on a V100 GPU.
2. We propose the novel dual-path diffusion (DPD) models to efficiently model coarse and fine acoustic information simultaneously with a particular semantic conditioning strategy.
3. We design an effective sampling scheme for DPD, which improves the generation quality over the previous sampling method in [23] proposed for this class of LDMs.
4. We reveal a successful audio VAE-GAN that effectively learns continuous latent representations, and is capable of synthesizing audios of competitive quality together with DPD.
Audio Generation Apart from the generation models shown in Table 1, there are also music generation models [28, 29] that can generate high-quality music samples at high speed, yet they cannot accept free-form text conditions and can only generate single-genre music, e.g., techno music in [29]. There also are some successful music generators in the industry, e.g. Mubert [30] and Riffusion [31], yet, as analyzed in [5], they struggled to compete with MusicLM in handling free-form text prompts. In a more general scope of audio synthesis, some promising text-to-audio synthesizers [12, 21, 22] trained with AudioSet [32] also demonstrated the ability to generate music from free-form text, but the musicality of their samples is limited.
Acceleration of Autoregressive Models WaveNet [33] is a seminal work that demonstrates the capability of autoregressive (AR) models in generating high-fidelity audio. It comes with the drawback of extremely high computational cost in sampling. To improve its practical feasibility, Parallel WaveNet [34] and WaveRNN [35] were separately proposed to accelerate WaveNet. With a similar goal, our proposed MeLoDy can be viewed as an accelerated variant of MusicLM, where we replace the last two AR models with a dual-path diffusion model. Parallel to our work, SoundStorm [36] also exceedingly accelerates the AudioLM with a mask-based non-AR decoding scheme [37]. While it is applicable to MusicLM, the sound quality of this model is still limited by the bitrate of the neural codec. In comparison, the proposed diffusion model in MeLoDy operates with continuous-valued latent vectors, which by nature can be decoded into music audios of higher quality.
<sup>1</sup>We focus on non-vocal music data by using an audio classifier [27] to filter out in-house music data with vocals. Noticeably, generating vocals and instrumental music simultaneously in one model is defective even in the SOTA works [5, 6] because of the unnaturally sound vocals. While this work aims for generating production-level music, we improve the fidelity by reducing the tendency of generating vocals.
Network Architecture The architecture designed for our proposed DPD was inspired by the dual-path networks used in the context of audio separation, where Luo et al. [38] initiated the idea of segmentation-based dual-path processing, and triggered a number of follow-up works achieving the state-of-the-art results [39-43]. Noticing that the objective in diffusion models indeed can be viewed as a special case of source separation, this kind of dual-path architecture effectually provides us a basis for simultaneous coarse-and-fine acoustic modeling.
3 Background on Audio Language Modeling
This section provides the preliminaries that serve as the basis for our model. In particular, we briefly describe the audio language modeling framework and the tokenization methods used in MusicLM.
3.1 Audio Language Modeling with MusicLM
MusicLM [5] mainly follows the audio language modeling framework presented in AudioLM [3], where audio synthesis is viewed as a language modeling task over a hierarchy of coarse-to-fine audio tokens. In AudioLM, there are two kinds of tokenization for representing different scopes of audio:
- Semantic Tokenization: K-means over representations from SSL, e.g., w2v-BERT [44];
- Acoustic Tokenization: Neural audio codec, e.g., SoundStream [45].
To better handle the hierarchical structure of the acoustic tokens, AudioLM further separates the modeling of acoustic tokens into coarse and fine stages. In total, AudioLM defines three LM tasks: (1) semantic modeling, (2) coarse acoustic modeling, and (3) fine acoustic modeling.
We generally define the sequence of conditioning tokens as  $\mathbf{c}_{1:T_{\mathrm{cond}}} \coloneqq [\mathbf{c}_1, \dots, \mathbf{c}_{T_{\mathrm{cond}}}]$  and the sequence of target tokens as  $\mathbf{u}_{1:T_{\mathrm{tgt}}} \coloneqq [\mathbf{u}_1, \dots, \mathbf{u}_{T_{\mathrm{tgt}}}]$ . In each modeling task, a Transformer-decoder language model parameterized by  $\theta$  is tasked to solve the following autoregressive modeling problem:
$$
p _ {\theta} \left(\mathbf {u} _ {1: T _ {\mathrm {l g t}}} \mid \mathbf {c} _ {1: T _ {\mathrm {c n d}}}\right) = \prod_ {j = 1} ^ {T _ {\mathrm {l g t}}} p _ {\theta} \left(\mathbf {u} _ {j} \mid \left[ \mathbf {c} _ {1}, \dots , \mathbf {c} _ {T _ {\mathrm {c n d}}}, \mathbf {u} _ {1}, \dots , \mathbf {u} _ {j - 1} \right]\right), \tag {1}
$$
where the conditioning tokens are concatenated to the target tokens as prefixes. In AudioLM, semantic modeling takes no condition; coarse acoustic modeling takes the semantic tokens as conditions; fine acoustic modeling takes the coarse acoustic tokens as conditions. The three corresponding LMs can be trained in parallel with the ground-truth tokens, but need to be sampled sequentially for inference.
3.1.1 Joint Tokenization of Music and Text with MuLan and RVQ
To maintain the merit of audio-only training, MusicLM relies on joint audio-text embedding model, termed as MuLan [46], which can be individually pre-trained with large-scale music data and weakly-associated, free-form text annotations. This MuLan model is learned to project the music audio and its corresponding text description into the same embedding space such that the paired audio-text embeddings can be as close as possible. In MusicLM, the embeddings of music and text are tokenized using a separately learned residual vector quantization (RVQ) [45] module. Then, to generate music from a text prompt, MusicLM takes the MuLan tokens from the RVQ as the conditioning tokens in the semantic modeling stage and the coarse acoustic modeling stage, following Eq. (1). Given the prefixing MuLan tokens, the semantic tokens, coarse acoustic tokens, and fine acoustic tokens can be subsequently computed by LMs to generate music audio adhering to the text prompt.
4 Model Description
The overall training and sampling pipelines of MeLoDy are shown in Figure [1] where, we have three modules for representation learning: (1) MuLan, (2) Wav2Vec2-Conformer, and (3) audio VAE, and two generative models: a language model (LM) and a dual-path diffusion (DPD) model, respectively, for semantic modeling and acoustic modeling. In the same spirit as MusicLM, we leverage LM to model the semantic structure of music for its promising capability of modeling complex relationships across long-term contexts [9-11]. We also similarly pre-train a MuLan model
Figure 1: The training and sampling pipelines of MeLoDy
to obtain the conditioning tokens. For semantic tokenization, after empirically compared against w2v-BERT [44], we employ a Wav2Vec2-Conformer model, which follows the same architecture as Wav2Vec2 [47] but employs the Conformer blocks [48] in place of the Transformer blocks. The remainder of this section presents our newly proposed DPD model and the audio VAE-GAN used for DPD model, while other modules overlapped with MusicLM are described in Appendix B regarding their training and implementation details.
4.1 Audio VAE-GANs for Latent Representation Learning
To avoid learning arbitrarily high-variance latent representations, Rombach et al. [2] examined a KL-regularized image autoencoder for latent diffusion models (LDMs) and demonstrated extraordinary stability in generating high-quality image [49], igniting a series of follow-up works [50]. Such an autoencoder imposes a KL penalty on the encoder outputs in a way similar to VAEs [51, 52], but, different from the classical VAEs, it is adversarially trained as in generative adversarial networks (GANs) [53]. In this paper, this class of autoencoders is referred to as the VAE-GAN. Although VAE-GANs are promisingly applied in image generation, there are limited comparable attempts in audio generation. In this work, we propose to use a similarly trained VAE-GAN for raw audio.
Specifically, the audio VAE-GAN is trained to reconstruct  $24\mathrm{kHz}$  audio with a striding factor of 96, resulting in a  $250\mathrm{Hz}$  latent sequence. The architecture of the decoder is the same as that in HiFi-GAN [54]. For the encoder, we basically replace the up-sampling modules in the decoder with convolution-based down-sampling modules while other modules stay the same. For adversarial training, we use the multi-period discriminators in [54] and the multi-resolution spectrogram discriminators in [55]. The implementation and training details are further discussed in Appendix B.
4.2 Dual-Path Diffusion: Angle-Parameterized Continuous-Time Latent Diffusion Models
The proposed dual-path diffusion (DPD) model is a variant of diffusion probabilistic models (DPMs) [1, 15, 56] in continuous-time [16, 57-59]. Instead of directly operating on raw data space  $\mathbf{x} \sim p_{\mathrm{data}}(\mathbf{x})$ , with reference to LDMs [2], DPD operates on a low-dimensional latent space  $\mathbf{z}_0 = \mathcal{E}_{\phi}(\mathbf{x})$  such that the audio can be approximately reconstructed from the latent vectors:  $\mathbf{x} \approx \mathcal{D}_{\phi}(\mathbf{z}_0)$ , where  $\mathcal{E}_{\phi}$  and  $\mathcal{D}_{\phi}$  are the encoder and the decoder in VAE-GAN, respectively. Diffusing the latent space could significantly relieve the computational burden of DPMs [2]. Also, sharing a similar observation with [2], we find that audio VAE-GAN performed more stabler than other VQ-based autoencoders [45, 60] when working with the outputs from diffusion models.
Formally speaking, DPD is a Gaussian diffusion process  $\mathbf{z}_t$  that is fully specified by two strictly positive scalar-valued, continuously differentiable functions  $\alpha_t, \sigma_t \in [16]$ :  $q(\mathbf{z}_t | \mathbf{z}_0) = \mathcal{N}(\mathbf{z}_t; \alpha_t \mathbf{z}_0, \sigma_t^2 \mathbf{I})$  for any  $t \in [0, 1]$ . In the light of [58], we define  $\alpha_t := \cos(\pi t / 2)$  and  $\sigma_t := \sin(\pi t / 2)$  to benefit from some nice trigonometric properties, i.e.,  $\sigma_t = \sqrt{1 - \alpha_t^2}$  (a.k.a. variance-preserving [16]). With this definition, the forward diffusion process of  $\mathbf{z}_t$  can be re-parameterized in terms of angle  $\delta \in [0, \pi / 2]$ :
$$
\mathbf {z} _ {\delta} = \cos (\delta) \mathbf {z} _ {0} + \sin (\delta) \boldsymbol {\epsilon}, \quad \boldsymbol {\epsilon} \sim \mathcal {N} (\mathbf {0}, \mathbf {I}), \tag {2}
$$
which implies  $\mathbf{z}_{\delta}$  gets noisier as the angle  $\delta$  increases from 0 to  $\pi /2$
To create a generative process, a  $\theta$ -parameterized variational model  $p_{\theta}(\mathbf{z}_{\delta - \omega} | \mathbf{z}_{\delta})$  is trained to reverse the diffusion process by enabling taking any step  $\omega \in (0, \delta]$  backward in angle. By discretizing  $\pi / 2$
Figure 2: The proposed dual-path diffusion (DPD) model
into  $T$  finite segments, we can generate  $\mathbf{z}_0$  from  $\mathbf{z}_{\pi /2}\sim \mathcal{N}(\mathbf{0},\mathbf{I})$  in  $T$  sampling steps:
$$
p _ {\theta} \left(\mathbf {z} _ {0} \mid \mathbf {z} _ {\pi / 2}\right) = \int_ {\mathbf {z} _ {\delta_ {1: T - 1}}} \prod_ {t = 1} ^ {T} p _ {\theta} \left(\mathbf {z} _ {\delta_ {t} - \omega_ {t}} \mid \mathbf {z} _ {\delta_ {t}}\right) d \mathbf {z} _ {\delta_ {1: T - 1}}, \quad \delta_ {t} = \left\{ \begin{array}{l l} \frac {\pi}{2} - \sum_ {i = t + 1} ^ {T} \omega_ {i}, & 1 \leq t <   T; \\ \frac {\pi}{2}, & t = T, \end{array} \right. \tag {3}
$$
where  $\omega_{1},\ldots ,\omega_{T}$ , termed as the angle schedule, satisfy  $\sum_{t = 1}^{T}\omega_{t} = \pi /2$ . Regarding the choice of angle schedule, Schneider et al. [23] proposed a uniform one, i.e.,  $\omega_{t} = \frac{\pi}{2T}$  for all  $t$ . Yet, we observe that noise scheduling in the previous works [61, 62] tend to take larger steps at the beginning of the sampling followed smaller steps for refinement. With a similar perspective, we design another linear angle schedule, written as
$$
\omega_ {t} = \frac {\pi}{6 T} + \frac {2 \pi t}{3 T (T + 1)}, \tag {4}
$$
which empirically gives more stable and higher-quality results. Appendix D presents the comparison results of this linear angle schedule against the uniform schedule used in [23].
4.2.1 Diffusion Velocity Prediction
In DPD, we model the diffusion velocity at  $\delta$  [58], defined as  $\mathbf{v}_{\delta} \coloneqq \frac{d\mathbf{z}_{\delta}}{d\delta}$ . It can be simplified as:
$$
\mathbf {v} _ {\delta} = \frac {d \cos (\delta)}{d \delta} \mathbf {z} _ {0} + \frac {d \sin (\delta)}{d \delta} \boldsymbol {\epsilon} = \cos (\delta) \boldsymbol {\epsilon} - \sin (\delta) \mathbf {z} _ {0}. \tag {5}
$$
When  $\mathbf{v}_{\delta}$  is given, we can easily remedy the original sample  $\mathbf{z}_0$  from a noisy latent  $\mathbf{z}_{\delta}$  at any  $\delta$ , since  $\mathbf{z}_0 = \cos (\delta)\mathbf{z}_{\delta} - \sin (\delta)\mathbf{v}_{\delta}$ . This suggests  $\mathbf{v}_{\delta}$  a feasible target for neural network prediction  $\hat{\mathbf{v}}_{\theta}(\mathbf{z}_{\delta};\mathbf{c})$ , where  $\mathbf{c}$  generally denotes the set of conditions controlling the music generation. In MeLoDy, as illustrated in Figure 1, the semantic tokens  $\mathbf{u}_1,\dots ,\mathbf{u}_{T_{\mathrm{ST}}}$ , which are obtained from the SSL model during training and generated by the LM at inference time, are used to condition the DPD model. In our experiments, we find that the stability of generation can be significantly improved if we use token-based discrete conditions to control the semantics of the music and let the diffusion model learn the embedding vector for each token itself. As in [23, 58], this velocity prediction network  $\theta$  can be effectively trained with a mean squared error (MSE) loss:
$$
\mathcal {L} := \mathbb {E} _ {\mathbf {z} _ {0} \sim p _ {\text {d a t a}} (\mathbf {z} _ {0}), \boldsymbol {\epsilon} \sim \mathcal {N} (\mathbf {0}, \mathbf {I}), \delta \sim \operatorname {U n i f o r m} [ 0, 1 ]} \left[ \| \cos (\delta) \boldsymbol {\epsilon} - \sin (\delta) \mathbf {z} _ {0} - \hat {\mathbf {v}} _ {\theta} (\cos (\delta) \mathbf {z} _ {0} + \sin (\delta) \boldsymbol {\epsilon}; \mathbf {c}) \| _ {2} ^ {2} \right], \tag {6}
$$
which forms the basis of DPD's training loss.
4.2.2 Multi-Chunk Velocity Prediction
With reference to [23], for long-context generation, we can incrementally appending new chunks of random noise to infinitely continue audio generation. To achieve this, the velocity prediction
network needs to be trained to handle the chunked input, where each chunk exhibits a different scale of noisiness. In particular, we define the multi-chunk velocity target  $\mathbf{v}_{\mathrm{tgt}}$  that comprises  $M$  chunks of velocities. Given  $\mathbf{z}_0, \mathbf{z}_{\delta}, \epsilon \in \mathbb{R}^{L \times D}$  with  $L$  representing the length of latents and  $D$  representing the latent dimensions, we have  $\mathbf{v}_{\mathrm{tgt}} := \mathbf{v}_1 \oplus \dots \oplus \mathbf{v}_M$ , where  $\oplus$  is the concatenation operation and
$$
\mathbf {v} _ {m} := \cos (\delta_ {m}) \boldsymbol {\epsilon} [ L _ {m - 1}: L _ {m},: ] - \sin (\delta_ {m}) \mathbf {z} _ {0} [ L _ {m - 1}: L _ {m},: ], \quad L _ {m} := \left\lfloor \frac {m L}{M} \right\rfloor . \tag {7}
$$
Here, we use the NumPy slicing syntax (0 as the first index) to locate the  $m$ -th chunk, and we draw  $\delta_{m} \sim \mathrm{Uniform}[0, \pi/2]$  for each chunk at each training step to determine the noise scale. The MSE loss in Eq. (6) is then extended to
$$
\mathcal {L} _ {\text {m u l t i}} := \mathbb {E} _ {\mathbf {z} _ {0}, \boldsymbol {\epsilon}, \delta_ {1}, \dots , \delta_ {M}} \left[ \| \mathbf {v} _ {\mathrm {t g t}} - \hat {\mathbf {v}} _ {\theta} \left(\bar {\mathbf {z}} _ {\delta_ {1}} \oplus \dots \oplus \bar {\mathbf {z}} _ {\delta_ {M}}; \mathbf {c}\right) \| _ {2} ^ {2} \right], \tag {8}
$$
$$
\bar {\mathbf {z}} _ {\delta_ {m}} := \cos \left(\delta_ {m}\right) \mathbf {z} _ {0} \left[ L _ {m - 1}: L _ {m}, : \right] + \sin \left(\delta_ {m}\right) \boldsymbol {\epsilon} \left[ L _ {m - 1}: L _ {m}, : \right]. \tag {9}
$$
Different from the original setting where we use a global noise scale for the network input [1, 61, 63], in the case of multi-chunk prediction, we need to specifically inform the network what the noise scales are for all  $M$  chunks. Therefore, we append an angle vector  $\delta$  to the set of conditions  $\mathbf{c} := \{\mathbf{u}_1, \dots, \mathbf{u}_{T_{\mathrm{ST}}}, \delta\}$  to record the angles drawn in all  $M$  chunks aligned with the  $L$ -length input:
$$
\boldsymbol {\delta} := \left[ \delta_ {1} \right] _ {r = 1} ^ {L _ {1}} \oplus \dots \oplus \left[ \delta_ {M} \right] _ {r = 1} ^ {L _ {M}} \in \mathbb {R} ^ {L}, \tag {10}
$$
where  $[a]_{r=1}^{B}$  denotes the operation of repeating a scalar  $a$  for  $B$  times to make a  $B$ -length vector.
4.2.3 Dual-Path Modeling for Efficient and Effective Velocity Prediction
To predict the multi-chunk velocity with  $\hat{\mathbf{v}}_{\theta}$ , we propose a dual-path modeling mechanism, which plays a prime role in DPD for efficient parallel processing along coarse and fine paths and effective semantic conditioning. Figure 2 presents the computation procedures of  $\hat{\mathbf{v}}_{\theta}$ , which comprises several critical modules that we present one by one below.
To begin with, we describe how the conditions  $\{\mathbf{u}_1,\dots ,\mathbf{u}_{T_{\mathrm{ST}}},\pmb {\delta}\}$  are processed in DPD:
Encoding Angle Vector First, we encode  $\delta \in \mathbb{R}^L$ , which records the frame-level noise scales of latents. Instead of using the classical positional encoding [1], we use a spherical interpolation [64] between two learnable vectors  $\mathbf{e}_{\mathrm{start}}$ ,  $\mathbf{e}_{\mathrm{end}} \in \mathbb{R}^{256}$  using broadcast multiplications, denoted by  $\otimes$ :
$$
\mathbf {E} _ {\delta} := \mathbf {M L P} ^ {(1)} (\sin (\delta) \otimes \mathbf {e} _ {\text {s t a r t}} + \sin (\delta) \otimes \mathbf {e} _ {\text {e n d}}) \in \mathbb {R} ^ {L \times D _ {\text {h i d}}}, \tag {11}
$$