Efficient Neural Music Generation
Authors: Max W. Y. Lam
Qiao Tian
Tang Li
Zongyu Yin
Siyuan Feng
Ming Tu
Yuliang Ji
Rui Xia
Mingbo Ma
Xuchen Song
Jitong Chen
Yuping Wang
Yuxuan Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Music Generation
Language Model
Diffusion Model
MusicLM
TL;DR: This paper presents MeLoDy (M for music; L for LM; D for diffusion), an LM-guided diffusion model that efficiently generates music audios of state-of-the-art quality from text faster than real-time on a V100 GPU.

Abstract: Recent progress in music generation has been remarkably advanced by the state-of-the-art MusicLM, which comprises a hierarchy of three LMs, respectively, for semantic, coarse acoustic, and fine acoustic modelings. Yet, sampling with the MusicLM requires processing through these LMs one by one to obtain the fine-grained acoustic tokens, making it computationally expensive and prohibitive for a real-time generation. Efficient music generation with a quality on par with MusicLM remains a significant challenge.
In this paper, we present **M**e**L**o**D**y (**M** for music; **L** for LM; **D** for diffusion), an LM-guided diffusion model that generates music audios of state-of-the-art quality meanwhile reducing 95.7\% to 99.6\% forward passes in MusicLM, respectively, for sampling 10s to 30s music. MeLoDy inherits the highest-level LM from MusicLM for semantic modeling, and applies a novel dual-path diffusion (DPD) model and an audio VAE-GAN to efficiently decode the conditioning semantic tokens into waveform. DPD is proposed to simultaneously model the coarse and fine acoustics by incorporating the semantic information into segments of latents effectively via cross-attention at each denoising step. Our experimental results suggest the superiority of MeLoDy, not only in its practical advantages on sampling speed and infinitely continuable generation, but also in its state-of-the-art musicality, audio quality, and text correlation.

Our samples are available at https://Efficient-MeLoDy.github.io/.

Efficient Neural Music Generation
Max W. Y. Lam, Qiao Tian, Tang Li, Zongyu Yin, Siyuan Feng, Ming Tu, Yuliang Ji, Rui Xia, Mingho Ma, Xuchen Song, Jitong Chen, Yuping Wang, Yuxuan Wang Speech, Audio & Music Intelligence (SAMI), ByteDance
Abstract
Recent progress in music generation has been remarkably advanced by the state- of-the-art MusicLM, which comprises a hierarchy of three LMs, respectively, for semantic, coarse acoustic, and fine acoustic modelings. Yet, sampling with the Mu- sicLM requires processing through these [.Ms one by one to obtain the fine-grained acoustic tokens, making it computationally expensive and prohibitive for real- time generation. Efficient music generation with a quality on par with MusicLM remains significant challenge. In this paper, we present MeLoDy (M for music; L for LM; D for diffusion). an LM-guided diffusion model that generates music audios of state-of-the-ae quality meanwhile reducing 95.7% 99. 99.6% forward passes in MusicLM, respectively, for sampling 10s to 30s music. McLoDy inherits the highest-level LM from MusicLM for semantic modeling. and applies a novel dual-path diffusion (DPD) model and an audio VAE-GAN to efficiently decode the conditioning semantic tokens into waveform. DPD proposed to simultaneously model the coarse and fine acoustics by incorporating the semantic information into segments of latents effectively via cross-attention at each denoising step. Our experimental results suggest the superiority of MeLoDy, not only in its practical advantages on sampling speed and infinitely continuable generation, but also its ofte-offtheeart musicality, audio quality, and text correlation.
Our samples are available at
MelloDy.githin ioo
1
Introduction
Music is an art composed of harmony, melody, and rhythm that permeates every aspect of human life. With the blossoming of deep generative models [H3] music generation has drawn much attention in recent years 4+3 As a prominent class of generative models, language models (LMs) 7 g showed extraordinary modeling capability in modeling complex relationships across long-term contexts
1-1 In light of this, AudioLM 3 and many follow-up works [5] 12 14
successfully applied
LMs to audio synthesis. Concurrent the LM-based approaches. diffusion probabilistic models (DPMs) [ [5 [6]. as another competitive class of generative models 2 have also demonstrated exceptional abilities in synthesizing speech
sounds 21 22 and music
However, generating music from free-form text remains challenging as the permissible music descrip- tions can be very diverse and relate to any of the genres, instruments, tempo, scenarios. or even some subjective feelings. Conventional text-to-music generation models are listed in Table D where both MusieLM [5] and Musse2Music
( were trained on large-scale music datasets and demonstrated the state-of-the -rt (SOTA) generative performances with high fidelity and adherence to various aspects of text prompts. Yet, the success of these two methods comes with large computational costs, which would be serious impediment to their practicalities. In comparison, Moñsai [23] building upon DPMs made efficient samplings of high-quality music possible. Nevertheless, the number of their demonstrated cases was comparatively small and showed limited in-sample dynamics. Aiming for feasible music creation tool, high efficiency of the generative model is essential since it facilitates interactive creation with human feedback being taken into account as in 24
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table I: A comparison of MeLoDy with conventional text-to-music generation models in the literature. We use AC to denote whether audio continuation is supported, FR to denote whether the sampling faster than real-time on a V100 GPU, VT to denote whether the model has been tested and demonstrated using various types of text prompts including instruments, genres, and long-form rich
descriptions, and MP to denote whether the evaluation was done by music producers.
Model
Prompts
Training Data
AC
FR
VT
MP
Moûsai
2
Text
2.5k hours of music
/
X
MusicLM
5
Text, Melody
280k hours of music
/
X
V
X
Moise2Music
Text
340k hours of music
/
MeLoDy (Ours)
Text, Audio
257k hours of musi
/
/
V
While LMs and DPMs both showed promising results, we believe the relevant question is not whether one should be preferred over another but whether can leverage both approaches with respect to their individual advantages, c.g., 125. After analyzing the success of MusicLM, we leverage the highest-level LM in MusicI.,, termed semantic I.M, to model the semantic structure of music, determining the overall arrangement of melody, rhythm, dynamics, timbre, and tempo. Conditional on this semantic LM, we exploit the non-autoregressive nature of DPMs to model the acoustics efficiently and effectively with the help of a successful sampling acceleration technique [26]. All in all, in this paper, we introduce several novelties that constitute our main contributions:
1. We present McLoDy (M for music; L for LM; D for diffusion), an LM-guided diffusion model that generates music of competitive quality while reducing 95.7% 5.7% and 99.6% iterations of MusicLM to sample 10s and 30s music, being faster than real-time on a V100 GPU.
2. We propose the novel dual-path diffusion (DPD) models to efficiently model coarse and fine acoustic information simultaneously with a particular semantic conditioning strategy.
3. We design an effective sampling scheme for DPD, which improves the generation quality over the previous sampling method in ( proposed for this class of LDMs.
4. We reveal a successful audio VAE-GAN that effectively Iearns continuous latent representa- tions, and is capable of synthesizing audios of competitive quality together with DPD.
2 Related Work
Audio Generation Apart from the generation models shown in Table D there are also music
generation models 128 29 that can generate high-quality music samples high speed, yet they cannot accept free-form text conditions and can only generate single-genre music, e.g., techno music 129] There also are some successful music generators in the industry, e.g. Mubert (0 and Riffusion 31. yet, as analyzed in [.] they struggled to compete with MusicL. in handling free-form text prompts.
more general scope of audio synthesis, some promising text-to-audio 21 22] trained with AudioSet [32) also demonstrated the ability to generate music from free-form text, but the musicality of their samples is limited.
synthesizers 212
+3 is seminal work that demonstrates the Acceleration of Autoregressive Models WaveNet
capability of autoregressive (AR) models in generating high-fidelity audio. It comes with the drawback of extremely high computational cost in sampling. To improve its practical feasibility, Parallel WaveNet (34] and WaveRNN (35 were separately proposed to accelerate WaveNet. With a similar goal, our proposed MeLoDy can be viewed as an accelerated variant of MusicLM, where we replace the last two AR models with a dual-path diffusion model. Parallel to our work, SoundStorm 361 also exceedingly accelerates the AudioLM with mask-based non-AR decoding scheme [37] While it applicable to MusicLM, the sound quality of this model still limited by the bitrate the neural codec. In comparison, the proposed diffusion model in MeLoDy operates with continuous- valued latent vectors, which by nature can be decoded into music audios of higher quality.
We focus on non-vocal music data by using an audio classifier 24 to filter out in-house music data with vocals. Noticeably, generating vocals and instrumental music simultaneously in one model is defective even in the SOTA works [ 6 because of the unnaturally sound vocals. While this work aims for generating production-level music, improve the fidelity by reducing the tendency of generating vocals.

---

Network Architecture The architecture designed for our proposed DPD was inspired by the dual-path networks used the context of audio separation, where Luo et al. G8 initiated the idea of segmentation-baset dual-path processing, and triggered number of follow-up works achieving the state-of-the-arr results 39 45 Noticing that the objective in diffusion models indeed can be viewed as special case of source separation, this kind of dual-path architecture effectually provides us basis for simultaneous coarse-and- fine acoustic modeling.
Background on Audio Language Modeling
This section provides the preliminaries that serve as the basis for our model. In particular, we briefly describe the audio language modeling framework and the tokenization methods used in MusicLM.
3.1 Audio Language Modeling with MusicLM
MusicI. S mainly follows the audio language modeling framework presented in AudioLM K where audio synthesis is viewed as a language modeling task over hierarchy of coarse-to-fine audio tokens. In AudioLM, there are two kinds of tokenization for representing different scopes of audio:
Semantic Tokenization: K-means over representations from SSL, e.g., w2v-BERT
Acoustic Tokenization: Neural audio codec, c.g., SoundStream
To better handle the hierarchical structure of the acoustic tokens, AudioL. further separates the modeling of acoustic tokens into coarse and fine stages. In total, AudioLM defines three LM tasks: (1) semantic modeling, (2) coarse acoustic modeling, and (3) fine acoustic modeling.
We generally define the sequence of conditioning tokens as CI: := [cI...., Craal and the sequence of target tokens u [u....; UT In each modeling task, Transformer-decoke language model parameterized by 0 is tasked to solve the following autoregressive modeling problem:
(ui:T C1:T..)
II po (uyl[cc........ [cl..... Craa: U1,.... U uy-1)), j=1
(1)
where the conditioning tokens are concatenated to the target tokens as prefixes. In Audiol semantic modeling takes no condition; coarse acoustic modeling takes the semantic tokens as conditions; fine acoustic modeling takes the coarse acoustic tokens as conditions. The three corresponding LMs can be trained in parallel with the ground-truth tokens, but need be sampled sequentially for inference.
3.1.1 Joint Tokenization of Music and Text with MuLan and RVQ
To maintain the merit of audio-only training, MusicLM relies on joint audio-text embedding model, termed as MuLan 46. which can individually pre-trained with large-scale music data and weakly- associated, free-form text annotations. This MuLan model is learned to project the music audio and its corresponding text description into the same embedding space such that the paired audio-text embeddings can be as close as possible. In MusicI.M, the embeddings of music and text are tokenized using a separately learned residual vector quantization (RVQ) 45 module. Then, to generate music from text prompt, MusicLM takes the MuLan tokens from the RVQ the conditioning tokens in the semantic modeling stage and the coarse acoustic modeling stage, following Eq. ① Given the prefixing MuLan tokens, the semantic tokens, coarse acoustic tokens, and fine acoustic tokens can be subsequently computed by LMs to generate music audio adhering to the text prompt.
Model Description
The overall training and sampling pipelines of MeLoDy are shown in Figure D where, we have three modules for representation learning: (1) MuLan, (2) Nav2Vec2-Confomman
and (3) audio VAE, and two generative models: a language model (LM) and a dual-path diffusion (DPD) model, respectively, for semantic modeling and acoustic modeling. In the same spirit as MusicI.,, we leverage LM to model the semantic structure of music for its promising capability of modeling complex relationships across long-term contexts M.
We also similarly pre-train MuLan model
3

---

stage
Sampling stage
Commentted audio
To get audior
Pucio prompt
Text promptt iucignienn
VAE-GAN
musii mach natutter Aie digte viless"
SIE SAN
Milan luucl (owsr))
Wax2VecZ-Co Tormar
Enooder
Decodar
I m-maree
Samsll
Mbp
Marian (audarfus towe))
Concinuation
Semantic Modeling
Bug
Chune-wise rosurigl
tollens
DDD
features
Mulan tokers
Semantic tokens
Latent Instures
Acoustic Modeling
Semantic Modeling
Modeling
Nose
Noke
Figure 1: The training and sampling pipelines of MeLoDy
to obtain the conditioning tokens. For semantic tokenization, after empirically compared against w2v-BERT 44 we employ a Vec2-CCofformer model, which follows the same architecture as Wav2Vec2 A but employs the Conformer blocks [48] in place of the Transformer blocks. The remainder this section presents our newly proposed DPD model and the audio VAE-GAN used for DPD model, while other modules overlapped with MusicLM are described in Appendix B regarding their training and implementation details.
4.1 Audio VAE-GANs for Latent Representation Learning
To avoid learning arbitrarily high-variance latent representations, Rombach et al. 2 examined a KL- regularized image autoencoder for latent diffusion models (LDMs) and demonstrated extraordinary stability in generating high-quality image 49), igniting a series of follow-up works [500] Such an autoencoder imposes KL penalty the encoder outputs in a way similar to VAEs [51 52 but, different from the classical VAES, it is adversarially trained as generative adversarial networks (GANs) (53 In this paper, this class of autoencoders is referred as the VAE-GAN. Although VAE-GANs are promisingly applied in image generation, there are limited comparable attempts in audio generation. In this work, we propose use similarly trained VAE-GAN for raw audio.
Specifically, the audio VAE-GAN is trained to reconstruct 24kHz audio with a striding factor of 96, resulting in a 250Hz latent sequence. The architecture of the decoder is the same as that in HiFi-GAN 5... For the encoder, we basically replace the up- samplingi modules in the decoder with convolution- based down-sampp sampling modules while other modules stay the same. For adversarial training, the multi perlid discriminators in [54] and the multi-resolution spectrogram discriminators in (55] The implementation and training details are further discussed in Appendix B.
4.2 Dual-Path Diffusion: Angle-Parameetrizzed Continuous- Tamiinuous-Time Latent Diffusson Models
The proposed dual-path diffusion (DPD) model a variant of diffusion probabilistic models (DPMs)
[56] in continuous-time 16 57459. Instead of directly operating on raw data space x ~
Pdata (x). with reference LDMs 2 DPD operates on a low-dimensional latent space Zu = (x).. such that the audio can be approximately reconstructed from the latent vectors: x = D& (zo), where Eg and D3 are the encoder and the decoder in VAE-GAN, respectively. Diffusing the latent space could significantly relieve the computational burden of DPMs 12). Also, sharing a similar observation we find that audio VAE-GAN performed more stabler than other VQ-based autoencoders when working with the outputs from diffusion models.
with
45
Formally speaking, DPD Gaussian diffusion process z, that is fully specified by two strictly positive scalar-valued, continuously differentiable functions oe,, a. [6]] g(zz (20) = =N(z;; az If for any E [0, 1]. In the light of (58). we define a, :- cos(ml/2) and o; sin() to benefit from some nice trigonometric properties, i.c., a, = /I - a (a.k.a. seriance-preservinn (16.. With this definition, the forward diffusion process of z; can be re-parameterized in terms of angle 8 E m/2]:
Zお = cos z) + sin( E E~~ (O,,I.
(2)
which implies 2, gets noisier as the angle increases from 0 7/2.
To create generative process, e--parameterized variational model po (as- (2s) trained to reverse the diffusion process by enabling taking any step w e (0, 6] backward in angle. By discretizing n/2

---

Anchtecuree
Excoding Conditoos
Dusl- -2th- Blocks
Angle vacor
Predicted Veloo ty
Learnabic Siorp
Add RMSSNomm
Lineer
RVSMNam
Stacked SRUS
Res rerige to Fine Pati
N
Add RMSNomm
Segmentation
Repear Segments
Aver age
RMSMMorm
Roformer
Linear
Learnable Embedding
Merge Segments
Nakr Latent Rs;
Mp. Magg
Figure 2: The proposed dual-path diffusion (DPD) model
into T finite segments, we can generate Zo from 2x/2 ~ (N,,,)) sampling steps:
T
Σ Whi 15t< (3)
II
po(zs, -w (26.) d28
8, =
1=
2
where w Wy, termed as the angle schedule, satisfy Σ -= wi = x/2. Regarding the choice of
angle schedule, Schneider al. (23] proposed uniform one, i.e., Wi = 2T for all Yet, we observe that noise scheduling in the previous works [6] 62 tend to take larger steps at the beginning of the
sampling followed smaller steps for refinement. With a similar perspective, we design another linear angle schedule, written
π
2t
We
6T
3T +1)
which empirically gives more stable and higher-quality results. Appendix presents the comparison results of this linear angle schedule against the uniform schedule used in 23
4.2.1 Diffusion Velocity Prediction
In DPD, we model the diffusion velocity at 58 defined as Va = dus d5
It can be simplified as:
d coos(ó)
6)
d e = cos( -
V5 dõ
Zo +
(5)
d6
When is given, easily remedy the original sample zo from noisy latent z5 any d, since Zn = cos (6)23 sin This suggests Va a feasible target for neural network prediction v (zzi c). where generally denotes the set of conditions controlling the music generation. In MeLoDy, as illustrated in Figure [ the semantic tokens uI...., U2TT which are obtained from the SSL model during training and generated by the LM inference time, used to condition the DPD model. In our experiments, we find that the stability of generation can be significantly improved if we use noken-based discrete conditions control the semantics of the music and let the diffusion model learn the embedding vector for each token itself. As in (23 58), this velocity prediction network can be effectively trained with a mean squared error (MSE) loss:
= Exg P-ra (P)) (1. (--iitmmm [[ccos(0e sin(6) ve(cos(6)z)) + (in(8)); ();;))
which forms the basis of DPD's training loss.
4.2.2 Multi-Chunk Velocity Prediction
With reference to 23], for long-context generation, we can incrementally appending new chunks of random noise infinitely continue audio generation. To achieve this, the velocity prediction
5

---

network needs to be trained to handle the chunked input, where each chunk exhibits a different scale of noisiness. particular, we define the multi-chunk velocity target Vigi that comprises M chunks of velocities. Given Zo; 25; C RLLxD with L representing the length of latents and D representing the latent dimensions, we have Vig :- V1 @ ... @ MM, where @ is the concatenation operation and
mL M
Vm == cos(6m (mmm)))mmmm)) Imms] sin(ōm) (oooTT [Tmm -1 Tmm::. :)
L
(7)
Here, we use the NumPy slicing syntax (0 as the first index) to locate the m-th chunk, and we draw 8m Uniform[0, r/2] for each chunk at each training step to determine the noise scale. The MSE
loss in Eq. 6 is then extended to
Lmulti Ezo,e, [Ivirr vo(Zs, + Zön c)]
(8)
p cos( ) [m L :1 + sin ) c[ -
Different from the original setting where we use global noise scale for the network input M| 661 53 in the case of multi-chunk prediction, we need specifically inform the network what the noise scales are for all M chunks. Therefore, we append an angle vector ò to the set of conditions , 63 to record the angles drawn in all M chunks aligned with the L-length input:
= {uI.... UTa
:= [6 | @ ....... @ [ E ERL,
(10)
where (a)/=] denotes the operation of repeating scalar a for B times make B-length vector.
4.2.3 Dual-Path Modeling for Efficient and Effective Velocity Prediction
To predict the multi-chunk velocity with Vg. we propose a dual-path modeling mechanism, which plays a prime role in DPD for efficient parallel processing along coarse and fine paths and effective semantic conditioning. Figure !
presents the computation procedures of Vo. which comprises several critical modules that we present one by one below.
To begin with, we describe how the conditions {u),.... UTST 8} are processed in DPD:
Encoding Angle Vector First, we encode ô E R which records the frame-level noise scales of latents. Instead of using the classical positional encoding MT we use spherical interpolation [64] between learnable vectors esturt: eend 256 using broadcast multiplications, denoted by @:
E. = MI.((1) (sin()) X estart + sin (8) @ Gend) E L.L.
(11)
where, for all i, MLP(0 RMSNorm W (1) GELU(x bb() by ((0) projects arbitrary
input x E DDD to RR Dad using RMSNorm 655 and GELU activation [66] with learnable w E
(6) D Duee ((( IR Dau (( b E Rai and Daia is hidden dimension.
The remaining conditions are the discrete tokens representing semantic information u1,..., UT Following the typical approach for embedding natural languages [8) directly use a lookup table of vectors to map any token u, E {l...., VsT } into real-valued vector E(u)) E R Das, where VsT denotes the vocabulary size of the semantic tokens, i.c., the number of clusters in k-means for Wav2Vec2-Confore
Encoding Semantic Tokens
By stacking the vectors along the time axis and applying another MLP block, we obtain EsT == MLP(2)
R Tex Das,
Conditional on the computed embeddings E, and EsT. next show how the network input, i.c. zs. for the case of having same noise scale 6, for all chunks and 2,, @ ..... + 28 for the case having different noise scales, is processed in DPD for velocity prediction. For the simplicity of notation, use 2s, to denote the network input here and below. 2.5 is first linearly transformed and added up with the angle embedding of the same shape: H RMSNorm (z,, Win Eş). where Wi. R D Dra is learnable. Then, a crucial segmentation operation is applied for fual-path modeling.
Segmentation As illustrated in Figure 3) the segmentation module divides 2-D input into S half-overlapping segments each of length K. represented by a 3-D tensor H == [0, H,,..., Hs, E Sx Dns,, where H. := H 1)K
1)K K, and zero-padded such that we have
2

---

Finc path
"SOS sordic
Stack chunks
Cousc aith
Figure 3: Diagram for visually understanding the segmentation operation for dual-path modeling
S s= + + 1. By choosing a segment size K R VI, the costs of sequence processing become sub-linear (O(VL)) as opposed to (O(L). This greatly reduces the difficulty of learning very long sequence and permits MeLoDy to use 250Hz latent sequences was used. In comparison, MusicLM [5] was built upon 50Hz codec.
latents for better audio quality. In this work,
Dual-Path Blocks After the segmentation, we obtain 3-D tensor input. shown in Figure 2 the tensor is subsequently passed to N dual-path blocks, where each block contains two processing stages corresponding to coarse-path (i.c., inter-segment) and fine-path (i.c., intra-segment) processing, respectively. Similar to the observations in 40 we find it superior to use attention-based network for coarse-pall path processing and to use a bi-directional RNN for fine-path processing. The goal of fine acoustic modeling is to better reconstruct the fine details from the roughly determined audio structure B At a finer scope, only the nearby elements matter and contain the most information needed for refinement, Specifically, we employ the Roformer network [77] for coarse-path processing, where we use a self-attention layer followed by cross-attention layer be conditional on EST with rotary positional embeddings. On the other hand, we use a stack of 2-layer simple recurrent units (SRUs) 68 for fine-path processing. A feature-wise linear modulation (FiLM) 699 layer is applied to the output of SRUS to assist the denoising with the angle embedding Es and the pooled EsT. The details of inner mechanism in each dual-path block is presented in Appendix B.
supported by the modeling perspectives in neural vocoding [33] 351
4.2.4 Music Generation and Continuation
Suppose we have a well-trained multi-chunk velocity model Vs. we begin with a L-length latent generation, where L is the length of latents we used in training. According Appendix A, the DDIM sampling algorithm 26 can be re- ee-ormulated by applying the trigonometric identities:
28, -5; cos(w))
sin(w)))o(zs,;
(12)
which, by running from t = T to t = 1 using the defined We in Eq. 44. we can generate sample of zo of length L. To continue generation, we append new chunk composed of random noises to the generated Zo and drop the first chunk in Zo- Recall that the inputs to va are the M concatenated noisy latents of different noise scales. The continuation of generation is feasible since the conditions (i.e., the semantic tokens and the angle vector) defined in DPD have an autoregressive nature at inference time. On one hand, the semantic tokens are generated by the semantic LM in an autoregressive manner, therefore chunk. On the other hand, since the multi-chunk model Vo trained to tackle chunks of different noise scales with respect to the angle vector, we can simply ignore the generated audio (on the first M - 1 chunks) by zeroing the respective values and setting ones for the newly appended chunk,
can continue the generation of semantic tokens for the new
[L/M], Then, the newly appended noise chunk can be transformed to meaningful music audio after [T/M] step of DDIM sampling. For more details of generation, we present the corresponding algorithms in Appendix C. Besides music continuation, based on MuLan, My also supports music prompts to generate music of a similar style, as shown in Figure Examples of music continuation, and music prompting shown our demo page.
i.e, 6 &nw : [0] -0
L/M L-M
+ [8,] -==
7

---

Table The speed and the quality our proposed MeLoDy on a CPU (Intel Xeon Platinum 8260 CPU @ 2.40GHz) or a GPU (NVIDIA Tesla V100) using different numbers of sampling steps.
Steps (T)
Speed on CPU (t) Speed on GPU (t)
FAD (
MCC (f)
(MusicCaps)
0.43
5
1472Hz (0.06x)
181.1kHz (7.5x)
7.23
0,49
893Hz (0.04x)
104.8kHz (4.4x)
5.93
0.52
20
498Hz (0.02x)
56.9kHz (2.4x)
5.41
0.53
Experiments
5.1 Experimental Setup
Data Preparation As shown in Table [ McLoDy was trained on 257k hours of music data (6.4M
24kHz audios), which were filtered with [27] to focus non-vocal music. Additionally, inspired by the text augmentation in 6 we enriched the tag-based texts to generate music captions by asking ChatGPT 770. This music description pool is used for the training of our 195.3M MuLan, where we randomly paired each audio with either the generated caption or its respective tags. In this way, we robustly improve the model's capability of handling free-form text.
Semantic LM For semantic modeling, we trained 429.5M ..M ILLMM with layers, 8 heads, and 2048 hidden dimensions, which has a comparable number of parameters to that of the MusicIM [5]. For conditioning. we set up the MuLan RVQ using 12 1024-sized codebooks, resulting in 12 prefixing tokens. The training targets were 10s semantic tokens, which are obtained from discretizing the 25Hz embeddings from a 199.5M Wav2Vec2-Confome with 1024-center k-means.
Dual-Path Diffusion For the DPD model, set the hidden dimension to Dmaa 768, and block number to N = 8, resulting in 296.6M parameters. For the input chunking strategy, we divide the 10s training inputs in fixed length of L 2500 into MM parts. For segmentation, segment size of = 64 (i.e., each segment 256ms long), leading to S = 80 segments. addition, applied the classifier-free guidance (CFG) 72 samples and conditions. During training, the cross-attention to semantic tokens is randomly replaced by self-attention with probability of 0.1. For sampling, the unconditional prediction Vuncond and the conditional prediction Voand are linearly combined: (Vooodd + (1 - p) Vuncond with scale of p 2.5.
used
DPD to improve the correspondence between
Audio VAE-GAN For audio VAE-GAN, used hop size of 96, resulting in 250Hz latent
sequences for encoding 24kHz music audio. The latent dimension D 16, thus we have a total compression rate of 6x. The hidden channels used in the encoder were 256, whereas that used in the decoder were 768. The audio VAE-GAN in total contains 100. parameters.
5.2 Performance Analysis
Objective Metrics We use the VGGish-based [iX]] Frechet audio distance (FAD) 74 between the generated audios and the reference audios from MusicCaps (5] as a rough measure of generation fidelity To measure text correlation, use the MuLan cycle consistency (MCC) (5). which calculates the cosine similarity between text and audio embeddings using a pre-trained MuLan
Inference Speed We first evaluate the sampling efficiency of our proposed MeLoDy. As DPD permits using different numbers of sampling steps depending on our needs, we report its generation speed in Table Surprisingly, MeDoDy steadily achieved higher MCC score than that of the reference set, even taking only 5 sampling steps. This means that (i) the MuLan model determined that our generated samples were more correlated MusicCaps captions than reference audios, and (ii) the proposed DPD is capable of consistently completing the MuLan cycle at significantly lower costs than the nested LMs in [5].
? Note that McLoDy was mainly trained with non-vocal music data, its sample distribution could not fit the reference one as well as in [5] K) since about 76% audios in MusicCaps contain either vocals or speech. SSnne MuLan model trained with different dataset, MCC results cannot be compared to 5
8

---

Table The comparison of MeLoDy with the SOTA text-to- text-to-music generation models. NFE the
number of function evaluations [58] for generating Teseoond audio
Audioo[Msiccaliiii Musicality, Quality, and Text Corr. are the winning proportions terms of musicality, quality, and text correlation, respectively.
Musicality (t)
Quality (t)
Text Corr. (1)
Model
NFE (L)
MLM
N2M
MLM
N2M
MLM
N2M
Music M 5
(25 + 200 + 400)7
0.541
0.465
0.548
Moise2Music
6
1000 + 800 + 800
0.555
0.436
0.572
MeL.oDy (20 steps)
25T + 20
0.459
0.445
0.535
0.564
0.452
0.428
Comparisons with SOTA models We evaluate the performance of MeLoDy by comparing it to MusicL. [5] and Noise2Music [6. which both were trained large-scale music datasets and
demonstrated SOTA results for wide range of text prompts. To conduct fair comparisons, we used
the same text prompts in their demos (70 samples from MusicLM; 41 samples from Noise2Music) R
and asked seven music producers to select the best out of pair of samples or voting for tie (both win)
in terms of musicality, audio quality, and text correlation. In total, we conducted 777 comparisons and collected 1,554 ratings. We detail the evaluation protocol in Appendix F. Table 3 shows the
comparison results, where each category of ratings is separated into two columns, representing the comparison against MusicLM (MLM) or Moise2MMusic (N2M), respectively. Finally, McLoDy consistently achieved comparable performances (all winning proportions fall into [0.4, 0.6]) in musicality and text correlation to MusicLM and 2uise2 Regarding audio quality, Mel.oDy outperformed MusicLM (p < 0.05) and 2usse2 (p < 0.01), where the p-values were calculated using the Wilcoxon signed-rank test. We note that, to sample 10s and 30s music, MeLoDy only takes 4.32% and 0.41% NFEs of MusicLM, and 10.4% and 29.6% NFEs of Moise2Music, respectively.
We conduct an additional experiment to study the diversity and validity of MeLoDy's generation given the same text prompt of open description, e.g., feelings scenarios. The sampled results were shown on demo page, in which we obtained samples with diverse combinations of instruments and textures.
Diversity Analysis Diffusion models are distinguished for its high diversity 25
Ablation Studies We also study the ablation on two aspects of the proposed method. In Appendix D. compared the uniform angle schedule in [X] and the linear one proposed in DPD using the MCC metric and case-by-case qualitative analysis. It turns out that our proposed schedule tends to induce fewer acoustic issues when taking a small number of sampling steps. In Appendix E, we showed that the proposed dual-path architecture outperformed other architectures [23] 31 used for
LDMs in terms of the signal-to-nove ratio (SNR) improvements using a subset of the training data.
6 Discussion
Limitation We acknowledge the limitations of our proposed MeLoDy. To prevent from having any disruption caused by unnaturally sound vocals, our training data was prepared to mostly contain non-vocal music only, which may limit the range of effective prompts for MeLoDy. Besides, the training corpus we used was unbalanced and slightly biased towards pop and classical music. Lastly, we trained the LM and DPD on 10s segments, the dynamics of long generation may be limited.
Broader Impact We believe our work has huge potential grow into music creation tool for music producers, content creators, or even normal users to seamlessly express their creative pursuits with low entry barrier. McLoDy also facilitates an interactive creation process, in Midjourney [2] to take human feedback into account. For a more precise tune of McLoDy on musical style, the LoRA technique
75 can be potentially applied to MeLoDy, as in Stable Diffusion 49
All samples for evaluation available at https://EfffiiientMaae MeieeeMeeLDDyiibbieb Note that samples were not cherry- picked, whereas the samples we compared were cherry-iicke [6] constituting very strong baselines. 'We use + to separate the counts for the iterative modules, i..., LM or DPM. Suppose the cost of each module is comparable, then the time steps taken by LM and the diffusion steps taken by DPM be fairly compared.

---

References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-683 2020.
[2] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[3] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: language modeling approach to audio generation. arXiv preprint arXiv:
2022.
[4] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: generative model for music. arXiv preprint arXiv:2005.00:: 2020.
[5] Andrea Agostinelli,, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXhv:2301.11323 2023.
[6] Qingqing Huang, Daniel S Park, Tao Wang, Timo Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang. Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.039/ 2023.
[7] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[8] Jacob Devlin, Ming- deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of
[9] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[10] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.088 200.292399 2022.
[11] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedhack. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[12] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209.155: 15352, 2022.
[13] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text 2023.
to speech synthesizers. arXiv preprint arXiv:2301.08
[14] Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi. Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. arXiv preprint arXiv:2302.03540, 2023,
[15] Jascha vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265, 2015.
Eric Weiss, Niru Maheswaranathhn, and Surya Ganguli. Deep unsuper-
[16] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21707 2021.
10

---

[17] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances neural information processing systems, 34, 2021.
[18] Rongjie Huang, Max WY Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. Fast- diff: A fast conditional diffusion model for high-quality speech synthesis. arXiv preprini arXiv:2204.09933 2022.
[19] Sungwon Kim, Heeseung Kim, and Sungroh Yoon. Guided-tts 2: A diffusion model for high-quality adaptive rext-to-speech with untranscribed data. arXiv preprint arXiv:2205.153700 2022.
[20] Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint urXiv:2304.091 16, 2023.
[21] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-an -o-uioo- generation with latent diffusion models. arXiv preprint arXiv:
2023.
[22] Rongjie Huang, Jiawei Huang, Dongchao Yang. Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. arXiv preprini arXiv:2301. 12661, 2023.
[23] Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf. Moñsai: Text-to-music generation with long-context latent diffusion. arXiv preprint a111772301.1175,, 2023.
[24] David Holz et al.
Midjourney. Artificial Intelligence platform. Accessible ar http:///www..miloooooee////////e November 1st, 2023.
[25] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:2112.0788 2021.
[26] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02550,, 2020.
[27] Qiuqiang Kong, Yin Cao, Turab lqbal, Yuxuan Wang. Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880-2894, 2020.
[28] Antoine Caillon and Philippe Esling. Rave: A variational autoencoder for fast and high-quality neural audio synthesis. arXiv preprint arXiv:2111.0555 2021.
[29] Marco Pasini and Jan Schlüter. Musika! fast infinite waveform music generation. arXiv preprint arXiv:2208.08706, 2022.
[30] Mubert Inc. Mubert. URL. https://mubert.com// 2023.
[31] S Forsgren and H Martiros. Riffusion stable diffusion for real-time music generation. URL https://riffusion.com/ 2023.
[32] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780. IEFE, 2017.
[33] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint 1rX.v:1609.034900 2016.
[34] Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukeuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. In International conference on machine learning, pages 3918-3926 3926. PMLR, 2018.

---

[35] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, pages 2410-2419. PMLR, 2018.
[36] Zalân Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.094 00966, 2023.
[37] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[38] Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path rnn efficient long sequence modeling for time-domain single-channel speech separation. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 46-50. IEEE, 2020.
[39] Jingjing Chen, Qirong Mao, and Dong Liu. Dual-path transformer network: Direct context- aware modeling for end-to-end monaural speech separation. arXiv preprint arXiv:2007.1397. 2020.
[40] Max WY Lam, Jun Wang, Dan Su, and Dong Yu. Effective low-cost time-domain audio separation using globally attentive locally recurrent networks. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 801-808. IEEE, 2021.
[41] Max WY Lam, Jun Wang, Dan Su, and Dong Yu. Sandglasset: A light multi-granullrity self- attentive network for time-domain speech separation. In ICASSP 202/-2021 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5759-5763, IEEE, 2021.
[42] Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong. Attention all you need in speech separation. In ICASSP 202 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 21-25. IEEE, 2021.
[43] Shengkui Zhao and Bin Ma. Mossformer: Pushing the performance limit of monaural speech separation using gated single-head transformer with convolution-augne augmented joint self-attention, In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 1-5. IEEE, 2023.
[44] Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244-250. IEEE, 2021.
[45] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
[46] Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel PW Ellis. Mulan: A joint embedding of music audio and natural language. arXiv preprint arXiv:2208. 12415. 2022.
[47] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 18:1241122460) 2020.
[48] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang. Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution augetetedd transformer for speech recognition. arXiv preprint arXiv:2005.088 2020.
[49] Robin Rombach and Patrick Esser.
Stable diffusion v2-1. 2023.
URI.
12

---

[50] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to- text-to-image diffusion models, arXiv preprint arXiv:2302.05543, 2023.
[51] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013.
[52] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and upproximate inference in deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.
[53] Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine, 35(1):53-66 2018.
[54] Jungil Kong, Jachyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 15:17022-170333 2020.
[55] Won Jang, Dan Lim, Jaesam Yoon, Bongwan Kim, and Juntae Kim. Univnet: A neural vocoder with
spectrogram discriminators for high-fidelity waveform generation. arXiv preprint urXiv:2106.07889 2021.
[56] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-88888 8171. PMLR, 2021.
[57] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13443 2020.
[58] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXhv:2202.00512, 2022.
[59] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint (xXiv:2206.00364 2022.
[60] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.133. 13438, 2022.
[61] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International conference on learning representations, 2020.
[62] Max WY Lam, Jun Wang, Dan Su, and Dong Yu. Bddm: Bilateral denoising diffusion models for fast and high-quality speech synthesis. In International Conference on Learning Representations, 2022.
[63] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwwave: A versatile diffusion model for audio synthesis. arXiv preprint wwXiv:200900976. 2020.
[64] Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pages 245-254, 1985.
[65] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
[66] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint Stfi::6006.04115. 2016.
[67] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2 104.09844 2021.
[68] Tao Lei, Yu Zhang, Sida I Wang, Hui Dai, and Yoav Artzi Simple recurrent units for highly parallelizable recurrence. arXiv preprint XXXv:1709002755 2017.
13

---

[69] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[70] OpenAl. Chatgpt. URL https::/chat.openaiiccomm
2023.
[71] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13:: 13971, 2023.
[72] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. wrXi::227.112598 2022.
arXiv preprint
[73] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Chan- ning Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. 2017 ieee international conference on acoustics, speech and signal processing (icassp), pages 131-135. IEEE, 2017.
[74] Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fréchet audio distance: reference- free metric for evaluating music enhancement algorithms. In INTERSPEECH, pages 2350-2354, 2354, 2019.
[75] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0900 2021.
14


Reviews:
[{"review_id": "hcD8NCHvLi", "reviewer": "Reviewer_hBX7", "summary": "The paper introduces MeLoDy, a music generation method combining Language Models (LMs) and Diffusion Probabilistic Models (DPMs). It addresses the challenge of generating music from diverse free-form text descriptions while mitigating high computational costs associated with current state-of-the-art models like MusicLM and Noise2Music. Key contributions include the development of MeLoDy, which significantly reduces computational requirements, the proposal of efficient Dual-Path Diffusion (DPD) models, an improved sampling scheme for DPD, and a successful implementation of audio VAE-GAN for effective continuous latent representation learning.", "questions": "What's the rule of thumbs for selecting the number of chunks when training?", "limitations": "The authors have discussed about the limitations of this work in the discussion section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The DPD model is particularly innovative as it is a variant of continuous-time diffusion probabilistic models that operates on low-dimensional latent representations.\n\nAs for significance, the paper offers a substantial contribution to the field of music generation by showcasing how complex relationships across long-term contexts can be modeled and by overcoming limitations of prior works. ", "weaknesses": "The proposed method builds upon existing research, employing the diffusion probabilistic model for music generation. Although the combination with the dual-path architecture is intriguing, it still largely borrows from previous architectures like AudioLM, MusicLM, and MuLan.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's an overall positive response. The question raised by the reviewer is addressed below.\n\n**Question:** What's the rule of thumbs for selecting the number of chunks when training?\n\n**Answer:** If we understand correctly, by \"the number of chunks\", the reviewer refers to the hyperparameter $M$, which defines the number of chunks for dividing the diffusion input with different noise scales. To understand the rule of thumbs for determining $M$, we first would like to re-visit the functionalities of using multi-chunk input. In essence, introducing such a input chunking technique leads to ``infinitely continuable generation'' for DPD. Recall that the network $v_\\theta(\\mathbf{z}\\_{\\text{noisy}}; \\mathbf{c})$ is conditional on with a noise scale vector ($\\boldsymbol\\delta\\in\\mathbb{R}^{L}$) that records the time-aligned noise scales of all elements in $\\mathbf{z}\\_{\\text{noisy}}\\in\\mathbb{R}^{L}$. Because of this time-aligned condition, the model is capable of de-noising partial regions of the noisy input.\n\nFor example, given a model trained on 10s of music audio and let M = 4, at the first run we can generate 10s sample. To extend for 2.5 more seconds, we can drop the first chunk of the generated sample (7.5s left) and append a new chunk of 2.5s white noise to the end of the sample. The resulted 10s is passed to the diffusion model as the input. Then, taking advantage of the input chunking scheme we used for training, we can intentionally set the first $\\frac{3L}{4}$ values in $\\boldsymbol\\delta$ to zeros and set the last $\\frac{L}{4}$ values in $\\boldsymbol\\delta$ to one. In this way, the model is capable of ignoring the first 3 chunks and only de-noises the the last appended chunk. At the same time, the de-noising process of the last chunk (2.5s) would depend on the first previously generated 7.5s for audio continuation.\n\nBy concerning the effect of $M$ on generation, we can select the number of chunks by considering 4 rules:\n- (1) **The length of training input**: MeLoDy was trained with 10s music clips, therefore it is better to select an $M$ such that $L$ is divisible by $M$. In our setting, 10s of music audio leads to $L=2500$ (250Hz * 10s). Setting $M=4$ leads to a length of $\\frac{L}{M}=625$ latents in each chunk.\n- (2) **The length of segment size in DPD**: In order to perform segmentation in DPD, we should ensure $\\frac{L}{M} \\geq K$. In our setting, $K=80$, therefore the constraint is satisfied.\n- (3) **The speed of continuation**: A larger $M$ leads to a smaller chunk $\\frac{L}{M}$, and the continuation would become slower.\n- (4) **The quality of continuation**: While different $M$ values correspond to different length of historical contents that assist the de-noising of the last chunk, when we set a smaller $M$ the length of previously generated contents used also becomes shorter.\n\nFrom rule (1), we have the possible values $M\\in\\set{1, 2, 4, 5, 10, 20, 25, 50, 100, 125, 250, 500, 625, 1250, 2500}$ (but $M=1$ is a dummy case that does not support continuation). From rule (2), we can further narrow the value set to $M\\in\\set{1, 2, 4, 5, 10, 20, 25}$. From rule (3), we conceive that it is more practically feasible to extend one-fifth to one-fourth of the generated audio at each run of the de-noising process. From rule (4), we experimented both $M=4$ and $M=5$ and assess their quality in continuation. We empirically found subtle differences in quality when using $M=4$ and $M=5$, therefore we finally opted for $M=4$ to enjoy a faster continuation."}, {"review_id": "WyGbgCPIiA", "reviewer": "Reviewer_ieEr", "summary": "This paper proposes MeLoDy, a diffusion-based text-to-music generation system. The system comprises four parts: 1) the MuLan contrastive model, 2) semantic language modeling using Wav2Vec2-Conformer, 3) a dual-path diffusion model to generate the latent, and 4) a 1D VAE-GAN autoencoder for learning the latent. The greatest contribution of this paper is the dual-path diffusion model, while other components follow a similar paradigm to that proposed in previous models, albeit with minor upgrades.\n\nIn the dual-path diffusion model, instead of applying diffusion on the full audio length, the paper suggests segmenting the target of length L into approximately sqrt(L) chunks. A RoFormer model is then used to process inter-segment, while an RNN is utilized to process the intra-segment. The dual-path diffusion model trains on a velocity objective with a novel linear angle schedule.\n\nA subjective listening test with music producers as listeners demonstrates that the proposed model outperforms previous methods in quality but falls short in musicality and text correlation.", "questions": "Typos:\nLine 277: The sentence appears to be incomplete: \"For sampling, the predicted velocity is linearly combined as .\"\n\nIssues with introducing the system:\nIn section 4.1.1, the concept of \"velocity\" should be introduced beforehand. Its sudden appearance made my initial reading confusing.\n\nAlso, I would like to know about the justification of the dual-path diffusion design choice. Please see the first point in the weakness section.", "limitations": "The limitations of the proposed system are adequately discussed in the final section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a novel dual-path diffusion model, which could be an effective alternative to the current design of the diffusion model. A new schedule for the diffusion process is also proposed. The model generates satisfactory results, as shown on the demo website, and the writing in the paper is clear.", "weaknesses": "From my perspective, there are two minor weaknesses:\n\n1. The authors should further justify the choice of the dual-path diffusion model in terms of its effectiveness. When compared to an alternative diffusion module for the proposed system, the prediction length in the dual-path diffusion is smaller. However, according to the paper, in the dual-path diffusion, at each diffusion (reverse) step, the input needs to run through two modules: RoFormer and SRU. While I can imagine that the RoFormer can be run in parallel for both inter-segment and intra-segment, I believe the SRU, as a type of RNN, still requires sequential computation. Therefore, an alternative diffusion model that works on 1D (waveform) or 2D (spectrogram) latent could leverage the benefits of parallel computing and run faster than the dual-path diffusion model on a GPU under the same sampling steps. I would appreciate the authors' insights on this point. It would be beneficial if this could be included in the paper to provide a more solid justification for the design choice.\n\n2. The experiments show that the proposed model does not outperform in terms of musicality and text correlation, which calls the design of the proposed model into question. However, I acknowledge that the data used in each model differ, making a controlled and reproducible comparison nearly impossible.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's overall positive response. The concerns and questions raised by the reviewer is addressed below.\n\n### Response to concern #1\n\n**Justification of the choice of the dual-path diffusion model**: Thanks for the insightful comments. We would like to first clarify one fact about dual-path network used in DPD. Suppose the input to the diffusion model is $\\mathbf{z}\\_\\text{noisy}\\in\\mathbb{R}^L$. After segmentation, there are two shortened sequences to be processed by the DPD blocks:\n- **Intra-segment sequences** (fine path): Each intra-segment sequence has a length of $K\\approx \\sqrt{L}$. The processing is parallelized across $S=\\lceil{\\frac{2L}{K}}\\rceil+1$ segments. In practice, suppose the input to DPD block has a shape of $[B, D, S, K]$, we can easily implement this parallelization in PyTorch by `.permute(0, 2, 3, 1).reshape(B * S, K, D)`, where $B$ is the batch size and $D$ is the hidden dimension. When using SRU, the computation cost is $\\mathcal{O}((BS)(D^2K))=\\mathcal{O}((B\\sqrt{L})(D^2\\sqrt{L}))$.\n- **Inter-segment sequences** (coarse path): On the other hand, each intra-segment sequence has a length of $S$. The processing is parallelized across $K$ frames (or merged frames). Let's first ignore the case of segment merging and repeating. The parallelization can be similarly implemented in PyTorch by `.permute(0, 3, 2, 1).reshape(B * K, S, D)`. When using attention-based sequence processing method, the computation cost is $\\mathcal{O}((BK)(D^2K+DS^2))=\\mathcal{O}((B\\sqrt{L})(D^2\\sqrt{L}+DL))$.\n\nAs mentioned by the reviewer, since the batched computation is specifically speeded up in GPU (with batch sizes of $BS$ and $BK$, respectively, for the above two sequences), we separate the considerations of batch axis and other axes in Big-O notation. On the other hand, considering the ``alternative diffusion model that works on 1D (waveform) or 2D (spectrogram) latent'' mentioned by the reviewer, we note that, without segmentation, an attention-based network operating on the full input sequence of length $L$ gives rise to a cost of $\\mathcal{O}((B)(D^2L+DL^2))$. In this sense, we can show in Big-O notation that DPD should be faster than any attention-based network operating on the full sequence of length $L$: \n- When $D$ is much larger than $L$, the dominant term in the attention network of interest would be $\\mathcal{O}((B)(D^2L))$. The dominant term in DPD would be $\\mathcal{O}((B\\sqrt{L})(D^2\\sqrt{L}))$, caused by SRU. Since the batch axis in DPD can processed much faster with GPU, we conclude that DPD is faster than than the attention network of interest.\n- When $L$ is much larger than $D$, the dominant term in the attention network of interest would be $\\mathcal{O}((B)(DL^2))$. The dominant term in DPD would be $\\mathcal{O}((B\\sqrt{L})(DL))$, caused by RoFormer. It is obvious that, even not considering the GPU acceleration on the batch axis, the cost of DPD in $\\mathcal{O}(BDL^{3/2})$ is still lower than that in the attention network of interest $\\mathcal{O}(BDL^2)$.\n\nWe also reach to a consistent conclusion in practice. During our ablation on different architectures, we also validated that the DPD model is faster than alternative diffusion models (i.e., 1D-UNet and 2D-UNet with cross-attention) for sampling one step, when both networks are constrained with the same model size. \n\nRegarding the particular choice of SRU and Roformer in DPD, we refer to the linkage between the de-noising task in diffusion to the separation problem, as stated in the main paper. In the context of separation, we accord to the findings in a separation work [2]: (1) in local modeling, the recurrent model performed better than the attention model; (2) in global modeling, the attention model performed better than the recurrent model. Following from these conclusions, we experimented through different RNNs (i.e., GRU, LSTM, and SRU) and attention networks (i.e., Transformer and Roformer), and eventually selected overall the best combinations (SRU+Roformer).\n\n---\n\n### Response to concern #2\n\n**``the proposed model does not outperform in terms of musicality and text correlation''**: First, we would like to note that the baseline samples we used for comparisons are (at least partially) cherrypicked, as stated in the main paper of Noise2Music. In contrast, we used the non-cherrypicked ones for all the text prompts taken from MusicLM and Noise2Music. In fact, the number of samples released on our demo page is more than 300, which prohibits the chance of cherrypicking sample for each prompt. Presumably, the sample variances of diffusion and LM could cover marginal differences in musicality and text correlation. In other words, from a practical sense, if we need to improve the musicality and text correlation given a text prompt, we can re-sample with the same prompt until we get a desired sample. Whereas, the generation speed and audio quality (denoting upper bound of the generation quality) aspects are rather steady -- they would not drastically vary even if we draw for more samples.\n"}, {"review_id": "7or0nOlvAA", "reviewer": "Reviewer_cbjn", "summary": "The authors introduce a framework (MeLoDy) for music generation. This new framework significantly improves the inference speed of MusicLM while producing  high-quality results. in terms of musicality, audio quality, and text correlation. The main reason for this speedup is that the authors replaced the coarse and fine acoustic stages in MusicLM and AudioLM from a language model to a dual-path diffusion model (DPD) in the latent space,  while keeping a LM to perform the high level semantic modeling process similar to MusicLM. The authors show that the resulting model can perform similar to MusicLM while being faster at sampling.", "questions": "A- Would you please address the weekness above?\nB- In line 158, you say: \"In our experiments, we find that the stability of generation can be significantly improved if we use token-based discrete conditions to control the semantics of the music and let the diffusion model learn the embedding vector for each token itself.\" For me, it's not clear what you mean here: are the tokens coming from the SSL Wav2Vec2-Conformer? How are you \"letting the diffusion model learn the embedding vector for each token\" if the embedding vectors are coming from the k-means clusters of the Wav2Vec2-Conformer?\n\nC-Line 183, The notation is not very clear for me; I assume the same MLP is performed in parallel for each token. and this is a different MLP than the one in equation 10? Is this correct? If so, I suggest stating the parameterization to make it more clear.\n\nD-Line 192, In order to reduce complexity, you set k = sqrt(L), which allows you to process long sequences with complexity that grows linearly with the sequence length L (after the self-attention over the coarse path); this is the reason you state in the abstract that \"infinitely continuable generation.\" Is that correct? Does not this come with the limitation that as this number K grows, the coarse path becomes even more coarse (more abstract)? Is this explained somewhere?\n\nE-Line 284, why did you choose to use an outdated VGGish model? Wouldn't the results be more accurate if one used more capable models, like a transformer (in AudioGen) or models that are trained to perform musical tasks, e.g., detect musical instruments?", "limitations": "The authors discussed the limitations of their work. One additional thing that comes to mind is copyright in music, which is a general issue for generative models. Is there a way to determine if the model reproduces copyrighted material from its training data?", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The authors address efficiency, which is a very important issue and can be easily overlooked in new research.\n2. I find the idea of combining LM and diffusion models very intuitive and has a lot of potential.\n3. The authors put a lot of effort into evaluating the generated samples by using human feedback.   \n4. The generated samples sound good (since they are not cherrypicked).", "weaknesses": "1. Complex pipeline system: The authors propose a complex pipeline with many components, moving parts, and many hyperparameters. This makes the system hard to reproduce, and the contribution of individual components is not clear. The authors provide a limited ablation study in the appendix. I think in a system like this, an extensive ablation study is needed. Examples: (a) What role does the encoder play? (b) Would different encoders affect the quality? (c) The authors used different Mulan audio and text towers compared to MusicLM; what effect does that have? (d) The authors mention they used attention on the course path and an SRU on the fine path; what role do these play? (e) What is the effect of merging and repeating?\nI think a lot of these decisions sound arbitrary making the work read like a technical report.\n2. Reproducibility, private dataset: the dataset is very vaguely described, making the proposed system even harder to reproduce and evaluate. The dataset plays a large role in the quality of the generation.\n3. Clarity: Section 4 in particular is very hard to read; the description of the system is not very clear. Additionally, some details are omitted, which can be in part because of the complexity of the pipeline. I recommend a restructure of this section. The appendix significantly helps in understanding the details.\n4. Evaluation: I think a comparison with the more specialized models can greatly improve the quality of the evaluation. Even if the specialized models (like genre) perform better under specific conditions. I believe it's important to illustrate the gap between these models and those that rely on free-form text generation. I think the current evaluation is limited.\n5. I believe the related work section is limited. If possible, I think this section should be extended to include related work and alternatives that can be used in the proposed pipeline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thorough review regarding our study. We provide detailed responses to all reviewer's concerns, as summarized below.\n\n### Response to concern #1\nThe reviewer concerned about the **complex system design** of the proposed MeLoDy with **``many components, moving parts, and many hyperparameters''**. We conceive that the term \"many\" used here could be arguably subjective without a suitable comparing target for comparison. Therefore, we would like to emphasize the following points:\n\n- (1) Since MeLoDy is proposed for an efficient generation of high-quality music, rationally the best comparing target would be the SOTA method/system proposed for music generation up to the date of this submission, i.e., MusicLM.\n\n- (2) Having confirmed a suitable comparing target, we now compare MusicLM against MeLoDy in terms of the number of components:\n  - **MusicLM**: 3 LMs (semantic, coarse & fine), a SSL module (w2v-BERT), a prompt encoder (MuLan) and an autoencoder (SoundStream). In total, there are **6 components** for training MusicLM.\n  - **MeLoDy**: 1 LM, 1 diffusion model (DPD), a SSL module (Wav2Vec2-Conformer), a prompt encoder (MuLan) and an autoencoder (VAE-GAN). In total, MeLoDy consists of **5 components**, which is less than that in MusicLM.\n\n- (3) In fact, we agree with the reviewer on the direction of making less complex music generation system for the ease of reproduction and analysis of individual components. However, it remains hypothetical whether a simple system can generate comparable or even better music audios without particular designs from different perspectives of understanding, e.g., semantic and acoustic perspectives.\n\n---\n\n### Response to concern #2\nThe reviewer also concerned the **contribution of individual components** and the **limitation of ablation studies**.\n\n- Firstly, the reviewer is right that we have experimented over many possible alternative modules for different sub-tasks, e.g., w2v-BERT v.s. Wav2Vec2-Conformer. In general, this can be casted into a searching problem for a best-performing model, which leads to a combinatorial cost. The ideal starting position of searching for us would be to replicate the settings in MusicLM.\n- Similar to many other research works, we used a leave-one-out comparison to greedily select the best module in each of the sub-task with an acceptable cost. It is noteworthy that the cost of training each possible recipe of our generation model is huge, as we used a large-scale training dataset (257k hours).\n- For each of the individual modules, we investigated suitable objective measures for rigorous comparisons, e.g., SI-SNR for diffusion architectures. However, for other modules like SSL, it is very hard to tell which one is better with an objective measure (if there is any, please let us know.). We therefore can only listen to some random samples (10-100 per system) led by different SSL modules (i.e., Wav2Vec2, Wav2Vec2-Conformer, and w2v-BERT), and subjectively vote for a better module. In common practice, this has been a simple yet effective way to select an audio generative model, since no objective metric can compete with human ears in audio quality assessment.\n- Response to **``(a) What role does the encoder play? (b) Would different encoders affect the quality?''**: If we understand correctly, the reviewer refers to the audio encoder in VAE-GAN. In this sense, the role of encoder is to construct a latent feature space that is robust to the generation errors, as discussed in [1]. We have tried a number of different configurations for VAE-GAN, and we can conclude that different encoders would certainly affect the quality. The output rate of the encoder (a.k.a. frequency of the latent sequence) is especially important, as the audio reconstruction from a higher-frequency latent sequence (e.g. 250Hz in MeLoDy) appears to be much better than a lower-frequency one (e.g. 50Hz in MusicLM). We will add this justification to our revised paper. \n- Response to **``(c) The authors used different Mulan audio and text towers compared to MusicLM; what effect does that have?''**: Since the code and training data of MuLan is not publicly available, the best we can do was to train the model from scratch following the network choices stated in their papers (AST for audio tower; BERT for text tower). We admit that the MuLan we trained on our data cannot be aligned with the one used in MusicLM. Yet, we actually used comparable network architectures for audio and text towers.\n- Response to **``(d) The authors mention they used attention on the course path and an SRU on the fine path; what role do these play?''**: If we understand the question correctly, the reviewer is confused about the design choice of applying attention and SRU for coarse-path and fine-path processing, respectively. In the main paper, we linked the de-noising task in diffusion to the separation problem. We referred to the conclusion in a separation work [2], where the authors found two consistent patterns: (1) in local modeling, the recurrent model performed better than the attention model; (2) in global modeling, the attention model performed better than the recurrent model.\n- Response to **``(e) What is the effect of merging and repeating?''**: Segment merging and repeating operations are originated from [3], where the authors stated two benefits: (1) Dynamic segment scale (termed as multi-granularity in [3]), and (2) speed up coarse-path processing. We will add more discussion on these effects in the revised paper.\n\n[1] R. Rombach, et al. High resolution image synthesis with latent diffusion models. In CVPR, pages 10684–10695, 2022.\n\n[2] M. WY Lam, et al. Effective low-cost time-domain audio separation using globally attentive locally recurrent networks. In 2021 IEEE SLT, pages 801–808. IEEE, 2021.\n\n[3] M. WY Lam, et al. Sandglasset: A light multi-granularity self-attentive network for time-domain speech separation. In ICASSP 2021, pages 5759–5763. IEEE, 2021."}, {"review_id": "jArqTURcu2", "reviewer": "Reviewer_4HjZ", "summary": "The authors present a text to music model, with the aim of achieveing faster than real time generation on a V100 gpu. The authors introduce a latent space diffusion model. The latent space is obtained with an adverdarial VAE (replacing the VQ VAE from MusicLM), and a first LM is trained on the semantic tokens just as in MusicLM. The LM is also conditioned on the quantized output of a Mulan model. Then a latent diffusion model is trained to predict the latent representation of the VAE, conditioned on the semantic tokens. In order to improve the speed of the model, the authors uses a dual path architecture, i.e. they alternate between modeling long range and short range dependencies. Short range dependencies are modeled with a simple RNN, while long range ones are modeled with an attention based model.\nThe model is trained on ~260k hours of music with tag labels, enriched with ChatGPT outputs into full descriptions.\n\nThe authors compare with the publicly available samples of MusicLM and Noise2Music. They show that their model achieves better quality but overall worse musicality and text fidelity. They also report the FAD on MusicCaps (5.1) which is worse than that of MusicLM (4.0) or Noise2Music (2.1).", "questions": "What are \"velocities\" ? this term is not properly defined when introduced.\n\nWhy are the authors using different noise levels for different segments? I see in the supplementary this is used for inpainting. Other than that regular sampling would always use the same noise level for all segments? Is there a benefit to this training method for regular sampling?\n\nIn Section 5.1, the authors mention they filter the data to focus on non vocal music. Is the 257k hours number given before or after this filtering?\n\nThe sample rate of the proposed of the VAE seems relatively large compared to the one used in MusicLM, is this something the authors experimented with?", "limitations": "The authors fail to discuss potential adverse effect of automation of creative jobs. The authors fail to discuss the potential breaches to intellectual property for the data used. In particular, the authors should discuss how the dataset was assembled.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- large improvement in terms of speed compared with the many models and diffusion steps required by Noise2Music.\n- improvements on the diffusion schedule.\n- interesting idea of using a dual path architecture to improve the runtime. This allows to take full advantage of the non causal aspect of the diffusion process used here instead of auto-regressive modeling.\n- Runtimes on Table 2 are impressive.\n- subjective evaluations show improved quality.\n", "weaknesses": "- still requires a large number of models to train, e.g. Mulan model, a semantic model w2v-bert + one LM for the semantic tokens, and one dual path diffusion model, without counting the VAE.\n- ablation studies are limited to the noise schedule. Not strong objective motivation for most of the design choices.\n- authors mention that the Mulan Cycle consistency score is worse for the test set MusicCaps than their model. However that could be an artefact of how the authors' Mulan model is trained, e.g. on weak labels generated from partial tags rather than full descriptions, in particular considering that their own model is trained with this Mulan model, so that it could overfit any bias it has.\n- when comparing the number of function calls in Table 3, the authors forget to mention that for MusicLM, each call require the evaluation of a single time step, while their own model require to process all the timesteps of the input sequence. In particular, in the context of batched generation, this can make a big difference between the two models.\n- objective metric FAD is worse than existing methods, and two of the subjective metrics (musicality and text correlation). Subjective evaluations are done on the samples released by the other methods which might be better than the average though. \n\nNote that the authors provide their supplementary material through a website, which might not be an acceptable way of doing so, as it doesn't enforce the deadline for the supplementary material submission.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's careful reading of our paper. We hope the following fully address all concerns mentioned by the reviewer:\n\n### Response to concern #1\n\n**``large number of models to train''**: The reviewer is correct that MeLoDy comprises **5 components** for a successful training, i.e., a semantic LM, a diffusion model (DPD), a SSL module (Wav2Vec2-Conformer), a prompt encoder (MuLan) and an autoencoder (VAE-GAN). Yet, in comparison to the SOTA MusicLM, the design MeLoDy is already simpler, since MusicLM consists of training **6 components**: 3 LMs (semantic, coarse & fine), a SSL module (w2v-BERT), a prompt encoder (MuLan) and an autoencoder (SoundStream). In this sense, we could reckon this work as one step towards a less complex system.\n\n---\n\n### Response to concern #2\n\n**Motivations behind the design choices**: Noticeably, reviewer **cbjn** and **KL1t** also share a similar argument. We take these comments seriously, and restructure Section 4 with several amendments:\n- (1) We re-organized Section 4 to make it more condense and self-contained. We added textual explanations for the design choice of each module (e.g. angle parameterization has been compared against noise parameterization and performed the best in [1]). A lot of mathematical details regarding network architecture has been moved to the Appendix.\n- (2) Some leaps of definition were avoided in the revised paper. For example, the diffusion velocity was ill-defined in Section 4, as pointed out by the reviewer.\n- (3) As we compress Section 4 to be more condense, the ablation studies is now moved from the Appendix to the main paper, as suggested the reviewer **KL1t**.\n\nBesides, the reviewer mentioned the **objective motivation** for the design choices. In fact, for each of the individual modules, we investigated suitable objective measures for rigorous comparisons, e.g., SI-SNR for diffusion architectures. However, for other modules like SSL, it is very hard to tell which one is better with an objective measure (if there is any, please let us know.) We therefore can only listen to some random samples (10-100 per system) led by different SSL modules (i.e., Wav2Vec2, Wav2Vec2-Conformer, and w2v-BERT), and subjectively vote for a better module. In common practice, this has been a simple yet effective way to select an audio generative model, since no objective metric can compete with human ears in audio quality assessment.\n\n[1] Flavio Schneider, et al. Moûsai: Text-to-music generation with long-context latent diffusion. 2023.\n\n---\n\n### Response to concern #3\n\n**Mulan Cycle Consistency**: The reviewer is right that there is an inductive bias in the MCC measurement. Yet, we note that the objective of using MCC here and in MusicLM was to examine whether the conditioning MuLan embeddings form a cycle in the conditional generation (MuLan -> MeLoDy -> MuLan). The MCC results in Table 2 was meant to confirm that our proposed DPD is capable of consistently completing the MuLan cycle at significantly lower costs. \n\n---\n\n### Response to concern #4\n\n**Fairness of comparing the number of function calls**: The reviewer's argument is thought-provoking. Yet, we note that each function call in an LM also need to take previous tokens into account. In the case of decoder-only LM, the computational cost of each call is $\\mathcal{O}(L^2)$ with $L$ being the length of tokens. It is actually comparable to the cost of each call of a diffusion model ($\\mathcal{O}(L^2)$ if we also use attention in diffusion). Let alone the big-O notation, in practice the cost of each call of a DPD is in fact much lower than that of a Transformer-based LM, since the segmentation leads to $L=L'/K\\approx \\sqrt{L'}$, where $L'$ is the original sequence length. The cost of attention in DPD thus becomes almost linear to the length of sequence.\n\n---\n\n### Response to concern #5\n\n**``objective metric FAD is worse than existing methods''**: Notably, it is unfair to directly compare the FAD of MeLoDy against MusicLM and Noise2Music, since MeLoDy was mainly trained on processed non-vocal data and most samples in MusicCaps contain speech or vocal.\n\n---\n\n### Response to concern #6\n\n**Worse subjective metrics (musicality and text correlation)**: First, we would like to note that the baseline samples we used for comparisons are (at least partially) cherrypicked, as stated in the main paper of Noise2Music. In contrast, we used the non-cherrypicked ones for all the text prompts taken from MusicLM and Noise2Music. In fact, the number of samples released on our demo page is more than 300, which prohibits the chance of cherrypicking sample for each prompt. Presumably, the sample variances of diffusion and LM could cover marginal differences in musicality and text correlation. In other words, from a practical sense, if we need to improve the musicality and text correlation given a text prompt, we can re-sample with the same prompt until we get a desired sample. Whereas, the generation speed and audio quality (denoting upper bound of the generation quality) aspects are rather steady -- they would not drastically vary even if we draw for more samples.\n\n---\n\n### Response to other questions\n\n- The definition of velocities ($\\mathbf{v}\\_t:=\\frac{\\partial \\mathbf{z}\\_t}{\\partial t}$) has been added to the main context.\n- Using different noise levels for different segments makes possible the infinitely continuable generation (see the detailed response to reviewer **hBX7**).\n- The 257k hours of data has already gone through the filtering.\n- Yes, we have tried a number of different configurations for VAE-GAN, and we can conclude that output rate of the encoder (a.k.a. frequency of the latent sequence) is especially important, as the audio reconstruction from a higher-frequency latent sequence (e.g. 250Hz in MeLoDy) appears to be much better than a lower-frequency one (e.g. 50Hz in MusicLM). We have added this justification in our revised paper."}, {"review_id": "0h5zK6cVpH", "reviewer": "Reviewer_KL1t", "summary": "This paper introduces an LM-guided diffusion model designed for the generation of music audios. The proposed system follows a three-step process, starting with the extraction of audio-text embeddings (MuLan tokens) using a pre-trained MuLan model. Next, semantic tokens (wav2vec tokens) are generated with the aid of a language model. Finally, the synthesized music audio is produced using a dual-path diffusion model.\n\nExperimental results highlight the superiority of the proposed model, showcasing its practical advantages in terms of sampling speed and the ability to generate music seamlessly. ", "questions": "NA", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper exhibits clear and well-written sections before the experiments, ensuring the reader’s comprehension and engagement.\n2. The paper addresses a highly meaningful topic and task, aligning with the current trends and hot topics in language modeling.\n3. The proposed method in this paper is well-designed, effectively integrating domain knowledge in music and demonstrating a deep understanding of the subject matter.", "weaknesses": "1. The overall structure of the paper’s writing could benefit from optimization. For example, important experimental results should ideally be included in the main body rather than being relegated to the appendix.\n2. The paper lacks a comprehensive objective evaluation of the main experiments, particularly in comparing the proposed method with the baseline approaches instead of only using objective metrics in the ablation study. \n3. Based on the findings in Table 3, the proposed method exhibits a noticeable drop in musicality and text correlation compared to the baseline approaches. This raises questions about the motivation behind the proposed method. Is the trade-off in performance for the sake of efficiency justified? Does the efficiency gained by the proposed method outweigh the potential loss in important aspects such as musicality and text correlation?\n4. The appendix file is difficult to locate, as there is no indication or hint for readers that it is available on the demo page. Clear guidance or references within the paper would greatly assist reviewers and readers in accessing the relevant appendix materials.\n5. The paper lacks a clear explanation for certain design choices in the proposed method, such as the motivation behind utilizing an angle parameterization for the diffusion schedule and the specific design of Equation 4. Without a detailed rationale and justification for these design decisions, it becomes difficult to comprehend their significance and how they improve the overall performance and effectiveness of the method. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the the reviewer's comments and suggestions. In the following, we address each of the concerns raised by the reviewer.\n\n### Response to concern #1\n\n**Optimizing overall structure of this paper**: We are thankful for the reviewer's constructive comment, which is taken seriously to revise this paper for a clearer presentation. Thanks to the concrete suggestions raised by all reviewers, we have already made several changes in the revised paper. Some of the amendments are described below:\n\n- (1) We have re-organized Section 4 to make it more condense and self-contained. A lot of details regarding network architecture has been moved to the Appendix. We put more efforts to summarize the idea in sentences instead of equations. Also, some leaps of definition were avoided in the revised paper. For example, the diffusion velocity was ill-defined in Section 4 (as pointed out by reviewer **4HjZ** and **ieEr**). We added a brief description in advance to make it self-contained.\n\n- (2) As we make the Section 4 to be more condense, the ablation studies is now moved from the Appendix to the main paper, as suggested the reviewer.\n\n- (3) We also extended the related work to add more discussion on audio/music generation methods as well as audio representation learning, as suggested the reviewer **cbjn**.\n\n---\n\n### Response to concern #2\n\n**Lack of comprehensive objective evaluation in the main experiments**: For each of the individual modules, we were eager to investigate suitable objective measures for leave-one-out rigorous comparisons, e.g., SI-SNR for different diffusion architectures. However, for an overall assessment on the whole generation system, since the competing systems are not publicly released and were trained with entirely different datasets, the fairness of objective comparison cannot be guaranteed.\n\nBesides, as insisted in the title of this paper, MeLoDy is proposed to solve the computational problem associated with the SOTA MusicLM, and to make efficient music generation from free-form text possible. We focus on the comparison against MusicLM in terms of various aspects -- speed, quality, musicality and text correlation. Among these 4 aspects, only the assessment of speed was objectively evaluated since, to the best of our knowledge, no known objective measures can be as precise as the subjective evaluations of music professionals when assessing the audio quality, musicality and text correlation of an arbitrary audio sample.\n\n---\n\n### Response to concern #3\n\n**``noticeable drop in musicality and text correlation''**: First, we would like to note that the baseline samples we used for comparisons are (at least partially) cherrypicked, as stated in the main paper of Noise2Music. In contrast, we used the non-cherrypicked ones for all the text prompts taken from MusicLM and Noise2Music. In fact, the number of samples released on our demo page is more than 300, which prohibits the chance of cherrypicking sample for each prompt. Presumably, the sample variances of diffusion and LM could cover marginal differences in musicality and text correlation. In other words, from a practical sense, if we need to improve the musicality and text correlation given a text prompt, we can re-sample with the same prompt until we get a desired sample. Whereas, the generation speed and audio quality (denoting upper bound of the generation quality) aspects are rather steady -- they would not drastically vary even if we draw for more samples.\n\n---\n\n### Response to concern #4\n\n**Appendix**: We would like to apologize for the missing appendix due to some sudden network issues happened around the appendix deadline. We chose to immediately upload it to the demo page (the commit time in Github is tractable). Although the supplementary material is regarded as optional, the reviewer is right that we strongly recommend the readers to look into the Appendix for a thorough understanding of this paper. We will definitely supplement the Appendix file if this paper can be accepted.\n\n---\n\n### Response to concern #5\n\n**Lack of clear explanation for certain design choices**: We acknowledge the reviewer's comment regarding presentation clarity. A similar argument has been made by reviewer **cbjn**. We take this seriously, and add more texts for explaining the motivations and the reasons behind the choice of each module (e.g. angle parameterization has been compared against noise parameterization and performed the best in [1]). As pointed out before, we have restructured Section 4 to provide more incentives in textual forms instead of mathematical forms. We believe our revisions made for this paper sufficiently improve the clarity and address the problems concerned by the reviewer.\n\n[1] Flavio Schneider, et al. Moûsai: Text-to-music generation with long-context latent diffusion. 2023."}]

Author rebuttal:

Meta-review: This paper introduces a text-to-music latent diffusion model, that conditions on LM generated semantic tokens, with a target latent space obtained from an adversarial VAE. To speed up decoding, the paper proposes a dual-path architecture for diffusion in the latent space which is then decoded using the audio VAE-GAN. 

The overall score leans positive. All of the reviewers found the proposed method to be innovative and significant, and that it addresses the important issue of efficiency. Clarification of how the model enables improved efficiency was needed, and the authors have addressed most of the reviewers' questions in the rebuttal.

A couple reviewers raised the need for ablation studies, as the method requires many components and design decisions. The authors have explained some of their choices and how they tested for various modeling alternatives. We encourage the authors to include those results in the final paper. 

The paper demonstrated improvement on metrics of audio quality, while some reviewers were concerned that the method performed worse on other objective and subjective metrics. The authors provided a reasonable explanation, and a few reviewers did find the samples from the demo page compelling.

Overall, this paper presents a novel and significant contribution that is well-executed. It offers a novel example of combining the strengths of LM and diffusion-based models, enabling  improved efficiency. Hence, the recommendation is to accept the paper.
Decision: Accept (poster)