EchoAudio: Efficient and High-Quality Text-to-Audio Generation with Minimal Inference Steps

Authors: huadai  liu
rongjie  huang
yang  liu
hengyuan  cao
jialei  wang
xize  cheng
siqi  zheng
zhou  zhao
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 17:20:43

Abstract: Recent advancements in Latent Diffusion Models (LDMs) have propelled them to the forefront of various generative tasks. However, their iterative sampling process poses a significant computational burden, resulting in slow generation speeds and limiting their application in text-to-audio generation deployment. In this work, we introduce AudioLCM, a novel consistency-based model tailored for efficient and high-quality text-to-audio generation. Unlike prior approaches that address noise removal through iterative processes, AudioLCM integrates Consistency Models (CMs) into the generation process, facilitating rapid inference through a mapping from any point at any time step to the trajectory's initial point. To overcome the convergence issue inherent in LDMs with reduced sample iterations, we propose the Guided Latent Consistency Distillation with a multi-step Ordinary Differential Equation (ODE) solver. This innovation shortens the time schedule from thousands to dozens of steps while maintaining sample quality, thereby achieving fast convergence and high-quality generation. Furthermore, 
to optimize the performance of transformer-based neural network architectures, we integrate the advanced techniques pioneered by LLaMA into the foundational framework of transformers. This architecture supports stable and efficient training, ensuring robust performance in text-to-audio synthesis. Experimental results on text-to-audio generation and text-to-music synthesis tasks demonstrate that AudioLCM needs only 2 iterations to synthesize high-fidelity audios, while it maintains sample quality competitive with state-of-the-art models using hundreds of steps. AudioLCM enables a sampling speed of 333x faster than real-time on a single NVIDIA 4090Ti GPU, making generative models practically applicable to text-to-audio generation deployment. Our extensive preliminary analysis shows that each design in AudioLCM is effective.
\footnote{Audio samples are available at \url{https://AudioLCM.github.io/.}}~\footnote{Code is Available at~\url{https://github.com/Text-to-Audio/AudioLCM}}

Main paper: 
 introduction: 
 Text-to-audio generation (TTA) [13, 20, 23] is a subfield of generative tasks that produces natural and precise audio from textual prompts. TTA has a wide range of applications, including sound effects creation, musical compositions, and synthesized speech. It is used in various domains, such as film post-production, video game development, and audio manipulation. Previous iterations of neural TTA models have been primarily categorized into two main types: language models [21, 29] and diffusion models [7, 12] .(1) Language models encode raw waveform data into discrete representations and employ autoregressive language models to predict audio tokens based on textual input features. (2) Diffusion models: This category involves generating audio representations, such as discrete codes [50] or mel-spectrograms [13] , from textual prompts using diffusion models. These representations are then converted into audio waveforms using separately trained vocoders. Although both methods have demonstrated the ability to produce high-quality audio samples, they are often limited by high computational costs, which can make it difficult to achieve both quality and efficiency in audio generation.Diffusion models [9, 16, 41] , particularly Latent Diffusion models (LDMs) [33, 34] , have yielded unprecedented breakthroughs across various fields including image synthesis [32, 34, 35] , video generation [11, 39] , and audio synthesis [17, 24, 54] . A key feature of diffusion models is their iterative sampling mechanism, which progressively refines random initial vectors, thereby mitigating noise and enhancing sample quality iteratively. This iterative refinement process facilitates a flexible trade-off between computational resources and sample quality; typically, allocating additional compute for more iterations results in the generation of higher-quality samples. However, the generation procedure of diffusion models typically requires more computational costs for sample generation, causing slow inference and limited real-time applications.To alleviate this computational bottleneck, advanced numerical solvers [27, 53] of Stochastic Differential Equations (SDE) or Ordinary Differential Equations (ODE) substantially reduce the required Number of Function Evaluations (NFE), further improvements are challenging due to the intrinsic discretization error present in all differential equation solvers. Recent endeavors have focused on enhancing sample efficiency through distillation models. These models [30, 31, 36] aim to distill the knowledge from a pre-trained diffusion model into compact architectures capable of conducting inference with minimal computational overhead. For instance, Meng et al. [31] proposed a two-stage distillation approach aimed at improving the sample efficiency of classifier-free guided models. However, this approach presents several challenges: (1) Computational Intensity: As estimated by Liu et al. [26] , the distillation process entails significant computational resources, necessitating at least 45 A100 GPU days for training 2-step student models. (2) Error Accumulation: The two-stage guided distillation process may inadvertently introduce accumulated errors, thereby compromising the performance of the distilled models, resulting in suboptimal results.In this work, we propose AudioLCM, a consistency model based method tailored for efficient and high-quality text-to-audio generation; 1) To avoid significant degradation of perceptual quality when reducing reverse iterations, AudioLCM integrates Consistency Models (CMs) [40] into the generation process, facilitating rapid inference through a mapping from any point at any time step to the trajectory's initial point; 2) To overcome the convergence issue inherent in LDMs with reduced sample iterations and accelerate the convergence speed, AudioLCM reduces the data variance in the target side via One-stage Guided Consistency Distillation with multi-step Ordinary Differential Equation (ODE) solver. Specifically, We can sample 𝒙 𝑡 +𝑘 from the transition density of the SDE, and then compute x𝜙 using 𝑘 (𝑘 > 1) discretization step of the numerical ODE solver. We also include the Classifier-free Guidance (CFG) [10] into the process of distillation and sampling to improve the audio quality.Moreover, as a monochromatic image, the mel-spectrogram lacks spatial translation invariance. The vertical axis of the melspectrogram corresponds to the frequency domain, implying that mel-spectrogram patches at different vertical positions can convey completely different information and should not be treated equivalently. In addition, the use of a 2D convolution layer and a spatial transformer-stacked U-Net architecture limits the model's ability to generate audio of variable length. Prior research [12] in TTA has demonstrated the effectiveness of addressing this issue. To enhance the performance of transformer-based neural network architectures, we integrate advanced techniques developed by LLaMA [45] into the foundational framework of transformers. The LLaMA methodology, characterized by its causal transformer architecture tailored for large language models, holds promise for enhancing the capabilities of transformer backbones. By seamlessly integrating these methodologies, AudioLCM aims to unlock heightened performance levels and facilitate a more stable training process.Experimental results from text-to-audio generation and textto-music synthesis tasks demonstrate that AudioLCM achieves high-fidelity audio synthesis in only 2 iterations, while maintaining sample quality comparable to state-of-the-art models that require hundreds of steps. In addition, AudioLCM achieves a sampling rate 333 times faster than real-time on a single NVIDIA 4090Ti GPU, making generative models practical for text-to-speech applications. Our comprehensive preliminary analysis demonstrates the effectiveness of each component within AudioLCM. The main contributions of this study are summarized below:  
preliminaries: 
 In this section, we briefly introduce the theory of Diffusion models and consistency models.Diffusion models [14, 42] , also known as score-based generative models, are designed to generate data by iteratively perturbing data with Gaussian noise and subsequently generating samples from the perturbed data using a reverse denoising process. Let 𝑝 𝑑𝑎𝑡𝑎 (𝑥) denote the original data distribution. Continuous-time diffusion models [15, 18, 43] conduct the forward process using a stochastic differential equation (SDE):EQUATIONwhere 𝒇 () and 𝒈 represent the drift and diffusion coefficients, respectively, while 𝒘 𝑡 denotes the standard Brownian motion. By considering the existence of an ordinary differential equation (ODE), also called the Proability Flow ODE (PF-ODE):EQUATIONwhere ∇𝑙𝑜𝑔𝑝 𝑡 (𝑥) is the score function of 𝑝 𝑡 (𝑥).We train the noise prediction model 𝜖 𝜃 (𝑥 𝑡 , 𝑡) to approximate the score function, enabling us to derive the following empirical PF-ODE:EQUATIONConsistency models [40] (CMs) represent a novel class of models facilitating few-step or even one-step generation. The fundamental principle underlying consistency models is the establishment of a mapping capable of connecting any point at any given timestep to the starting point of a trajectory. This mapping is encapsulated by the consistency function, denoted as 𝒇 : (𝒙 𝑡 , 𝑡) → 𝒙 𝜖 , where 𝑥 𝑡 represents a predetermined trajectory of the PF-ODE and 𝜖 is a predefined small positive value. A distinctive feature of the consistency function is its self-consistency property, where points on the same trajectory converge to the same starting point. This property is expressed mathematically as follows:EQUATIONThe objective of a consistency model 𝒇 𝜃 is to estimate the consistency function 𝒇 from data, thereby enforcing the self-consistency property. For any consistency function, it holds that 𝒇 (𝒙 𝜖 , 𝜖) = 𝒙 𝜖 , termed as the boundary condition. To satisfy this boundary condition, the consistency model 𝒇 𝜃 is parameterized as follows:EQUATIONwhere 𝑐 𝑠𝑘𝑖𝑝 (𝑡) and 𝑐 𝑜𝑢𝑡 (𝑡) are differentiable functions, with 𝑐 𝑠𝑘𝑖𝑝 (𝜖) = 1 and 𝑐 𝑜𝑢𝑡 (𝜖) = 0. Additionally, 𝑭 𝜃 (•, •) represents a deep neural network. Consistency models can be trained either by consistency distillation or from scratch.In the former scenario, consistency models distill insights from pre-trained diffusion models into a target neural network tailored for sampling. Given a data point 𝒙, an accurate estimation of 𝒙 𝑡 𝑛 from 𝒙 𝑡 𝑛+1 is achieved by executing a single discretization step using a numerical ODE solver. Mathematically, this is expressed as:EQUATIONwhere Φ(•, •; 𝜙) signifies a one-step ODE solver applied to the empirical PF-ODE. Afterward, the objective is to minimize discrepancies within the pair ( x𝜙 𝑡 𝑛 , 𝒙 𝑡 𝑛+1 ) by 𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦𝑑𝑖𝑠𝑡𝑖𝑙𝑙𝑎𝑡𝑖𝑜𝑛𝑙𝑜𝑠𝑠 defined as:EQUATIONwhere x𝜙 𝑡 𝑛 corresponds to Equation 6, 𝜃 -represents a running average of past 𝜃 values during the optimization process, and 𝑑 (•, •) denotes a metric function used to quantify the distance between the pair. 
audiolcm: 
 This section presents our proposed AudioLCM, a few-step and onestep consistency model for high-fidelity text-to-audio generation. The framework of the proposed method is shown in Figure 1 . We begin by outlining the motivation behind each design choice in AudioLCM. Next, we detail the process of selecting a diffusion teacher model and performing Guided Consistency Distillation from it. We then discuss the model architecture and training loss used in AudioLCM, followed by a discussion of the training and inference algorithms.Diffusion models have made significant progress in various domains such as audio and image generation. However, several challenges hinder their industrial deployment: 1) The iterative nature of the prevailing latent diffusion text-to-audio models requires a considerable amount of computational resources for sample generation, resulting in slow inference times and limited real-time applicability. This limitation is a barrier to the practical use of these models in real-world scenarios. 2) Diffusion models typically train noise prediction models using extended time-step schedules. For example, models such as AudioLDM employ a time-step schedule of length 1,000, resulting in slower convergence rates and increased computational requirements.Recently, Consistency Models [40] have emerged as a novel generative approach to support few-step and even one-step generation. Unlike previous methods that rely on iterative processes for noise removal, AudioLCM integrates consistency models (CMs) into the generation process. This integration enables rapid inference by mapping any point at any time step to the starting point of the trajectory. To address the convergence challenges inherent in Latent Diffusion Models [34] (LDMs) with reduced sample iterations, we introduce Guided Consistency Distillation with a multi-step ODE solver. This approach significantly reduces the schedule from thousands to tens of steps while maintaining sample quality, resulting in faster convergence and high-quality generation. Furthermore, to improve the performance of transformer-based neural network architectures, we integrate advanced techniques pioneered by LLaMA into the basic framework of transformers.In conclusion, AudioLCM leverages a consistency model in the audio latent space to achieve rapid convergence and generation while ensuring sample quality. This approach is suitable for interactive, real-world applications. In this subsection, we describe the teacher model we select. As a blossoming class of generative models, many text-to-audio generation systems apply latent diffusion models to generate high-quality audio. However, the teacher model is supposed to achieve fast and high-quality text-to-audio generation, and thus the distilled student could inherit its powerful capability. As such, we choose the Make-An-Audio 2 [12] and modify its feed-forward transformer [47] with our enhanced transformer-based Backbone while keeping other designs as our teacher model to strike a proper balance between perceptual quality and sampling speed.Enhanced Transformer-based Backbone. Previous research [12] on TTA synthesis treated the mel-spectrogram as a single-channel image, similar to text-to-image synthesis, but the mel-spectrogram lacks spatial translation invariance due to its representation of the frequency domain. Consequently, patches at different heights carry distinct meanings and should not be treated equally. Furthermore, the use of a 2D-convolution layer and spatial transformer-stacked U-Net architecture limits the model's ability to generate variablelength audio. According to the practice of Make-An-Audio 2, they use a modified audio VAE that uses a 1D-convolution-based model and a feed-forward Transformer-based diffusion denoiser backbone. Inspired by the success achieved by LLaMA [45] , we integrate the advanced techniques pioneered by LLaMA into the foundational framework of feed-forward transformers. We draw the main improvement with the original architecture:• Pre-normalization: We use the RMSNorm normalizing function [52] and normalize the input of each transformer layer for the training stability. • Rotary Embeddings: We replace the absolute positional embeddings with rotary positional embeddings [44] (RoPE) at each layer of transformer.• SwiGLU Activation: We remove the SiLU function, and instead add the SwiGLU activation function [38] to improve the performance.Given the audio latent space established by the teacher model, aimed at mitigating computational overhead and enhancing performance, we proceed to redefine the PF-ODE governing the reverse diffusion process, denoted as Equation 3:EQUATIONTo preserve the property of self-consistency, we introduce the consistency function 𝒇 𝜃 : (𝒛 𝑡 , 𝒄, 𝑡) → 𝒛 0 , which maps points from any trajectory of the PF-ODE to the origin of the trajectory. We then parameterize the consistency function using the noise prediction model ε𝜃 to satisfy the boundary condition:EQUATIONwhere 𝑐 𝑠𝑘𝑖𝑝 (0) = 1, 𝑐 𝑜𝑢𝑡 (0) = 0, 𝛼 𝑡 and 𝜎 𝑡 specify the noise schedule, and ε𝜃 (𝒛, 𝒄, 𝑡) is a noise prediction model initialized with the parameters of the teacher model.To adapt to the discrete-time schedule in the teacher model, we utilize DDIM as the ODE solver Ψ(𝒛 𝑡 , 𝑡, 𝒄) to obtain an accurate estimate of 𝒛 𝑡 𝑛 from 𝒛 𝑡 𝑛+1 . Note that we only use the solver in training, not in inference. Then we estimate the evolution of the PF-ODE from 𝑡 𝑛+1 → 𝑡 𝑛 using one-step ODE solver:EQUATIONFinally, we calculate the consistency distillation loss to optimize the AudioLCM model:EQUATIONClassifier-free Guidance (CFG). The effectiveness of classifierfree guidance has been demonstrated in the synthesis of highquality text-aligned audio [10] . To facilitate one-stage guided distillation, we integrate CFG into the distillation process using an extended PF-ODE. Specifically, we replace the noise prediction model ε𝜃 (𝒛, 𝒄, 𝑡) with ε𝜃 (𝒛, 𝜔, 𝒄, 𝑡), which incorporates the learnable guidance scale embedding 𝜔. Consequently, we introduce an extended consistency function 𝒇 𝜃 : (𝒛 𝑡 , 𝒘, 𝒄, 𝑡) → 𝒛 0 :EQUATIONSimilarly, the consistency loss remains the same as Equation 11, with the only difference of consistency function.Latent diffusion models typically train noise prediction models with a long time-step schedule to achieve high-quality generation results.The formula of the teacher model has the same problem, which needs to sample across all 1,000 timesteps. However, the extended time steps are highly time-consuming and computer-consuming for applying the guided consistency distillation. Consistency models conduct one-step ODE solvers to estimate 𝒛 𝑡 𝑛 from 𝒛 𝑡 𝑛+1 , while they are close to each other and incurring a small consistency loss.To achieve fast convergence while preserving generation quality, we introduce multi-step ODE solvers to considerably shorten the length of time schedule (from thousands to dozens). Instead of ensuring consistency between adjacent time steps 𝑡 𝑛+1 → 𝑡 𝑛 , we perform a k-step discretization step of the ODE solver to obtain 𝒛 𝑡 𝑛 from 𝒛 𝑡 𝑛+𝑘 . The small 𝑘 step leads to slow convergence while very large 𝑘 may incur large approximation errors of the ODE solvers. Therefore, it is critical to choose a proper 𝑘 which achieves fast convergence while preserving the sample quality. Results in Section 4.2 show the effect of different 𝑘 values and reveal that the multi-step ODE solvers are crucial in accelerating the distillation process. Specifically, ẑΨ 𝑡 𝑛 is an estimate of 𝒛 𝑡 𝑛 using numerical multistep ODE solvers:ẑΨ 𝑡 𝑛 -𝒛 𝑡 𝑛+𝑘 = (1+𝜔)Ψ(𝒛 𝑡 𝑛+𝑘 , 𝑡 𝑛+𝑘 , 𝑡 𝑛 , 𝒄)-𝜔Ψ(𝒛 𝑡 𝑛+𝑘 , 𝑡 𝑛+𝑘 , 𝑡 𝑛 , ∅) (13)The modification of the consistency distillation loss in Equation 9 ensures consistency from 𝑡 𝑛+𝑘 to 𝑡 𝑛 .EQUATIONThe training and sampling algorithms of AudioLCM have been outlined in Algorithm 1 and Algorithm 2, respectively.Training. All consistency models are initialized with their corresponding pre-trained diffusion models. To ensure a more robust model fit, we employ the Huber loss L 𝜂 as the distance function in Equation 15:EQUATIONwhere 𝜂 is a hyperparameter used to regulate the sensitivity of the loss function to outliers. 𝜃 -← stopgrad(𝜇𝜃 -+ (1 -𝜇)𝜃 ) 10: until AudioLCM converged Inference. During inference, AudioLCM generates samples by first sampling from the initial distribution ẑ𝑇 ∼ N (0,𝑇 2 𝑰 ) and then evaluating the consistency model for 𝒛 = 𝒇𝜃 ( ẑ𝑇 , 𝜔, 𝒄,𝑇 ). The LCM models can be evaluated with either a single forward pass or multiple passes, alternating denoising and noise injection steps to enhance sample quality. As outlined in Algorithm 2, this multi-step sampling approach offers the flexibility to balance computational resources with sample quality. Ultimately, the generated audio latents 𝒛 are transformed into mel-spectrograms and subsequently used to generate waveforms using a pre-trained vocoder.  
experiments 4.1 experimental setup: 
 Dataset. For text-to-sound generation, we use a diverse combination of datasets to train our teacher model, while the Audio-Caps dataset [19] is specifically utilized for training our AudioLCM model. For text-to-music generation, we exclusively employ the LP-Musicaps dataset for both teacher and AudioLCM training endeavors. More details about dataset can be found in Appendix A. Model configurations. Our teacher model is originally trained with 𝜖-prediction, we utilize its pre-trained VAE, a continuous 1Dconvolution-based network. This VAE is used to compress the melspectrogram into a 20-channel latent representation with a temporal axis downsampling rate of 2. Training AudioLCM involves 15,000 iterations on an NVIDIA 4090Ti GPU, with a batch size of 8 per GPU. We use the AdamW optimizer with a learning rate of 9.6e-5 and an exponential moving average (EMA) rate of 𝜇 = 0.95. For the ODE solver, we use the DDIM solver with a multi-step parameter 𝑘 = 20. The control scale range is defined as [𝜔 𝑚𝑖𝑛 , 𝜔 𝑚𝑎𝑥 ] = [4, 12] . Our choice for the vocoder is BigVGAN [22] , which is known for its universal applicability to different scenarios. We train the vocoder on the AudioSet dataset to ensure robust performance. More details on the model configuration can be found in Appendix B.Evaluation Metrics. Our models conduct a comprehensive evaluation [5] using both objective and subjective metrics to measure audio quality, text-audio alignment fidelity, and inference speed. Objective assessment includes Kullback-Leibler (KL) divergence, Frechet audio distance (FAD), and CLAP score to quantify audio quality. The Real-time Factor (RTF) is also introduced to measure the system's efficiency in generating audio for real-time applications. RTF is the ratio between the total time taken by the audio system to synthesize an audio sample and the duration of the audio. In terms of subjective evaluation, we conduct crowd-sourced human assessments employing the Mean Opinion Score (MOS) to evaluate both audio quality (MOS-Q) and text-audio alignment faithfulness (MOS-F). Detailed information regarding the evaluation procedure can be accessed in Appendix C.2.We conduct a comparative analysis of the ODE Solver with varying estimate step 𝑘 and explore the impact of the guidance scale 𝜔 in LCM. To assess the Multi-step ODE schedule, we evaluate the convergence speed and identify the optimal performance among different 𝑘 values using training iterations and Frechet audio distance (FAD). In analyzing classifier-free guidance, we examine the sample quality across different inference steps using the guidance scale and FAD.Multi-step ODE solver. The results are shown in Figure 2(a) . Several observations emerge from these results: 1) Compared to the one-step ODE solver (k=1), the multi-step ODE solver exhibits significantly faster convergence, underscoring the effectiveness of using multiple steps in accelerating convergence speed. 2) Increasing the k value from 1 to 20 results in faster convergence and improved performance. However, using a larger step size, such as k = 50, may lead to suboptimal results. Therefore, we choose k = 20 to strike a balance between sample quality and convergence speed.Classifier-free Guidance. The results are summarized and presented in Figure 2(b) . Here are our main observations: 1) As expected, larger inference steps show superior performance compared to smaller steps of the same size, highlighting the effectiveness of multi-step sampling in improving sample quality. 2) The performance of the Classifier-Free Guidance (CFG) model peaks at a certain value. Specifically, increasing the scale from 1 to 5 significantly enhances audio quality. However, increasing the scale further to 9 results in a gradual decline in performance. Therefore, we opt for a scale of 5 as a guideline. 3) The performance disparities among inference steps of 2, 4, and 8 are minimal, indicating the effectiveness of LCM in the 2-8 step range. However, a significant performance gap is evident in one-step inference, suggesting areas for potential improvement.We conduct a comparative analysis of the quality of generated audio samples and inference latency across various systems, including GT (i.e., ground-truth audio), AudioGen, Make-An-Audio, AudioLDM-L, TANGO, Make-An-Audio 2, ConsistencyTTA, and AudioLDM 2, utilizing the published models as per the respective paper and the same inference steps of 100 for a fair comparison. The evaluations are conducted using the AudioCaps test set and then calculate the objective and subjective metrics. The results are compiled and presented in Table 1 . From these findings, we draw the following conclusion:Audio Quality In terms of audio quality, our proposed system, AudioLCM, demonstrates outstanding performance, particularly when configured with 4 inference steps. With a Fréchet Audio Distance (FAD) of 1.56 and Kullback-Leibler Divergence (KL) of 1.30, requiring only 2 iterations to synthesize high-fidelity audio samples, which translates to a remarkable speed of 333x times faster than real-time on a single NVIDIA 4090Ti GPU. This significant reduction in inference time positions AudioLCM as a leading solution, substantially outperforming dominant diffusion-based models.Model NFE Objective Metrics Subjective Metrics FAD (↓) KL (↓) CLAP (↑) RTF (↓) MOS-Q(↑) MOS-F(↑) GT / / / 0In order to deepen our understanding of the correlation between inference latency and audio quality, we visualize the performance metrics, particularly focusing on FAD, concerning Make-An-Audio 2, TANGO, and AudioLDM 2. Our analysis, depicted in Figure 3 , reveals several key insights: 1) Diffusion-based models exhibit considerable performance degradation as inference time is shortened (i.e., the number of sampling steps is halved). In contrast, AudioLCM demonstrates minimal degradation until the RTF drops to 0.0015 (i.e., one-step inference), highlighting its robustness in achieving high-quality audio generation even with few inference steps. 2) Among the compared models, Make-An-Audio 2 demonstrates superior performance under low latency conditions, underscoring its suitability as a teacher model for knowledge distillation purposes. The distilled student models can inherit the powerful capabilities of Make-An-Audio 2, thereby enhancing overall model performance. 3) After approximately 10 steps, the trade-off between sample quality and inference speed inherent in consistency models becomes less apparent. This phenomenon arises due to the accumulation of discrete errors throughout the multi-step sampling process, leading to deviations of the generated audios from the target distribution.We defer the resolution of this challenge to future investigations and refinements.Zero-shot Evaluation To further investigate the generalization performance of the models, we additionally test the performance of the models on the Clotho-evaluation dataset in the zero-shot setting. As illustrated in Table 2 , AudioLCM with 4 sampling steps has significantly better results than AudioLDM 2-Large and Make-An-Audio 2 in IS and FAD, attributing to the scalability in terms of data usage. Furthermore, our 2 steps model continues to demonstrate competitive performance with baselines. In this section, we conduct a comparative analysis of the audio samples generated by AudioLCM against a range of established Music Generation systems. These include: 1) GT, the ground-truth audio; 2) MusicGen [4] ; 3) MusicLM [1] ; 4) Mousai [37] ; 5) Riffusion [6] ; 6) MusicLDM [3] ; 7) AudioLDM 2 [25] . The results are presented in Table 3, and we have the following observations: 1) In terms of audio quality, our teacher model consistently outperforms all diffusionbased methods and language models across a spectrum of both objective and subjective metrics. Furthermore, AudioLCM demonstrates competitive performance even when compared to diffusion models that require a significantly greater number of iterations, while also surpassing auto-regressive models. This underscores AudioLCM's effectiveness in producing high-quality music samples, establishing it as a highly capable model in the realm of audio synthesis. 2) In terms of sampling speed, AudioLCM stands out for its exceptional efficiency. It requires a mere 2 iterations to produce high-fidelity music samples, illustrating its potent capability to strike an optimal balance between the quality of the samples and the time required for inference. This efficiency is not just incremental but rather monumental when compared to the inference times of its contemporaries. 
related works 5.1 text-to-audio generation: 
 Text-to-Audio Generation is an emerging task that has witnessed notable advancements in recent years. For instance, Diffsound [49] leverages a pre-trained VQ-VAE [46] on mel-spectrograms to encode audio into discrete codes, subsequently utilized by a diffusion model for audio synthesis. AudioGen [20] frames text-to-audio generation as a conditional language modeling task, while Make-An-Audio [13] , AudioLDM 2 [23] , and TANGO [8] are all founded on the Latent Diffusion Model (LDM), which significantly enhances sample quality. However, a notable drawback of diffusion models lies in their iterative sampling process, leading to slow inference and restricting real-world applications. Unlike the diffusion model mentioned above, AudioLCM introduces a novel consistency-based models to support few-step and even one-step inference without sacrificing sample quality.Consistency Models (CMs) [40] have emerged as a promising approach for efficient sampling without compromising quality. They leverage consistency mapping to enable fast one-step generation, as demonstrated in various tasks such as image generation [28] , video generation [48] , and audio synthesis [2, 51] . For example, Luo et al. [28] propose the latent consistency models demonstrates the potential of CMs to generate higher-resolution images. And Bai et al. [2] apply the consistency model into text-to-sound generation. However, their potential in high-quality text-to-audio generation has not been fully explored. In this study, we incorporate CMs into the text-to-sound and text-to-music generation tasks to uncover their capabilities. 
conclusion: 
 Latent Diffusion Models (LDMs) have made significant strides in generative tasks but are hindered by slow generation due to their iterative sampling process. In this work, we introduced AudioLCM, a latent consistency model tailored for efficient and high-quality text-to-audio generation. Unlike prior approaches that addressed noise removal through iterative processes, AudioLCM integrated Consistency Models into the generation process, facilitating rapid inference through a mapping from any point at any time step to the trajectory's initial point. To overcome the convergence issue inherent in LDMs with reduced sample iterations, we proposed the Guided Latent Consistency Distillation with a multi-step ODE, which shortened the time schedule from thousands to dozens of steps while maintaining sample quality. Furthermore, we incorporated the methodologies of LLaMA and Diffusion transformer to enhance the performance and support variable-length generation. This architecture supported stable and efficient training, ensuring robust performance in text-to-audio synthesis. Experimental results on text-to-audio generation and text-to-speech synthesis tasks demonstrated that AudioLCM required only 2 iterations to synthesize high-fidelity audios, while maintaining sample quality competitive with state-of-the-art models using hundreds of steps. AudioLCM enabled a sampling speed of 333x faster than real-time on a single NVIDIA 4090Ti GPU, making generative models practically applicable to text-to-audio generation deployment. Our extensive preliminary analysis showed that each design in AudioLCM was effective. 
 
 Appendix: Audio samples are available at https://AudioLCM.github.io/.Code is Available at https://github.com/Text-to-Audio/AudioLCM 


Decision: Initially reviewers have mixed scores. Some reviewers have the concerns on the novelty of the proposed method, as well as the lack of some strong baseline comparisons. After the author rebuttal, most of the concerns are addressed. All reviewers changed their scores to borderline accept. The AC agrees with the reviewers, and thinks the work has enough contributions, thus recommending accept for this submission.

Reviews: review: paper_summary: paper_summary: This paper presents EchoAudio, a novel consistency-based model designed for efficient and high-quality text-to-audio generation. By integrating Consistency Models (CMs) into the generation process and employing a multi-step Ordinary Differential Equation (ODE) solver, the authors aim to address the computational burden associated with Latent Diffusion Models (LDMs). The proposed method claims to achieve high-quality audio generation with significantly reduced inference steps. Experimental results demonstrate that EchoAudio achieves competitive performance with state-of-the-art models while being substantially faster.
strength_weakness: strength_weakness: 1. Well-Written: The paper is clearly written and easy to follow, with a well-structured presentation of the proposed method and experimental results.

2. The method proposed by the authors is novel, particularly in how it integrates Consistency Models with a multi-step ODE solver to enhance the efficiency of text-to-audio generation.

3. The experimental results are comprehensive, showing that EchoAudio achieves competitive quality in text-to-audio generation with a much higher efficiency, which is a significant contribution to the field.
limitations: limitations: 1. The related work section does not adequately cover existing works on consistency models in the audio and speech generation domains. Specifically, it lacks references to [1] and [2], which are closely related to the proposed method. The authors need to properly cite these works and discuss their relevance to the proposed method.

2. The paper states, "However, their potential in high-quality text-to-audio generation has not been fully explored." This claim is misleading as [2] also applies consistency models to text-to-audio generation. The authors should clarify the main differences and contributions of their method compared to this existing work and provide a thorough comparison in the experimental section.

3. Baseline Comparisons: The proposed method is claimed to be independent of the teacher model architecture. However, the authors have only conducted experiments using a single baseline, Make-An-Audio 2. It would be more convincing if the authors tested their method with multiple baselines to fully validate its effectiveness and generality.

[1] CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model, ACMMM 23
[2] ACCELERATING DIFFUSION-BASED TEXT-TO-AUDIO GENERATION WITH CONSISTENCY DISTILLATION
paper_summary: paper_summary: The author proposes an audio generation model that integrates Consistency Models to decrease the number of NFE steps required for the diffusion model generation process. To tackle the convergence problem, the author further suggests a distillation method with a multi-step Ordinary Differential Equation (ODE) solver. The author also includes the mature architecture of the Llama model to enhance the model's structure. This results in a 2 NFE step student model that enables real-time generation for deployment.
strength_weakness: strength_weakness: 1. The results are indeed impressive. The teacher model, with its improved architecture borrowed from Llama, already exhibits low FAD/KLD. With the distillation training for the Consistency Models, the student model manages to maintain a low FAD/KLD while only requiring 2 iterations during inference. 

2. Audio generation modeling is a trending research area. A low RTF paves the way for the real-time deployment of these generation models.

3. The author conducts an ablation study to demonstrate the design choices of the ODE solver and Classifier-free Guidance, as well as the benefits of a Llama-like model architecture.

4. In addition to the FAD score, the author also provides audio generation samples, demonstrating that the model's generation is acceptable and closely matches the teacher model's generation.
limitations: limitations: 1. The paper has limited novelty. The main concepts, such as Consistency Models, ODE solver, distillation of diffusion models, classifier-free guidance, and the Llama architecture, have already been explored in the image generation area or LLMs. The authors are primarily combining these elements in audio generation models.

2. For a fair comparison of quality and inference speed, the authors could include or discuss each model's size in the main table or abalation study. Is the teacher/student model superior simply because it's larger in scale? Moreover, training efficiency is another metric that the authors could compare, in addition to inference efficiency, especially considering the cost of training the teacher model and the distillation training.

3. Will the code/model be open-sourced if accepted?
paper_summary: paper_summary: The authors introduce EchoAudio, a model for efficient text-to-audio generation. Unlike previous approaches, EchoAudio integrates Consistency Models (CMs) into the generation process, allowing for rapid inference. To address convergence issues, the authors propose Guided Latent Consistency Distillation with a multi-step ODE solver. Additionally, they integrate advanced techniques from LLaMA into transformer-based neural network architectures to support stable and efficient training for text-to-audio synthesis.
strength_weakness: strength_weakness: EchoAudio introduces a unique approach by integrating Consistency Models (CMs) into the text-to-audio generation process. This novel integration allows for rapid inference. The presentation of EchoAudio is clear and well-structured. The authors systematically describe the problem, propose their solution, and validate their approach with detailed experiments.
limitations: limitations: While EchoAudio introduces the novel integration of Consistency Models (CMs) and a multi-step ODE solver, some model aspects build upon existing techniques without substantial innovation.
paper_summary: paper_summary: This paper proposes EchoAudio which is a new model designed for efficient and high-quality text-to-audio generation. The model addresses the computational challenges of diffusion models by integrating consistency models (CMs) into the generation process, which allows for rapid inference.
The paper also integrated some techniques like pre-layer norm, rotary embedding and SwiGLU from LLaMA into the foundational framework of transformers to optimize the performance of transformer-based neural network architectures. 

Experimental results show that EchoAudio needs only 2 iterations to synthesize high-fidelity audios, maintaining sample quality competitive with teacher models using hundreds of steps. EchoAudio enables a sampling speed of 333x faster than real-time on a single NVIDIA 4090Ti GPU.
strength_weakness: strength_weakness: 1. Apply the CM to audio generation tasks.
2. The generation inference speedup up to 333X seems impressive.
limitations: limitations: 1. Insufficient novelty. CM was proposed for image generation. It was later also used in audio generation e.g. arXiv:2309.10740
2. The baselines and teachers looks weak on audiocaps evaluation data. Audiobox got FAD 0.78 much lower than the teacher model in the paper. Additionally audiobox uses 64 NFE.
3. The paper mentioned about the techniques from Llama. But there isn't ablation study to show the effectiveness these techniques to the audio generation tasks.