Tango 2: Aligning Diffusion-based Text-to-Audio Generative Models through Direct Preference Optimization

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 12:32:02

Abstract: Generative multimodal content is increasingly prevalent in much of the content creation arena, as it has the potential to allow artists and media personnel to create pre-production mockups by quickly bringing their ideas to life. The generation of audio from text prompts is an important aspect of such processes in the music and film industry. Many of the recent diffusion-based text-to-audio models focus on training increasingly sophisticated diffusion models on a large set of datasets of prompt-audio pairs. These models do not explicitly focus on the presence of concepts or events and their temporal ordering in the output audio with respect to the input prompt. Our hypothesis is focusing on how these aspects of audio generation could improve audio generation performance in the presence of limited data. As such, in this work, using an existing text-to-audio model Tango, we synthetically create a preference dataset where each prompt has a winner audio output and some loser audio outputs for the diffusion model to learn from. The loser outputs, in theory, have some concepts from the prompt missing or in an incorrect order. We fine-tune the publicly available Tango text-to-audio model using diffusion-DPO (direct preference optimization) loss on our preference dataset and show that it leads to improved audio output over Tango and AudioLDM2, in terms of both automatic- and manual evaluation metrics.

Main paper: 
 introduction: 
 Generative AI is increasingly turning into a mainstay of our daily lives, be it directly through using ChatGPT [24] , GPT-4 [23] in an assistive capacity, or indirectly by consuming AI-generated memes, generated using models like StableDiffusion [27] , DALL-E 3 [1, 22] , on social media platforms. Nonetheless, there is a massive demand for AI-generated content across industries, especially in the multimedia sector. Quick creation of audio-visual content or prototypes would require an effective text-to-audio model along with text-to-image and -video models. Thus, improving the fidelity of such models with respect to the input prompts is paramount.Recently, supervised fine-tuning-based direct preference optimization [26] (DPO) has emerged as a cheaper and more robust alternative to reinforcement learning with human feedback (RLHF) to align LLM responses with human preferences. This idea is subsequently adapted for diffusion models by Wallace et al. [32] to align the denoised outputs to human preferences. In this work, we employ this DPO-diffusion approach to improve the semantic alignment between input prompt and output audio of a text-to-audio model. Particularly, we fine-tune the publicly available text-to-audio latent diffusion model Tango [5] on our synthesized preference dataset with DPO-diffusion loss. This preference dataset contains diverse audio descriptions (prompts) with their respective preferred (winner) and undesirable (loser) audios. The preferred audios are supposed to perfectly reflect their respective textual descriptions, whereas the undesirable audios have some flaws, such as some missing concepts from the prompt or in an incorrect temporal order or high noise level. To this end, we perturbed the descriptions to remove or change the order of certain concepts and passed them to Tango to generate undesirable audios. Another strategy that we adopted for undesirable audio generation was adversarial filtering: generate multiple audios from the original prompt and choose the audio samples with CLAP-score below a certain threshold. We call this preference dataset Audio-alpaca. To mitigate the effect of noisy preference pairs stemming from automatic generation, we further choose a subset of samples for DPO fine-tuning based on certain thresholds defined on the CLAP-score differential between preferred and undesirable audios and the CLAP-score of the undesirable audios. This likely ensures a minimal proximity to the input prompt, while guaranteeing a minimum distance between the preference pairs.We experimentally show that fine-tuning Tango on the pruned Audio-alpaca yields Tango 2 that significantly surpasses Tango and AudioLDM2 in both objective and human evaluations. Moreover, exposure to the contrast between good and bad audio outputs during DPO fine-tuning likely allows Tango 2 to better map the semantics of the input prompt into the audio space, despite relying on the same dataset as Tango for synthetic preference data-creation.The broad contributions of this paper are the following:(1) We develop a cheap and effective heuristics for semi automatically creating a preference dataset for text-to-audio generation;(2) On the same note, we also share the preference dataset Audio-alpaca for text-to-audio generation that may aid in the future development of such models; (3) Despite not sourcing additional out-of-distribution textaudio pairs over Tango, our model Tango 2 outperforms both Tango and AudioLDM2 on both objective and subjective metrics;(4) Tango 2 demonstrates the applicability of diffusion-DPO in audio generation. 
related work: 
 Text-to-audio generation has garnered serious attention lately thanks to models like AudioLDM [17] , Make-an-Audio [9] , Tango [5] , and Audiogen [14] . These models rely on diffusion architectures for audio generation from textual prompts. Recently, AudioLM [2] was proposed which utilizes the state-of-the-art semantic model w2v-Bert [4] to generate semantic tokens from audio prompts. These tokens condition the generation of acoustic tokens, which are decoded using the acoustic model SoundStream [34] to produce audio. The semantic tokens generated by w2v-Bert are crucial for conditioning the generation of acoustic tokens, subsequently decoded by SoundStream. AudioLDM [17] is a text-to-audio framework that employs CLAP [33] , a joint audio-text representation model, and a latent diffusion model (LDM). Specifically, an LDM is trained to generate latent representations of melspectrograms obtained using a Variational Autoencoder (VAE). During diffusion, CLAP embeddings guide the generation process. Tango [6] utilizes the pre-trained VAE from AudioLDM and replaces the CLAP model with a fine-tuned large language model: FLAN-T5. This substitution aims to achieve comparable or superior results while training with a significantly smaller dataset.In the realm of aligning generated audio with human perception, Liao et al. [16] recently introduced BATON, a framework that initially gathers pairs of audio and textual prompts, followed by annotating them based on human preference. This dataset is subsequently employed to train a reward model. The reward generated by this model is then integrated into the standard diffusion loss to guide the network, leveraging feedback from the reward model. However, our approach significantly diverges from this work in two key aspects: 1) we automatically construct a pairwise preference dataset, referred to as Audio-alpaca, utilizing various techniques such as LLM-guided prompt perturbation and re-ranking of generated audio from Tango using CLAP scores, and 2) we then train Tango on Audio-alpaca using diffusion-DPO to generate audio samples preferred by human perception. 
background 3.1 overview of tango: 
 Tango, proposed by Ghosal et al. [5] , primarily relies on a latent diffusion model (LDM) and an instruction-tuned LLM for text-toaudio generation. It has three major components:(1) Textual-prompt encoder (2) Latent diffusion model (LDM) (3) Audio VAE and VocoderThe textual-prompt encoder encodes the input description of the audio. Subsequently, the textual representation is used to construct a latent representation of the audio or audio prior from standard Gaussian noise, using reverse diffusion. Thereafter, the decoder of the mel-spectrogram VAE constructs a mel-spectrogram from the latent audio representation. This mel-spectrogram is fed to a vocoder to generate the final audio.Tango utilizes the pre-trained LLM Flan-T5-Large (780M) [3] as the text encoder (𝐸 𝑡𝑒𝑥𝑡 ) to acquire text encoding 𝜏 ∈ R 𝐿×𝑑 𝑡𝑒𝑥𝑡 , where 𝐿 and 𝑑 𝑡𝑒𝑥𝑡 represent the token count and token-embedding size, respectively.3.1.2 Latent Diffusion Model. For ease of understanding, we briefly introduce the LDM of Tango in this section. The latent diffusion model (LDM) [27] in Tango is derived from the work of Liu et al. [18] , aiming to construct the audio prior 𝑥 0 guided by text encoding 𝜏. This task essentially involves approximating the true prior 𝑞(𝑥 0 |𝜏) using parameterized 𝑝 𝜃 (𝑥 0 |𝜏).LDM achieves this objective through forward and reverse diffusion processes. The forward diffusion represents a Markov chain of Gaussian distributions with scheduled noise parameters 0 < 𝛽 1 < 𝛽 2 < • • • < 𝛽 𝑁 < 1, facilitating the sampling of noisier versions of 𝑥 0 :EQUATIONEQUATIONwhere 𝑁 is the number of forward diffusion steps, 𝛼 𝑛 = 1 -𝛽 𝑛 , and 𝛼 𝑛 = 𝑛 𝑖=1 𝛼 𝑛 . Song et al. [29] show that Eq. ( 2) conveniently follows from Eq. ( 1) through reparametrization trick that allows direct sampling of any 𝑥 𝑛 from 𝑥 0 via a non-Markovian process:EQUATIONwhere the noise term 𝜖 ∼ N (0, I). The final step of the forward process yields 𝑥 𝑁 ∼ N (0, I).The reverse process denoises and reconstructs 𝑥 0 through textguided noise estimation ( ε𝜃 ) using lossEQUATIONwhere 𝑥 𝑛 is sampled according to Eq. (3) using standard normal noise 𝜖 𝑛 , 𝜏 represents the text encoding for guidance, and 𝛾 𝑛 denotes the weight of reverse step 𝑛 [8] , interpreted as a measure of signalto-noise ratio (SNR) relative to 𝛼 1:𝑁 . The estimated noise is then employed for the reconstruction of 𝑥 0 :EQUATIONEQUATIONEQUATIONEQUATIONThe parameterization of noise estimation ε𝜃 involves utilizing U-Net [28] , incorporating a cross-attention component to integrate the textual guidance 𝜏.3.1.3 Audio VAE and Vocoder. The audio variational auto-encoder (VAE) [11] compresses the mel-spectrogram of an audio sample, 𝑚 ∈ R 𝑇 ×𝐹 , into an audio prior 𝑥 0 ∈ R 𝐶 ×𝑇 /𝑟 ×𝐹 /𝑟 , where 𝐶, 𝑇 , 𝐹 , and 𝑟 denote the number of channels, time-slots, frequency-slots, and compression level, respectively. The latent diffusion model (LDM) reconstructs the audio prior x0 using input-text guidance 𝜏. Both the encoder and decoder consist of ResUNet blocks [13] and are trained by maximizing the evidence lower-bound (ELBO) [11] and minimizing adversarial loss [10] . Tango utilizes the checkpoint of the audio VAE provided by Liu et al. [18] . As a vocoder to convert the audio-VAE decoder-generated melspectrogram into audio, Tango employs HiFi-GAN [12] which is also utilized by Liu et al. [18] .Finally, Tango utilizes a data augmentation method that merges two audio signals while considering human auditory perception. This involves computing the pressure level of each audio signal and adjusting the weights of the signals to prevent the dominance of the signal with higher pressure level over the one with lower pressure level. Specifically, when fusing two audio signals, the relative pressure level is computed using the following equation:EQUATIONHere 𝐺 1 and 𝐺 2 are the pressure levels of signal 𝑥 1 and 𝑥 2 . Then the audio signals are mixed using the equation below:EQUATION)The denominator is to account for the fact that the energy of a sound wave is proportional to the square of its amplitude as shown in Tokozume et al. [30] . Note that in this augmentation, textual prompts are also concatenated.Tuning Large Language Models (LLMs) to generate responses according to human preference has been a great interest to the ML community. The most popular approach for aligning language models to human preference is reinforcement learning with human feedback (RLHF). It comprises the following steps [26] :Supervised Fine Tuning (SFT). First, the pre-trained LLM undergoes supervised fine-tuning on high-quality downstream tasks to obtain the fine-tuned model 𝜋 𝑆𝐹𝑇 . Reward Modeling. Next, 𝜋 𝑆𝐹𝑇 is prompted with an input 𝜏 to generate multiple responses. These responses are then shown to human labelers to rank. Once such a rank is obtained, 𝑥 𝑤 ≻ 𝑥 𝑙 | 𝜏 indicating 𝑥 𝑤 is preferred over 𝑥 𝑙 , the task is to model these preferences. Among several popular choices of preference modeling, Bradley-Terry (BT) is the most popular one which relies on the equation below:EQUATIONThe overall idea is to learn the human preference distribution 𝑝 * . 𝑟 * (𝜏, 𝑥) is a latent reward function that generates the preferences. With a static dataset created by human annotators, D =𝜏 (𝑖 ) , 𝑥 𝑤 (𝑖 ) , 𝑥 𝑙 (𝑖 ) 𝑁 𝑖=1, one can train a reward model 𝑟 𝜙 (𝜏, 𝑥) using maximum likelihood estimation. The negative log-likelihood loss of this training can be written as follows:L 𝑅 (𝑟 𝜙 , D) = -E (𝜏,𝑥 𝑤 ,𝑥 𝑙 )∼D log 𝜎 (𝑟 𝜙 (𝜏, 𝑥 𝑤 ) -𝑟 𝜙 (𝜏, 𝑥 𝑙 )) (12)This formulation considers framing the problem as a binary classification problem.RL Optimization. The final step is to leverage 𝑟 𝜙 (𝜏, 𝑥) to feedback the language model. As explained by Rafailov et al. [26] , this can be embedded into the following learning objective: maxEQUATIONHere, 𝜋 ref represents the reference model, which in this context is the supervised fine-tuned model denoted as 𝜋 SFT . 𝜋 𝜃 stands for the policy language model, intended for enhancement based on feedback from 𝑟 𝜙 (𝜏, 𝑥). 𝛽 governs 𝜋 𝜃 to prevent significant divergence from 𝜋 ref . This control is crucial as it ensures that the model stays close to the distributions upon which 𝑟 𝜙 (𝜏, 𝑥) was trained. Since the outputs from LLM are discrete, Eq. ( 13) becomes non-differentiable, necessitating reinforcement learning methods like PPO to address this objective. 
methodology: 
 The two major parts of our approach (i) creation of preference dataset Audio-alpaca and (ii) DPO for alignment are outlined in Fig. 1 .4.1.1 Audio Generation from Text Prompts. Our first step is to create audio samples from various text prompts with the pre-trained Tango model. We follow three different strategies as follows:Strategy 1: Multiple Inferences from the same Prompt. In the first setting, we start by selecting a subset of diverse captions from the training split of the AudioCaps dataset. We use the sentence embedding model gte-large1 [15] to compute dense embedding vectors of all the captions in the training set. We then perform K-Means clustering on the embedded vectors with 200 clusters. Finally, we select 70 samples from each cluster to obtain a total of 14,000 captions. We denote the selected caption set as T 1 .The captions selected through the above process constitute the seed caption set. Now, we follow two settings to generate audio samples from these captions:(1) Strategy 1.1: Prompt Tango-full-FT with the caption to generate four different audio samples with 5, 25, 50, and 100 denoising steps. All samples are created with a guidance scale of 3. (2) Strategy 1.2: Prompt Tango-full-FT with the caption to generate four different audio samples each with 50 denoising steps. All samples are created with a guidance scale of 3. In summary, we obtain (𝜏, 𝑥 1 , 𝑥 2 , 𝑥 3 , 𝑥 4 ) from Strategy 1, where 𝜏 denotes the caption from T 1 and 𝑥 𝑖 denotes the audios generated from 𝜏.Strategy 2: Inferences from Perturbed Prompts. We start from the selected set T 1 and make perturbations of the captions using the GPT-4 language model [23] . For a caption 𝜏 from T 1 , we denote 𝜏 1 as the perturbed caption generated from GPT-4. We add specific instructions in our input prompt to make sure that 𝜏 1 is semantically or conceptually close to 𝜏. We show an illustration of the process in Table 1 . In practice, we create five different perturbed 𝜏 1 for each 𝜏 from GPT-4, as shown in Table 1 .We then prompt Tango-full-FT with 𝜏 and 𝜏 1 to generate audio samples 𝑥 𝜏 and 𝑥 𝜏 1 . We use 50 denoising steps with a guidance scale of 3 to generate these audio samples.To summarize, we obtain (𝜏, 𝑥 𝜏 , 𝑥 𝜏 1 ) from Strategy 2. Note that, we considered 𝜏 1 only to generate the audio sample 𝑥 𝜏 1 . We do not further consider 𝜏 1 while creating the preference dataset.Strategy 3: Inferences from Temporally Perturbed Prompts. This strategy is aimed at prompts that describe some composition of sequence and simultaneity of events. To identify such prompts in AudioCaps' training dataset, as a heuristics, we look for the following keywords in a prompt: before, after, then, or followed. We denote the set of such prompts as T 2 .For each caption 𝜏 2 in T 2 , we then prompt GPT-4 to create a set of temporal perturbations. The temporal perturbations include changing the order of the events in the original caption, or introducing a new event or removing an existing event, etc. We aim to create these temporal perturbations by providing specific instructions to GPT-4, which we also illustrate in Table 1 .We denote the temporally perturbed caption as 𝜏 2 . We then follow the same process as mentioned earlier in Strategy 2 to create the audio samples 𝑥 𝜏 and 𝑥 𝜏 2 . Finally, we pair the (𝜏, 𝑥 𝜏 , 𝑥 𝜏 2 ) samples from this strategy. Analogous to the previous strategy, the 𝜏 2 is only used to create the 𝑥 𝜏 2 , and is not used anywhere else for preference data creation.We collect the paired text prompt and audio samples from the three strategies and denote it overall as (𝜏, ⟨𝑥⟩), where ⟨𝑥⟩ indicates the set of 4 or 2 generated audio samples depending upon the corresponding strategy. 1. Crowd applauding and bicycles speeding past.People cheering and race cars racing by. i) Would be closely related to the audio clip and the original caption.2. Spectators clapping and trains rushing by. ii) Would contain inaccuracies in terms of describing the audio clip i.e. they would be somewhat wrong captions for the audio clip.3. Audience cheering and horses galloping past.iii) However, the new captions should not be completely unrelated. Always keep some concepts from the original caption in the new one.4. Fans shouting and airplanes flying by. iv) Would be of similar length to the original caption.5. Group celebrating and motorcycles revving past. Generate only the captions in separate lines so that I can programmatically extract them later.I have an audio clip for which the original caption is as follows: A man is speaking then a sewing machine briefly turns on and off. Can you generate five candidate captions that would satisfy the following requirements:1. A sewing machine briefly starts, then a man begins speaking.A man is speaking then a sewing machine briefly turns on and off.i) Would be closely related to the audio clip and the original caption.2. The sound of a sewing machine is heard after a man's voice.ii) Would have a change of order of the events described in the original caption.3. A man's voice is followed by the noise of a blender.iii) Would contain inaccuracies in terms of describing the audio clip i.e. they would be somewhat wrong captions for the audio clip.4. A woman speaks and then a sewing machine is turned on. iv) However, the new captions should not be completely unrelated. Always keep some concepts from the original caption in the new one.5. The noise of a sewing machine is interrupted by a man talking. v) Would be of similar length to the original caption. Generate only the captions in separate lines so that I can programmatically extract them later.For Strategy 1. Let's assume we have an instance (𝜏, ⟨𝑥⟩) from Strategy 1. We first compute the CLAP matching score following Wu et al. [33] between 𝜏 and all the four audio samples in ⟨𝑥⟩. We surmise that the sample in ⟨𝑥⟩ that has the highest matching score with 𝜏 is most aligned with 𝜏, compared to the other three audio samples that have a relatively lower matching score. We consider this audio with the highest matching score as the winning sample 𝑥 𝑤 and the other three audio samples as the losing sample 𝑥 𝑙 . In this setting, we can thus create a pool of three preference data points: (𝜏, 𝑥 𝑤 , 𝑥 𝑙 ), for the three losing audio samples 𝑥 𝑙 .For Strategy 2 and 3. Let's assume we have an instance (𝜏, ⟨𝑥⟩) from Strategy 2 or 3. We compute the CLAP matching score between i) 𝜏 with 𝑥 𝜏 , and ii) 𝜏 with the 𝑥 𝜏 1 or 𝑥 𝜏 2 , corresponding to the strategy. We consider only those instances where the CLAP score of i) is higher than the CLAP score of ii). For these instances, we use 𝑥 𝜏 as the winning audio 𝑥 𝑤 and 𝑥 𝜏 1 or 𝑥 𝜏 2 as the losing audio 𝑥 𝑙 to create the preference data point: (𝜏, 𝑥 𝑤 , 𝑥 𝑙 ).Final Selection. We want to ensure that the winning audio sample 𝑥 𝑤 is strongly aligned with the text prompt 𝜏. At the same time, the winning audio sample should have a considerably higher alignment with the text prompt than the losing audio sample. We use the CLAP score as a measurement to fulfill these conditions. The CLAP score is measured using cosine similarity between the text and audio embeddings, where higher scores indicate higher alignment between the text and the audio. We thus use the following conditions to select a subset of instances from the pool of preference data:(1) The winning audio must have a minimum CLAP score of 𝛼 with the text prompt to ensure that the winning audio is strongly aligned with the text prompt. (2) The losing audio must have a minimum CLAP score of 𝛽 with the text prompt to ensure that we have semantically close negatives that are useful for preference modeling.(3) The winning audio must have a higher CLAP score than the losing audio w.r.t to the text prompt. (4) We denote Δ to be the difference in CLAP score between the text prompt with the winning audio 2 and the text prompt with the losing audio. The Δ should lie between certain thresholds, where the lower bound will ensure that the losing audio isn't too close to the winning audio, and the upper bound will ensure that the losing audio is not too undesirable.We use an ensemble filtering strategy based on two different CLAP models: 630k-audioset-best and 630k-best [33] . This can reduce the effect of noise from individual CLAP checkpoints and increase the robustness of the selection process. In this strategy, samples are included in our preference dataset if and only if they satisfy all the above conditions based on CLAP scores from both of the models. We denote the conditional scores mentioned above as 𝛼 1 , 𝛽 1 , Δ 1 , and 𝛼 2 , 𝛽 2 , Δ 2 for the two CLAP models, respectively. Based on our analysis of the distribution of the CLAP scores as shown in Figure 2 , we choose their values as follows: 𝛼 1 = 0.45, 𝛼 2 = 0.60, 𝛽 1 = 0.40, 𝛽 2 = 0.0, 0.05 ≤ Δ 1 ≤ 0.35, and 0.08 ≤ Δ 2 ≤ 0.70.Finally, our preference dataset Audio-alpaca has a total of ≈ 15k samples after this selection process. We report the distribution of Audio-alpaca in Table 2 .As opposed to RLHF, recently DPO has emerged as a more robust and often more practical and straightforward alternative for LLM alignment that is based on the very same BT preference model (Eq. ( 11)). In contrast with supervised fine-tuning (SFT) that only optimizes for the desirable (winner) outputs, the DPO objective also allows the model to learn from undesirable (loser) outputs, which is key in the absence of a high-quality reward model, as required for RLHF. To this end, the DPO objective is derived by substituting We see that for an unfiltered dataset: i) the winner audio sample is not always strongly aligned to the text prompt in the 𝛼 1 plot; ii) winner and loser audio samples can be too close in the Δ 1 plot. We thus choose the values of our 𝛼 1 , Δ 1 and other selection parameters to ensure the filtered dataset is less noisy with more separation between the winner and loser audios.the globally optimal reward-obtained by solving Eq. ( 13)-in the negative log-likelihood (NLL) loss (Eq. ( 12)). This success spurred on Wallace et al. [32] to bring the same benefits of DPO to diffusion networks. However, unlike DPO, the goal for diffusion networks is to maximize the following learning objective (Eq. ( 14)) with a reward (Eq. ( 15)) defined on the entire diffusion path 𝑥 0:𝑁 : maxEQUATION)EQUATIONSolving this objective and substituting the optimal reward in the NLL loss (Eq. ( 12)) yields the following DPO objective for diffusion:EQUATION)Now, applying Jensen's inequality by taking advantage of the convexity oflog 𝜎 allows the inner expectation to be pushed outside. Subsequently, approximating the denoising process with the forward process yields the following final form in terms of the L2 noise-estimation losses from LDM (Eq. ( 4)):EQUATIONwhere D pref := {(𝜏, 𝑥 𝑤 0 , 𝑥 𝑙 0 )} is our preference dataset Audio-alpaca, 𝜏, 𝑥 𝑤 0 , and 𝑥 𝑙 0 being the input prompt, preferred, and undesirable output, respectively. Furthermore, 𝑛 ∼ U (0, 𝑁 ) is the diffusion step, 𝜖 * 𝑛 ∼ N (0, I) and 𝑥 * 𝑛 are noise and noisy posteriors, respectively, at some step 𝑛. 𝜆 𝑛 is the signal-to-noise ratio (SNR) and 𝜔 (𝜆 𝑛 ) is a weighting function defined on SNR. We use Tango-full-FT as our reference model through its noise estimation εref . 
experiments 5.1 datasets and training details: 
 We fine-tuned our model starting from the Tango-full-FT checkpoint on our preference dataset Audio-alpaca. As mentioned earlier in Section 4.1.2, we have a total of 15,025 preference pairs in Audio-alpaca, which we use for fine-tuning. We use AdamW [20] with a learning rate of 9.6e-7 and a linear learning-rate scheduler for fine-tuning. Following Wallace et al. [32] , we set the 𝛽 in DPO loss (Eq. ( 17)) to 2000. We performed 1 epoch of supervised finetuning on the prompt and the preferred audio as training samples, followed by 4 epochs of DPO. The entirety of the fine-tuning was executed on two A100 GPUs which takes about 3.5 hours in total. We use a per GPU batch size of 4 and a gradient accumulation step of 4, resulting in an effective batch size of 32.We primarily compare Tango 2 to three strong baselines:(1) AudioLDM [17] : A text-to-audio model that uses CLAP [33] , a joint audio-text representation model, and a latent diffusion model (LDM). Specifically, the LDM is trained to generate the (2) AudioLDM2 [19] : An any-to-audio framework which uses language of audio (LOA) as a joint encoding of audio, text, image, video, and other modalities. Audio modality is encoded into LOA using a self-supervised masked auto-encoder. The remaining modalities, including audio again, are mapped to LOA through a composition of GPT-2 [25] and ImageBind [7] . This joint encoding is used as a conditioning in the diffusion network for audio generation. (3) Tango [5] : Utilizes the pre-trained VAE from AudioLDM but replaces the CLAP text-encoder with an instruction-tuned large language model: FLAN-T5. As compared to AudioLDM, its data-augmentation strategy is also cognizant of the audio pressure levels of the source audios. These innovations attain comparable or superior results while training on a significantly smaller dataset. Baton [16] represents another recent approach in human preference based text-to-audio modeling. It trains a reward model to maximize rewards through supervised fine-tuning, aiming to maximize the probability of generating audio from a textual prompt. As discussed in Section 2, Baton's reward model is not trained using the pairwise preference objective presented in Equation (12) . In this approach, each text (𝜏) and audio (𝑥) pair is classified as 1 or 0, indicating whether human annotators favored the text-audio pair or not. Subsequently, this reward is incorporated into the generative objective function of the diffusion. This methodology stands in contrast to the prevailing approach in LLM alignment research. As of now, neither the dataset nor the code has been made available for comparison.Objective Metrics. We evaluated the text-to-audio generation using the standard Frechet Audio Distance (FAD), KL divergence, Inception score (IS), and CLAP score [17] . FAD is adapted from Frechet Inception Distance (FID) and measures the distribution-level gap between generated and reference audio samples. KL divergence is an instance-level reference-dependent metric that measures the divergence between the acoustic event posteriors of the ground truth and the generated audio sample. FAD and KL are computed using PANN, an audio-event tagger. IS evaluates the specificity and coverage of a set of samples, not needing reference audios. IS is inversely proportional to the entropy of the instance posteriors and directly proportional to the entropy of the marginal posteriors. CLAP score is defined as the cosine similarity between the CLAP encodings of an audio and its textual description. We borrowed the AudioLDM evaluation toolkit [17] for the computation of FAD, IS, and KL scores.Subjective Metrics. Our subjective assessment examines two key aspects of the generated audio: overall audio quality (OVL) and relevance to the text input (REL), mirroring the approach outlined in the previous works, such as, [5, 31] . The OVL metric primarily gauges the general sound quality, clarity, and naturalness irrespective of its alignment with the input prompt. Conversely, the REL metric assesses how well the generated audio corresponds to the given text input. Annotators were tasked with rating each audio sample on a scale from 1 (least) to 5 (highest) for both OVL and REL. This evaluation was conducted on a subset of 50 randomlyselected prompts from the AudioCaps test set, with each sample being independently reviewed by at least four annotators. Please refer to the supplementary material for more details on the evaluation instructions and evaluators.We report the comparative evaluations of Tango 2 against the baselines Tango [5] and AudioLDM2 [19] in Table 3 . For a fair comparison, we used exactly 200 inference steps in all our experiments. Tango and Tango 2 were evaluated with a classifier-free guidance scale of 3 while AudioLDM2 uses a default guidance scale of 3.5. We generate only one sample per text prompt.Objective Evaluations. Tango 2 achieves notable improvements in objective metrics, with scores of 2.69 for FAD, 1.12 for KL, 9.09 for IS, and 0.57 for CLAP. While FAD, KL, and IS assess general naturalness, diversity, and audio quality, CLAP measures the semantic alignment between the input prompt and the generated audio. As documented in Melechovsky et al. [21] , enhancing audio quality typically relies on improving the pre-training process of the backbone, either through architectural modifications or by leveraging larger or refined datasets. However, in our experiments, we observe enhanced audio quality in two out of three metrics, specifically KL and IS. Notably, Tango 2 also significantly outperforms various versions of AudioLDM and AudioLDM2 on these two metrics.On the other hand, we note a substantial enhancement in the CLAP score. CLAP score is particularly crucial in our experimental setup as it directly measures the semantic alignment between the textual prompt and the generated audio. This outcome suggests that DPO-based fine-tuning on the preference data from Audio-alpaca yields superior audio generation to Tango and AudioLDM2.Subjective Evaluations. In our subjective evaluation, Tango 2 achieves high ratings of 3.99 in OVL (overall quality) and 4.07 in REL (relevance), surpassing both Tango and AudioLDM2. This suggests that Tango 2 significantly benefits from preference modeling on Audio-alpaca. Interestingly, our subjective findings diverge from those reported by Melechovsky et al. [21] . In their study, the authors observed lower audio quality when Tango was fine-tuned on music data. However, in our experiments, the objective of preference modeling enhances both overall sound quality and the relevance of generated audio to the input prompts. Notably, in our experiments, AudioLDM2 performed the worst, with the scores of only 3.56 in OVL and 3.19 in REL, significantly lower than both Tango and Tango 2.Results on Temporal and Multi-concept Prompts. We analyze the performance of Tango and AudioLDM2 models in audio generation when text prompts contain multiple sequential events. For instance, "Two gunshots followed by birds flying away then a boy laughing" consists of three distinct events in a sequence. In contrast, "A man is snoring" lacks any temporal sequence. We Table 3 : Text-to-audio generation results on AudioCaps evaluation set. Due to time and budget constraints, we could only subjectively evaluate AudioLDM 2-Full-Large and Tango-full-FT. Notably these two models are considered open-sourced SOTA models for text-to-audio generation as reported in [31] . partition the AudioCaps test set based on temporal identifiers such as "while, " "as, " "before, " "after, " "then, " and "followed, " creating two subsets: one with multiple sequential events and the other with no temporality. Our objective evaluation results for these subsets are shown in Table 5 . Tango 2 achieves the highest scores of KL, CLAP, and IS on both temporal and non-temporal events, indicating a consistent trend of performance improvement. Similar trends are observed in subjective evaluations. We attribute this improvement to the augmentation of both temporal and non-temporal inputs in constructing Audio-alpaca. Additionally, we partition the prompts based on the presence of multiple concepts, such as "A woman speaks while cooking". In general, Tango 2 outperforms AudioLDM2 and Tango across most objective and subjective metrics.EQUATION 
conclusion: 
 In this work, we propose aligning text-to-audio generative models through direct preference optimization. To the best of our knowledge, this represents the first attempt to advance text-to-audio generation through preference optimization. We achieve this by automatically generating a preference dataset using a combination of Large Language Models (LLMs) and adversarial filtering. Our preference dataset, Audio-alpaca, comprises diverse audio descriptions (prompts) paired with their respective preferred (winner) and undesirable (loser) audios. The preferred audios are expected to accurately reflect their corresponding textual descriptions, while the undesirable audios exhibit flaws such as missing concepts, incorrect temporal order, or high noise levels. To generate undesirable audios, we perturb the descriptions by removing or rearranging certain concepts and feed them to Tango. Additionally, we employ adversarial filtering, generating multiple audios from the original prompt and selecting those with CLAP scores below a specified threshold. Subsequently, we align a diffusion-based text-to-audio model, Tango, on Audio-alpaca using DPO-diffusion loss. Our results demonstrate significant performance leap over the previous models, both in terms of objective and subjective metrics. We anticipate that our dataset, Audio-alpaca, and the proposed model, Tango 2, will pave the way for further advancements in alignment techniques for text-to-audio generation.  
 
 Appendix: hf.co/thenlper/gte-largeIn our paper, we employ the terms "winner" and "preferred" interchangeably. Likewise, we use "loser" and "undesirable" interchangeably throughout the text. 


Decision: The quality of the paper is good. It is well-written with some original contributions, such as utilizing diffusion-DPO for audio generation, and a carefully curated audio preference dataset. The authors have done a good job to address reviewers' critiques during the rebuttal period. Hope the authors can take the reviewers' comments into consideration in the camera-ready version, if this paper is accepted.

Reviews: review: paper_summary: paper_summary: This paper proposes TANGO 2, a fine-tuned version of text-to-audio model TANGO using diffusion-DPO.
Based on a carefully curated caption dataset, the author proposes three different strategies to build caption-audio pairs with different perturbation methods.
strength_weakness: strength_weakness: - The paper is very well written and easy to follow
- The paper presents the first work on utilizing diffusion-DPO for audio generation.
- The paper presents a carefully curated audio preference dataset.
- The experiment has shown that the proposed method can greatly benefit the audio quality and text relevance.
limitations: limitations: - The preference dataset is curated based on the CLAP score. Instead of using the proposed method, I’m curious about the model's performance if the author directly used TANGO to generate multiple samples and filter the best result using the CLAP score.
- The most vital limitation of the proposed method is the heavy reliance on the CLAP score during the data curation process. Still, considering the use of the ensembling filtering strategy and strict threshold set, it’s generally acceptable.
- The presentation could benefit from including case studies or a web page for demo purposes.
paper_summary: paper_summary: This paper construct a text-audio preference alignment dataset and uses DPO to finetune TANGO model to improve the temporal and content-aspect model performance. The paper demonstrate a good pipeline about how to create such a dataset and share both the subjective and objective evaluation results, where TANGO 2 performs better than previous SOTA models.
strength_weakness: strength_weakness: - The introduction of a novel preference dataset, Audio-alpaca, specifically designed for text-to-audio generation.

- This paper also shows that using DPO to improve the performance of pretrained text-to-audio model works well.
limitations: limitations: Pipeline:
- The pipeline strategy 3 mentioned using temporally perturbing to create prompt pairs, but CLAP more focuses on the semantic similarity; from the original paper, there is no evidence that CLAP embedding contains temporal information, which makes the pipeline and ablation in Section 5.5 not convincing. In this case, the paper should consider using some improved version, such as T-CLAP, to update the pipeline.

Evaluation

Actually the evaluation section is my main concern, since reinforcement learning and DPO finetuning is sensitive to the entire pipeline and the evaluation results should show a reasonable improvement to make the method convincing.

- This paper uses CLAP to pick audio samples, therefore the finetuned TANGO model is affected by CLAP's inductive blas. In this case, evaluating the model with CLAP score is inappropriate. We need to realise that CLAP score is more like a biased model instead of oracle, the further evaluation should be conducted.

- In the evaluation section, the author also points out that TANGO 2 "significantly" performs better than TANGO and AudioLDM 2, but the significance test is not performed at the evaluation stage. Is the improvement really significant? I encourage the paper can add a t-test to show the significant level.

- The subjective evaluation can be a good addition to the objective evaluation, especially when the main metric FAD score increases after DPO finetuning. I believe a significance test is still necessary, and from the original paper, OVL and REL score is not a 5-point metric, it is better to re-scale it to 0-100 scale.

Related work

The paper can consider add the literature below:

[1] Cideron, G., Girgin, S., Verzetti, M., Vincent, D., Kastelic, M., Borsos, Z., ... & Agostinelli, A. (2024). MusicRL: Aligning Music Generation to Human Preferences. arXiv preprint arXiv:2402.04229.
paper_summary: paper_summary: This paper focuses on text to audio generation with an emphasis on enhancing multiple concepts and their temporal ordering accuracy by fine-tuning an existing model using Direct Preference Modelling (DPO). It introduces a method for curating a synthetic preference dataset using the Tango model, a SOTA text-to-audio generator.  The dataset consists of pairs of preferred and undesirable audio. The data curation strategies involve multiple inference and re-ranking, as well as generating audio from perturbed prompts compared to the original prompts. After that, Tango is fine-tuned on the preference dataset using DPO for diffusion models proposed by Wallace et al. Experiments demonstrate that the fine-tuned model achieves higher text-to-audio generation quality in varied aspects, including scenarios with multiple concepts and ordering constraints.
strength_weakness: strength_weakness: Based on the experimental results, the proposed Tango2 model achieves competitive text-to-audio performance with existing SOTAs.
limitations: limitations: This paper is overall well proposed but the reviewer has some questions/concerns as follows:

* Line 801 states “AudioLDM2 … significantly lower than both Tango and Tango2.” Although there is indeed a good margin in this case, the statistical significance is not overall tested in the experiments. The author should be more cautious when using such a term.

* It is not clear to the reviewer if the metrics applied in this paper are discriminative enough to the temporal ordering of sound objects. In Table 2 of Appendix, KL and CLAP barely show any difference with or without Strategy 3 (which introduces temporal perturbation in the text prompts). Is it due to inefficiency of Strategy 3, or insensitivity of these metrics to the temporal ordering? Since enhancing temporal ordering accuracy is a key proposal in this paper, the reviewer deems it necessary to investigate this aspect more thoroughly.

* Based on the proposed synthetic dataset and curation methods, are there any more general insights into the fields such as multimedia and generative modelling? To the reviewer’s understanding, the main technical method regarding DPO is completely adopted from Wallace et al. The technical contribution of this paper is seemingly limited.

A few more suggestions for writing: 

* Section 3.1 introduces Tango in great detail, which is, however, potentially distracting (in particular, Eq. (9)-(10) seems not relevant at all to this paper). The reviewer recommends delete Section 3.1 or put it to the Appendix. If DPO is a universal approach, then it need not rely on any specific base model.

* The claims in Line 94 and 101 are written in an uncertain tone (using words such as “likely”). It is not encouraged. If it is not an assumption, then it should be validated by experiment.

* Eq. (12), (15), and (16) use undefined variables $\sigma$, $R$ and $x^*$.