ETTA: Elucidating the Design Space of Text-to-Audio Models
Authors: Sang-gil Lee
Zhifeng Kong
Arushi Goel
Sungwon Kim
Rafael Valle
Bryan Catanzaro
Venue: ICML 2025 poster (2025)
Keywords: audio generation
text-to-audio
synthetic data
diffusion
flow matching
Primary area: applications->everything_else
TL;DR: We elucidate the design space of text-to-audio and present ETTA with state-of-the-art result and improved abilities to generate creative audio.

Abstract: Recent years have seen significant progress in Text-To-Audio (TTA) synthesis, enabling users to enrich their creative workflows with synthetic audio generated from natural language prompts. Despite this progress, the effects of data, model architecture, training objective functions, and sampling strategies on target benchmarks are not well understood. With the purpose of providing a holistic understanding of the design space of TTA models, we set up a large-scale empirical experiment focused on diffusion and flow matching models. Our contributions include: 1) AF-Synthetic, a large dataset of high quality synthetic captions obtained from an audio understanding model; 2) a systematic comparison of different architectural, training, and inference design choices for TTA models; 3) an analysis of sampling methods and their Pareto curves with respect to generation quality and inference speed. We leverage the knowledge obtained from this extensive analysis to propose our best model dubbed Elucidated Text-To-Audio (ETTA). When evaluated on AudioCaps and MusicCaps, ETTA provides improvements over the baselines trained on publicly available data, while being competitive with models trained on proprietary data. Finally, we show ETTA's improved ability to generate creative audio following complex and imaginative captions -- a task that is more challenging than current benchmarks.

ETTA: Elucidating the Design Space of Text-to-Audio Models
Sang-gil Lee"
Zhifeng Kong"
Arushi Goel
Sungwon Kim Rafael Valle* Bryan Catanzaro
Abstract
research community has attempted understand this design space and the contribution of cach factor, drawing conclu- sions between experiments is challenging. Our goal in this work is not to explore novel model designs or methodolo- gies. Instead, we aim to provide a holistic understanding of existing paradigms for building TTA models, to identify important aspects that allow for improving results, and to assess scalability with respect to data and model size.
Recent years have seen significant progress in Texi-To-Audio (TTA) synthesis, enabling users to enrich their creative workflows with synthetic audio generated from natural language prompts. Despite this progress, the effects data, model ar- chitecture, training objective functions, and sam- pling strategies target benchmarks understood. With the purpose of providing holis- tic understanding of the design space of TTA mod- els. we set up a large-scale empirical experiment focused on diffusion and flow matching models. Our contributions include: 1) AF-Synthetic, a large dataset of high quality synthetic captions obtained from audio understanding model; 2) systematic comparison of different architectural, training, and inference design choices for TTA models; 3) an analysis of sampling methods and their Pareto curves with respect to generation qual- ity and inference speed. We leverage the knowl- edge obtained from this extensive analysis pro- pose our best model dubbed Elucidated Text-To- Audio (ETTA). When evaluated on AudioCaps and MusicCaps, ETTA provides improvements over the baselines trained on publicly available data, while being competitive with models trained on proprietary data. Finally, we show ETTA's im- proved ability to generate creative audio following complex and imaginative captions- task that is more challenging than current benchmarks
well
In this paper, aim to elucidate the design space of TTA model with respect to training data, model architecture and implementation. model capacity, training objective, and sampling methods during inference. In a controlled scenario and with vast sweep over factors, we offer insights on the contribution of each factor. In addition elucidating the de- sign space of TTA models, our best configuration produces model namely Elucidated Text-to-Audie (ETTA) that significantly improves over open-sourced baselines on both AudioCaps (Kim et al., 2019) and MusicCaps (Agostinelli et al., 2023) benchmarks with a single model.
Recent research has shown that scaling dataset size, com- bined with a careful data filtering strategy, can yield sizable improvements on benchmarks in other domains (Radford et al., 2019; Betker al., 2023). Comparatively, the datasets used in TTA are generally much smaller, and their cap- tions of varying quality, thus posing a challenge to scaling datasets (Liu et al., 2023b; Huang et al., 2023c). In order to circumvent these challenges, we introduce a large-scale and high-qualit dataset of synthetic captions, and show that it possible leverage synthetic captions to obtain significant improvements.
While Transformers (Vaswani, 2017) have become the de facto architecture choice in many domains, sometimes their efficiency and stability, specially in larger models, are severely impaired by implementation details related to nu- merical precision and weight initialization." We improve on several implementation details of the Diffusion Transformer (DiT) (Peebles & Xie, 2023) in the area of TTA genera- tion, and provide insights on which details are important for improving benchmark scores.
1. Introduction
The design space of text-to-audio (TTA) models is com- plex, including a myriad of correlated factors. While our
'Equal contribution @oviiaa.@iiiii...m......... Zhifeng Kong <zkong monng@@vidiacomm.... Rafael Valle <rafaelvall www.dvaallleiiiiicoomm.
'NVIDIA. Conrespondence to:
Sang-gil Lee
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In tandem, current trends have shown the benefits scaling
Demo: https: /opa:://reeerchh comidabscommaabs adlr/ETTA/, Code: https: https/////gtttt/t/////e//tttettt cwt.b//DI/////AA..... elucidated
FF.g, see https: / //nn///ti 1 oth. i /bl og/ geemaa buuga for the importance of implementation details.

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
model size (OpenAI, 2024; Chung et al,, 2024; Radford et al., 2019), including better result on henchmarks and the appearance of emergent capabilities. While increasing ca- pacity overall can yield improvements, it is important to strategically allocate capacity in a way that is Pareto opti- mal, maximizing scores and alleviating inference costs. In addition to increasing the decoder's capacity, the community has compared CLAP (Wu et al., 2023) and T5-based (Raffel et al., 2020; Chung et al., 2024) text encoders (Liu et al., 2023b; Ghosal et al., 2023; Liu et al., 2024), but the results seem mixed and strongly dependent on the data and decoder capacity at hand. We show in our experiments that, although improvements can be obtained by scaling model size, some strategies for increasing capacity yield better returns than others.
Kingma al., 2021; Dhariwal & Nichol, 2021) are type of deep generative models that learn the data distribution with optional conditions (e.g. text-to-X generation). They learn a reverse stochastic process that gradually transforms the Gaussian noise into clean data. The training objective of diffusion models is to predict the score function, i... the gra- dient of the oog-likelihood with respect to data, via a neural network. Alternatively, some flow matching models predict the vector field related to the optimal transport between dis- tributions (Lipman et al., 2022; Tong et al., 2023). These models can also be trained in the latent space (Rombach et al., 2022; Liu et al., 2023b) for better efficiency, scala- bility, and quality. Appendix B includes the mathematical details.
Text-to-Audio Models There are two main streams of text-to-audio (TTA) models (including both audio and music generation) in the research community. One line of work uses diffusion and flow matching- -based models. These works proposed numerous architectural and training designs for audio generation (Liu et al., 2023b; Ghosal et al., 2023; Huang et al., 2023c;a; Liu et al., 2024; Kong et al., 2024b; Xue et al., 2024; Haji-Ali et al., 2024; Hai et al., 2024; Vyas et al., 2023) and music generation (Melechovsky et al., 2023; Huang et al., 2023b; Evans et al., 2024a;b;c; Lam et al., 2024; Schneider et al., 2024; Lan et al., 2024; Li et al., 2024b;a; Fei et al., 2024). However, there is no systematic study on their design choices, and a main challenge is that the design space has too many variables to investigate. Our work falls in this category and aims at conducting the first systematic study on the design space of diffusion and flow matching based TTA models, and we choose to use the latest Stable Audio Open (Evans et al., 2024c) as our base model to investigate. Another line of research focuses on the language model approach and uses next token prediction to train a language model on discrete token representation of audio (Kreuk et al., 2022; Borsos et al., 2023; Agostinelli et al., 2023; Copet et al., 2024). These works are orthogonal to our study.
Finally, the diffusion model literature (Ho et al., 2020; Song et al., 2021) includes a wide range of training and sam- pling methods on the shelf (Kingma et al., 2021; Salimans & Ho, 2022; Lipman et al., 2022; Ho & Salimans, 2022; Karras et al., 2022; Tong et al., 2023; Karras et al., 2024). Through comprehensive experiments across various training objectives and sampling methods, we determine the most effective training method for our setting. In addition, we provide deeper insights into how to optimally select the sam- pling method for the best results by drawing Pareto curves across various evaluation metrics.
We summarize our contributions below:
We introduce a large-scale and high-quality synthetic caption dataset called AF-Synthetic, and show that it can significantly improve text-to-audio generation quality on benchmarks.
We ablate on major design choices in the text-to-audio space, and elucidate the importance of each compo- nent with respect to improving scores on benchmarks with an emphasis on data, architectural design, training objectives, and sampling methods.
AudioSet (Gemmeke et al., 2017) pioneered large-scale audio-text dataset with labels for about 2M audio segments. AudioCaps (Kim et al., 2019) and MusicCaps (Agostinelli et al., 2023) are sub- sets of AudioSet with high-quality human- auman-annotated cap- tions. They are among the most common benchmarks for text-to-audio and text-to-music generation. With the rapid progress in large language models (LLMs) in recent years, LLM--nhanced audio-caption datasets such as WavCaps (Mei et al., 2024) and Laion-630K (Wu et al., 2023) were proposed, enabling large-scale audio-language models in- cluding TTA and other tasks. However, the captions can be noisy as the caption generation process does not depend on the audio signals. In the domain of TTA, recent works have used different collections of audio-caption pairs (mostly by
Audio-Caption Datasets
We introduce an improved implementation of diffusion transformer (DiT) for text-to-audio.
We present ETTA, the state-of-the-ar text-to-audio model trained on publicly available datasets. ETTA is also comparable with models trained on much larger proprietary data.
We show ETTA's improved ability to generate creative audio following complex and imaginative captions.
2. Related Works
Diffusion models (Ho et al., 2020; Song et al., 2021; Kong et al., 2021;
Diffusion and Flow Matching Based Models
2

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
combining existing datasets) in order train powerful TTA models. Examples include TangoPromptBank (Ghosal et al., 2023), AudioLDM (Liu et al., 2024), and Make-an-Audio (Huang et al., 2023c). However, these works mostly consti- tute combination and/or augmentation of existing data.
We follow and improve the caption synthesis pipeline from AF-AudioSet. We use Audio Flamingo (Kong et al., 2024a) to generate ten captions for each audio sample and store the caption с with the highest CLAP similar- ity cos( (LLPP CL.APaccoo()) CLAP (c)) to the audio a (Wu et al., 2023). We discard the caption if the similarity is be- low 0.45, the optimal threshold according to AF-AudioSet (Kong et al., 2024b). In addition, there are challenges when applying this pipeline to larger-scale synthesis (be- yond AudioSet), such as extremely long, homogeneous, or low-quality audio. To address these challenges, we caption each non-overiapping ten-second segment to obtain as many captions as possible. We then use keywords, e.g. "noisy", "low quality", or "unknown sounds", to detect low-quality audio. Finally, we also sub-sample long audio segments except for music and speech. With this strategy, we are able to generate 1.35M high-quality captions using audio from AudioCaps (Kim et al., 2019), AudioSet (Gemmeke et al., 2017), VGGSound (Chen et al., 2020), WavCaps (Mei et al., 2024), and Laion-630K (Wu et al., 2023). 3 We name our synthetic dataset AF-Synthetic.
Synthetic Data for Improved TTA Very recently, several concurrent works have studied using audio captioning mod- els to generate synthetic captions of unlabeled audio. This leads to more accurate audio-caption pairs that could be used to train better TTA models. In detail, (Yuan et al., 2024) uses CogVLM (Wang et al., 2023) to generate visual descriptions and EnClap (Kim et al., 2024) to generate sound descriptions, and then use ChatGPT to condense into captions. This approach does not apply to audio data without video, and the captions may contain excessive visual information that does not exist in audio. Tango-AF is trained on AF-AudioSet (Kong et al., 2024b) generated with Audio Flamingo (Kong et al., 2024a). It has very high quality, but is very small in scale. GenAU (Haji-Ali et al., 2024) is a concurrent study to ours, trained on captions generated with AutoCap (Haji-Ali et al., 2024). All these studies demonstrate synthetic captions could lead to significant improvement of TTA generation quality. In- spired by these pioneering studies, we propose a large-scale synthetic dataset of captions leveraging an audio language model followed by filtering that ensures high quality cap- tions.
Table summarizes the comparison between AF-Synthetic and existing synthetic datasets. Our dataset is both large- scale (over IM captions) and high-quality (CLAP 2 0.45), We further apply our CLAP-similarity filtering to Sound- VECaps (denoted as Sound-VECaps A -0.45) and find that over 90% of the captions are rejected. Figure I displays the distributions of CLAP similarities. Our AF-Synthetic is over 8x larger than Sound-VECapa VECapsA and AF-AudioSet, and has systematically higher CLAP similarities (about 3.8% absolute improvement the median) than these two datasets. Table 9 in Appendix C.2 further shows that AF- Synthetic sigmificantly improves text adherence of the audio compared to the haselines from human evaluations.
3. Methodology
In Section 3.1, we introduce our method for building large-scale, high-quality synthetic dataset used to train our TTA models. In Section 3.2, we describe our ETTA model, including architectural design, training objectives, and train- ing methods of the variational autoencoder (VAD) and latent diffusion model (LDM). Section 3.3, we describe the sampling algorithms that we will study in our experiments.
We then investigate the distributions of CLAP-similarity scores between our synthetic captions and AudioCaps and MusieCaps, two benchmarks we will use to eval- uate our TTA. For each caption e in AudioCaps or MusicCaps, we find its most similar caption from AF-Synthetic via the max-similarity max-sim (X,c) max cos (CLAPtexd (eext.)) (I), CLAPteet (c)). We plot the distributions of max-sim in Table 2. We find AF-Synthetic has captions that are more similar to MusicCaps than Au- dioCaps, possibly due to caption lengths. We also find most max-sim scores less than 0.9, indicating AF-Synthetic captions are quite different from these two datasets. We display some examples of most similar caption pairs in Ap- pendix C.3. In summary:
3.1. AF-Synthetic
Inspired by the recent success of synthetic captions in the text-to-image domain (Betker et al., 2023; Nguyen et al., 2024), we aim to build a large-scale and high-quality synthetic captions dataset for better text-to-audio models. While there are several in the-hhewwld datasets with paired text and audio data, they have certain limitations that we aim to overcome. Captions in WavCaps (Mei et al., 2024) and Laion-630K (Wu et al., 2023) are noisy because they are produced from text metadata only, not considering the actual audio. Sound-VECaps VECaps does not apply to audio data without video, and the captions may contain excessive visual infor- mation that does not exist in audio. AutoCap (Haji-Ali et al., 2024) and AF-AudioSet (Kong et al., 2024b) are closest to ours; Synthetic constitutes scaling this approach.
Ouur datasct has no overlap with MusicCaps (Agostinelli et al., 2023), which is also derived from AudioSet.
3

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
Table 1: Overview of our proposed AF-Synthetic dataset compared to existing synthetic captions datasets. AF-Synthetic improves the caption generation pipeline in AF-Audioset, and applies it to a variety of datasets, leading to a large-scale and high-quality synthetic dataset of captions. is the first million-size synthetie captions dataset with strong audio-caption correlations (1.35M captions with CLAP similarity 2 0.45). f After CLAP-similarity filtering.
Dataset
Generation Model
Filtering Method
# Hours
# Captions
TrngoPromp
Collected
None
3.5K
1.21M
Sound- V
CogVLM+ EnClap
Removing visual-only data
14.3K
1.66M
Sound-VECapp A
CogVLM EnClap
CLAP 0.45 Removing music or speech
448
161K
AutoCap
AutoCap
8.7K
761K
AF-AudioSet
Audio Flamingo
CLAP 2 0.45
255
161K
AF-Synthetic (ours)
Audio Flamingo
CLAP 2 0.45 and others
3.6K
1.35M
Distribution CLAP simllarties between audio and captions
Distribution of (max miaiammmttteeiiice///0 ()
hee0 copiisss
Caupe
GT=AudioCaps
1e1K coptien
AF AudioSet
1.22 captions
GT-MusicCaps
AF SySttthetic (ours)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.6
0.7
0.8
0.9
Distributions of CLAP similarities between au- dio and caption c, Cos(CL.AP (a), CT.APtext text (c)), existing datasets and our AF-Synthetic. Empirically, we consider CLAP score of 0.4 as meaningful correlation, 0.45 stronger, and below 0.3 as weak. AF-Synthetic has > IM strongly correlated audio-caption pairs.
Figure I:
Figure 2: Distributions of max-similarities, max-sim (X,,) maX x Cos(CT.AP Pestt CLAPPtext text (c)), between AF-Synthetic and real datascts. Results indicate AF- Synthetic captions are quite different from AudioCaps and MusicCaps because most max-sim scores are below 0.9.
(c)): in
Ay-Sytthetic is the first million-size synthetic caption dataset with strong audio correlations.
shown in Table 33 and Table 34 (Appendix G).
ETTA-LDM Next, we train a text-conditional latent gen- erative model for TTA synthesis. The latent model can be either diffusion model (Ho et al., 2020; Song et al., 2021; Salimans & Ho, 2022) or a flow matching model (Lip- man et al., 2022; Tong et al., 2023). We parameterize our model using the Diffusion Transformer (DiT) (Peebles Xie, 2023) architecture based on Evans al. (2024c) and Lan et al. (2024), with 24 layers, 24 heads, and a width of 1536 as the default choices. We condition our model on the outputs of the T5-base (Raffel et al., 2020) text encoder, which outputs embeddings for variable-length text. In our experiments, we also explore other common choices and combinations of different text encoders - including T5- based (Raffel et al., 2020; Chung et al., 2024) and CLAP models (Wu et al., 2023) study the effect of this compo- nent.
3.2. ETTA
Our TTA model, dubbed Elucidated Text-To-Audio (ETTA), is built upon the LDM (Rombach et al., 2022) paradigm and its application audio generation. First, variational autoencoder (VAE) (Kingma & Welling, 2014) is trained to compress waveform into a compact latent space. Once the VAE is trained, we freeze it and train a latent genera- tive model in the VAE latent space. See Appendix B for mathematical details. We conduct our experiments based on the stable-audii oolele--udio-tooooo library.* which provides the most recent practices in building TTA models.
For training the VAE, we adopt 44kHz stereo Audio-VAR with 156M parameters using the same de- fault configuration used in stable wewllllaado-ttee
ETTA-VAE
s with a latent frame rate of 21.5Hz. We refer to (Evans et al., 2024c) and Appendix B for details. The Audio-VAE is trained from scratch on our large-scale collection of pub- licly available datasets (see Table 12). In terms of quality, our Audio-VAR matches or exceeds Stable Audio Open, as
ETTA-DIT Finally, we provide several key improvements to the DiT implementation in Evans et al. (2024c), and call our implementation ETTA-DIT. Through experiments, we find that solely replacing their architecture with ETTA-DiT leads to improved training losses and evaluation results. Our
'Sinee our dataset includes speech data, it is noticeably better in reconstructing speech signals.
https:: / //ggtttuut com/ Stability-ATI stable-aude o- tools a commit id: 7311840

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
improvements include:
4. Experiments
1) Adaptive layer normalization (AdaLN); We switch from prepending to AdaLN timestep embedding and ap- ply AdaLN. 6 Contrary to the baseline, we apply AdaLN to all inputs: self-attention, cross-attention, and feed-forward layer. The AdaLN parameters are initialized with scale = and bias 0 so that AdaLN does not modulate the feature at initialization. When applying AdaLN, we enforce FP32, use
Our experiments thoroughly evaluate our framework ETTA on benchmark datasets (AudioCaps and MusicCaps). We start with a systematic comparison to elucidate the design space of TTA in four major aspects: 1) training data, 2) train- ing objectives, 3) architectural design and model sizes, and 4) sampling methods. Furthermore, we show ETTA's im- proved ability to generate creative audio following complex and imaginative captions, a task that is more challenging than current benchmarks. In our commitment to fully elu- cidate all aspects of our investigation, we also document the additional directions we explored, including numerous additional ablations (in Appendix D and E) and mixed or negative results (in Appendix F). We train all models using 8 A A100 GPUs.
for numerical precision. Contrary to the baseline, we use bias term for the linear layer and use unbounded gating (i.e. no sigmoid).
2) Final layers: Compared to the baseline, we initialize the final projection layer of DiT to output zeros. This matches the mean of the VAE latent distribution, and therefore leads to improved stability and convergence rate. We also use AdaLN in the final projection layer.
4.1. Training Data
3) Other changes: we use the tanh approximation mode of the GELU activation (Hendrycks & Gimpel, 2016). We use rotary position embedding (RoPE) (Su et al., 2024) in the self-attention layer, with rope_base = 16384 to inject relative positional information. We also ensure that RoPE operates in FP32. We additionally apply dropout with Paropout = 0.1 for all modules to enhance robustness in parameter estimation.
We train ETTA on four different training datasets to as- sess TTA quality: AudioCaps (50K captions), AF-AudioSet (161K captions), TangoPromptBank (1.2 captions), and our AF-Synthetic (1.35M captions). We fix audio length to 10 seconds and sampling rate to 44. IkHz in all these datasets.
4.2. Training Objective and Sampling
3.3. Training objective and Sampling
Audio VAE We train a 44. IkHz stereo Audio-VAE based stable-audio-tooll with our collection of unla- beled and public audio datasets (Table 12). We train the Audio- VAE using AdamW (Loshchilov, 2017) with a peak learning rate of 1.5 x 10 with exponential decay for 2.8M steps, with a total batch size of 64 with 1.5 seconds per sample. We train with full precision (FP32) to make the waveform compression model as accurate as possible. The latent dimension is 64 and the frame rate is 21.5 Hz.
Training For the diffusion model training objective, we use the u-prediction loss function (Salimans & Ho, 2022). For the flow matching training objective, we use the optimal transport conditional flow matching (OT-CFM) loss func- tion (Lipman et al., 2022; Tong et al., 2023). We refer to Appendix B for details of these methods. Prior works also found sampling / more often on intermediate steps leads to better results (Esser et al., 2024; Lan et al., 2024). We follow their approach and sample from logit-normal dis- tribution, in practice t ~ (N, 1)), when training ETTA with OT-CFM.
Training Objective and Architecture We train ETTA- LDM with ETTA-DIT as the backbone. We use the T 75-base text embedding with max.length=512 trun- cation to accommodate longer captions in AF-Synthetic.? We train with both u-diffusion and OT-CFM objectives, where we additionally apply logit-normal t-sampling for OT-CFM (see Section 3.3). Our final model is trained for IM steps using Adam with peak learning rate of 10-4 with exponential decay and total batch size of 128 with 10 seconds per sample. For ablation studies, we train each model for 250k steps unless otherwise stated. We use BF 6 mixed-precision training (Micikevicius et al., 2017) and Baashatttem
Sampling We consider Euler and 2nd -oddrr Heun (Karras et al., 2022) methods for solving the ODE parameterized by ETTA. We conduct an extensive sweep over hyperparame- ters focusing on two major design choices: the number of function evaluations (NFE) and the classifier-fre free guidance (CFG) (Ho & Salimans, 2022) scale Wefg. We draw Pareto curves across benchmark datasets and metrics to discover the optimal choice for ETTA. In addition, we also explore the effectiveness of recently proposed guidance method, autoguidance (Karras et al., 2024), in TTA applications.
(Dao et al., 2022) to maximize training throughput.
'In our preliminary study using tubble--udiottooo with its vanilla implementation. switching from prepending to AdaIN resulted in worse results.
'Our reproduction Stable Audio Open using AF-Synthetic datasel also uses the same lexgt h= for a fair compari- son.
5

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
Sampling For diffusion models, following (Evans al,, 2024c) we use the dpmpp-3m-sde sampler and CFG scale Wofk = 7. For OT-CFM models, we compare between Euler and 2nd ooddrr Heun samplers and draw Pareto curves for each method with respect to the number of function eval- uations (NFEs) and CFG scale. After this extensive sweep, we choose Euler sampling with NFE = 100, Wrfy = 3.5 for main results, and Wefg = 1 (no classifier-free guidance) for ablation studies unless otherwise stated.
ated samples. Objective scores on MusicCaps are signifi- cantly better than previous models using public datasets and comparable to music specialist models (Li et al., 2024a;b; Fei et al., 2024) that use proprietary data. Since FDo can measure stereo audio, Stable Audio Open and ETTA are noticeably better than previous mono models. Both CLL and CLas show a preference towards ETTA, where our improvements on CLM more salient. Subjective scores (OVL/REL) of ETTA are competitive with outperforms baselines and are consistent with the objective evaluation.
4.3. Results
We then fine-tune ETTA on the AudioCaps training set (FT-AC) for 50k and 100k additional steps. We find ETTA can quickly adapt to the target distribution via fine-tuning, Table 2 shows that ETTA keeps approximating the target distribution with better FDp, which is close to Audiobox Sound (Vyas al., 2023) trained proprietary dataset. It is noteworthy that this also comes at cost of shifting to the target distribution as evidenced by Table 3, where TTTA-FT-AC-100E starts to show noticeable degradation for music generation. In summary, our results show that:
Metries We use a collection of established objective met- rics for systematic evaluation. 1) Fréchet distance (FD) mea- sures the distributional gap between generated and ground truth audios using features extracted from an audio classi- fier. We consider PANNs (Kong et al., 2020) (FDp) and OpenL. (Cramer et al., 2019) (FDo). 9 2) Kullback-Leibles divergence (KL) is an instance-level metric that measures the difference between the posterior distributions of audio events for the ground truth and generated audio samples. This metric helps assess how close the generated audio aligns with the ground truth on the single-sample level. We report KL using PaSST (Koutini et al., 2022) (KLs) and PANNs (KLp). 3) Inception Score (IS) evaluates the di- versity and specificity of the generated samples without requiring ground truth. IS is calculated from the entropy of instance posteriors and the entropy of marginal posteri- ors, where higher score reflects both better diversity and sharper class predictions. We use PANNs for IS (IS 4) Finally, CLAP score measures the cosine similarity between text and audio embeddings, which indicates the correlation between the generated sample and the given prompt. For extensive evaluation, we use two CLAP models: CL. for LAION's 630k best checkpoint (Wu et al., 2023) fol- lowing Vyas et al. (2023), and CLas for MS-CLAP 2023 version (Elizalde et al., 2023). We also perform 5-scale subjective evaluation from mechanical turk following con- ventional metrics: 1) OVL: an overall quality of sample without sceing captions, and 2) REL: a relevance of the sample to the provided caption.
ETTA is the SOTA text-to-audio and text-to-music gen- eration model using only publicly available data. It is also comparable to models trained with proprietary and/or licensed data.
Design Improvements Tables 4 summarize important design choices that lead to significant improvements. We use FDp, KLs, and CLys on MusicCaps as summary (Full results in Tables 13 and 14, Appendix D). First, we reproduce Stable Audio Open using AFF-Synthetic without other modification (+AF-Synthetic). Results show notice- able improvements from training data. Then, we switch the DiT implementation to ours (+ETTA-DiT). Results again show significant improvements. Next, we switch the train- ing method from diffusion to OT-CFM (+OT-CFM) with conventional uniform timestep sampling (1 ~ U(0 1)). Em- pirically, although OT-CFM slightly degrades some met- rics without CFG, we find OT-CFM is more stable to train, more consistent in quality especially with CFG, and more robust under fewer sampling steps in agreement with pre- vious works. Finally, we adopt logit-normal t-sampling (t ~ (N 1))) (Esser et al., 2024) and find it marginally improves FDp. Therefore, we conclude:
Main Results Tables 2 and 3 present our main objective results on AudioCaps and MusicCaps, respectively. Overall, ETTA shows significant improvements compared to Stable Audio Open (the base model) for both benchmarks with a single model. Compared to other works, ETTA shows competitive FDp, FDo, and KL scores. Notably, it shows exceptionally high ISp for both general sounds and music, demonstrating improved diversity and clarity of the gener-
Our AF-Synthetic leads to the most significant improve- ments in ETTA. Our improved ETTA-DIT, the OT-CFM objective, and logit-normal f-smmllnng lead to further im- provements.
Scalability with Data We assess the scalability of TTA models with respect to training data in Table (Full results in Tables 21 and 22 in the Appendix E). First, AudioCaps lacks in quantity: ETTA trained solely on AudioCaps sig- nificaly underperforms on MusicCaps. TangoPromptBank
'Implementation available in httts:://tttittio com/ crowsonkb/ dddiiffusiin
OpenL.3 the latest model with better embedding quality, and FDo can measure up to 48klIz stereo quality (Evans et al., 2024c).
6

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
Table 2: Main results of ETTA compared to SOTA baselines on AudioCaps. FT-AC-m: fine-tuned on AudioCaps training set for m iterations. Best reported numbers. i Uses proprietary data.
FD
F
Klg 4
Klpp
18p
Cl.
Cla 1
OVL1
REL-1
Ground Truth
13.49
0.62
0.38
3.43
0.11
3.62 0.10
Audiobox yVyas ot 2023)"1
10.14
1.19
0.70
Audiobox Sound yas et al., 2023y"
8.30
1.15
12.70
0.71
Make-An-Audio (Huang et 2023c)
18.32
1.61
Muke-An-Audio (Hunng al, 2023a)"
11.75
1.32
11,16
(Liu ot 2023b)
23.31
1.59
8.13
0.43
AMdioLDM2
2024)
26.44
AudioLDM2 large (Liiu et al., al. 2024)
32,50
170.31
1.57
1.54
8.55
0.45
IM2-large
2024)
3,00 0.11
SANGO-FulI-T-AA. (Ghosal et al., 2023)*
18.47
1.20
LIS
8.80
0.56
MANGO-AF&AC-FT-- (Kong et al., 2024b)
17.19
11.04
0.53
TANGO2 (Majumder al., 2024))
1,12
9.09
3.08 0.10
3,66 0.09
GenAU-L (Haji-Ali ot all, 2024)*
16.51
11.75
Stable Audio Open (Evans 2024c)*
Stable Audio Open (Evans al,, 2024c)
38.27
105.88
2.23
2.32
12.09
0.35
0.34
3.29 1: 0.11
3.15 1. 0.11
ETTA
13.12
80.13
1.22
1.42
14.36
0.54
0.43
3,43
3.68 = 0.10
STTA-FT-AC-S.
11.13
65.35
1.12
1.26
15.05
0.59
0.43
STTA-FT-AC-100
10.10
61.79
113
1.24
14.29
0.60
0.43
3.26
2 0.10
3.77 =:010
(AudioCups only)
12.21
Table Main results of ETTA compared SOTA baselines on CasicCaps. FT-AC-m: fine-tuned on AudioCaps training set for m iterations. Best reported numbers. t Uses proprietary or licensed data.
FDp
FDo
Kls.
KLp
18p
Cla.
CLM 1
OVL
REL.
Ground Truth
4.49
0.53
0.45
3,88
0,10
3,90 +: 0.30
(L al,, 20246)"
1.29
QA-MDT 20244))
FluxMusic (Fiei al, 2024)*
1.25
2.98
MusicGen-medium (Copet et al., 2024)
1.23
1.22
AMdioLDM-M (Lio
20236)
AudioLDM2 (Liu ot 2024)
1.20
1.20
AudioLDM2
2024)
21.39
0.45
Audiel large (I 2024)
16.34
190.16
1.00
1.40
2.59
0.48
0.47
3.25 0.10
3.15
TANGO-AF (Kong et al., 2024b)
22.69
270.32
1,26
2.79
0.51
3.38
0.09
3.31
Stable Audio Open (Evans al. 2024c)
36.42
127.20
1.32
1.56
2.93
0.48
0.49
3,92
3.35
ETTA
10.06
92.18
0.84
1.04
3.32
0.51
0.53
3.52 - 0.10
3.57
P77AFT..C FT-AC-500
SITA-FT-AC-I0C
13.49
89.56
1.07
1.15
2.77
0.49
0.52
0.10
3.44 + 0.12
Table Improvements by adding each of the major design choice of ETTA (evaluated MusicCaps)
comparable FDp, other metrics (KLs and CLAr) are much worse, suggesting that the quality of their music captions is not as good as AF-Synthetic. AF-AudioSet contains high-quality synthetic captions: it competitive with AF- Synthetic, emphasizing the importance of data quality. We further evaluate
Ablation
FDr
KLs +
CL 1
Stable Audio Open
39.96
1.81
0.41
Ay-Synthetic
26.22
1.57
0.43
ETTA-DiT
20.48
1.38
0.45
cattof-distribution (OOD) dataset in Table 24 (Appendix E), and AF-Synthetic results consis- tently better than AF-AudioSet. The results highlight that AF--Sytthetic is powerful dataset that is comprehensive in both quantity and quality. As such, we conclude:
+ OT-CFM, L 1)0,1)
22.16
1.35
0.45
o(V(0.
21.59
1.41
0.45
Table 5: Ablation study on the results of ETTA trained on different datasets (evaluated on MusicCaps).
Both training data sizes and quality have positive effect on the results, where quality matters more.
Dataset (million captions)
FDp J
KLs
CL t
AudioCaps (0.05)
76.14
3.20
0.27
TangoPromptBank (1.21)
24.72
1.73
0.38
Scalability with Model Size Table provides the sum- mary scaling behavior of ETTA with respect its model size. We explore different depths, widths, and the convolu- tional feed-forward layer kernel sizes (kcouvFP) of ETTA- DIT. We use Weig to eliminate the effect of CFG.
AuAAuiee (0.16 00.66)
21.40
1.45
0.44
AF-Synthetic (1.35)
21.59
1.41
0.44
is simillar to AF-Synthetic in quantity: while it scored
As expected, most metrics show consistent improvements segments a long audio.
I'In practice, we used 2.33M audio-caption pairs for Tango- PromptBank due to repetitive captions for multiple 10-second

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
Table 6: Ablation study on the results of ETTA with different depths, widths, and kernel sizes (evaluated on AudioCaps). * Our best model choice.
lead a trade-off between diversity and accuracy. Detailed results the choices of sampler and NFE are provided in Table 19 (Appendix D). In summary:
Model
Size(B)
FDp...
KLs.J
CLM f
Heun's sampler is better than Euler at lower NFE. Wefg = 3.5 provides the best overall metrics, and one should be cautious that higher CFG scale potentially leads to lower diversity.
depth =4
0.38
36.46
2.15
0.30
depth = 12
0.81
29.48
2.05
0.32
depth = 24"
1.44
28.46
2,00
0.32
depth = 36
2.08
27.08
1.95
0.32
width 384
0.28
35.97
2.14
0.30
width = 768
0.52
31.03
2.04
0.32
Table 7: Subjective Evaluation Result of Creative Audio Generation with 95% Confidence Interval.
width = 1536"
1.44
28.46
2.00
0.32
FF I'
1.44
28.46
2.00
0.32
RF
2.34
28.72
2.04
0.31
Model AMdioLDM2
TANGO2
Stable Audio Open
ETTA
OVL 3,95 + 0.05
3.82 + 0.05
3.94 + 0.05
3.99 0.05
REL1 3.79 = 0.06
3.94 = 0.05
3,95 + 0.05
4.05 : 0.05
as we grow depth or width of ETTA-DiT. We find the 1.44B model with h=pth=22 and width=1536 leads to an op- timal balance between model size and quality. On the other hand, increasing keonvFF does not bring clear im- provements, suggesting that allocating the model capacity to self-attention parameters is more important. See tables 17 and (Appendix D) for extended results. In summary,
Creative Audio Generation We test ETTA's abilities to generate creative audio and music samples that do not exist in the real world, especially for complex and imaginative captions. We ask ChatGPT to generate hard captions that require blending and transformation of various sound ele- ments towards creative audio. See Table 20 (Appendix D) for the imaginative captions. We generate 20 samples for each model and invite human listeners to measure 5-scale rating of OVL and REL. Table 7 shows that ETTA signifi- cantly improves its ability to follow the complex captions as measured by the REL score (p < 0.05 from Wilcoxon signed-rank test). We strongly encourage the readers to listen to the audio samples in the demo page (Appendix A). Therefore, we claim:
In TTA tasks, increasing model size helpful via in- creasing depth and width of DiT's self-attention block. However, increasing the kernel size of the convolutional feed-forward layer is not helpful.
Euler
Heun
18
14
ETTA shows an improved ability to generate audio that follows complex and imaginative captions.
e
12
10
0
NFE (MusicCaps)
25
5D
75
100
2
CFG (MusieCaps)
4
5. Conclusion
Figure 3: The effect of different sampling methods on the generation quality of ETTA MusieCaps. We investigate Eu- ler and Heun solvers. NFE: number of function evaluations. CFG: classifier-free guidance scale.
In this paper, we setup a large-scale empirical experiment to comprehensively understand the design space of modern text-to-audio models. We provide insights on data scaling, architectural design, model scaling, training methods, and inference strategies. Based on our findings, we present ETTA, state-of-the-ar text-to-audio model that results from large-scale and high-quality synthetic captions, a better DIT implementation, and a better VAE.
Choice of Sampler and its Impact on Metrics Figure 3 and Figure 5 (Appendix D) present a comprehensive anal- ysis of the impact of sampler choices. The results reveal several key insights: 1) All metrics improve as the number of function evaluations (NFE) increases, as expected. 2) At lower NFE, the Heun sampler is noticeably better than Euler; as NFE increases, they converge to similar results. 3) FD behaves as a convex function with respect to the CFG scale, indicating that FD penalizes low diversity caused by CFG's over-emphasis on text condition and/or distortion caused by high CFG scales. 4) Metrics such as KI., IS, and CL (Figure 5) show continuous improvement with higher CFG scales, suggesting their preference for accuracy over diversity. Therefore, one should be cautious when selecting the CFG scale, as optimizing for these metrics alone may
Future work While this work aims to elucidate the de- sign space of TTA with large-scale experiments. there are still several unexplored problems we plan to study in future work. (1) We plan to improve data augmentation with caption rephrasing and audio re-mixing (Melechovsky et al., 2023; Liu et al., 2023b; 2024; Huang et al., 2023c;a) and systematically study the effect of data augmentation. (2) We plan to investigate better evaluation methods and benchmarks for rext-to-audio generation that could reflect both the accuracy and diversity of TTA models in way that corresponds with human perception.
8

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
Impact Statement
Défossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. Transactions on Ma- chine Learning Research, 2023.
This paper aims to contribute to the advancement gen- erative modeling of audio by introducing a method that enhances the quality corresponding better to the input text prompt. The approach has broad applicabilityy across in- dustries, such as media production and music composition. However, responsible usage is crucial to ensure adherence to copyright regulations in specific contexts.
Dhariwal, P. and Nichol, A. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-877 8794, 2021.
Elizalde, B., Deshmukh, S., and Wang, H. Natural lan- guage supervision for general-purpose audio represen- tations, 2023. URL https: /ttp/aaxvvvv////// org abs/
References
Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M.. Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.1132. 2023.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference Machine Learning, 2024.
Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., al. Improving image generation with better captions. Computer Sci- ence. https://cdn. openai. com/paperstdall-e-3 pdf, 2(3): 8, 2023.
Evans, Z., Carr, C.,, Taylor, J., Hawley, S. H., and Pons, J. Fast timing-conditioned latent audio diffusion. In Forty-first International Conference Machine Learn- ing, 2024a.
Borsos, 7., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grang- ier, D., Tagliasacchi, M., et al. Audiolm: a language modeling approach to audio generation. JEEE/ACM trans- actions on audio, speech, and language processing, 31: 2523-2533, 2023.
Evans, Z., Parker, J. D., Carr, C., Zukowski, Z., Taylor, J., and Pons, J. Long-form music generation with latent diffusion. arXiv preprint MxXiv:2404.13000 2024b.
Evans, Z. Parker, J. D., Carr, C., Zukowski, Z., Taylor, J., and Pons, J. Stable audio open. arXiv preprint arXiv:2407. 14358, 2024c.
Chen, H., Xie, W., Vedaldi, A., and Zisserman, Vg- gsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020.
Fei, Z., Fan, M., Yu, C., and Huang, J. Flux that plays music. arXiv preprint (Tti:24090008877 2024.
Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and labelel dataset for audio events. 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani. M., Brahma, S., et al. Scaling instruction finittund language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and Défossez, A. Simple and controllable music generation. Advances Neural Information Pro- cessing Systems, 36, 2024.
Ghosal. D., Majumder, N., Mehrish, A., and Poria. S. Text- to-audio generation using instruction guided latent diffu- sion model. In Proceedings of the 31 ACM International Conference on Multimedia, pp. 3590-3598, 2023.
Cramer, A. L., Wu, H.-H., Salamon, and Bello, J. Look, listen, and learn more: Design choices for deep audio embeddings. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3852-3856. IEEF, 2019.
Hai, J.. Xu, Y., Zhang, H., Li, C., Wang, H., Elhilali, M., and Yu, D. Ezaudio: Enhancing text-to-audio genera- tion with efficient diffusion transformer. arXiv preprint arXiv:2409, 10819, 2024.
Dao. T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory memory-efficient exact attention with io-awareness. Advances in Neural Information Process- ing Systems, 15:16344-166 16359, 2022.
Haji-Ali, M., Menapace, W., Siarohin, A., Balakrish- nan, G., Tulyakov, S., and Ordonez, V. Taming data and transformers for audio generation. arXiv preprint arXiv:2406. 19388, 2024.

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
Hang. T., Gu, S., Li, C., Bao, J., Chen, D., Hu, H., Geng, X., and Guo, B. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7441- 7451, 2023.
Kingma, D. P. and Welling, M. Auto-Encoding Variational Bayes. In 2nd International Conference Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Kong, Q. Cao, Y., Iqbal, T., Wang, Y., Wang, W., and Plumbley, M. D. Panns: Large-scale scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Process- ing, 28:2880-2894, 2020.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint trXiv:1606.08417 2016.
Ho, J. and Salimans, Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:684006851, 2020.
Kong, Z.., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. In International Conference Learning Representations, 2021.
Huang. J., Ren, Y., Huang, R., Yang, D., Ye, Z., Zhang, C., Liu, J.. Yin, X., Ma. Z., and Zhao, Z. Make-an-audie 2: Tempporl--enhanced text-to-audio generation. arXiv preprint (rXXi:23355.84777 2023a.
Kong, Z., Goel, A., Badlani, R., Ping, W., Valle, R., and Catanzaro, B. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. In- ternational Conference on Machine Learning, 2024a.
Huang, Q. Park, D. S., Wang. T., Denk, T. I., Ly, A., Chen, N., Zhang, Z., Zhang, Z., Yu, J., Frank, C., et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03917, 2023b.
Kong, Z., Lee, S.-g., Ghosal, D., Majumder, N., Mehrish, A., Valle, R., Poria, S., and Catanzaro, B. Improving text- to-audio models with synthetic captions. arXiv preprint 24XX::2406..58877 2024b.
Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., Ye, Z., Liu, J., Yin, X., and Zhao, Z. Make-an- Text-to- audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pp. 13916-13932, PMLR, 2023c.
Koutini, K., Schlüter, J., gehbal-zadehl H., and Wid- mer, G. Efficient training of audio transformers with patchout. In Interspeech 2022, 23rd Anmual Confer- ence of the International Speech Communication As- sociation, Incheon, Korea, 18-22 September 2022, pp. 2753-2757. ISCA, 2022. doi: 10.21447/Imterspeechh wtt.iioogg/1..1133333
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-basee generative models. Advances in neural information processing systems, 35: 26565-26577, 2022.
2022-227. URL Interspeech, 2022-227.
Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., Défossez, A., Copet, J., Parikh, D., Taigman, Y., and Adi, Y. Audio- gen: Textually guided audio generation. arXiv preprint MXXh:2209.153522 2022.
Karras, T., Aittala, M., Kynkäänniemi, T., Lehtinen, J., Aila, T., and Laine, S. Guiding a diffusion model with a bad version of itself. arXiv preprint wxXiv2206.002507 2024.
Kim, C. D., Kim, B., Lee, H., and Kim, G. Audiocaps: Gen- erating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long and Short Papers), pp. 119 132, 2019.
K Kynkäänniemi, T., Aittala, M., Karras, T., Laine, S., Aila, T., and Lehtinen, J. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXiv preprint 2xxxi:2404.07724. 2024.
Lam, M. W., Tian, Q., Li, T., Yin, Z., Feng, S., Tu, M., Ji, Y., Xia, R., Ma, M., Song, X., et al. Efficient neural music generation. Advances in Neural Information Processing Systems, 36, 2024.
Kim, J., Jung, J., Lee, J., and Woo, S. H. Enclap: Combin- ing neural audio codec and audio-text joint embedding for automated audio captioning. In ICASSP 2024-2024
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6735-6739. IEBE, 2024.
Lan, G. L., Shi, B., Ni, Z., Srinivasan, S., Kumar, A., El- lis, B., Kant, D., Nagaraja, V., Chang, E., Hsu, W.-N, et al. High fidelity text-guided music generation and editing via single-stage flow matching. arXiv preprint arXiv:2407.nm 03648, 2024.
Kingma, D., Salimans, T., Poole, B., and Ho, J. Varia- tional diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
10

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
Lee, S.-g., Ping, W., Ginsburg, B., Catanzaro, B., and Yoon, S. Bigygan: A universal neural vocoder with large- scale training, In The Eleventh International Confer- ence on Learning Representations, 2023, URL https: // //openreview ew, net forum? 1
Melechovsky, J., Guo, Z., Ghosal, D., Majumder, N., Herremans, D., and Poria, S. Mustango: Toward controllable texi-to-music generation. arXiv preprint arXi::231 2023.
Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. Mixed precision training. arXiv 2017.
Li, C., Wang, R., Liu, L., Du, J., Sun, Y., Guo, Z., Zhang, Z., and Jiang, Y. Quality-aware masked diffusion trans- former for enhanced music generation. arXiv preprint arXiv:2405. 15863, 2024a.
preprint wrXiv:1710.0377
Nguyen, T., Gadre, S. Y., Ilharco, G., Oh, S., and Schmidt, L. Improving multimodal datasets with image captioning. Advances in Neural Information Processing Systems, 36, 2024.
Li, P. P., Chen, B., Yao, Y., Wang, Y., Wang, A., and Wang, A. Jen-1: I: Text-guided universal music generation with omnidirectional diffusion models. In 2024 IEEE Confer- ence on Artificial Intelligence (CAI), pp. 762-769, IEEE, 2024b.
OpenAL.
Gpt-4o: A powerful multimodal lan- guage model. https: /tt///ppeaa.... com/ research/ hello-gpt-4o, 2024. Accessed: 2024-09-21.
Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint 1Xiv:2210.027477 2022.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEF/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Liu, A. H., Lc, M., Vyas, A., Shi, B., Tjandra, A., and Hsu, W.-N. Generative pre-training for speech with flow matching. arXiv preprint urXiv:2310.16338 2023a.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. Audioldm: Text-to- audio generation with latent diffusion models. In Inter- national Conference on Machine Learning, pp. 21450- 21474. PMLR, 2023b.
Raffel, C., Shazeer, N., Roberts, A., Lee. K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q.. Tian, Q., Wang, Y., Wang, W., Wang, Y., and Plumbley, M. D. Audioldm 2: Learning holistic audio generation with sule-supe pretraining. IEEE/ACM Transactions Audio, Speech, and Language Processing, 2024.
Rafi, Z., Liutkus, A., Stöter, F.-.,, Mimilakis, S. I,, and Bittner, R. The musdb18 corpus for music separation. 2017.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05100) 2017.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Majumder, N., Hung, C.-Y., Ghosal, D., Hsu, W.-., Mihal- cea, R., and Poria, S. Tango 2: Aligning diffusion- based text-to-audio generations through direct preference opti- mization. arXiv preprint arXiv:2404.09955 2024.
Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXi:22202.055/2 2022.
Manco, I., Weck, B., Doh, S., Won, M., Zhang, Y., Bog- danov, D., Wu, Y., Chen, K., Tovstogan, P., Benetos, E., Quinton, E., Fazekas, G., and Nam, J. The song describer dataset: corpus of audio captions for music- and-language evaluation. In Machine Learning for Audio Workshop at NeurIPS 2023, 2023.
Schneider, F., Kamal, O., Jin, 7., and Schölkopf, B. Moûsai: Efficient text-to-music diffusion models. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 0050-8063 2024.
Mei, X., Meng, C., Liu, H., Kong, Q., Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., and Wang, W. Wavcaps: A chstgpt-assisted -cakly-labelled audio captioning dataset for ludio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Process- ing, 2024.
Song, Y., mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
J., Kingma, D. P., Kumar, A., Er-

---

ETTA: Elucidating the Design Space of Texioto-Audio Models
Steinmetz, C. J. and Reiss, D. auraloss: Audio focused
loss functions in pytorch. In Digital music research net- work one-day workshop (DMRN+ 15), 2020.
Steinmetz, C. J., Pons, J., Pascual, S., and Serra, Au- tomatic multitrack mixing with differentiable mixing console of neural audio effects. In ICASSP 2021-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 71-75. IEEE, 2021.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing 568:127066 2024.
Tong, A., Malkin, N., Huguet, G., Zhang, Y., Rector--rrood
J., Fatras, K., Wolf, G., and Bengio. Y. Conditional flow matching: Simulation-free dynamic optimal transport. arXiv preprint arXiv:3302.00482, 2023.
Vaswani, Attention is you need. Advances in Neural Information Processing Systems, 2017.
Vyas, A., Shi, B.. Le. M., Tjandra, A., Wu, Y.-C., Guo, B., Zhang, J., Zhang, X., Adkins, R., Ngan, W., et al. Au- diobox: Unified audio generation with natural language prompts, arXiv preprint arXiv:2312.1582// 2023.
Wang, W. Lv, Q., Yu, W., Hong. W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079 2023.
Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg- Tong-Kirkpatrick T., and Dubnov, S. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to co-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Xue, J., Deng, Y., Gao, Y., and Li, Y. Auffusion: Leveraging the power of diffusion and large language models for text- to-audio generation. arXiv preprint 4rX1v:2401.01044, 2024.
Yuan, Y., Jia, D., Zhuang, X., Chen, Y., Liu, Z., Chen, Z., Wang, Y., Wang, Y., Liu, X., Plumbley, D., al. Improving audio generation with visual enhanced caption. arXiv preprint arXiv:2407.044 044.6. 2024.
Zen, H., Dang, V., Clark, R. Zhang, Y., Weiss, R. J., Jia, Y., Chen, Z. and Wu, Libritts: A corpus de- rived from librispeech for text-to-speech, arXiv preprint arXiv: 1904. 02880, 2019.
12

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
A. Links
LstereoMRSTTT (2, LMRSTET ( (T Fsum)
+ -MRSSFF (Fair, Bair).
(2)
The link to our demo page is: https: happ:://reseaacee wwwiiaiiiiiiii..ii wwt.caas/m/////// ETTA/
where T is the number of STFT frames, each stft, is the STFT transformation with resolution i, and
The link to our code repository is:
https: / com com/NVIDTA/
suum Fleft + Frights diff
Zleft
Fright-
(3)
2. An adversarial hinge loss and feature matching loss from Encodec (Défossez et al., 2023):
B. Mathematical Background
Let Pdata be the data distribution and X Pdata be N ii.d training samples drawn from the data distribution. In (unconditional) audio synthesis, we assume Pdata is on [] where L = 441000 is fixed length for 10 seconds of audio at 44. 1kHz sampling rate. A generative model on X aims to model (x) R Paata(z) and draw samples from it. In text-to-audio synthesis, each sample = (a, c) is composed of an audio a E []+ and corresponding caption in the natural language space. In this case, we aim given
Ladv (2,
m 1001-1100 mmax0,11-x(z))
+ +
(4)
K L
-
Lreat (I.,
(5)
KL
mean (())
to model pa (alc) and draw samples conditioned caption c. For conciseness, we introduce all the mathemati- cal background the unconditional setting, and these can be translated into the conditional setting by conditioning all distributions on C.
where Dj is the I-th layer of the k-th discriminator Da.
3. The KL divergence loss:
KL(ge(z[>) | (W
(6)
B.1. Variational Auto Encoders
The VAE is trained using randomly chunked unlabeled audio data without captions.
Variational auto encoders (VAEs) (Kingma & Welling, 2014) include an encoder and a decoder D. aims to encode sample into a nower-dimensional space, and aims to reconstruct (ar) to the original space with minimal infor- mation loss. The training loss is
B.2. Diffusion Models
Diffusion models (Ho et al., 2020; Song et al., 2021) include two processes:
LVAF Ex (R))RR((((EE)))))))LL
(N(, I)),
fixed Markov chain diffusion process
dx f(x, t) + g(t)dw,
where R is a reconstruction loss that measures the distance between the original sample r and the reconstructed sample D(E(x)). (=(z/²) the approximate posterior distribution of the latent variable z given using E, and the KL diver- gence loss measures how close the posterior distribution is to the prior I).
where x represents data, / [0, 1] represents time, f the drift term, g is the diffusion term, and dw is the standard Brownian motion.
2. A learned Markov chain reverse process
Stable Audio Open's 's VAE (Evans et al., 2024c) is trained with combination of below losses:
dr [f(xtt)
(z) g(t)dw,
where dw is the reverse Brownian motion.
A stereo sum and difference multi-resolution STFT loss (Steinmetz & Reiss, 2020; Steinmetz al., 2021) that com- putes distances in the spectrogram space with different reso- lutions:
A neural network (a t) is used to substitute the score function V. (xr) and therefore trained to approximate the true score function V, log q( (z)))) at time t, leading to training objective
m [s) (+)le
(MRSTFT(E,8)
[x)
Ex-2((0,1) E1-2(00.1) prg V., log q( (20))
+ log siftu(())11))
(1)
where we could write 2, terms of noise & ~ AO, I) 2t Varao + vi a45t for pre-defined schedule os
st.ft,
13

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
Table 8: Detailed breakdown of our proposed AF-Synthetic dataset compared to existing datasets.
Total Hours
Number of captions
Dataset
Total
AudioCaps
AudioSet
Laion-630K
WavCaps
VGGSound
TrmgoPromptBank
3,5K
1.21M
45K
108K
1.55M
14.3K
1.66M
1.66M
AutoCap
8.7K
761K
339K
295K
127K
AF-AudioSet
255
161K
161K
AF- Synthhett
3.6K
1.35M
33K
165K
282K
783K
92K
and Vz, log (org)o))))) = - oz. For this reason, the standard loss function called the c-prediction.
C. Dataset Details
C.1. AF-Synthetic Details
One can predict other quantities to train diffusion models as well. One example is the x-diffusion, where we train network to predict 21 (Iz - vI vI-acca)/ Vai. Another example is the u-diffusion (Salimans & Ho, 2022), where the network predicts 02 vate - vI- açzo. -
Table 8 provides detailed breakdown of sources of data from which each audio-caption dataset is built. Compared to previous datasets, AF-Synthetic include diverse data source to construct synthetic captions, which enables strong gen- eralization to numerous audio types when training TTA model.
B.3. Optimal Transport Conditional Flow Matching
We use Laion-CLAP checkpoint to compute CLAP similarity (Wu et al., 2023), We use the following keywords to filter low-quality audio samples:
Optimal Transport Conditional Flow Matching (OT-CFM) (Lipman et al., 2022; Tong et al., 2023) is an alternative method to train diffusion models via flow matching. Instead of predicting it directly predicts the vector field f(x, t) V log (tr), leading to the following loss function:
amb ambiguous, art fact, background noise, broken up, buzzing, choppy, clipping, compremised, crackling, deficient, distant, distorted, dropout, echo, faint, faulty, feedback, flawed, fluctuating, fuzzy, garbled, gibberish, glitch, hissing, imprecise, inadequate, inaudible, incoherent, indiatinet, interior, inautricient, interference, Irregular, irrelevant, lacking, 1ew quality, low volume, low-quality, madiecre, misheard, misinterpretationn mutcled, nurmur, nolse, nolay, off-mic, overlapping speech, overmodulated, pooz, popping, reverberation, scrambled, second zale, sibilance, skipped, skipping, static, suboptimal, substandard, uncertain, uncleas, undermodsl ated, unintelligible, unknown sounds, unreliable, unsatisfactory, unspecifle, vague.
LOTCFM = [1
||vo(xe, (f - Vtt)2 log (1: (x)))))))))
B.4. Latent Diffusion Models
Latent diffusion models (LDMs) (Rombach al., 2022; Liu et al., 2023b) combine VAE with diffusion models, training the diffusion models within the latent space of the VAE. In this approach, the VAB's latent variable z serves as the target for generation. Rather than directly modeling Pdata, LDMs model the pushforward distribution PaPtata: utilizing the frozen encoder and decoder from the VAE to transition between the original data space and the latent space.
C.2. Subjective Evaluation of AF-Synthetic Captions
B.5. Classifier-Free Guidance
Table 9 shows the 5-scale REL scores of AF-Synthetic com- pared to the original baseline captions for three subsets: 1) AudioSet, where the baseline captions are drawn from Laion-633 660k, FreeSound subset of Laion-630k, along with the original captions, and 3) WavCaps, where the captions are from LLM rephrasing of the original metadata. We ran- domly sampled 100 audio-caption pairs for each subset and used the same human evaluation protocol to measure the REL scores. For all subsets we consider, AF-Synthetic gives significant improvements in text adherence by human raters, consistent with the objective results using CLAP similarity.
Classifier-Fres Guidance (CFG) (Ho & Salimans, 2022) ad- justs the balance between diversity and quality in generative models by over-emphasizing conditioning. The model is trained both conditionally and unconditionally by randomly replacing the condition e with a null embedding 0. During sampling, the guided output is given by:
vo( t|c) vg t) + weig (vo(Ie, t|e) (1 t)),
where Welg is a guidance scale. Welg = 1 disables guidance and Welg > amplifies the conditioning.
14

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
C.3. Most Similar AF-Synthetic Captions to AudioCaps and MusicCaps
D. Additional Ablation Study on ETTA-DiT
Table 13 and 14 show an extended ablation study of our ar- chitectural design from ETTA-DiT. Figure 4 shows training loss comparisons to further justify the main design choices of ETTA. Tables 15 and 16 discuss additional setups explored: setting RoPE frequency base, the use of dropout. Tables 17 and 18 provide extended study regarding model capacity.
In Table 10 and Table 11 show some captions from AudioCaps MusicCaps and their most similar captions from AF-Synthetic. These examples, together with Figure 2, demonstrate that AF-Synthetic captions are quite dif- ferent from these two datascts, which further proves the generalization ability of our ETTA that is only trained on AF-Synthetic.
Training loss comparison Figure 4 displays training loss plots over several configurations to illustrate the rationale behind each of the design choices of ETTA. The result shows that: 1) Stable Audio-DiT plateaus around 300K steps and starts to diverge early. ETTA-DiT continues to improve its quality with better loss for the same training steps. This shows clear benefits of the ETTA-DiT architecture. 2) For prolonged training (c.g. over 500K steps), v-diffusion starts to be unstable, whereas OT-CFM provides better stability up 1M steps. This shows practical advantages of OT- CFM over v-diffusion and is motivation to use for ETTA. 3) AF-AudioSet quickly diverges around 250K steps and is unable to continue its training, whereas AF-Synthetic provides better convergence with continued improvements up to IM steps, meaning that AF-Synthetic helps ETTA converge better from its scale.
Table 9: REL Scores for Ay-Stthhetic Caption Quality with 95% Confidence Interval.
RELt
AudioSet
FreeSound
WavCaps
Baseline
3.82 + 0.11
3.68 + 0.13
3.92 + 0.11
AF-Synthetic
4.04 + 0.10
3.83 0.11
4.02 + 0.10
C.4. Training Data for ETTA-VAE
Table 12 shows the training data of our ETTA VAE.
We decide to rope bape.base-16884 which can be considered as significantly "longer" than the length ETTA would usually be exposed to (up to 512 for text token embedding, and 215 for the VAE latent window). This design is inspired by recent trends in LLM where applying longer rope base during training helps improving extrapolation to longer sequence generation. Considering usual I/O length of ETTA, also tried using shorter cope_basess....... find that the early training loss is slightly better but the difference in objective metrics is small, mostly within an expected margin of error. While the shorter rase may have been sufficient, our final model uses the longer one towards scalability to longer text and audio window beyond what we have explored in this work.
RoPE frequency base
We
Different RoPE frequency base does not affect the results significantly. However, we conjecture longer value can help for models with longer window.
Dropout Although turning off dropout Paropout = 0.0 yields slightly better benchmark scores (FD scores and KLs. for example) measured at 250k training steps, we decide use Paropout 0.1 for the final model where speculate that it may provide improved generalization and enhance robustness in parameter estimation, leading to a more robust model in real-world captions beyond benchmark datasets. We do not draw a conclusion that turning off dropout is better or worse this work, and it remains to be seen if it
15

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
Table 10: Examples of captions from AudioCaps and their most similar caption from AF-Synthetic.
AudioCaps caption An airplane engine ronning.
Most similar Sy-Syathetic caption
The audio primarily features the continueos roar aireraft engine, with high-pitchod whoosh, swoosh, swish sound also present.
Multiple cars are racing, spooding and roar- ing in the distance.
The audio features distinct sounds of a race car and other racing vehicles. The race car engine the dominant sound throughout the audio, while the other racing vehicles can be heard intermittenly.
A consistent, loud mechanical motor
The audio features an aircraft engine, which produces loud, continuous, mechanical sound. The wind sound is also audible throughout the audio.
A small tool motor buzzes and an adult
The audio features man speaking internittentl, with the sound of an electric shaver running throughuus. There are also instancess of a high-pitched beoping sound.
male speaks.
mid-size motor vehicle engine is idling.
The audio primarily consists of the sound of large trick engine idlingg, with occasional engine revving sounds. There is also high frequency, sandom-frqquene content present throughout the audio.
Insect noises with people talkinng
The audio teatures a child speaking, with the sound of insects and background noise throughout. There's is also a brief sound of a buzzing, repetitive cricket.
A very short spray and then silence after
The audio contains the sound of a spark and a hiss, which are often heard when a spark is created in a gas or fluid.
Multiple dogs bark, people speak.
The audio features dog barking and yipping, along with the sound of a television playing in the background. There's is also a conversation happening, with a woman speaking at certain intervals, Additionally, there are instances a human voice and laughter.
Table 11: Examples of captions from AudioCaps and their most similar caption from AF- AF-Synthetic.
MusicCaps caption
Most similar A AF-Sytthetic caption
The low quality recording features a ballad song that contains sustained strings, mellow piane melody and soft female vocal singing over it. It sounds sad and soulful, like something you would hear at Sunday services.
The audio Teatures calming piano melody and soft vocals.
A male voice is singing a melady with changing tempos while snipping his fingers thythmically. The recording sounds like
The audio features a male voice, which is singing a cutchy melody with folk style.
been recorded in empty rom. This song may be playing, practicing snipping and singing along. This song contains digital drums playing simple gruove along with two guitars. ne strumming chords along with the snare other one playing melody on top. An e-bass is playing the footnote while piano playing a major and minor chord progression. A trumpet is playing loud melody alongside the guitar. All the instruments sound Hatt and are being played by a key Loadd There are little bongo hits in the background panned to the left side of the speakers, Apart from the music you can bear cating sounds and a stomach rumbling. This song may playing for an udvertisement.
The audio features a syuth, drums, and guitar. The synth playing repetitive melody, the drums are play- a beat, and the guitar is strumming chords.
This clip is three tracks playing consecutively. The first one an electric guitar lead harmony with a groovy bass line, followed by white noise and then female vocalisation vivacious melody with a keyboard harmony, slick drumming. lunky bass lines and male backup. The three songs are unrelated and unsynced.
The audio contains a distorted rock song, playing on top of acoustic drums. There are also sounds of crowd and clapping, which contribute to the overall energetic and lively feel of the music.
A male singer sings this grovyy melody. The song is techno dance song with a gronvy hass line, strong drumming
The audio features a strong bass and electronic drum beats, which are characteristic of this genre. There's s also the sound of a female voice singing, which adds unique element to the overall sound.
rhythm and a keyboard acompaniment. The song is so groovy and serves dance track for the dancing children. The audio quality very poor with high gains and hissing noise.
Someone is playing high pitched melody on a steel drum. die is of poor audio-quality.
The audio features a steelpan being played to music.
Low lidelity andio from live performance leaturing solo direct input acoustic guilur strummning airy, suspended open chords. Also present are occasional amhient sounds, perhups papers being shuffled.
The audio features the sustained, mellow strumming of a nylon string guitar in free time. There are also high pitched, thin strings heing plucked.
The instrumental music features an ensemble that resembles the orchestra. The melody is being played by brass
The audio features variety of strings and brass instru- ments playing a fast melody.
section while strings provide harmonic accompaniment. end the music excerpt one can hear double hass playing long note and then pereussive noise.
0.71
0.89
0.7
Stable Audio Open DIT
ETTA-DIT
1.05
1.1
1.14
1.12
1.16
AF AudiaSel
AF Synthatteic
0.66
0.35
0.67
0.9
1.08
0.66
0.35
1.06
0.65
(ateess early)
0.0
1.04
WDierress unable to bontine training)
0.81
0.75
(Unstable)
1.02
0.65
0.7
0.62
0.58
v-d ffusion
0.96
0.8
OT-CFM o992.1
200 3OOK--5KKKKKKBB 500
100k 200.200 500K 80OK 700k BOOK 9O0K
300K 400k BOOK BOOK 700K BOOK
Figure 4: Training loss comparison across three setups: (left) Stable Audio Open-DiT vs. ETTA-DiT, using the same v-diffusion objective and Ay-Sttthttic training dataset. (center) v-diffusion vs. OT-CFM, using the same ETTA-DIT architecture and AF-Synthetic training dataset. (right) AF-AudioSet vs. AF-Synthetic, using the same OT-CFM training objective and ETTA-DiT architecture.
would help or not as we scale data and model further.
Scalability with Model Size while CFG turned on Ta- bles 17 and 18 show additional result on the model size scaling experiment using Wefz = or Wefz = 3. Compared to Wefg = 1, the difference of metrics between model of dif- ferent sizes is smaller. This suggests that while the quality
Dropout does not affect the overall results significantly. We speculate that adding dropout could enhance robust- ness in parameter estimation as we scale the TTA models.
16

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
Table 12: Datasets used for training ETTA-VAE.
Dataset
URL
HIFL-TTS
ptt /wwp/////ww///////////// ogeneeee-eeeleeeoo.eo.ooe. -org///g/ 109/
ASS-PODCAAT-PPulis
https: 2022://cce /ecs ut.aallaaa sdu/rssearch http///sssccchhaaaeceeeeee
STWIS
httpp: /dsteeeeeeeeeeee.ee ad,ac, wk/handl e/1lle/022225 83/2533333
Spanish-HQ
TTS- TTS-Portugume Corpus
https: http::///e /gtthu/ com/ Sasresson/II----- www.ii----tugeeeeeeeee- Corpae
VCTK
hit httpp:: 1/ ////////aaa the ed.ac ac. ok/ wik/har 2/102///0/////
hi pappps thattth-oooooo - www///ybyynggggggoocoooo Fess10
indic Saggages--ts
nttp:// /ttp:///s.eee.ooe/ wtt/dstabaees////eoooo.t/
12anatic
pa //tp////////// engr- corr-tom amm edda/ 1t//112-mmmmm co-oopuss
CREMA-D
attps: /tttt///gthh.../ B000aacc-.o0
emow-dh
nttps: httgg/hthhhoo/o
ji-oepus
htt httpps//ggtht..../
Corpus
ravdess
https: www.kkkg.....oooooo wwt.//c.cc.//////.ooo wtttttttttttttsoos http///tttttt//t//eeeee/e// wwwpkaggios/raaddseeese petedddos-onoiiionooooo inntional wonoonappppeeeeeo aposct-auddo
less
nttpp: /wwss Kagg 10 wwt.///tttsaoo.oooos sst//ttt///ssssessse/sssss 1okkI/toront o-enot i onn speech- set- teae
Andioser
httpp:: http://reseaacccco gong www./adi/iaoooooooooooooo oset /down bod. htm)
Laion630k (2022)
httpps//gittua.... wwtoo///oO-A///////
dacaset
Clothe-A2 AQA
nttps://200 http:///e0000.0o00 org/
http:oooorrgr//ooooooooo rt/or//cood///e 732gV
Clofhe-22
http:://gtee http:///gtths... http:tthsccco//////o www./////////di 6- cept toni loni ng///l ael ot det sieet /tst//t ree /naster
CochiSonee
bttps:/ http://gtttttttte httpshusocccoccccoooo
DCASEI
nttps://dese http:///aaecccmmmmttt//aoooaeoo/
//taat--a-t -atge--arge-----e-eeee sos eourd-event pereet--eeectioo--eeeeee
ESC-50 50
https://citttbb. httpshhwwccco//sooooppaaaaaaaaaa
FMA
attps: http::///gtttaa..o wtt.////////////n/mm
FSDAOK
httpp: http:///ee000..ro/
JEMOCAP
MACS
MELD
bttps://olthus
MU-LLAMA
coceoonne------- ov-tile
Algnn
https//hhhogggaaggttaaaee dacaset
Medley solos DB
bttps:
Music-AVQA
MusicNet
betps: http://wwagaocco//aaa22///// htt.://ccce/.aacattt//aaaaaaaaaa
date-datasas http:///wwo///aa////////////////
NSynth
nttps: /ttp///ttt........ooooooooo
NonSpecch7
nttps:/ /ttp:///ooooorrg/eeooo//e http://ecoddd/////////
6961442
OMGEmotion
bttps: wtt.::/......maaaaa..... wnd-hanbuo.
OpenAQA
https:/ httpp////othtttaccooot wttp://ooooooooooceccooee
readne-ov- -ot--oooooooooogg------.. wtrrigg---tttt-e
SONYC-LST
https: http://eeooddocco 2two/d//o////////// 3566543
SesedDeses
https: /ttp:/ttt.ccoooooo watoooo////ddde (eoodd--2200 sotvieval
-benchnark.
UrbanSwurd8KK
bttps: /ttp///aanssossaattaaaa woc.s..........o........
wcn./abbanooundk.cco
VocalSound
bttps://oitie htt.stttaacoooo/// wttpa////aaaoooa/////aaaaaaaa
Way TexLSK
AudhoCaps
htttps://gtthaa.. http:/cccccccccccc
bttps:// www.soouodddoooooooo
sk/p feot honee-hooe--ee wewhoooon-tttt dataset-aat-ooo -ele--nnnntttitt tow----aaaa----aattttoo-------e
https:/ httppa///hugcaacaac... http///aaattt//////o//
common-accant
meestrov33
Lplpp: /tt.///agg/....eeeeeee Leaate Clow
mag janejonndddo
https: /tttha/ ctctstaaauccoo//// www./oonn//////ee/t /ttp//////atteeeeeeeee/eeeeeeeee Janendo- dataset.
MUSDB
bttps:/ /ttpp/////ccoooo.oo wttp:///////////////
Table 13: Improvements by adding each of the major design choices of ETTA (evaluated on AudioCaps).
Ablation
FDp |
FDo
KLs 4
KLp L
ISpt
CL,
CLM 1
Stable Audio Open
47.10
127.82
3.14
3.13
6,81
0.18
0.24
+ Sy-Stnthctic
37.40
125.33
2.45
2.69
5.37
0.28
0.29
+ ETTA-DiT
28.20
92.31
2.07
2.19
6.04
0.37
0.33
+ OT-CFM, 1(0,1)
30.39
89.44
2.03
2.26
5.48
0.37
0.31
+ (NV(0
28.46
89.60
1.99
2.21
5.64
0.37
0.32
Table 14: Improvements by adding each of the major design choices of ETTA (evaluated on MusicCaps).
Ablation
FDp |
FDo
KLs 4
KLp 1
ISpt
CL,1
CLM
Stable Audio Open
39.96
119.54
1.81
2.11
3.19
0.34
0.41
AF-Synthetic
26.22
127.90
1.57
1.73
2.37
0.39
0.43
+ EITA-DIT
20.48
100.53
1.38
1.50
2.21
0.42
0.45
+ OT-CFM, 1(0
22.16
98.84
1.35
1.49
2.10
0.42
0.45
~~ ((((0
21.59
92.30
1.41
1.51
2.20
0.41
0.45
of model grows with its total size, small models can also generate high-quality samples with CFG at cost of having potentially lower diversity.
Classifier- guidance helps smaller models to be closer to large models in objective metrics.
Full results on the sampling methods Table 19 and Fig- ure show the full results on the effect of sampling methods,
17

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
Table 15: Ablation study on the effect of other architectural designs of ETTA on generation quality (evaluated on AudioCaps).
Ablation
FDp
FDo
KLs 4
KLp
1Sp
CL.11 t
CLar
ETTA
13.01
81.23
1.29
1.50
12.42
0.52
0.41
ETTA + rope base 512
12.64
79.49
1.32
1.51
12.45
0.52
0.41
ETTA Páropout. 0.0
13.04
76.30
1.28
1.50
12.27
0.53
0.41
Table 16: Ablation study on the effect of other architectural designs ETTA on generation quality (evaluated on MusicCaps).
Ablation
FDp
FDo +
KTs
KT.p..
1Sp
C
CLar
ETTA
12.15
96.46
0.88
1.08
2.93
0.51
0.52
ETTA + rope base = 512
12.11
95.04
0.84
1.08
2.94
0.51
0.52
ETTA + Pilropont = 0.0
11.75
88.74
0.75
1.10
2.97
0.51
0.51
Table 17: Ablation study the results of ETTA with different depths, widths, and kernel sizes (evaluated AudioCaps). The classifier-frve guidance Wefg 1. * Our best model choice.
Model
Size(B)
FDp |
FDo
KLs 4
KLp
1Sp
CLE f
CLM f
depth = 4
0.38
36.46
103.35
2.15
2.39
4.88
0.33
0.30
depth 12
0.81
29.48
93.13
2.05
2.28
5.73
0.36
0.32
depth 24"
1.44
28.46
89.61
2.00
2.22
5.65
0.37
0.32
depth 36
2.08
27.08
82.60
1.95
2.18
5.87
0.38
0.32
width 381
0.28
35.97
100.58
2.14
2.43
4.99
0.33
0.30
width 768
0.52
31.03
93.74
2.04
2.29
5.49
0.36
0.32
width 1536*
1.44
28.46
89.61
2.00
2.22
5.65
0.37
0.32
ReonvET = 1'
1.44
28.46
89.61
2.00
2.22
5.65
0.37
0.32
RF =
2.35
28.72
82.49
2.04
2.28
5.84
0.36
0.31
Table 18: Ablation study the results of ETTA with different depths, widths, and kernel sizes (evaluated AudioCaps). Our best model choice.
The classifier-free guidance Wefg 3.
Model
Size(B)
FDp |
FDo
KLs
KLp 4
ISp
CLL f
CLM f
depth =
0.38
15.81
81.71
1.41
1.53
11.35
0.50
0.40
depth 12
0.81
13.88
83.62
1.36
1.54
12.52
0.52
0.41
depth 24"
1.44
13.01
81.23
1.29
1.50
12.42
0.52
depth 36
2.08
12.37
75.51
1.30
1.49
12.24
0.52
0.40
width 384
0.28
16.08
76.01
1.40
1.59
11.31
0.49
0.39
width 768
0.52
14.32
77.97
1.33
1.55
12.62
0.51
0.40
width 1536"
1.44
13.01
81.23
1.29
1.50
12.42
0.52
RconWPF =
1.44
13.01
81.23
1.29
1.50
12.42
0.52
0.41
ReoNEEF = 3
2.35
13.87
78.74
1.38
1.56
11.71
0.49
0.40
Table 19: Results choice of sampler and number sampling steps using AudioCaps test model trained for IM steps and Welg
We used the main ETTA
3.
Sampler
Steps
NFE
FDp J
FDo
KLs
KLp 1
ISp
CLL1
CLM
Heun
100
199
12.09
79.60
1.20
1,41
13.57
0.55
0.42
Heun
50
99
12.00
79.24
1.19
1.41
13.61
0.55
0.43
Heun
49
12.20
80.06
1.18
1.40
13.64
0.55
0.43
Heun
10
12.22
85.27
1.21
1.40
13.23
0.55
0.43
Heun
9
13.27
97.45
1.27
1.43
12.22
0.52
0.42
Euler
200
200
12.26
79.77
1.19
1.41
13.56
0.55
0.43
Euler
100
100
12.10
80.67
1.18
1.42
13.90
0.55
0.43
Fuler
50
50
11.83
81.49
1.18
1.39
13.45
0.55
0.43
Euler
20
20
12.36
90.85
1.19
1.39
13.15
0.54
0.42
Euler
10
10
14.74
112.65
1.31
1.43
11.46
0.50
0.42
including the solver, number of function evaluations, and classifier-free guidance.
Heun sampler is better than Fuler at lower NFE under all metrics. Increasing Wefg improves most objective metrics (KL, IS, and CL) except for FD.
18

---

ETTA: Elucidating the Design Space of Texi-to-Audio Models
Euler
Heun
+14
13
14
12
20
24
16
15
18
12
12
10
0
25
50
75
100
25
50
75
100
2
6
2
NFE (AutliCCaps)
NFE (MusicCaps)
CFG (AudioCups)
CFG (MusicCups)
0.9
2
1.3
1.4
1.5
1.1
1.2
0.8
0.8
0
50 NFFE (AutliCaaps)
25
75
100
0
25
50
75
100
2
6
2
6
NFE (MusicCups)
CFG (AudioCaps)
CFG (MusicCaps)
16
14
3.1
13
3.5
2
12
11
2.9
3
10
2.5
3
2
25
50
75
100
25
50
75
100
2
2
NPE (AudioCaps)
NPE (MusicCaps)
CFG (AudioCaps)
CFG (MusicCaps)
0.07
0.51
0.56
0.51
0.054
0.5
T 0.61
0.48
0.45
0.45
0.4
0.42
0.48
0.45
0
25
50
75
100
0
25
50
75
100
2
4
2
6
NIFE (AudioCaps)
NFE (uusicCaps)
CFG (AudioCaps)
CFG (MusicCaps)
Figure 5: The effect of different sampling methods on the generation quality of ETTA on AudioCaps and MusicCaps. We
investigate both Buler and Heun solvers. NFE: number of function evaluations. CFG: classifier-free guidance scale.
Table 20: List of imaginative captions used to generate creative audio.
Caption
A hip-hop track using sounds from a construction site hammering nails as the heat, drilling sounds as scratches, and metal clanks as rhythm accents.
A saxophone that sounds like meuwing of cal.
A techno song where all the electronic sounds are generated from kitchen acises blender whirs, toaster pops, and the sizzle cooking.
Dogs barking, birds chirping, and clectronic dance music.
Dog barks a beautiful and fast- paced folk melody while several cats sing chards while meowing.
A time lapse of city evalving over a thousand years, represented through shilting musical genres blending seamlessly from ancient to futuristic sounds.
An underwater city where buildings hum melodies currents pass through them, accompanied by the distant drumming of biolumineseent sea creatures.
A factory machinery that sereams in metallic agony.
lullaby sung by robotic voices, accompanied by gemtle of electric currents and beeping machines.
A soundscape with choir of alarm siren from an ambulance car but to produce a lush and calmn choir composition with sustained chords.
The sound of ocean waves where each crash is infused with a musical chord, and the calls of seagulls are transformed into flute melodies.
Mechanical flowers blooming dawn, each petal unfolding with soll chime, orchestrated with the gentle ticking of gears.
The sound metoor shower where each falling star cmits unique musical note, creating a celestial symphony night sky.
A elock shop where the ticking and chiming of various timepieces synchronize into complex polyrhythmic composition.
An enchanted library where each book opened releases sounds of its story adventure tales bring drum heats, romances evoke violin strains.
A rainstorm where each raindrop hitting different surfaces produces unique musical pitches, forming an unpredictable symphony.
A carnival where the laughter of children and carousel music interwine, and the sound of games and rides blead into festive overture.
A futuristic rainforest where holographic animals emit digital soundscapes, and virtual raindrops produce glitchy electronic rhythms.
echo inside a cave where droplets water create a cascuding kylophone melody, and bats' echolocation forms ambient harmonies.
A steampunk cityscape where steam engines pufl' in thmm, and metallic gears turning produce mechanical melodies.
Creative captions Table 20 contains the creative captions for subjective evaluation.
19

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
E. Extended Evaluations
Pretraining with the audio inpainting task produces mixed results, possibly due to data imbalance or sub-optimal implementation details.
In this section, we present an extended model evaluation including additional out-of-distribution (OOD) benchmark on SongDescriber dataset (Manco et al., 2023). Table 21 and 22 show an extended ablation study on the choice of training dataset. Table 23 includes the OOD evaluation results compared to baseline models. ETTA achieves the lowest FDo and FDp as well as the highest CLAP scores compared to baseline models. With only synthetic captions, ETTA matches the REI. of Stable Audio Open which is trained on numerous music data and captions. Table 24 compares the AF-AudioSet and AF-Synthetic training sets on the OOD benchmark. Results indicate that ETTA trained with the larger AF-Synthetic leads to significantly better OOD generation results than the smaller AF-AudioSet.
While the current experimental setup did not bring posi- tive result, we believe that introducing multiple tasks into single model will enable a generalist model. We leave exploring alternative ways to ingest the inpainting task into better TTA to future work.
Choice of Text Encoder Many previous works have im- plemented different text encoders for TTA, but the results are mixed. Researchers have experimented with various models such as BERT, T5, and CLAP to find the optimal text encoder for improving TTA result (Liu et al., 2023b; 2024; Ghosal et al., 2023; Melechovsky et al., 2023; Ma- jumder et al., 2024; Huang et al., 2023c;a). We also explore the text encoder choice in controlled environment, where we train multiple models with different text encoders. We consider T5-base, FLAN-T5-large T5- large, FLAN-T5 T--bas,, and In addition, we experiment with a dual text encoder setup (Huang et al., 2023a; Liu et al., 2024; Haji-Ali al., 2024) by using CLAP as additional global text embedding. We use a different CLAP checkpoin (LAION's music music_audionn awwiosettttt............ wwwioset.epoch.................. 15 eesc_90...... .141) to the benchmark CLAP models (CL,, and CL.) to rule out possibility of inflated result from the same representa- tion. In this experiment, we start training with a pretrained weight from the inpainting task for 700k training steps, II and trained each model with different text encoder for 300k steps.
F. Mixed or Negative Results
In this section, we document additional directions we plored when building ETTA inspired by previous work, but resulted in mixed or worse results in our study. Our goal is not to claim that the methods described below don't 't work: again, we aim to provide a holistic understanding of design choices commonly found in the TTA literature and speculate that these have been ineffective specific to our experimental setup. We believe that below methods we explored hold the potential to improve results further in future work.
Pretraining TTA with audio inpainting This experiment is inspired by SpeechFlow (Liu et al., 2023a) that presented improvement of various speech tasks (e.g.. Text-to-Speech (TTS)) by pretraining the flow matching model with an inpainting task using unlabeled data.
We follow the masking method in (Liu et al., 2023a) and concatenate the masked feature with the noisy input. Note that we do not feed the masked feature to cross -ooss-attention input, so the cross-attention parameters are not activated dur- ing pretraining. We pretrain the model with this inpainting task for 700k steps. Then, we reset the first input projection layer of DiT and optimizer, and switch to the main TTA task starting from the pretrained weight. We observe that the training loss starts much lower for the pretrained model.
Table 26 summarizes the result on different text encoder choices evaluated on AudioCaps. Unfortunately, we were not able to discover noticeably better choice compared to others. Nevertheless, we find interesting observations: 1) TTAN-T5-bass scores relatively better than T5-base for FDo. but the opposite can be observed for other metries such as KLs. 2) for our setup, we have not found strong evidence that dual text encoder with CLAP is better; it worsened most metrics except for PLAN-T5-1arge 3) larger T5 encoder may not necessarily be better in improving results, where base model generally scored better metrics than 1 large model. T5-1arge showed surprisingly worse result compared to others for two independent training runs (with or without CLAP). While this seems counter-intuitive. it also suggests that the optimal choice of text encoder would depend on other factors such as training dataset and the main TTA model capacity at hand.
Table 25 summarizes the benchmark results with or without the inpainting as pretraining task. We find that the result is mixed where AudioCaps result worsened and some Mus- icCaps metrics improved such as FD and IS. We speculate that the pretrained weight focuses more on the music sig- nal because of our unlabeled audio collection has higher proportion of music compared to speech. We also conjec- ture that the result would be different if we use the masked feautre to the cross-attention input in pretraining stage in- stead of concatenation.
We launched this experiment based on the preliminary obser- vation of the lower training loss. We speculate that the observation would not change if we train the models from scratch.
20

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
Table 21: Ablation study on the results of ETTA trained on different datasets (evaluated on AudioCaps).
Dataset (million captions)
FDp
FDo +
KLs
KLp
ISp t
CL
CLy
AudioCaps (0.05)
22.60
95.99
1.49
1.63
6.73
0.48
0.35
TangoPronptBank (1.21)
33.44
77.07
2.39
2.72
4.64
0.29
0.27
AF-AudioSet (0.16)
25.06
108.31
1.81
2.01
6.32
0.42
0.34
AF-Synthetic (1.35)
28.46
89.60
2.21
5.64
0.37
0.32
Table 22: Ablation study on the results of ETTA trained on different datasets (evaluated on MusicCaps).
Dataset (million captions)
FDp
FDo +
Kl..
KT.
1Sp t
CL
CLy
AudioCaps (0.05)
76.14
279.44
3.20
3,63
2.05
0.12
0.27
BangoPromptBank (1.21)
24.72
86.17
1.73
2.02
2.27
0.35
0.38
AF-AudioSet (0.16)
21.40
107.00
1.45
1.52
2.36
0.40
0.44
AF-Synthetic (1.35)
21.59
92.30
1.41
1.51
2.20
0.41
0.44
Table 23: Additional results of ETTA compared to SOTA baselines on SongDescriber.
Model
FD+1 |
FDo
KLs
KLp |
ISp f
CLi,
CLM
OVLt
REL1
Ground Truth
1.88
0.48
0.38
4.37 + 0.07
4.26 + 0.09
AMd
16.02
335.37
0.74
0.78
1.93
0.42
0.45
AM2raL.DM2--arge
10.50
324.38
0.67
0.75
1.95
0.44
0.48
3
3.38 + 0.11
TANGO-AF
21.49
233.32
0.79
0.88
1.96
0.43
0.44
3.32 1.32+0.10 0.10
3.36 + 0.11
Stable Audio Open
34.76
129.88
0.99
1.01
2.19
0.42
0.47
3,92 192+0.0
3.80
+ 0.10
ETTA
9.98
95.66
0.80
0.76
2.15
0.44
0.53
3.70 0.09
3.79 0.10
Table 24: Ablation study on the results of ETTA trained on different datasets (evaluated on SongDescriber).
Dataset (million captions)
FDp J
FDo |
KLs
KLp 1
1Sp
CL,,7
CLM
AF-AudieSet (0.16)
12.97
125.16
1.03
0.89
2.36
0.41
0.50
AF-Synthetic (1.35)
10.29
104.16
0.80
0.76
2.06
0.43
0.51
Table 25: Results on pretraining with the audio inpainting task vs. training from scratch. In either case, ETTA is trained on the TTA task for 250k steps.
Dataset
Pretrain
FDr
FDo4
KLs
KLp
ISp
CL,
CLy f
12.90
90.77
1.40
1.54
11.87
0.49
0.40
AudioCaps
x 13.01
81.23
1.29
1.50
12.42
0.52
0.41
10.87
81.19
0.91
1.10
3.03
0.51
0.50
MusicCaps
X
12.15
96.46
0.88
1.08
2.93
0.51
0.52
Table 26: Effects of different text encoders in ETTA (evaluated on AudioCaps). We initialize weights from checkpoint that
pretrained on the audio inpainting task for 700k steps and train each model on the TTA task for 300k steps.
Eners
Encalap
FDp +
FDo
KLs
KLP +
ISp t
CL.,² 1
CLM 1
75 base
12.93
91.81
1.42
1.57
12.57
0.49
0..0)
75 ('5-base
12.87
85.40
1.42
1.58
11.91
0.48
0.39
1rrg
16.80
213.71
1.61
11.91
0.45
0.36
75-Large
V
18.73
219.93
1.67
1.74
9.44
0.43
0.35
FLAN-T5-ban
X
13.01
80.67
1.50
1.61
13.26
0.48
0.40
13.51
84.08
1.50
1.62
11.37
0.47
0.39
FLAN-T5-large
15.94
103.15
1.63
1.70
10.60
0.45
0.38
FLAN-T5 LLAN-T5-large 1arge
13.28
81.64
1.52
1.63
11.80
0.47
0.39
No single text encoder consistently outperformed others. The effectiveness of text encoders seems depend on specific metries and setup. Larger text encoders do not always lead to better results.
Autoguidance Recently, (Karras al., 2024) showed that the improvement in perceptual quality of CFG stems from its ability to eliminate unlikely outlier samples, but it may reduce diversity from over-emphasis. They proposed a new way of guiding the model, called autoguidance, that uses bad version of the same model (either by under-
21

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
Table 27: Results on AutoGuidance (evaluated on AudioCaps). We use our best 1.44B ETTA model (trained for IM steps). Modelag denotes the bad model used for AutoGuidance. Same: same 1.44B model architecture as ETTA. XS: the smallest 0.28B model using w 3
Modelag (steps)
Wiefg
Wag
FDp J.
FDo
KLs 4
KLp
ISpt
CL,
CLM 1
25.33
93.03
1.77
2.00
6.41
0.42
0.34
XS (50k)
1
2
14.51
86.14
1.63
1.73
9.10
0.51
0.38
XS (50k)
3
2
12.15
80.24
1.20
1.41
13.64
0,55
0.43
XS (10Dk)
2
14.19
83.52
I.63
1.72
8.54
0.50
0.38
XS (10DR)
3
2
14.15
94.08
1.37
1.49
13.83
0.55
0.42
Same (10Dk)
1
2
16.49
91.79
1.58
1.78
7.63
0.48
0.37
Same (100k)
3
2
12.72
81.85
1.27
1.50
13.80
0.56
0.42
3
12.10
80.67
1.18
1.42
13.90
0.55
0.43
Table 28: Results on AutoGuidance (evaluated on MusicCaps). We use our best 1.44B ETTA model (trained for 1M steps). We report the results using the best combination according to Table 27.
Modelag
Wefa
Wag
FDp J
FDo
KLs
KLp
1Spt
CL.f
CLM f
19.89
101.15
1.28
1.43
2.21
0.42
0.46
XS (50k)
1
2
13.17
104.43
1.12
1.32
2.59
0.48
0.49
XS (50k)
3.
2
9.83
97.63
0.78
1.03
3.19
0,50
0.53
3
9.82
98.19
0.78
1.03
3.18
0.50
0.53
training and/or with smaller model) that increases diversity while ensuring high-quality output as follows (omitting the condition e for brevity):
so we believe autoguidance holds its potential towards cre- ativity. We leave exploring recently proposed methods for sampling from the model for better TTA results in future work.
AutoGuidance increases diversity but does not consis- tently outperform CFG objective metrics. It shows potential for diversity, though its effectiveness is sensitive model and scale choices.
vo(2,, t) = U9 (Ie, t) + Wag (uo(xe,)) t) U8As (xes t)),
where Oug denotes a bad model and Wag is the scale for autoguidance. Same as CFG, Wag = 1 disables the guidance and Wag > 1 amplifies the main model's prediction.
Min-SNR-y training strategy (Hang et al., 2023) We use
7 5 per convention, and trained ETTA-DiT for 250k steps with the u-diffusion objective. Results can be found in Table 29 and 30 (weig = 3). Most metrics became worse, but and FD o and P are slightly better music generation.
We conducted experiments applying autoguidance to eval- uate its effectiveness to our TTA setup. The results are in Tables 27 and 28. From our grid search of Wag from I to 2.5 with 0.25 interval, Wag 2 provided the best possible metrics.
CFG on a limited interval (Kynkäääniemi al., 2024) We remove CFG for the initial 40% of diffusion steps and then apply CFG for the remaining 60% steps. We inference and evaluate both with and without AutoGuidance. Results can be found in Table 31 and 32. Most metrics became worse, but is slightly better in audio generation using the small (XS) model.
Subjectively, we observed that while autoguidance could produce more diverse audio samples corroborating (Karras et al., 2024), but these samples sometimes lacked realism. We find that the method is sensitive to the choice of the bad model and its guidance scale Wag. In terms of improv- ing benchmark results, despite our best efforts and various combinations including different bad models (either under- trained versions or smaller models) and guidance scales, we were unable to identify a setup that clearly outperforms plain CFG with Weig 3. Similar benehmark metrics could only be achieved by combining both CFG and autoguidance, but at an increased cost with 2x NFE.
We conjecture that our search space may have been incom- plete. However, we do observe noticeable increase in diver- sity from autoguidnace where the same ETTA checkpoint can sometimes generate even more "interesting" samples,
22

---

ETTA: Elucidating the Design Space of Text-to-Audio Models
Table 29: Additional Results on training strategies (evaluated on AudioCaps).
Ablation
FDp
FDo
KLs
KLp
ISpt
CL
CLy
Stable Audio Open
38.27
105.88
2.23
2.32
12.09
0.35
0.34
+ AF-S AF-Synthetic
18.50
86.13
1.58
1.74
14.96
0.47
0.40
ETTA-DiT
16.43
90.26
1.29
1.47
14.49
0.53
0.42
Min-SNN-- (r 5)
18.00
100.86
1.36
1.56
13.85
0.52
0.40
Table 30: Additional Results on training strategies (evaluated on MusicCaps).
Ablation
FDp
FDo
KLs
KTp
ISp
CL
CL
Stable Audio Open
36.42
127.20
1.32
1.56
2.93
0.48
0.49
+ AF-Synthetic
14.59
103.59
1.00
1.20
3.19
0,50
0.52
ETTA-DiT
12.48
98.19
0,82
1.06
3.30
0.50
0.52
Min-SNR-7 ( 5)
13.04
97.44
0.89
1.12
3.66
0.50
0.50
Table 31: Additional Results on Guidance on Limited Interval (evaluated on AudioCaps).
Modelag (steps)
Wefg
Wag
FDp
FDo
KLs
KLp
ISp 1
CL.
CLM f
25.33
93.03
1.77
2.00
0.42
0.34
XS (50k) CFG @ [0, 0.6]
2
12.15
80.24
1.20
1.41
13.64
0.55
0.43
2
11.74
89.18
1.44
1.59
10.59
0.54
0.40
12.10
80.67
1.18
1.42
13.90
0.55
0.43
+ CFG [0, 0.6]
16.13
93.28
1.45
1.66
8.35
0.48
0.38
Table 32:
Additional Results on Guidance on Limited Interval (evaluated on MusicCaps).
Modelag
Wefg
Fag
FDr
FDo
KL.
KT.
ISp t
CL.
CLy f
19.89
101.15
1.28
1.43
2.21
0.42
0.46
XS (50k) CFG [0, 0.6]
9.83
97.63
0.78
1.03
3.19
0.50
0.53
11.40
102.66
1.02
1.26
2.79
0.49
0.50
9.82
98.19
0.78
1.03
3.18
0.50
0.53
CFG @ [O, 0.6]
16.43
100.18
1.12
1.27
2.37
0.44
0.48
G. Vocoder/Auttoencoder Reconstruction Results
Table 33 and 34 show objective results of our VAE we used (ETTA-VAE) in this work. Our 44kHz stereo VAE is identical to the one used in Stable Audio Open (Evans et al., 2024c), but trained from scratch using our large-scale unlabeled audio collection based on public datasets. We also attach Big VGAN-v2 (Lee et al., 2023), the state-of-the-art mel spectrogram vocoder in 44kHz mono, reference of waveform reconstruction quality from the models.
Despite being 4x lower in latent frame rate (21.5Hz) com- pared to conventional mel spectrogram vocoder (86Hz), ETTA-VAE shows competitive reconstruction quality. It matches the quality of Stable Audio Open-VAr VAE on music data (MUSDB 18-HQ (Rafi al., 2017)) and outperforms on speech data (LibriTTS (Zen et al., 2019)), because our dataset contains considerably high portion of speech signals.
Our ETTA-VAE matches or exceeds the reconstruction quality of Stable Audio Open's 's VAE. This is because we use larger-scale public audio datasets.
23

---

ETTA: Elucidating the Design Space of Texioto-Audio Models
Table 33: Comparison of waveform vocoder/auto--encoeer on LibriTTS (dev-clean and dev-other).
Model
Framerate
PESQ1
UTMOS1
VisQOL
M-STET
SI-SDR1
Ground Truth
4.64
3.86
4.73
BigVGAN
86 Hz
4.14
3.73
4.69
0.71
-7.86
Stable Audio Open- VAE
21.5
2.75
3.13
4.31
7.15
ETTA-VAL VAE
21.5 Hz
3.18
3.76
4.37
0.79
9.92
Table 34: Comparison of waveform vocoder//autoenccodrr on MUSDBI8-HO test set.
Model
Framerate
ViSQOLT
M-STET,
S1-SDRI
Ground Truth
4.73
BigVGAN-v2
86 Hz.
4.63
0.94
222.66
Stable Audio Open-VAE VAE
21.5 Hz
4.25
1.00
9.34
ETTA-VAL
21.5 Hz
4.27
0.95
10.59
24


Reviews:
[{"review_id": "ztNQd92cUX", "reviewer": "Reviewer_8RJt", "summary": "The paper provide an extensive analysis on the design choices of TTA models, achieving much superior quantitative performance to baselines across most metrics. The authors provided extensive results showing the superiority and generalization of their method as well as extensive ablations justifying their choices. Additionally, the paper present a large scale synthetic dataset which the authors has verified its effectiveness in TTA generation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review and appreciating our large-scale extensive study on the design choices of TTA to reach state-of-the-art results.\n\n**Q: AutoCap has open-sourced a dataset of size 40+ millions.**\n\nA: Thank you for mentioning the status of this concurrent work. We will discuss this concurrent work in the final version of the paper.\n\n**Q: Subjective evaluation is close or underperform baselines (e.g Table 3, Table 22).**\n\nA: For music, Stable Audio Open is trained on high quality music from Freesound and Free Music Archive (FMA) datasets, but many of these samples are not available as of today. In contrast, ETTA is not trained on any studio-quality music datasets (not even MusicCaps). As a result, ETTA has a slightly worse OVL (which only measures audio quality and does not consider text) on MusicCaps and SongDescriber.\n\nWhile only trained on synthetic captions, ETTA outperforms Stable Audio Open on the REL (audio-text relevance) score on AudioCaps and MusicCaps (Table 2-3), and matches it on an OOD testset SongDescriber (Table 22). \n\n**Q: The authors have not promised the release of their checkpoints or code.**\n\nA: We will release the model code and data preparation methods to reproduce the result of ETTA models.\n\n**Q: The paper lacks a clear discussion on the difference between the dataset collection pipeline of AF-Audioset and AF-Synthetic.**\n\nA: We scaled-up the data collection method proposed in AF-AudioSet [1], and added several filtering methods (Section 3.1 and Appendix C.1).\n\n[1] Kong, Zhifeng, et al. \"Improving text-to-audio models with synthetic captions.\" arXiv preprint arXiv:2406.15487 (2024).\n\n**Q: How much of the initial dataset is filtered with CLAP and others?**\n\nA: The total filtering rate is about 73.5% (acceptance rate 26.5%) under a CLAP threshold of 0.45 and other filtering methods.\n\n**Q: AF-audioset performance is comparable with AF-synthetic in Table. 5, despite having a smaller size. While AF-synthetic achieves better performance in Tab. 23.**\n\nA: This is a great question. First of all, we find a scaled-up dataset with AF-Synthetic shows much better results for out-of-distribution data (Table 23) (Note that AudioCaps and MusicCaps are considered in-distribution, derived from AudioSet).\n\nAF-Synthetic brings other practical benefits during training as well: we found that training ETTA using AF-AudioSet diverged after 250k training steps, whereas AF-Synthetic provides stable training up to 1M steps without instabilities.\n\nWe added the training loss curves of AF-AudioSet vs. AF-Synthetic along with the generated samples using the OOD challenging captions to the demo webpage, to better showcase the necessity of AF-Synthetic for generalization. Link: https://anonymous.4open.science/r/etta_demo-778A/index.md", "claims_and_evidence": "While the paper does not propose any methodological components, the authors supported all of their claims on the best practices to train TTA with convincing experiments.", "methods_and_evaluation_criteria": "The authors evaluated their design choices using standard evaluation protocols of TTA on ambient sounds (AudioCaps) and music generation (MusicCaps), reporting metrics that shows the superiority of their methods in sound generation as well as text-audio alignment.", "theoretical_claims": "The paper does not present any theoretical claims.", "experimental_designs_or_analyses": "The author analyses and experimental designs are sound.", "relation_to_broader_scientific_literature": "While the paper does not propose any novel methodological components,  the authors has done extensive study on the design choices of TTA generation, showing superior performance. Such studies are missing from most prior work and is very useful to the community.", "essential_references_not_discussed": "The authors has discussed the most essential references in TTA, and dataset creation.", "other_strengths_and_weaknesses": "In summary, the following are the main strength and weaknesses of the paper\n \n**Strengths** \n- The authors provided extensive analysis on the design choices of TTA generation\n- The paper presents significantly improved performance in TTA generation compared with the open-sources models. \n- The authors presented a large-scale synthetic dataset which will be useful to the community if released.\n- The paper is well written.  \n\n**Weakness**\n- Despite providing many analyses on design choices of TTA, the paper lacks any significant novel contribution to TTA. \n- Some claim in the papers are not well addressed. The authors claim that AF-Synthetic is the first million-size synthetic caption dataset, while AutoCap [1] has open-sources a dataset of size 40+ millions.\n- While the authors has presented significant quantitative improvements, their subjective evaluation is close on underperform baselines (e.g  Table 3, Table 22) .\n- The authors has not promised the released of their checkpoints or code. Considering that the paper is focused around building high quality audio generation model by adapting the best practices, this could largely impact the level of the contribution of the paper.\n- The paper lacks a clear discussion on the difference between the dataset collection pipeline of AF-Audioset and AF-Synthetic.", "other_comments_or_suggestions": "The conclusions of the papers could be summarized better in experiment sections, while leaving details on hyperparameters to the supplementaries. This could make the technical details of the papers easier grasp.", "questions_for_authors": "- How much of the initial dataset is filtered with CLAP and others? \n- AF-audioset performance is comparable with AF-synthetic in Table. 5, despite having a smaller size. While AF-synthetic achieves better performance in Tab. 23. I am wondering what would be the performance of AF-synthetic at a similar scale. Have the authors perform any experiments that shows the impact of data quantity over the generation quality?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a4g8SAKHTE", "reviewer": "Reviewer_okEU", "summary": "This paper explores the design space affecting text-to-audio generation models. Specifically, the authors analyze the effects of dataset quality and scale, architectural and training/inference design choices, and sampling methods during inference. For this purpose, a new large-scale synthetic dataset, AF-Synthetic, is constructed, and several architectural ablations are performed based on existing text-to-audio models. Extensive experiments indicate that both dataset size and quality are important, with quality having more impact. Additionally, improving DiT and applying OT-CFM result in more stable training and improved outcomes.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We address your concerns as follows:\n\n**Q: Table 5 indicates that performance with AF-AudioSet is comparable or even superior… It remains challenging to conclude that AF-Synthetic is crucial for model improvement… This observation questions the necessity of AF-Synthetic compared to AF-AudioSet.**\n\nA: We showed that a scaled-up dataset with AF-Synthetic shows much better overall results for out-of-distribution data (Table 23). Note that AudioCaps and MusicCaps are considered in-distribution as both are derived from AudioSet. This means that using larger-scale training data (AF-Synthetic) provides better OOD generalizability.\n\nAF-Synthetic brings other practical benefits during training as well: we found that training ETTA using AF-AudioSet diverged after 250k training steps, whereas AF-Synthetic provides stable training up to 1M steps without instabilities. This means AF-Synthetic helps ETTA converge better from its scale.\n\nWe added the training loss curves of AF-AudioSet vs. AF-Synthetic (Figure-Rebuttal-3) along with the generated samples using the OOD challenging captions to the demo webpage, to better showcase the necessity of AF-Synthetic for generalization. Link: https://anonymous.4open.science/r/etta_demo-778A/index.md \n\n**Q: The rationale behind adding each component is unclear… Additional analyses—such as human perception studies, loss curves, or convergence times—would better illustrate each component's impact.**\n\nA: To address this concern, we added loss curves to the updated demo webpage to better illustrate the design choices: https://anonymous.4open.science/r/etta_demo-778A/index.md \n\nFigure-Rebuttal-1: Stable Audio-DiT vs. ETTA-DiT, under the same v-diffusion objective and AF-Synthetic training dataset for both. We find that Stable Audio-DiT plateaus around 300K steps and starts to diverge early. ETTA-DiT continues to improve its quality with better loss for the same training steps. This shows clear benefits of the ETTA-DiT architecture.\n\nFigure-Rebuttal-2: v-diffusion vs. OT-CFM training objective, under the same ETTA-DiT architecture and AF-Synthetic dataset for both. Note that the loss scale of OT-CFM is different to v-diffusion. For prolonged training (e.g. over 500K steps), v-diffusion starts to be unstable, whereas OT-CFM provides better stability up to 1M steps. This shows practical advantages of OT-CFM over v-diffusion and is a motivation to use it for ETTA.\n\nFigure-Rebuttal-3:  AF-AudioSet vs. AF-Synthetic training dataset, under the same OT-CFM objective and ETTA-DiT architecture. AF-AudioSet quickly diverges around 250K steps and is unable to continue its training, whereas AF-Synthetic provides better convergence with continued improvements up to 1M steps.\n\nWe believe these additional loss plots better illustrate the rationale behind each of the design choices of ETTA.\n\n**Q: Why are the results of Stable Audio Open different between Table 3 and Table 4? Are these different models?**\n\nA: Table 4 turned off the classifier free guidance (CFG) for all configs as an ablation study, including the original Stable Audio Open, so the difference compared to Table 3 (which uses a default CFG scale of the baselines) is expected. \n\n**Q: Table 2 reports human scores for both ETTA and ETTA-FT-AC-100k, while Table 3 only includes ETTA.**\n\nA: Thank you for mentioning this, we indeed measured ETTA-FT-AC-100k on musiccaps as well: OVL: 3.30 ± 0.10, REL: 3.44 ± 0.12. This shows that fine-tuning ETTA on AudioCaps (general sound) degrades the music generation quality in subjective evaluation, as expected. We will improve the writing to avoid confusion.\n\n**Q: In Section 3.1, when generating synthetic captions, if all generated captions have CLAP similarity below the threshold, is the audio discarded?**\n\nA: Yes, the audio is discarded in that case.", "claims_and_evidence": "- The authors claim that the proposed AF-Synthetic significantly improves text-to-audio generation quality. While Table 4 shows improvement for the existing Stable Audio Open trained on AF-Synthetic, Table 5 indicates that performance with AF-AudioSet is comparable or even superior. Similarly, Table 20 suggests using AF-AudioSet is generally preferable. Although the reviewer acknowledges the authors' efforts in building a large dataset, it remains challenging to conclude that AF-Synthetic is crucial for model improvement.\n\n- The authors also argue that their proposed design choices effectively enhance model performance. Tables 5, 12, and 13 demonstrate improvements with AF-Synthetic. However, simply adding components does not fully support the effectiveness of each design choice, as results fluctuate. Additional analyses—such as human perception studies, loss curves, or convergence times—would better illustrate each component's impact.-", "methods_and_evaluation_criteria": "- The rationale behind adding each component is unclear. Additionally, comparisons seem unfair when datasets differ. For instance, the authors should provide ablations modifying each design choice while training on the same dataset as in Table 4. Currently, all models trained with the new large-scale dataset potentially mask the true effects of added modules relative to the original Stable Audio Open.", "theoretical_claims": "No theoretical claims are made in the main paper.", "experimental_designs_or_analyses": "- Table 4 makes it difficult to interpret the contributions of proposed components, indicating that dataset scale has the largest effect, while other components yield mixed results. Clarifying effects via human evaluations, additional datasets, or deeper analyses would help. Tables 12 and 13 similarly show mixed results, suggesting dataset choice as the most influential factor.\n\n- Why are the results of Stable Audio Open different between Table 3 and Table 4? Are these different models?\n\n- In Table 5, AF-AudioSet achieves comparable performance with only one-tenth of AF-Synthetic's data, suggesting dataset quality outweighs dataset scale. This observation questions the necessity of AF-Synthetic compared to AF-AudioSet.\n\n- Comparing baseline models versus ETTA using the same training dataset (e.g., all trained on AF-Synthetic or AudioCaps) would better validate ETTA's design choices.\n\n- The reason for not reporting some human scores in Tables 2 and 3 is unclear. Table 2 reports human scores for both ETTA and ETTA-FT-AC-100k, while Table 3 only includes ETTA.", "relation_to_broader_scientific_literature": "No relation to broader scientific literature is identified.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- Limitations have not been discussed in this paper.", "other_comments_or_suggestions": "- In Section 3.1, \"Table 2\" should be denoted as \"Figure 2.\"", "questions_for_authors": "- In Section 3.1, when generating synthetic captions, if all generated captions have CLAP similarity below the threshold, is the audio discarded?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wSWZYpDTfr", "reviewer": "Reviewer_wC5z", "summary": "This paper proposes ETTA, a state-of-the-art text-to-audio model trained on public data. Its innovations include:\n- A new dataset called AF-Synthetic that follows the audio captioning pipeline from AF-AudioSet but scales up to million-scale. This is done by captioning or re-captioning audio in AudioCaps, AudioSet, VGGSound, WavCaps, and LAION-630K datasets.\n- A latent diffusion/flow matching model that builds on Stable Audio and makes several architectural changes.\n- An analysis on the scalability with respect to model size and training data size.\n- An ablation study on diffusion sampler and steps (NFEs).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review. We address your questions as follows:\n\n**Q: according to Table 6, the 2.08B 36-layer model seems to produce better results. So why is the balance achieved by the 1.44B model optimal?**\n\nA: We observed improvements for the deeper model (2.08B), but the gap narrows with CFG (Table 16 vs. 17). The subjective quality has been similar from the preliminary study. For this work we consider 1.44B a good balance between quality and speed, but we also expect that larger models will become better as we scale up the synthetic dataset.\n\n**Q: CFG could suffer FD penalties via other mechanisms, such as distortion caused by CFG's extrapolation.**\n\nA: This is a very good point. We also notice oversmoothing and/or distortion when CFG is set too high. This can be another source of its impact on FD. We will update the description accordingly to reflect this.\n\n**Q:  I suggest letting human raters compare the captions in AF-Synthetics and those in the source dataset + An alternative and potentially more straightforward metric would be Frechet embedding distance.**\n\nA: Thank you for your suggestion to evaluate the subjective quality of the synthetic caption from AF-Synthetic. We also agree CLAP does provide a good proxy that quantifies the caption quality, as evidenced by ETTA’s generation results by using AF-Synthetic as the training dataset. \n\nWe are happy to conduct human evaluation of AF-Synthetic captions. Since it is a very large scale, we will randomly sample a subset of captions and report human results in the final version.\n\nThe purpose of the max-sim metric is to assess to which extent we generate novel captions in AF-Synthetic. A max-sim close to 1.0 indicates that generated captions are copied from the source captions, which is not desired. We show in Figure 2 that max-sim is not close to 1.0, suggesting the generated captions are novel and diverse (see Table 9 and 10 for some qualitative examples).\n\n\n**Q: It would also be nice to include AudioCaps, WavCaps, and Laion-630K.**\n\nA: Nice suggestion. We will include the full results in the paper.\n\n**Q: The authors make several architectural changes to the DiT model (AdaLN, GELU, etc.) How were these modifications decided? Were they added all at once, or one by one? + AdaLN didn't work with vanilla implementation but worked for ETTA? Why?**\n\nA: ETTA-DiT modifications have been made entirely, inspired by recent best practices. We conjecture adding AdaLN to both self-attention & cross-attention input with unbounded gating provides stronger conditioning signals.\n\nWe provide training loss curves between Stable Audio-DiT vs. ETTA-DiT (Figure-Rebuttal-1), link: https://anonymous.4open.science/r/etta_demo-778A/index.md . ETTA-DiT provides better convergence whereas Stable Audio Open-DiT plateaus early.\n\n**Q: To my knowledge, score-prediction diffusion models are also very popular and are known to produce high performance. Has it been considered to compare with them?**\n\nWe did not consider the score (epsilon) prediction diffusion but used velocity (v) prediction in this work, as v-prediction has been the default choice of our baseline (Stable Audio Open). To the best of the author's knowledge, v-prediction has been favored by practitioners due to its better stability during training. We also find that OT-CFM provides even better stability.\n\n\n**Q:  Make-An-Audio 2, TANGO-AF&AC-FT-AC, TANGO2, and GenAU-L all seem strong. Are these models comparable with \"ETTA (AudioCaps only)\"? Were they trained on additional data?**\n\nA: Baselines models are from their own official results and/or the public checkpoints trained with different datasets. ETTA (AudioCaps only) in Table 2 is also to illustrate improved results by fine-tuning ETTA on AudioCaps resuming from a pre-trained ETTA with AF-Synthetic. We’ll add the extra training data column to the main tables.\n\n**Q: Is there any chance that the AF-Synthetic dataset can be open-sourced?**\n\nA: We will release the model code and data preparation methods to reproduce the result of ETTA models.\n\n**Q: I found it interesting that FLAN-T5-Large is not universally better than T5-Base.**\n\nA: Thank you for mentioning this. While we are not drawing a conclusive claim, it suggests that an optimal choice may also depend on other factors (i.e. “quirks”). We also speculate that another possible reason is the difference in text embedding variance between the encoders [1].\n\n[1] Xie, Enze, et al. \"Sana: Efficient high-resolution image synthesis with linear diffusion transformers.\" arXiv preprint arXiv:2410.10629 (2024).\n\n**Q: What does \"Finally, we also sub-sample long audio segments except for music and speech\" mean in Section 3.1?**\n\nA: For long sound (non-music or speech) data, we found most of them to be homogenized (e.g. an hour of continuous engine sound). For these samples, we sub-sample a few ten-second segments instead of adding the entire 360 segments to AF-Synthetic.", "claims_and_evidence": "Yes in general, but I have minor questions about some details.\n\n- The authors claim that \"We find the 1.44B model with depth=24 and width=1536 leads to an optimal balance between model size and quality.\" However, according to Table 6, the 2.08B 36-layer model seems to produce better results. So why is the balance achieved by the 1.44B model optimal?\n- The authors use experiments to show that an excessively high CFG scale is suboptimal in terms of FD, claiming the compromised diversity as the reason for such a suboptimality. I found this under-motivated, as CFG could suffer FD penalties via other mechanisms, such as distortion caused by CFG's extrapolation.", "methods_and_evaluation_criteria": "Yes in general. I would like to point out several minor things.\n\n- The authors use CLAP score to show the superiority of AF-Synthetic captions. While I generally trust CLAP and am convinced about the quality of the proposed data captions, I believe it would benefit from a small-scale human listening test to verify AF-Synthetic's superiority. That is, for a small set of audio pieces shared between AF-Synthetic and AudioCaps or WavCaps or Laion-630K, I suggest letting human raters compare the captions in AF-Synthetics and those in the source dataset.\n- It would also be nice to include AudioCaps, WavCaps, and Laion-630K in Figure 1 (can be random subsets of these datasets).\n- When evaluating the distributional similarity between AF-Synthetic captions and other caption datasets, the authors considered a CLAP-score-based max-similarity metric. An alternative and potentially more straightforward metric would be Frechet embedding distance. That is, we can use a similar approach to computing FAD using a CLAP backbone, except now we use CLAP text embeddings instead of audio embeddings. A smaller distance between two caption sets means more similar captions.\n- The authors make several architectural changes to the DiT model (AdaLN, GELU, etc.) How were these modifications decided? Were they added all at once, or one by one? If they were added to the baseline DiT one by one, which modification made the largest impact?\n- The authors compare flow-matching models with $v$-prediction diffusion models. To my knowledge, score-prediction diffusion models are also very popular and are known to produce high performance. Has it been considered to compare with them?", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "Yes. They are sound.", "relation_to_broader_scientific_literature": "See summary.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Overall, this paper is comprehensive, thorough, well-written, and worth accepting.** It shares important insights into various dimensions of TTA model design. I had a good laugh when I listened to the demo.\n\n**As mentioned above, several things that can be improved are:**\n- Stronger motivations and explanations for architectural changes.\n- Human evaluation on the captions of AF-synthetic.\n- (Stretch goal) compare with a score-prediction diffusion setup. This paper is already quite comprehensive, and I am aware that training a new model from scratch could be time-consuming and infeasible to complete within the rebuttal time frame. So this is optional.\n\n**If the above can be addressed, I recommend accepting this paper.**", "other_comments_or_suggestions": "- There are a lot of interesting results buried in the appendix. It would be nice to briefly mention some of them in the main paper body.\n- I found it interesting that FLAN-T5-Large is not universally better than T5-Base. Not asking for additional experiments, but do you think this is a \"quirk\" of this particular model, or is it something likely shared across different types of conditional diffusion/flow matching models?\n- In Table 2, there are several models comparable with (and even better than) \"ETTA (AudioCaps only)\" in some metrics. For example, Make-An-Audio 2, TANGO-AF&AC-FT-AC, TANGO2, and GenAU-L all seem strong. Are these models comparable with \"ETTA (AudioCaps only)\"? Were they trained on additional data? Would be nice to add an \"extra training data\" column to that table.", "questions_for_authors": "- Is there any chance that the AF-Synthetic dataset can be open-sourced?\n- What does \"Finally, we also sub-sample long audio segments except for music and speech\" mean in Section 3.1?\n- What does \"We switch from prepending to AdaLN timestep embedding and apply AdaLN\" mean in Section 3.2? I'm also confused with the footnote \"in our preliminary study using stable-audio-tools with its vanilla implementation, switching from prepending to AdaLN resulted in worse results.\" So AdaLN didn't work with vanilla implementation but worked for ETTA? Why?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dyaz1OnNDm", "reviewer": "Reviewer_gQwi", "summary": "This paper presents a high-quality audio-caption dataset containing 1.35M pairs of data, named as AF-Synthetic. The dataset is established through the state-of-the-art (SoTA) audio-language model, Audio Flamingo. In addition, the paper introduces a text-to-audio system based on the Diffusion Transformer (DiT) based Latent Diffusion Model. The experimental results demonstrate that the proposed system, named Elucidated Text-To-Audio (ETTA), achieves SoTA performance on both the natural audio and music.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review. We address your concerns as follows:\n\n**Q: The paper lacks sufficient experiments directly comparing the dataset's contribution, such as showing how other SoTA models, such like Tango and AudioLDM, perform when trained with AF-Synthetic. What is the performance of other text-to-audio systems that train on AF-Synthetic? Such as AudioLDM, Tango, Make an Audio.**\n\nA: We acknowledge your concern; current main results (Table 2 and 3) are structured to represent prior works in their original form by using the official results and/or their public checkpoints. \n\nStable Audio Open is our most modern and direct baseline, and we did perform the ablation study in AF-Synthetic dataset’s direct impact while keeping the baseline model identical in Table 4 (+AF-Synthetic). Compared to the public Stable Audio Open checkpoint, AF-Synthetic provides significant improvements from the training dataset alone. \n\nIn our preliminary study, we also did our best effort to train previous models (other than Stable Audio Open) using AF-Synthetic, but except for Stable Audio Open we have faced difficulties using their original training recipe (e.g. model not converging, loss instabilities, etc.). We speculate these models would need non-trivial optimization in the training recipe to accommodate significantly larger-scale data such as AF-Synthetic.\n\n\n**Q: The main weakness of the paper is that using audio-language model, such as Audio Flamingo to generate the caption making the result more like a conversation, not just normal captions. Most of the captions will begin with ”the audio consists”, “the audio features”,”There is a”. Which is actually different to the real-world cases. + Did the audio-language format of the caption lead to some problems? Is there any way to avoid such structure and re-caption into a more formal structure?**\n\nA: Thank you for asking this important question. While AF-Synthetic captions mostly start with a conversational prefix from the audio language model (Audio Flamingo), we empirically find that the model is robust in generating good quality samples using plain captions, as evidenced by our main benchmarks that use the original captions (Tables 2, 3, and 22) along with the OOD imaginative prompts in the demo webpage.\n\nWe also tried using an external LLM (Nemotron-340B [1]) to rephrase and shorten the AF-Synthetic captions to be more caption style (like AudioCaps). We found ETTA has similar quality when trained on this rephrased dataset.\n\n[1] Adler, Bo, et al. \"Nemotron-4 340b technical report.\" arXiv preprint arXiv:2406.11704 (2024).\n\n**Q: What is the difference between ETTA and Stable-Audio(Despite the DIT module)?**\n\nA: Other than the improved DiT backbone, ETTA uses an improved VAE, a better training objective (OT-CFM w/ logit-normal t-sampling), and is trained on AF-Synthetic. \n\n**Q: Why choose the current CLAP threshold, will the dataset be improved if raises the score?**\n\nA: The CLAP threshold >= 0.45 is set following a suggestion in existing work [2]. Improving the CLAP threshold will retain higher quality subset at a cost of reduction of the dataset size.\n\n[2] Kong, Zhifeng, et al. \"Improving text-to-audio models with synthetic captions.\" arXiv preprint arXiv:2406.15487 (2024).\n\n\n**Q: Why the ETTA does not outperform previous models in some MOS metrics?**\n\nA: For music, Stable Audio Open is trained on high quality music from Freesound and Free Music Archive (FMA) datasets, but many of these samples are not available as of today. In contrast, ETTA is not trained on any studio-quality music datasets (not even MusicCaps). As a result, ETTA has a slightly worse OVL (which only measures audio quality and does not consider text) on MusicCaps and SongDescriber.\n\nWhile only trained on synthetic captions, ETTA outperforms Stable Audio Open on the REL (audio-text relevance) score on AudioCaps and MusicCaps (Table 2-3), and matches it on an OOD testset SongDescriber (Table 22). \n\n\n\n**Q: Could you also provide some demos on music?**\n\nA: We have added several MusicCaps benchmark samples to the updated demo webpage. Link: https://anonymous.4open.science/r/etta_demo-778A/index.md", "claims_and_evidence": "The authors claim the effectiveness of the proposed dataset by demonstrating the SoTA performance of the ETTA system trained on AF-Synthetic. However, the paper lacks sufficient experiments directly comparing the dataset's contribution, such as showing how other SoTA models, such like Tango and AudioLDM, perform when trained with AF-Synthetic.", "methods_and_evaluation_criteria": "The main weakness of the paper is that using audio-language model, such as Audio Flamingo to generate the caption making the result more like a conversation, not just normal captions. Most of the captions will begin with ”the audio consists”, “the audio features”,”There is a”. Which is actually different to the real-world cases.", "theoretical_claims": "None", "experimental_designs_or_analyses": "The paper lacks more experiments to illustrate the effectiveness of AF-Synthetic dataset on other systems. \nIn addition, as an audio-language dataset, experiments that present the contribution of the dataset on other tasks, such as audio retrieval and audio captioning can enhance the significant usefulness of the proposed dataset.", "relation_to_broader_scientific_literature": "The paper mainly provides a larger-scale audio dataset, which is useful for audio-related tasks", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The proposed system, named as ETTA, more like a model applying Stable Audio as the backbone. Not too much novel techniques are proposed in this system.", "other_comments_or_suggestions": "Please see questions.", "questions_for_authors": "Overall, this is a high-quality paper and the idea of creating an audio-caption dataset is interesting. I am happy to change the score if the author can solve the following questions or experiments. \n\nQuestion1: What is the performance of other text-to-audio system that train on AF-Synthetic? Such as AudioLDM, Tango, Make an Audio.\n \nQuestion2:What is the difference between ETTA and Stable-Audio(Despite the DIT module)?\n\nQuestion3:Why choose the current CLAP threshold, will the dataset be improved if raises the score?\n\nQuestion4:Did the audio-language format of the caption lead to some problems? Is there any way to avoid such structure and re-caption into a more formal structure?\n\nQuestion5:Why the ETTA does not outperform previous models in some MOS metrics? \n\nQuestion6: Could you also provide some demos on music?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers recommend accepting the paper (3 accept and 1 weak accept). 

The rebuttal did a good job of addressing the concerns that the reviewers had. 
One reviewer (okEU) raised the score. 
While all the reviewers mentioned the lack of novelty in the method, they value comprehensive large-scale experiments.
Also, the release of the checkpoints or code would be a valuable asset for the community.

The AC thus follows this unanimous recommendation by the reviewers and recommends accepting the paper.
Decision: Accept (poster)