Elucidating the Design Space of Text-to-Audio Models

Authors: Sang-gil Lee, Zhifeng Kong, Arushi Goel, Sungwon Kim, Rafael Valle, Bryan Catanzaro
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: audio generation, text-to-audio, synthetic data, diffusion, flow matching

Abstract:
Recent years have seen significant progress in Text-To-Audio (TTA) synthesis, enabling users to enrich their creative workflows with synthetic audio generated from natural language prompts. Despite this progress, the effects of data, model architecture, training objective functions, and sampling strategies on target benchmarks are not well understood. With the purpose of providing a holistic understanding of the design space of TTA models, we setup a large-scale empirical experiment focused on diffusion and flow matching models. Our contributions include: 1) AF-Synthetic, a large dataset of high quality synthetic captions obtained from an audio understanding model; 2) a systematic comparison of different architectural, training, and inference design choices for TTA models; 3) an analysis of sampling methods and their Pareto curves with respect to generation quality and inference speed. We leverage the knowledge obtained from this extensive analysis to propose our best model dubbed Elucidated Text-To-Audio (ETTA). When evaluated on AudioCaps and MusicCaps, ETTA provides improvements over the baselines trained on publicly available data, while being competitive with models trained on proprietary data. Finally, we show ETTA's improved ability to generate creative audio following complex and imaginative captions — a task that is more challenging than current benchmarks.

Decision: reject
Reviewer scores: 5, 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1729430342871

thread_type:
official_review

<thread_object>
creator: reviewer_5eah

<content>
summary:
This paper presents a 1.35M high-quality audio-caption dataset utilizing the state-of-the-art (SoTA) audio-language model, Audio Flamingo, referred to as AF-Synthetic. Additionally, the paper introduces a text-to-audio system based on the Latent Diffusion Model (LDM) with a Diffusion Transformer (DiT) backbone. Experimental results demonstrate that the proposed Elucidated Text-To-Audio (ETTA) system achieves SoTA performance across multiple metrics on both the AudioCaps and MusicCaps datasets.

strengths:
The paper introduces a novel filtering pipeline to select the best captions using the CLAP score, developing one of the largest high-quality audio-language datasets. The proposed ETTA generation model is trained across a wide range of diverse audio datasets and achieves competitive scores on both AudioCaps and MusicCaps, demonstrating state-of-the-art performance in both text-to-audio and text-to-music tasks.

External ablation studies investigate the system's performance across different model sizes and training/sampling strategies. Subjective evaluations highlight significant improvements in the proposed ETTA system over baseline models, proving its enhanced ability to generate complex and imaginative captions.

Overall, this paper presents interesting work in the field of audio generation. The authors first introduce a large-scale dataset and then present a state-of-the-art generation system trained using this dataset. Various experiments demonstrate the effectiveness of different methods or modules within the system, concluding with an analysis of the ETTA system's limitations.

weaknesses:
1. The authors claim the effectiveness of the proposed dataset by demonstrating the SoTA performance of the ETTA system trained on AF-Synthetic. However, the paper lacks sufficient experiments directly comparing the dataset's contribution, such as showing how models like Tango or AudioLDM perform when trained with AF-Synthetic.
2. All the metrics used for evaluating the text-to-audio system are objective, which is generally sufficient but may not always reflect real-world performance. Therefore, including subjective evaluations like the Mean Opinion Score (MOS) would be beneficial, particularly for comparing models that achieve top performance across some metrics.
3. The authors attempt to compare the performance between different baseline models. While ablation studies already demonstrate the effectiveness of the proposed ETTA strategies, the paper lacks experiments where the proposed model is trained on the same dataset (e.g., AudioCaps) to clearly illustrate the improvements contributed solely by the system itself.
4. The ETTA model mainly builds on the backbone of the Stable Audio system. Beyond the training and sampling strategies (Flow Matching and ODE solvers, which are standard improvements in current LDM systems), the key enhancement appears to be the use of Adaptive Layer Normalization (AdaLN) within the DiT structure and the T5-base model for text embedding. These techniques are already implemented in the original DiT paper [1] and other baseline models. As a result, the model seems more like an engineering application with limited novel contributions.

Overall, this is an interesting paper with great effort, I am willing to change the score if the author can solve the questions and fulfil the following experiments:
1. Train baseline models like Tango or AudioLDM on AF-Synthetic and compare to their original performance.
2. Train ETTA on other datasets like AudioCaps or TangoPromptBank and compare performance to when trained on AF-Synthetic. These experiments would help separate the contributions of the model architecture from the dataset.
3. Develop human evaluations on a subset of generated samples, comparing ETTA to top baseline models, such as Mean Opinion Score (MOS) or other equivalent subjective ratings on audio quality, text relevance and so on.
4. More clearly articulate the novel aspects of their model architecture compared to existing work like DiT and Stable Audio. 

[1]. Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

questions:
1. For the comparison of the proposed AF-Synthetic dataset, the paper indicates that both AF-AudioSet and Sound-VECaps have around 161K samples remaining after filtering with a CLAP score of 0.45. Since both datasets are primarily developed from AudioSet, is there a significant overlap in the subset that achieves a higher CLAP score? If so, could this raise concerns regarding the reliability of using the CLAP score as a filtering metric?
2. What are the key differences between AF-AudioSet and the proposed AF-Synthetic, apart from the fact that the latter generates ten captions per sample and selects the one with the highest CLAP score?
3. What distinguishes the AdaLN layer in the proposed ETTA-DiT structure from the one used in the original DiT model mentioned in a previous section?
4. How does the ETTA model perform with improvements limited to the ETTA-DiT component (e.g., when trained solely on AudioCaps)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730527894668

thread_type:
official_review

<thread_object>
creator: reviewer_raqm

<content>
summary:
The paper focuses on pushing the performance of diffusion-based text-to-audio models by introducing a new synthetic dataset and revisiting design choices. 
- New synthetic dataset (AF-Synthetic): The authors improve the pipeline from AF-AudioSet to generate 1.35M captions using audio samples from multiple datasets. The result is a large dataset with high CLAP score, while existing datasets are either small or have simple to no filtering method. The authors also point out that AF-Synthetic captions are different than existing datasets.
- The model is built upon Audio-VAE (from stable-audio-tools), Diffusion Transfromer (Peebles & Xie, 2023) with some changes in the architecture.
- The authors compare across datasets and metrics to discover the optimal choice for the number of function evaluations and classifier-free guidance.

strengths:
The work aims to push the performance of diffusion-based text-to-audio models and show better results than open source models. The improvements come from a better dataset and an extensive evaluation of design/hyperparameter choices built on top of SOTA models. I appreciate the amount of work put into creating the dataset and try many different hyperparameters and design choices, and the commitment to open source the code that appears to push the field ahead.

weaknesses:
Since this is a purely empirical paper, my main concerns are mostly about evaluation and results.

- Since the dataset is curated using CLAP scores, I find the CLAP score not a reliable indicator, while FAD and KL use old models to extract features.
- There are no subjective scores in the results except for Table 9 which compares the final model with others.
- The improvements in ETTA-DiT, OT-CFM and t-sampling are not consistent across metrics.
- Using AF-AudioSet and AF-Synthetic yields similar results in AudioCaps and MusicCaps despite much larger size (Table 6-7)

Overall, I find that a few conclusions in the paper are not very helpful (e.g. increasing model size improves the performance), especially when we have inconsistent metrics and lack of subjective scores. While it's good to see many hyperparameters being assessed, the contribution is lack of novelty since it does not propose new techniques or reveal surprising findings. The lack of reliable metrics, which authors also admit in the final section, also weaken claims and conclusions.

questions:
- What can be the reason behind mode-collapsed models consistently produce good scores across multiple metrics, while the backbone for these metrics are different?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649912024

thread_type:
official_review

<thread_object>
creator: reviewer_gse7

<content>
summary:
The primary focus of this paper is not to explore novel model designs, but rather to provide a comprehensive understanding of the current paradigms in TTA models. It seeks to identify critical factors that contribute to performance improvements and to evaluate scalability concerning data and model size. Additionally, the paper introduces AF-Synthetic, the first million-size synthetic caption dataset with strong audio correlations.

strengths:
1. The contribution of the dataset: Experiments demonstrate that AF-Synthetic significantly improves performance.
2. It provides a practical guide for hyperparameter tuning in the field of TTA.
3. Each experimental conclusion is highlighted with a purple-bordered box, which makes the paper very reader-friendly and pleasant to navigate.

weaknesses:
1. As the authors themselves mentioned, the paper doesn't propose a novel method. Moreover, as a paper based on extensive experiments, while it offers valuable conclusions, it lacks an innovative insight or discovery that stands out. Therefore, I believe this paper might be a better fit for the Dataset & Benchmark track.  
2. I would also suggest that the authors include the following three points:  
   - Could the authors further analyze the "potentially mode-collapsed mode" part? In theory, the FD metric should be able to measure sample diversity.  
   - Since the authors have already tried logit-normal *t-sampling*, could they also test the *Min-SNR Weighting Strategy*?  
   - Regarding the Auto-guidance section, it might be worth experimenting with removing CFG (Classifier-Free Guidance) for the first 40% of steps and then adding CFG for the remaining 60% of steps to see if this improves the FD score.

questions:
Please refer to Weaknesses#2.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705903800

thread_type:
official_review

<thread_object>
creator: reviewer_dkey

<content>
summary:
This paper empirically studies the way to improve the current text-to-audio generation system, including creating a new synthetic audio-caption paired dataset, an improved architecture, and other related settings.

strengths:
1. The author proposed a synthetic data generation pipeline and created the first million-size dataset with high audio-text correlation.
2. The author has implemented a comprehensive list of objective evaluation metrics and shows interesting results during comparison.
3. The author mentioned they will open-source their code for reproducibility.

weaknesses:
1. As there are a large number of evaluation metrics used. It would be helpful to explain what each evaluation metrics focus on. For example, there are three types of Frechet Distance - what are the differences?
2. It is not clear how ETTA perform without pretraining on the large-scale datasets. For example, what would the result look like if the ETTA model is trained on AudioCaps from scratch (I assume this is a common setup)?
3. Lack of subjective evaluation. As the author mentioned, there is not yet an effective objective evaluation metric for the TTA task. Out of the seven main conclusions in the paper, only one is backed by the subjective evaluation. This can make the conclusion less convincing. 
4. The architectural improvement seems a bit marginal. The improvement of ETTA seems to come from the new synthetic dataset mostly.

questions:
1. “We initialize the final projection layer of DiT to output zeros.” Do you mean zero-initialize all the weight and bias in the projection layer? Or do you use gated operation? More explanations are welcomed.
2. Please consider adding the result of Make-an-Audio and AudioLDM in Table 2 and Table 3, as they are also very important baselines to compare with.
3. In Table 6, the ETTA trained on AudioCaps shows 3.00 FAD on the audiocaps evaluation set. This is significantly lower than the current state-of-the-art. It would be helpful if the author could explain this result.
4. Line 510: Please say more about the loss divergence issue? It seems contradictory as the evaluation metrics still look normal. More explanations would be helpful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732573079603

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
## (1) Subjective evaluations for benchmark datasets

As several reviewers have asked for subjective evaluation results on the main benchmarks, we additionally conducted MOS (OVL = overall audio quality where raters do not see captions, REL = audio-text relevance). The results are shown below (also shown in Table 30 & 31, page 27 in the updated pdf).

On AudioCaps, ETTA has significantly better quality and relevance. On MusicCaps, while Stable Audio Open has better music quality – likely because it is trained with massive music data including FMA – ETTA has better music-text relevance, indicating better caption-following ability.

### AudioCaps

|      Model       |        GT        | AudioLDM2-large | TANGO2 | Stable Audio Open |   ETTA    | ETTA-FT-AC-100k |
|------------------|------------------|------------------|--------|-------------------|-----------|------------------|
| **OVL**          | 3.43 ± 0.11     | 3.00 ± 0.11     | 3.08 ± 0.10 | 3.29 ± 0.11       | **3.43 ± 0.11** | 3.26 ± 0.10     |
| **REL**          | 3.62 ± 0.10     | 3.11 ± 0.10     | 3.66 ± 0.09 | 3.15 ± 0.11       | 3.68 ± 0.10      | **3.77 ± 0.10** |

---

### MusicCaps

|      Model       |        GT        | AudioLDM2-large | TANGO-AF | Stable Audio Open |   ETTA    |
|------------------|------------------|------------------|----------|-------------------|-----------|
| **OVL**          | 3.88 ± 0.10     | 3.25 ± 0.10     | 3.38 ± 0.09 | **3.92 ± 0.10**   | 3.53 ± 0.10  |
| **REL**          | 3.90 ± 0.10     | 3.15 ± 0.10     | 3.31 ± 0.10 | 3.35 ± 0.11       | **3.57 ± 0.10** |


## (2) Novelty and Contribution

We believe our work offers significant contributions to advance the field of text-to-audio generation. While studying novel model architecture is not the focus of our work, we do present an improved DiT implementation that leads to better quality. Here is the summary of our contributions:

* We create the first million-size synthetic dataset with strong audio-text correlations.

* For the first time, we show it is possible to achieve SOTA generation results on multiple benchmarks (AudioCaps, MusicCaps, SongDescriber) using **only synthetic captions and at scale**. 

* We systematically investigate the impact of data and model scaling, architectural choices, and training and sampling strategies, providing valuable insights to guide future research in this area.

* This work represents the most comprehensive suite of exploration and evaluation in the text-to-audio literature.

* We not only achieve SOTA generation results on standard audio and music generation benchmarks, but also show an advanced ability to synthesize novel audio that does not exist in the real world.

see our demo website at: https://anonymous.4open.science/r/etta_demo-72C4/index.md 

In summary, we believe that **elucidating the existing paradigm to push the practical boundaries in this domain is as equally important as exploring novel architecture**, and our contributions collectively offer valuable insights to the research community.


## (3) Clarification on potential mode-collapsing issue of ETTA-FT-MC

Several reviewers asked questions about our discussion in Section 5. We thank the reviewers for highlighting this point as an interesting observation worth further explanation. We aim to describe our observation and analysis to the best of our ability. 

**Context**: The discussion only applies to the model fine-tuned on about half of MusicCaps samples and evaluated on the other half, corresponding to the setting in Tango-AF. It does not apply to our main audio and music generation results in Section 4 as well as the Appendix. 

**Observation**: We listened to ETTA-FT-MC generated samples with different random seeds as well as slightly different captions. We found the model has reduced diversity, similar to but not as severe as the typical mode collapse phenomenon shown in GANs. We do not observe this in our pretrained ETTA model or ETTA-FC-AC that is fine-tuned with AudioCaps. 

**Analysis**: We believe it is a new bias that is undefined in the literature and likely cannot be identified by the existing quality metrics, and it is related to **small and homogeneous** fine-tuning sets. It is possible that Frechet Distance does not measure diversity well for small and homogeneous test sets. We also conducted MOS evaluation on quality and relevance, and the scores are high and consistent with the objective evaluations.
* OVL=$3.77\pm0.10$ where GT=$3.88\pm0.10$, REL=$3.73\pm0.09$ where GT=$3.90\pm0.10$.

**Why we discuss these results**: We aim to call the community’s attention to further study this bias that happens to small and homogeneous fine-tuning sets and motivate better evaluation metrics tailored for this situation.
</content>
</thread_object>

</thread>


Metareview:
metareview:
> The paper focuses on pushing the performance of diffusion-based text-to-audio models by introducing a new synthetic dataset and revisiting design choices.
> * New synthetic dataset (AF-Synthetic): The authors improve the pipeline from AF-AudioSet to generate 1.35M captions using audio samples from multiple datasets. The result is a large dataset with high CLAP score, while existing datasets are either small or have simple to no filtering method. The authors also point out that AF-Synthetic captions are different than existing datasets.
> * The model is built upon Audio-VAE (from stable-audio-tools), Diffusion Transfromer (Peebles & Xie, 2023) with some changes in the architecture.
> * The authors compare across datasets and metrics to discover the optimal choice for the number of function evaluations and classifier-free guidance.

The objective evaluations only give a partial view:
> CLAP scores, I find the CLAP score not a reliable indicator, while FAD and KL use old models to extract features.

The rebuttal period was productive and 3 out of 4 reviewers increased their score. Notably, the authors added subjective evaluations. Weaknesses that were addressed in the rebuttal: no subjective evaluations, and disentangling more clearly (in the paper) the effect of the methods vs. of the dataset. This is overall a good paper. However, those changes make the paper (which has 2 contributions: data and method) harder to follow. While reviewers increased their score, the paper falls a bit short of the required quality to be published at ICLR.