RiTTA: Modeling Event Relations in Text-to-Audio Generation

Authors: Yuhang He, Yash Jain, Xubo Liu, Andrew Markham, Vibhav Vineet
Primary area: generative models
Keywords: Text-to-Audio Generation, Audio Events Relation, Benchmark, Dataset Corpus

Abstract:
Despite significant advancements in Text-to-Audio (TTA) generation models achieving high-fidelity audio with fine-grained context understanding, they struggle to model the relations between audio events described in the input text. However, previous TTA methods have not systematically explored audio event relation modeling, nor have they proposed frameworks to enhance this capability. In this work, we systematically study audio event relation modeling in TTA generation models. We first establish a benchmark for this task by: (1) proposing a comprehensive relation corpus covering all potential relations in real-world scenarios; (2) introducing a new audio event corpus encompassing commonly heard sounds; and (3) proposing new evaluation metrics to assess audio event relation modeling from various perspectives. Furthermore, we propose a finetuning framework to enhance existing TTA models' ability to model audio events relation.

Decision: reject
Reviewer scores: 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730040798332

thread_type:
official_review

<thread_object>
creator: reviewer_bn9e

<content>
summary:
This paper proposed an audio event relation corpus for text-to-audio generation and a new evaluation benchmark. TTA models finetuned in this corpus got better generation of the audio event relations.

strengths:
The proposed corpus is important for research on audio event relations, which can be used for TTA with good quality and comprehension.

weaknesses:
I have some concerns about this work:

1. The corpus only contains 25 categories, which seems to not easily generate to other categories with relations. Previous training metarials like AudioCaps and Clotho have over 100 classes.

2. The paper targets on mono-channel audio, but in this case, the spatial distance makes little sence. How can you accurately measure whether the event is from 1m or 7m? I think multi-channel audio would also be interesting, e.g. a car driving from left to right.

3. The FAD and FD used feature embeddings from pretrained on VGGish model. However, this model is known as its poor performance. Even though some baselines may use this tool, I think modern backbone models should be considered.

questions:
1. Except the 25 categories, what are the results of other audio events with relations?  TTA is more like an open-set problem, so unseen labels' evaluation is needed.

2. What are the results on the previous test data from the baseline models? Will the finetuning affect the original results?

3. Can the model handle other lengths of audio instead of 10s?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730392123736

thread_type:
official_review

<thread_object>
creator: reviewer_ypb8

<content>
summary:
1.	Benchmark Construction: Created an audio event relation corpus for TTA tasks, covering common spatial, temporal, count, and compositional relations. Additionally, an audio event category corpus encompassing various everyday sounds was established to provide foundational data for studying event relation modeling in TTA.
2.	Evaluation Framework: Proposed a multi-stage relation-aware evaluation framework (MSR-RiTTA) that quantifies a model’s ability to capture audio event relations from multiple perspectives, offering more targeted metrics than existing TTA evaluation standards.
3.	Fine-Tuning Framework: Developed a fine-tuning strategy aimed at enhancing current TTA models’ capacity to model audio event relations, showing significant improvement in benchmark tests.

strengths:
The paper presents a substantial contribution to the field of Text-to-Audio (TTA) generation by introducing the RiTTA framework, which systematically addresses the modeling of audio event relations—a previously underexplored area. It proposes an innovative multi-stage, relation-aware evaluation framework (MSR-RiTTA) to more accurately assess the performance of TTA models in capturing complex spatial, temporal, count, and compositional relationships. The research is rigorous in its dataset construction and experimental validation, demonstrating clear improvements in TTA models’ relation modeling abilities. This work stands out for its originality, methodological clarity, and significant potential impact, expanding TTA applications in areas like virtual reality and immersive media.

weaknesses:
The paper benchmarks seven recent TTA models (such as AudioLDM and Tango) but focuses primarily on single-strategy models, and without testing in more complex scenarios like audio events in virtual reality or interactive environments. This limits the framework’s applicability across different model types and environments. A wider variety of model types, such as multimodal generation models or models optimized for extended audio sequences, and more diverse application datasets, like multi-event VR audio environments are recommended.

The description of the fine-tuning strategy is too general, lacking detailed information on training settings, parameter tuning strategies, and specific experimental procedures. The lack of details such as specific hyperparameter settings, number of iterations, and choice of loss function also weakens persuasiveness.

questions:
Refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730466170629

thread_type:
official_review

<thread_object>
creator: reviewer_rsts

<content>
summary:
This paper primarily introduces a benchmark for the relationships between multiple audio events, with its main contribution being the creation of a new dataset and the establishment of several novel evaluation metrics.

strengths:
1. Compared to previous work, the authors' most significant contribution lies in the development of a comprehensive dataset that assesses the relationships between multiple audio events.
2. Using temporal relationships as an example, the authors provide a detailed explanation of the dataset construction process and propose new evaluation metrics. They also conduct experiments using both the pre-trained TTA model and the fine-tuned version.

weaknesses:
1. I am unclear as to why RiTTA was submitted to the generative model track, as the primary contribution clearly stems from the new dataset and evaluation metrics related to the benchmark, rather than from advances in generative modeling.
2. In terms of innovation, the authors are not the first to explore audio event relationship modeling. RiTTA extends temporal relationships to four types of relationships, but the overall process appears to be data-driven. However, merely using the TTA model to model complex relationships is insufficient. As demonstrated in Figure 7, there is a performance decline with "not" relationships. I was expecting a novel TTA framework capable of jointly modeling relationships between different audio events. Unfortunately, RiTTA only fine-tunes the TTA model, which limits its novelty.
3. Regarding the core contribution—the dataset—there are several potential shortcomings. First, the dataset construction process does not seem to require significant effort. Second, the dataset itself feels somewhat toy. While the authors attempt to enhance its randomness, expanding each example to five events via ChatGPT is far from sufficient (only 5?); 500 events might be a more appropriate target. Additionally, the use of only one-channel audio for spatial relationships could limit the dataset’s diversity.

questions:
1. The authors provide a detailed explanation of the dataset construction process using temporal relationships as an example, but I am still curious about how other relationships, such as spatial relationships, were constructed. How were the corresponding text-audio pairs created for these? Further clarification would be beneficial.
2. Should out-of-domain scenarios be considered when forming the test set?
3. I could not find the RiTTA dataset in the supplemental materials.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732880205053

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Feedback to All Reviewers

comment:
We appreciate the reviewers' constructive comments and feedback.  We have provided feedback to each reviewer separately. Here we further provide a general feedback to all reviewers.

1. Benchmark scale and comparision with previous discussion on audio events relation.

   **Feedback:** **First**, comparing with prior work (like CompoA) that just discussed temporal relation and 400 and 200 data size, our introduced benchmark incorporated four more main relations, 20 more subrelations, and the corresponding data size is several magnitudes larger (currently, 8,000, and can further be scaled up). **Second**, we extensively show in the paper that all existing TTA models fail on the benchmark we have presented. So the benchmark in its current format is already beneficial enough for TTA research community to investigate deeper for the audio events relation modeling in TTA task. **Third**, Adaptivity and Scalability. We decoupled audio event relation from audio events category, the benchmark construction process can be easily scaled up to incorporate extra relation and audio events. 

2. RiTTA dataset is missing in supplementary (as per to Reviewer rsts).

   In the updated supplementary material, we have provided RiTTA **1. seed audios,  2. created events relation aware reference audios, 3. GPT-augmented text prompts.**. With the updated material, we hope the reviewer can have a direct understanding of our benchmark.

3. More experiment of our fine-tuned model (as per to Reviewer Bn9E).

   Following the requirements, we have run two extra experiments on the fine-tuned model, which provide reviewers with in-depth understanding of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper's main contribution is a new audio dataset and benchmark, which can be used for "text to audio" (TTA) experiments and evaluation. In conjunction with the benchmark, the paper discusses dataset construction, and some analysis that shows that existing text to audio models fail on the new benchmark. Challenging issues (which the reviewers would like to see), such as generation of unseen audio events, more complex event relationship patterns, etc are left for future work.

The main contributions are:
- Created an audio event relation corpus for TTA tasks, covering common spatial, temporal, count, and compositional relations. Additionally, an audio event category corpus encompassing various everyday sounds was established to provide foundational data for studying event relation modeling in TTA.
- Proposed a multi-stage relation-aware evaluation framework (MSR-RiTTA) that quantifies a model’s ability to capture audio event relations from multiple perspectives, offering more targeted metrics than existing TTA evaluation standards.
- Developed a fine-tuning strategy aimed at enhancing current TTA models’ capacity to model audio event relations, showing significant improvement in benchmark tests.

Strengths
- The paper treats an interesting problem and any progress towards complex audio generation (and this includes datasets, evaluation method, and running benchmarking) is a worthwhile contribution
- The paper is generally well written and easy to understand, the paper is accessible

Weaknesses
- it is not clear if the paper should not be in the dataset/ evaluation track, since its technical contributions are slim (even if not entirely absent)
- Audio Event Relationship modeling is not entirely new, authors are not the first to propose that. RiTTA presents incremental improvements. The paper would be much stronger if the dataset had bigger complexity, bigger size, and/ or a better model that can solve the described challenges

Summary
While reviewers are sympathetic to the papers basic premise, they agree that the presented contributions do not quite make the bar for publication at ICLR, for both (generative) modeling as well as dataset/ evaluation track.