T2A-Feedback: Improving Basic Capabilities of Text-to-Audio Generation via Fine-grained AI Feedback

Authors: Zehan Wang, Ke Lei, Chen Zhu, Jiawei Huang, Luping Liu, Xize Cheng, Shengpeng Ji, Rongjie Huang, Zhenhui Ye, Xiaoda Yang, Tao Jin, Zhou Zhao
Primary area: datasets and benchmarks
Keywords: text-to-audio generation; preference learning

Abstract:
Text-to-audio (T2A) generation has achieved remarkable progress in generating a variety of audio outputs from language prompts. However, current state-of-the-art T2A models still struggle to satisfy human preferences for prompt-following and acoustic quality when generating complex multi-event audio. To improve the performance of the model in these high-level applications, we propose to enhance the basic capabilities of the model with AI feedback learning. First, we introduce fine-grained AI audio scoring pipelines to: 1) verify whether each event in the text prompt is present in the audio (Event Occurrence Score), 2) detect deviations in event sequences from the language description (Event Sequence Score), and 3) assess the overall acoustic and harmonic quality of the generated audio (Acoustic & Harmonic Quality). We evaluate these three automatic scoring pipelines and find that they correlate significantly better with human preferences than other evaluation metrics. This highlights their value as both feedback signals and evaluation metrics. Utilizing our robust scoring pipelines, we construct a large audio preference dataset, T2A-FeedBack, which contains 41k prompts and 249k audios, each accompanied by detailed scores. Moreover, we introduce T2A-EpicBench, a benchmark that focuses on long captions, multi-events, and story-telling scenarios, aiming to evaluate the advanced capabilities of T2A models. Finally, we demonstrate how T2A-FeedBack can enhance current state-of-the-art audio model. With simple preference tuning, the audio generation model exhibits significant improvements in both simple (AudioCaps test set) and complex (T2A-EpicBench) scenarios. The project page is available at \url{https://T2Afeedback.github.io}

Decision: reject
Reviewer scores: 6, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1729114568842

thread_type:
official_review

<thread_object>
creator: reviewer_rkpb

<content>
summary:
This paper introduces an audio preference dataset, named T2A-Feedback, designed to enhance text-to-audio generation systems by improving both prompt adherence and acoustic quality. The dataset leverages three distinct scoring mechanisms: (1) Event Occurrence Score, which assesses the accurate presence of events specified in the prompt; (2) Event Sequence Score, which evaluates the correct sequencing of these events; and (3) Acoustic Harmonic Quality, which measures the overall audio quality based on harmonic consistency. Furthermore, the authors propose a benchmark that evaluates the system's ability to handle long-form captions, including complex multi-event and narrative-driven scenarios.

strengths:
The paper explains all three proposed pipelines in detail and presents corresponding experiments to illustrate the advantage of the proposed scoring metrics.

weaknesses:
1. The concept of using an audio separation model to detect event occurrence is intriguing. However, relying on a CLAP-based separation model to address the limitations of the CLAP model itself seems somewhat unconvincing.
2. The rationale behind determining the event occurrence score by selecting the lowest score needs further clarification.
3. For the event sequence score, identifying the correct sequence based solely on volume levels appears challenging. Additional strategies are warranted, especially for handling events that occur simultaneously.
4. In the event sequence pipeline, the criteria for determining concordant (C) or discordant (D) event pairs are unclear. It’s ambiguous whether only the correct ordering of immediate event pairs is needed or if all events must be in the correct sequence. More explanation and justification for this approach would be beneficial.
5. Only two annotators were involved in scoring the audio sample quality, which raises concerns about the robustness of the experimental results.
6. The paper initially claims that current text-to-audio systems exhibit low performance. However, the authors also used three different text-to-audio systems to generate the proposed T2A-Feedback dataset, which appears somewhat contradictory and requires further justification.
7. The proposed T2A-EpicBench dataset, with an average of 54.8 words per sample, presents a challenge for current text-to-audio generation systems. Given the difficulty in generating long stories within just 10-second audio clips, practical applications are unclear.
8. The experiments for each scoring pipeline, particularly the Acoustic Harmonic Quality (AHQ) evaluation, require further explanation. It is not immediately evident why or how AHQ contributes more effectively.
9. All preference tuning systems discussed in Section 5.2 report higher FAD scores, which warrants additional discussion.
10. Overall, this paper offers an interesting approach to using LLMs for evaluating audio clips and corresponding captions. While the authors attempt to develop a relevant caption dataset and benchmark for their scoring pipelines, the paper lacks sufficient detail and explanation for some experimental aspects. Furthermore, it leans more toward engineering work with limited novelty.

questions:
1. In the event occurrence pipeline, AudioSep is utilized to isolate sub-audio events to facilitate the identification of multiple events through the CLAP model. However, as AudioSep also employs the CLAP model as the text/audio encoder, this approach does not fully circumvent the limitations of CLAP for multi-event scenarios. The authors should further elaborate on the rationale behind this choice and clarify the benefits of combining AudioSep with CLAP in this context.
2. The decision to select the lowest value among all matching scores for event occurrence needs further justification. For example, if there are ten events and only one yields a significantly lower score, this could skew the overall assessment. The authors should validate whether this is an optimal approach for capturing overall performance.
3. Regarding the Event Sequence Score, how does the system handle cases where multiple events occur simultaneously, or when an event spans the entire audio sample? Clarification on these scenarios would strengthen the robustness of the sequence evaluation.
4. In Section 3.3, the authors state that current audio generation models frequently produce low-quality and noisy results (line 247). This claim would benefit from supporting evidence to substantiate it.
5. Further details are needed on the linear predictor mentioned in Section 3.3. Specifically, is this model trained from scratch, or is it fine-tuned from a pre-existing model? A description of its structure would provide better insight.
6. The statement on line 294, “We prompt the LLM with randomly selected events to create varied and natural multi-event audio descriptions,” could use more clarity, especially regarding temporal information. Based on the prompt shown in Appendix C, it appears that the LLM is instructed to assign distinct orders to events. How does the model address cases where audio events actually occur simultaneously?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730171516976

thread_type:
official_review

<thread_object>
creator: reviewer_3tsj

<content>
summary:
This work constitues three metrics and two datasets for text-to-audio (T2A):

* Event occurance score (EOS), Event sequence score (ESS), and Acoustic & harmonic quality (AHQ). EOS and ESS are algorithmic based on off-shelf models, and the final one AHQ is done by training a quality predictor using a manually annotated dataset.

* T2A-Feedback, a 40k captions, 250k generated audio clips with annotated scores.

* T2A-EpicBench, more challenging benchmark captions with longer, story-telling like captions.

T2A-Feedback improves existing T2A models (for example, Make-an-Audio 2) and is claimed to elicit emergent abilities to handle longer text and follow the event order measured by T2A-EpicBench.

strengths:
* A model-based, scalable apparoach to generate large-scale preference dataset is an important direction worth exploring, and T2A-Feedback is one of the early endeavor which warrants credit.

weaknesses:
* For evaluation and dataset papers like this, the authors can consider having more scrutiny in stating significance of the proposed metric's reliability. For example, a chi square test on the confusion matrix and reporting its p-value. Same goes to the benchmarks.

* The scope of verificiation of the proposed metric's robustness is tied to AudioCaps. The readers may question the reliability of the metrics to other audio datasets across different types: Clotho and MusicCaps to name a few.

* The reviewer is not certain about how we should utilize and scale T2A-Feedback because there was no analysis on the impact of the proposed dataset's scale to the model's improved quality. For example, how minimal amount of T2A-Feedback data is needed to elicit such ability? Does the improvement in quality exhibits a ceiling past the scale presented in current experiment? Since T2A-Feedback is one of the first effort in automated preference data generation (which is important direction, granted), I'd like to see deeper insights that can be learned from the proposed method.

questions:
* After the preference tuning (either with Audio-Alpaca or T2A-Feedback), FAD seems to suffer noticeable degradation. As FAD has been considered a go-to metric in T2A community, this may come as a surprise. Can the authors provide explanations and/or analysis on why FAD fails to capture the T2A model's true quality?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730445345206

thread_type:
official_review

<thread_object>
creator: reviewer_xq56

<content>
summary:
This paper addresses the challenge of aligning Text-to-Audio generation models with human preferences, especially for complex audio prompts involving multiple events. The authors first introduce three scoring pipelines, Event Occurrence Score, Event Sequence Score, and Acoustic Harmonic Quality, that measure the basic capabilities of TTA generation. Then construct a T2A-Feedback dataset. Also, a benchmark T2A-EpicBench is derived to evaluate the advanced capabilities of T2A models. In the experimental section, the paper first verifies the effectiveness of the proposed evaluation measures and then uses the constructed dataset for preference tuning, thereby enhancing the performance of state-of-the-art models.

strengths:
This paper is well-formulated and clear in structure, starting from the current deficiencies in TTA models, such as event occurrence, sequence prompt-following, and quality issues. It then progresses to the construction of the scoring pipelines, the introduction of the dataset and benchmark, and finally, demonstrates how preference-tuning with the dataset improves model performance. Each AI feedback scoring metric is described in detail, making it easy to replicate. The clear presentation of quantitative and qualitative analyses helps intuitively showcase the effectiveness of these innovations in enhancing the T2A model.

weaknesses:
1.	The paper does not mention the impact of the validation dataset on other models, such as AudioLDM 2 or Tango 2, to ensure the dataset’s generalizability. Additionally, the benchmark has not been tested on other models, making it difficult to determine the benchmark’s discriminative power and effectiveness.
2.	Lines 513 to 515 lack further analysis on why the model performs well in T2A-EpicBench’s long-text scenarios, despite T2A-Feedback focusing more on short-text and single-event descriptions.

questions:
1.	In lines 223-225, could you explain how, after determining the threshold value, the separated event’s occurrence time in the original audio is identified?
	2.	In line 372, the paper is positioned to address the sequence prompt-following issue for complex audio prompts involving multiple events. Were experiments conducted on samples with more than two events?
	3.	In lines 428-429, why were DPO and RAFT chosen for preference tuning instead of other preference tuning methods (such as PPO)?
	4.	Could you analyze whether the increase in FAD after fine-tuning in Table 4 contradicts the improvement in the acoustic quality metric?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655129030

thread_type:
official_review

<thread_object>
creator: reviewer_la3e

<content>
summary:
This paper focuses on constructing a preference feedback dataset for Text to Audio (T2A). It proposes a fine-grained scoring pipeline that relies on both AI models and human annotations to collaboratively establish an evaluation metric. This metric assesses both prompt-following (including event occurrence and sequence) as well as acoustic and harmonic quality. Based on this framework, the authors have developed the T2A-Feedback dataset. Additionally, they introduce T2A-EpicBench, a challenging set of evaluation samples.

strengths:
1. The motivation presented in the paper is very clear. Effectively evaluating the generation quality of TTA, aligning generated audio with human perceptual systems, and designing reliable evaluation sets are all crucial issues in the TTA and audio generation fields.

2. The methodology is reasonable. Event occurrence and sequence, as well as acoustics and harmonic quality, are indeed three important dimensions in the TTA problem. Utilizing AI models to assess event occurrence and sequence, while employing human scoring for acoustic and harmonic quality evaluation, creates a hybrid assessment system that is robust and lends itself to automated scaling.

3. The experimental results are good. The results of preference alignment and the samples on the demo page demonstrate the effectiveness of the T2A-Feedback dataset.

weaknesses:
1. My primary concern pertains to the scoring pipeline for event occurrence and sequence. In the current design, audio source separation is a critical component. From my experience, audio events in TTA datasets are often quite mixed, with multiple events potentially occurring simultaneously. The existing source separation models seem to struggle with effectively isolating various events. Furthermore, these separated results need to be accurately matched with the multiple event descriptions generated by the large language model (LLM). The authors do not appear to showcase any examples of audio separation, nor do they demonstrate how these separations match with the multiple captions generated by the large language model (LLM) on the demo website or in the supplementary materials. Additionally, the experimental setups presented in Tables 1 and 2 are overly simplistic; for instance, Table 1 only compares CLAP, and Table 2 utilizes samples with only two events. This lack of complexity seems insufficient to substantiate the reliability of the AI models currently in use.

2. Regarding the assessment of acoustic and harmonic quality, the authors suggest that only 1,000 labeled samples are necessary to train a satisfactory subjective quality predictor. Intuitively, this number seems too small. In such a limited training/testing experiment, even if the predictor performs well, it may simply be due to the overly simplistic distribution of the existing TTA dataset.

3. Concerning practical applicability in real-world scenarios, I acknowledge that the alignment learning via T2A-Feedback has led to some improvements in model performance as observed on the demo website. However, the overall quality of the models remains inadequate and far from sufficient to support applications in real-world settings.

questions:
I hope to see a response during the rebuttal phase addressing my concerns regarding the reliability of the event separation and event sequence produced by the AI model. Specifically, I would like to see more concrete qualitative or quantitative results. If the authors can adequately address this issue, I would be willing to reconsider my rating to accept.
</content>
</thread_object>

</thread>


Metareview: