ADIFF: Explaining audio difference using natural language

Authors: Soham Deshmukh, Shuo Han, Rita Singh, Bhiksha Raj
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: audio processing; audio-language; multimodal learning

Abstract:
Understanding and explaining differences between audio recordings is crucial for fields like audio forensics, quality assessment, and audio generation. This involves identifying and describing audio events, acoustic scenes, signal characteristics, and their emotional impact on listeners. This paper stands out as the first work to comprehensively study the task of explaining audio differences and then propose benchmark, baselines for the task. First, we present two new datasets for audio difference explanation derived from the AudioCaps and Clotho audio captioning datasets. Using Large Language Models (LLMs), we generate three levels of difference explanations: (1) concise descriptions of audio events and objects, (2) brief sentences about audio events, acoustic scenes, and signal properties, and (3) comprehensive explanations that include semantics and listener emotions. For the baseline, we use prefix tuning where audio embeddings from two audio files are used to prompt a frozen language model. Our empirical analysis and ablation studies reveal that the naive baseline struggles to distinguish perceptually similar sounds and generate detailed tier 3 explanations. To address these limitations, we propose ADIFF, which introduces a cross-projection module, position captioning, and a three-step training process to enhance the model’s ability to produce detailed explanations. We evaluate our model using objective metrics and human evaluation and show our model enhancements lead to significant improvements in performance over naive baseline and SoTA Audio-Language Model (ALM) Qwen Audio. Lastly, we conduct multiple ablation studies to study the effects of cross-projection, language model parameters, position captioning, third stage fine-tuning, and present our findings. Our benchmarks, findings, and strong baseline pave the way for nuanced and human-like explanations of audio differences.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730562850757

thread_type:
official_review

<thread_object>
creator: reviewer_6ejz

<content>
summary:
The authors propose a new derived dataset from AudioCaps and Clotho for describing differences between audios in three different tiers corresponding to the complexities of outputted descriptions. They also propose ADIFF, an architecture improving upon baseline by adding a separator token and cross-projection layer for various captioning metrics.

strengths:
- The proposed task in audio difference is novel and interesting. This is a good contribution to the audio reasoning research community. The three-tier approach makes sense and these generated descriptions can be utilized in various research topics.
- The authors provide thorough ablation studies in proposed architectures including architectural choices, language model sizes, and training scenarios, etc.

weaknesses:
- The choices of captioning metrics seem insufficient. As also observed by the authors in section 5.1, these metrics can be biased to the words irrelevant to the differences in audio. To complement the metrics, it might worth adding other benchmark such that the differences in sounding objects can be captured from the SED output from each audio, or the characteristics of audio can be derived from other signal processing techniques?
- Showing the complete captioning metrics in all the tables (2, 4, 5, 6) does not seem to help in the narratives, especially not many of them are referred in the text. It might worth consolidate and only show a couple, for example, in the audio captioning research, the community is starting to converge to several more representative metrics that are more relevant to the task 1. Also it might be helpful to bulk the high performant numbers that are also referred to in the text.
- In table 1, is the number of examples from the number of audio in each dataset? It seems that the proposed task requires two audio, therefore the dataset should be consisted of {a_i, a_j, e_i}? If this is the case, how many pairs are included in each of the split? Also, in the original dataset, each audio can include more than 1 caption (except the train split in AudioCaps), how are these captions selected as the seed for LLMs to generate descriptions, these are not clearly explained in the main paper nor in the appendix.

1 https://dcase.community/challenge2024/task-automated-audio-captioning#evaluation

questions:
- What are the types of differences described in different tiers? Is there a distribution around differences in sounding objects, differences in sounding characteristics within the same category of sound? It would be really helpful to provide these analysis from LLM generated captions.
- For table 3, is the human evaluation also based on the difference between two audios? If so, other than the three datasets utilized (Studio, FSD50K, GTZAN), is there a reason why proposed ACD and CLD are not included in the human evaluation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644942851

thread_type:
official_review

<thread_object>
creator: reviewer_syvc

<content>
summary:
The authors introduces a new task, to explain differences between audio recordings using natural language. The authors show that current audio language models cannot complete this task, and propose AudioCaps and Clotho "Difference" sets, generated by LLM and verified by human in test set. These sets are trained using ADIFF, the proposed model, to bring performance boost.

strengths:
The paper proposes a valid methodology in generating "audio difference" captions through LLMs using human captions; the methodology is verified through experiments.

weaknesses:
1. The three-tier captioning process seems arbitary; no ablation study is conducted for this (i.e. if trained only on tier-3, can the models have good performance on tier-1? If they are purely hierarchical, then training on tier-3 should in theory yield good results on lower tiers.)
2. Both AudioCaps and Clotho's captioning quality is questionable; WavCaps and other larger caption datasets contain more diversity, yet no experiment is conducted on them.
3. The SOTA model, Qwen-Audio, seems to not be finetuned for this task as well. It prompts the question that finetuning a stronger model with out ADIFF may solve the problem entirely just through the proposed dataset.

questions:
I suggest the authors complete the ablation experiments and additional finetuning experiments suggested.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709673737

thread_type:
official_review

<thread_object>
creator: reviewer_uxzd

<content>
summary:
This paper introduces a novel task focused on understanding and explaining the differences between audio recordings through various human perception primitives. It proposes a language model-based (LM-based) approach to generate explanations for the differences between two audio samples, establishing a pipeline that utilizes a large language model and existing audio metadata to create different tiers of audio differences. The experiments evaluate various configurations of the proposed audio difference model, considering factors such as text-encoder size, model components, and multi-stage training paradigms. Additionally, the paper discusses the challenges associated with creating annotations and explores potential real-world applications for this task.

strengths:
The paper contains three strengths:

1.  The paper introduces a considerable novelty by defining a new audio task inspired by the development of multimodal language models. The approach to explaining audio differences effectively enhances the LLM's ability to understand audio inputs at both the semantic and acoustic levels. Additionally, the method incorporates primitives of human perception into the dataset creation process, progressively decomposing audio elements from audio events to acoustic scenes, signal-level analysis, emotions, and other properties. This enriches the data quality and practical usability.

2. The evaluation is thorough, primarily focusing on ablation studies. The experimental framework is well-structured, starting with a baseline model and a reference state-of-the-art model (Qwen-Audio Chat), followed by ablation studies that investigate different design choices within the proposed model.

3. The paper includes thoughtful discussions regarding the hallucination tendencies of the LLM when generating annotations. It provides insights into how the authors filter these data using existing methods to enhance dataset quality.

Overall, I find this paper valuable and believe it makes significant contributions to the audio community.

weaknesses:
There is one weakness of this paper:

1. The presentation of the experimental section requires further refinement. The experimental tables are somewhat difficult to follow, as the model names are not consistently listed in most of them. Additionally, highlighting or bolding the best values for each metric would enhance clarity and emphasize the differences more effectively.

questions:
1.Regarding the structure of the sequence sent to the cross-projection layer in Figure 3, what is the length of the audio prefixes (i.e., [prefix 1] and [prefix 2])? From my understanding, HTS-AT provides only one audio embedding for the entire segment of audio samples. Do [prefix 1] and [prefix 2] consist of just a single token representing the latent embedding from HTS-AT?

I suggest including a more detailed explanation of this aspect, rather than solely focusing on the right side of Figure 3.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730847108611

thread_type:
official_review

<thread_object>
creator: reviewer_edmr

<content>
summary:
This paper proposes a new task  - Audio Difference Explanation, which involves generating natural language description of differences between two audio recordings. They motivate this new task by pointing its significance in fields like audio forensics, audio quality assessment, and synthetic audio generation. They point out that  the current landscape of Audio-Language models like SALMONN and GAMA are geared towards applications like audio question answering, audio captioning, etc but havent addressed audio difference explanation. 

The authors introduce new datasets Audio Caps Difference (ACD) and Clotho Difference (CLD). These contain pair of audios with text representing difference between the audios generated by an LLM. The text descriptions are at three levels of detail: concise, brief, detailed. These descriptions are human verified and hallucination corrected for the test subset. 

The authors propose a model ADIFF to perform this task. This model encodes the two audios, followed by a projection layer to project these embeddings in LLM space. inserts a separator token in the latent space and concatenates the BPE tokenized representations from a question of the form "Explain the difference between the two audios". They pass this through a cross-projection layer, the output of which serves as an prompt to the LLM backbone (GPT2 in their work), which in turn produces a textual description of the difference between the audios. The authors indicate a 3 stage training of their model - 1. unimodal training of audio encoder, LLM (using pretrained models suffice), 2. audio grounding in language and 3. finetuning all components.

The authors provide ablations on how the cross projection layer is important. As a baseline, authors propose a "naive" baseline - which is similar to existing work in literature. The authors also perform both objective and subjective evaluations of their proposed model, and perform various ablations to understand it better.

strengths:
- Introduction of a new task - The paper addresses a previously underexplored area by defining the audio exploration task and the paper motivates this task well.

- Dataset creation and availability -  The authors took great care in creating new datasets ACD and CLD for this task - which is a significant contribution for motivating further research on this task. 

- Novelty in Model Architecture - the authors build upon existing literature to introduce a new cross projection layer that helps compare the two audios properly - a component which is absent in existing literature on audios.

- Comprehensive Evaluation - The authors do both subjective and objective evaluations to establish the models performance. The authors also include an ablation with random weights for Audio encoder - thus establishing a lower bound on the objective evaluations.

- Hallucination detection tools - Authors provide means to detect hallucinations in the models responses using audio event predictions via a HTSAT encoder.

weaknesses:
- Absence of a concrete baseline - the baseline the authors compare ADIFF against is a clearly inferior version of the same model, and thus guaranteeing that ADIFF would perform better than this baseline. This baseline is more of an ablation of the components of the model.


- A major contribution of this work is "cross projection" layer that distinguishes ADIFF from baseline and existing literature. However there is Insufficient evidence of importance of the cross projection layer due to two factors:

1. The authors motivate that cross projection layer is very important by utilizing the bottom two sections of Table 4 where middle section represents  baseline model with frozen LLM, and lower section indicate baseline model + cross projection layer and sep token. But in the main model ADIFF, they finetune the LLM too, which could also learn what cross-projection learns, hence making having a cross projection layer redundant. If the comparison was between  baseline + LLM finetuned vs baseline + cross projection layer + LLM finetuned, it would indicate more conclusively that cross posting is useful.

2. Authors use GPT2 as the LLM model which ranges from a few hundred million params to 1.5 billion params. Its unclear which LLM was used for the cross projection ablation. Arguably, a larger LLM + finetuning could learn the task of cross projection layer. Its been shown that prompt tuning is very powerful for models like Llama3 8B models, and this ablation might not yield any difference there.


- The paper’s structure and presentation can be improved. There is redundancy and lack of detail in tables and captions —such as the overlapping results in Tables 2 and 4, missing key information about the LLM model used, and insufficient explanations of scores presented in Table 3, which collectively make it challenging to interpret the results accurately.

- The paper inaccurately equates compute across different-sized models in section 5.3 by training them for the same number of epochs, failing to account for the higher per-epoch computational cost of larger models. This undermines the  validity of their scaling comparisons in this section.

questions:
- Its unclear from the paper why  Qwen-Audio Chat is a valid baseline for subjective evaluation in Table 3. The authors mention that its the only ALM that supports two audio inputs, but thats not a reason enough to use this as a baseline for subjective evals if it is not designed to do audio difference explanation task? 

- Insufficient information on human evaluation - its unclear how many evaluations were done here, how many reviewers were involved and how it was ensured that the evaluation is fair. Its unclear what the numbers in Table 3 mean, and hence deriving insights out of them is hard.

- Authors mention "ADIFF sees the largest improvement in granularity metric", but this is not very clear from the table 3 itself, a more detailed table, maybe in appendix that highlights this could be more readable.

- Its unclear why the datasets used for Table 3 different from ACD and CLD datasets?

- Its unclear how the LLM is finetuned. Is a full finetune performed? How about using PEFT  instead and utilizing larger models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731598558159

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Rebuttal

comment:
We sincerely thank all the reviewer for recognizing our contribution and providing constructive feedback, especially for acknowledging that "**The paper introduces a considerable novelty by defining a new audio task**" (Reviewer uxzd), "**The proposed task in audio difference is novel and interesting and good contribution to the audio reasoning research community.**" (Reviewer 6ejz), "**Dataset creation and availability ... which is a significant contribution for motivating further research on this task**" (Reviewer EdMr), "**The authors do both subjective and objective evaluations to establish the models performance.**" (Reviewer EdMr), and **thorough ablation studies in proposed architectures**" (Reviewer 6ejz).

---
We would like to re-emphasize the novelty and technical contributions of this work. We present the task of explaining audio differences and subsequently propose benchmarks and baselines for the task. Our key contributions are: (1) New Audio Task and Datasets: we introduce a novel audio difference explanation task, motivating its relevance and need within the field. Creation of new datasets (ACD and CLD) with three tiers which are verified by human annotators for the test set. (2) Introduce ADIFF, which extends the naive prefix tuning baseline with a novel cross projection layer, three-step finetuning process, leading to consistent improvement across all metrics. (3) Thorough evaluation including subjective and objective methods, ablation studies, tools for detecting hallucinations, and presenting our findings for motivating subsequent works in small audio-language models. 

We summarize the two main questions brought up by the reviewers and address them here: \
**Question 1: Stronger baseline to compare against ADIFF. Finetuning the SOTA model Qwen-Audiofor this task using PEFT and full-finetuning.**\
We appreciate the reviewer's suggestion and assistance in improving our work. We selected Qwen-AC as the SoTA baseline because it is the only ALM in the literature that supports two audio inputs. However, we agree that it makes sense to fine-tune Qwen-AC on the proposed dataset and include it as a baseline. We have conducted the suggested experiments, fine-tuning Qwen-AC on the proposed data using Parameter-Efficient Fine-Tuning (LoRA) and full fine-tuning. The details have been added to Section 4.1, with the test set results of the proposed dataset included in Table 2 and the results with subjective evaluation presented in Table 3. Additionally, we expanded Section 4.2 to provide an analysis of these results and compare them against ADIFF. We hope these additional experiments address the concern about stronger baseline.

**Question 2: The paper’s result presentation can be improved. Showing the complete captioning metrics in all the tables (2, 4, 5, 6) does not seem to help in the narratives, especially not many of them are referred in the text. It might worth consolidate and only show a couple. Also it might be helpful to bold the high performant numbers that are also referred to in the text.**\
We appreciate the reviewer’s suggestion. We have updated the paper, specifically the result tables (Table 2, 4, 5, 6), to improve readability. First, we retained only the BLUE_4, METEOR, and SPIDEr metrics in the main paper's result tables as suggested by reviewer 6ejz. We did not include SPICE and CIDEr in the main result table as its linear combination is SPIDEr. The complete tables including SPICE and CIDEr metrics are available in the appendix and are referenced in the main paper's result tables. Second, in the new format, we bolded the best-performing numbers for each metric across models on the ACD and CLD tasks for each Tier. We hope these changes significantly improve the readability and ease of interpreting the results.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper deals with explaining audio differences between pairs of recordings. It presents a new audio difference explanation task, trained using two datasets. The explanation was designed according to three tiers and verified by human annotators. The audio difference explanation system ADIFF  takes two audio recordings and text prompt as input, encoding and projecting the audio embeddings to a latent space through a cross-projection layer, which is then used as a prompt to a transformer decoder to generate natural language explanation. The experimental section includes subjective and objective evaluation, ablation studies, and other special tasks. Issues of hallucinations are mentioned as concern to be addressed.