ADIFF: Explaining audio difference using natural language
Authors: Soham Deshmukh
Shuo Han
Rita Singh
Bhiksha Raj
Venue: ICLR 2025 Spotlight (2025)
Keywords: audio processing; audio-language; multimodal learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A new task, dataset and model for explaining audio differences using natural language

Abstract: Understanding and explaining differences between audio recordings is crucial for fields like audio forensics, quality assessment, and audio generation. This involves identifying and describing audio events, acoustic scenes, signal characteristics, and their emotional impact on listeners. This paper stands out as the first work to comprehensively study the task of explaining audio differences and then propose benchmark, baselines for the task. First, we present two new datasets for audio difference explanation derived from the AudioCaps and Clotho audio captioning datasets. Using Large Language Models (LLMs), we generate three levels of difference explanations: (1) concise descriptions of audio events and objects, (2) brief sentences about audio events, acoustic scenes, and signal properties, and (3) comprehensive explanations that include semantics and listener emotions. For the baseline, we use prefix tuning where audio embeddings from two audio files are used to prompt a frozen language model. Our empirical analysis and ablation studies reveal that the naive baseline struggles to distinguish perceptually similar sounds and generate detailed tier 3 explanations. To address these limitations, we propose ADIFF, which introduces a cross-projection module, position captioning, and a three-step training process to enhance the model’s ability to produce detailed explanations. We evaluate our model using objective metrics and human evaluation and show our model enhancements lead to significant improvements in performance over naive baseline and SoTA Audio-Language Model (ALM) Qwen Audio. Lastly, we conduct multiple ablation studies to study the effects of cross-projection, language model parameters, position captioning, third stage fine-tuning, and present our findings. Our benchmarks, findings, and strong baseline pave the way for nuanced and human-like explanations of audio differences.

Published as a conference paper at ICLR 2025
ADIFF: EXPLAINING AUDIO DIFFERENCE USING NAT- URAL LANGUAGE
Soham Deshmukh
Shuo Han
Rita Singh
Bhiksha Raj
Carnegie Mellon University
(adeshmuk, shuchan, rsing rig, bhi kssarr Band cmu edu
ABSTRACT
Understanding and explaining differences between audio recordings is crucial for fields like audio forensics, quality assessment, and audio generation. This in- volves identifying and describing audio events, acoustic scenes, signal character- istics, and their emotional impact on listeners. This paper stands out as the first work comprehensively study the task of explaining audio differences and then propose benchmark, baselines for the task. First, we present two new datasets for audio difference explanation derived from the AudioCaps and Clotho audio cap- tioning datasets. Using Large Language Models (LLMs), we generate three levels difference explanations: (1) concise descriptions of audio events and objects, (2) brief sentences about audio events, acoustic scenes, and signal properties, and (3) comprehensive explanations that include semantics and listener emotions. For the baseline,
prefix tuning where audio embeddings from two audio files are used to prompt a frozen language model. Our empirical analysis and ablation studies reveal that the naive baseline struggles to distinguish perceptually similar sounds and generate detailed tier 3 explanations. To address these limitations, we propose ADIFF, which introduces a cross-projecdt pross-prroectiion module, position captioning. and three-step training process to enhance the model's ability to produce de- model using objective metrics and human
tailed explanations. We evaluate evaluation and show our model enbancements lead to significant improvements performance over naive baseline and SoTA Audio-Language Model (ALM) Qwen Audio. Lastly, we conduct multiple ablation studies to study the effects of cross-projection, language model parameters, position captioning. third stage fine- tuning, and present our findings. Our benchmarks, findings, and strong baseline pave the way for nuanced and human-like explanations of audio differences.
INTRODUCTION
In the Boston Marathon bombing of 2014, forensic in- vestigators were faced with challenge were the var- ious audio recordings purportedly captured of the event (and put up social media) by various people record- ings of the same event, or were they mistaken or fraudu- lent uploads actually from different events? This anec- dote highlights the complexity of audio analysis. One have different sounding recordings of the same event, similar sounding recordings of different events, and other combinations. To know the relation, one must be able describe the recordings in relative or comparative terms. This natural human ability used various contexts- audio forensics to verify recording authenticity (Maher 20099; Kraetzer et al. (2007)), crucial for legal investi- gations. In production and broadcasting, it aids in Audio Quality Assessment (Campbell et al. (2009); Deshmukh et al. (0244a) to detect subtle variations. In audio gener- ation [Liu et al. y; Kreuk et al.). it helps create realistic synthetic audio. Currently, identifying
Listens to both audio files
dog barking, cat mecwing
dog barking, door closing Indoor, kitchen
Acoustic
charadteritics
low intensity
loud and pitched
Sequence
bark afler meow
bark before door dosing
Emotion evoked
calming
intimidating
Deduction
The audio clips not belong the because
Figure I: Humans use auditory information to compare scenes and make deductions.
audio differences requires a human listener with expertise in phonetics (Johnson & Johnson 2004

---

Published as a conference paper at ICLR 2025
the acoustic-phonetic approach (Stevens (2000),, ) and spectrogram analysis to examine recordings and identify differences across various parameters.
Recent advancements in audio-text learning has enabled new applications such as text-to-audio
trieval (Koepke et al. (2022); Deshmukh et al. (2023b)), audio captioning (Mei et al. (2022a); Desh-
mukh et al.] (2024b)), and audio question answering (Lipping al. (2022): :: Gong al. (024
To excel in these applications, models must possess both fine-grained and boare-graimed under- standing of audio, enabling them to describe audio content in human-like natural language. By pretraining models on multiple audio-text tasks, individual audio-text tasks see a benefit in perfor- mance improvement (Deshmukh et al.) (2023a)). Therefore, the field has shifted towards training audio models capable of performing the atorementioned audio-text tasks using single model. Ex- amples of such Audio-Language Models (ALM) include SALMONN (Tang et al. (024b)); GAMA (Ghosh et al. (2024a)), and LTU (Gong et al. 2024, 2023)) However, the current literature has not addressed the task of audio difference explanation. This is due to the challenges in generating coherent paragraph-Ievel descriptions, the lack of cognitive processing and understanding similar to human auditory perception, and the absence of annotated datasets specifically designed for this task.
Explaining the differences between two audio samples also acts as an effective benchmark for as- sessing a model's comparative reasoning and its ability to integrate audio information with world knowledge. When model is tasked with distinguishing between two audios, it must employ com- parative reasoning to identify both subtle and significant details. This requires the model to un- derstand the fundamental properties of audio signals, such frequency, amplitude, and temporal patterns, as well as determining pitch, timbre, and loudness. Beyond these signal characteristics, the grasp contextual elements, like the genre of music or the
model must also use world-knowledge type of environment where the audio was recorded. Finally, the model must use comparative and deductive reasoning to interpret and compare these features, identifying nuanced differences and similarities, and draw conclusion. This makes audio difference explanation an effective benchmark for evaluating audio-text models and exploring methods for integrating audio information, world knowledge, and comparative reasoning.
In this paper, our main contributions are:
Introduce the audio difference explanation task which aims to provide natural language explanation for the differences in two audio files. To build and benchmark models for this task, we create two datasets ACD and CLD, where the difference explanation is generated by LLM using human annotated captions and later verified by human annotators (test set). To mimic human explanations, each dataset contains three tiers of explanation ranging from one- sentence (audio events) to detailed explanations (scene, semantic, listener emotions).
We propose a naive baseline for this task using prefix-tuning. In this approach, audio em- beddings from the two audio files are used to prompt frozen language model. To address the limitations of this naive baseline, we introduce the ADIFF model, which incorporates cross- projection layer and undergoes a three-stage training process. These enhancements improve the model's 's ability to identify detailed differences and distinguish perceptually similar sounds, resulting in a performance improvement over the naive base- line and SoTA ALM (Section 4 The checkpoint will be publicly released
separator token, cross
Under the proposed framework, we conduct several ablation studies to understand the im- pact of cross projeetro,,,,oon language model scaling, audio- groundigo and fine-tuning. Our findings reveal that (1) cross-projection aids in utilizing text prefixes to store difference at- tributes, (2) under limited compute and data, smaller language models are easier to ground in audio with proper training and (3) position-basee audio captioning with multiple audio inputs enhances the model's performance on similar-sounding acoustic sources. Lastly, we address language model hallucinations by enabling users to detect inaccuracies through comparisons between generated descriptions and predicted audio event probabilities.
AUDIO DIFFERENCE EXPLANATION
The task of explaining audio differences consists of finding the difference between two audio files and explaining the difference in natural language. Human explanations draw on various sources of information, including acoustic details and human perception, well as linguistic nuances. From
Dataset and pretrained model available at httpp/ps/gggitubbccc///eeeeeeeee
2

---

Published as a conference paper at ICLR 2025
an acoustic standpoint, explanations range from broad audio event differences to finer- finerggrined signal variations. They can also be based on objective facts and world knowledge or on human perceptual differences. Linguistically, these explanations be concise, like single sentence, or more detailed, like a paragraph. This allows us to segregate audio difference explanations that vary in detail, focus, and style:
Concise:
This description is concise and straightforward, briefly mentioning the key characteristics of each audio without much elaboration. It highlights the main differences in ambiance and focus between the two audios. We refer to this as the lier-1 explanation.
This answer provides more detail and context. It not only describes the sounds but also includes audio events, sound sources, the nature of the sounds, making it slightly more analytical. It also describes the sequence of sounds (temporal order) and potential scene differences across audios. We refer to this as the tier-2 explanation.
Brief:
Detail: This is the most detailed and descriptive answer. It delves into the sonic qualities of each audio, including potential sources and the listener's experience. It compares the two audios in terms of complexity and engagement, providing a richer and more immersive description. It analyzes the audio events, acoustic scenes, sound sources, signal characteristics, tonal differences, and overall feel of each audio. We refer to this as the tier-3 explanation.
The first-tier explanation is concise and to the point, the second-ties -ien explanation provides more context, and the third explanation offers a thorough, immersive analysis. A secondary benefit of dividing the explanation into tiers allows us do fine-grained analysis of model performance.
2.1 AUDIO DIFFERENCE DATASET
In this section, we outline the development steps for the AudioCaps Difference (ACD) and Clotho Difference (CLD) datasets.
Concise
Rade Tisttrns dorenart speach signal with occaniene belches while commonee speech
Griet
Audio recordings. We source the audio recordings from the AudioCaps and V21 datasets. The AudioCaps dataset comprises 46,000 audio samples, each lasting 10 seconds, sourced from AudioSet, and includes human- annotated captions that describe the audio con- tent. Annotators had access both audio and visual cues during the annotation process. On the other hand, the Clotho dataset, though smaller, features audio samples ranging from 15 to 30 seconds, sourced from the Freesound platform, and includes five human-annotated captions per sample. These annotators only had access to audio information and followed de- tailed protocol to minimize errors and ensure diversity in the captions. By utilizing both the AudioCaps and Clotho, we ensure variability in audio content, duration, and annotation style.
The phrnayy offivernce behwear the hou audi be that the first cito mediame a clear andil riall dide candeno spokan nerraiie paraiaker auztle bedigronn rodue
seajjrning
loassing
and wird blowing which alguticcaityy mty weteen ounce funnic terten and letereek parraptiion
Detei
ioet the wo audios, not that the fint is leater ag a mon saking felowed by soriee suinees and feqquency trage 11at spons from low high pliches The be thes that brlow have evem auster mread and duratin,, cheree "'uft"
bucaing and wid blowing which cruples more oor occurdo endiroemant The modero hurm idbaddy with higher ireuusncy Thes
seil clear, siihtly. mutted by ar artseett noies mator and wind, creatrn a muroe dieaacc tradig round. more sbout the intapplay between the mant voice and the surrs riling an anvrrme vial soundds.
Figure 2: A random sample from the ACD dataset is displayed across three levels of explanation. The top pane provides concise explanation, the middle pane offers brief explanation, and the bottom pane presents a detailed explanation.
Difference explanations. Large Language Models (LLMs) have been effectively utilized gen- erate text descriptions for audio across various tasks, such as question-answering (Gong et al. (2044) ) compositional reasoning (Ghosh al. (2024b)a). and deductive reasoning (Deshmukh et al. (024c)). We adopt a similar approach for generating explanations for the audio difference task. The process involves three main steps: data sources, explanation generation, and explanation verification.
For data sources, we limit the audio recordings and human-anmotated descriptions to those from Au- dioCaps and Clotho To generate explanations, we prompt an LLM to describe the differences between two audio recordings using the provided human--nnotated descriptions. Our prompting setup is similar to (Deshmukh et al. (2024c)), with two key modifications. First, we instruct the LLM to incorporate knowledge about sound sources, materials, acoustics, and emotions when gen- erating explanations. Second, we define the tier of explanation by restricting the sources the LLM can use and the length of the explanation. Therefore, we sample two annotated human descriptions and prompt L.LM generate different tiers of explanations. Finally, human annotators verify the
3

---

Published as a conference paper at ICLR 2025
difference explanations. If an explanation is found to be inaccurate, the human annotators manually remove the hallucinated audio event and add necessary details. Due to the cost of this process, we restrict verification to the test set of three tiers. Details on prompting setup and data creation The resulting dataset comprises of audio recordings paired with the
available in the Appendix H three tiers of generated explanations.
Tier 1
Tier 2
Tier 3
Examples per tier
Data
Split
Med.
Max
Vocab.
Med.
Max
Vocab.
Med.
Max
Vocab.
CLD
Train
19195
27
49
6528
51
92
6462
155
221
10818
CLD CL.D
Val
5225
27
46
3743
89
4024
154
223
7026
Text
5225
27
28
5225
52
86
4059
156
219
7152
ACD
Train
48660
29
3287
104
8483
155
12891
ACD
Val
2456
28
46
2350
52
87
2563
154
227
4566
ACD
Test
4680
29
47
3287
95
3329
154
220
5489
Table 1: Dataset statistics of AudioCaps Difference (ACD) and Clotho Difference (CLD) dataset
Dataset statisties. The Audio difference dataset consists of {az; ( (i,e)) where o; is first audio, aj is the second audio, and 6gj is explanation belonging to one of the three tiers. The statistics of AudioCaps Difference (ACD) and Clotho Difference (CLD) datasets across three tiers: Train, Validation, and Test splits are presented in Table For example, in Tier 1, the ACD Train split has 48k examples with explanations having a median of 27, a maximum of 49, and a vocabulary size of 6528, while the CLD Train split has 19k examples and the explanations have median of 51, maximum of 92, and a vocabulary size of 6462. A randomly sampled example is shown in Figure
2.2 EVALUATION
For objective evaluation, we use audio captioning metrics- BLEU (Papineni et al. (2002) ). METEOR
(Banerjee & Lavie 2005)) ). SPICE (Anderson al. (2016)) CIDEr (Vedantam et al. (2015) and
SPIDEr (Liu et al. (017)). BLEU measures n-gram precision, while METEOR combines preci- sion and recall with features like stemming and synonymy matching. ROUGE focuses recall of n-grams and sequences, CIDEr measures consensus in audio descriptions, and SPICE evaluates semantic content. SPIDEr combines SPICE and CIDEr for a balanced assessment, making it our primary metric for model comparison. More details on metric choice are available in Appendix [.]
3 MODEL
In this section, describe our proposed model ADIFF, which employs prefix tuning prompt frozen Language Model. The model accepts three inputs: audio 1, audio 2, and a user prompt, and generates free-form text as output. The model architecture is covered in Section 3.1 the training process in Section 3.2
the three stages of training in Section 3.3 and lastly inference in Section 3.4
3.1 ARCHITECTURE
ADIFF consists of four main components: an audio encoder, an audio projection layer, a cross- projection layer, and decoder-only language model.
Audio Encoder. The audio encoder is used for extracting general-purpose purpose audio representations for both the audios. We use HTSAT (Chen et al. (2022) which is pretrained on AudioSet (Gemmeke et al. (2077) and achieves SoTA on identifying various sound events and acoustic scenes.
Projection. The audio projection is used to project the audio embeddings from audio encoder to latent space of Language model. The audio projection converts single embedding into a sequence of latent tokens [s. d]. The projection first expands the hidden dimension to a larger hidden dimen- sion k, which is then split to form [s, d] where k ** d. This is followed by concatenating with a learnable constant, resulting in [s + c, d). This output is passed to the transformer, followed by clipping of the learnable constant output
The resulting output of audio projection is of shape [s, d.. This projection architecture is shown to perform well for prefix-tuning architectures (Desh-) mukh et al. (2023a) 2024b); Mokady al. (2021)). The text projection translates text embeddings with learnable transtormer layers. The architecture is similar to audio projection (Fig 3) without the first linear layer.
Cross Projection. Once the audio embeddings are in the same latent space of Language model, the cross-projection layer is used to improve the model's ability to highlight differences. The cross-
4

---

Published as a conference paper at ICLR 2025
Audio 1 features a dominant low-frequency rumble (sizzling) and male voice, while Audio 2 has a high-pitched, playful laughter and a woman's gentle voice.
Audio projection
Prefix
Transformer Decoder
Clip
Cross projection
Transformer
[prefix| [|epp]])
Learnable constant
Audio
Audio Projection
Text Projection
Concat
Projection
Linear
Audio Encoder
Audio Encoder
Text Embedder
Embedding
Explain the difference between the two audios in few words
Figure 3: ADIFF takes two audio recordings and text prompt as input and generates free-form text as output. The two audios and prompt independently encoded by the audio encoder and text embedder respectively, followed by projection layers to project the emboddings the latent space of the transformer decoder. The two audio latent are separated by a separator token in latent space. The prefix formed by audio latent 1, SEP, audio latent and text prompt prefix is fed to the cross-projection layer. The output of the cross-proeection layer is used to prompt the transformer decoder to generate natural language explanations.
projection layer adds a separator token in latent space between the two audio embeddings followed by transformer layer to learn differences.
Language Model. The language model is a decoder-only language model which is used to auto- regressively generate text conditioned the output of cross-projection and user prompt. We use GPT2 as the language model following prefix tuning literature (Deshmukh et al. (202a 2024b); Mokady et al. (202T)) and due compute limitations.
3.2 TRAINING
The training process the next-token prediction objective learn unfrozen (learnable) parame- ters. A sample input to the model is {ai. ,c') where ui is the first audio, -r is the second audio, t' is the text prompt, and c is the difference explanation respectively. The user provides Audio (x²) and Audio 2 (zz) along with prompt (t') specifying the desired level of explanation. Each audio file independently encoded by the audio encoder (a²), creating separate audio embeddings. Si- multaneously, the user's ' text prompt and separator token (s) is tokenized using a BPE tokenizer and embedded with the language model's vocabulary embeddings (gy). These embeddings including the audios, separator embedding, and text prompt embeddings are then projected into the language model's 's latent space using a cross-projection layer (mc) layer to form a prefix.
of concat mm (x()), (s). m (as((a))) 9o))
pi..., pt = hs(c')
(2)
The combined prefix f of length (tokens) A: in the language models space. This prefix is used to prompt the language model (ff) to generate text explanation highlighting the differences between the two audios. The model trained as a typical captioning system, learning to predict caption (text tokens) of based on the prefix p in an autoregressive manner. The loss function is Cross-Entropy per token:
N
C
log 1ogyy(of (o of., 05-1)
where yy denotes model's 's trainable parameters and consists of o, Ç. tb B. Out of all the parameters- G are always trained while the rest are determined per stage (Section 3.3
5

---

Published as a conference paper at ICLR 2025
3.3 TRAINING STAGES
The model's 's training process involves several stages to preserve audio information and prevent the destructive gradient updates during unaligned full finetuning. These stages are: unimodal pretrain- ing, multimodal alignment training, and finetuning.
Unimodal Pretraining. Unimodal pretraining entails training the audio encoder and language model on tasks specific to their respective modalities. For the audio encoder, this means training on AudioSet to predict audio events. For the language model, involves pretraining on large cor- pus of text. For the audio difference task, we skip independent modality pretraining and instead use pretrained unimodal models like HTSAT and GPT-2 which have performed the unimodal training. Multimodal grouding. After unimodal pretraining, the language model must be grounded in the audio to generate audio-conditioned text. During this stage, both the audio encoder and language model are frozen, and the model learns the audio projection and cross projection using cross-entropy loss for each predicted next text token. Freezing the audio encoder and language model ensures that the initial gradient updates do not cause the loss of modality-spcciffe information. This approach is similar to the -ttniig literature, where the language model remains frozen (Deshmukh et al. (023a); Mokady et al. (202T).
Finetuning. In this final stage, we finetune the audio-grounde language model by unfreezing all the modules. This allows us retain audio-specific information and better steer the language model to produce the necessary descriptive answers using acoustic terminology. This stage uses a low learning rate with warmup and minimum training steps minimize catastrophic forgetting.
3.4 INFERENCE
During inference, the prefix created from the two test audios and the user-provided text prompt. The decoder (language model), generates the next token in sequence based this prefix. At each step, the model assigns probabilities to all vocabulary tokens, which are then used to select the next token according to the chosen decoding method. In our experiments, we employed top-k and top-p decoding for better performance across all experiments. The details on experimental setup, training, inference, and implementation can be found in Appendix G
Tier
Tier 2
Tier 3
Tusk
Models
BLEU,
METEOR
SPIDEr
BLEU,
METEOR
SPIDEr
BLEU
METEOR
SPIDEr
Baseline
0.118
0.220
0.163
OwenAC (L)
0,132
0,214
0.235
0.166
0.204
0.212
0.165
0,202
0.173
ACD
QwenAC (F)
ADIFF
0.135
0,221
0,303
0.180
0.197
0.345
0.171
0,208
0.183
Bascline
0.128
0.237
0.212
0.233
0.234
0.641
0.157
0.199
0.166
OwenAC (L)
0.140
0,285
0,230
0.232
0.236
0.756
0.155
0.200
0.182
CLD
QwenAC (F)
ADIFF
0.203
0.302
0.652
0.213
0.235
0.692
0.191
0.220
0.417
Table 2:
Benchmarking different models on Audio Difference task. The top half of the table shows the
performance of different models on the task of ACD, while The bottom half of the table shows the performance of different models the task of CLDD All the models trained ACD and CLD train split. The Tier 2,
and 3 classifications correspond the explanation tiers detailed in Section 2 the average score are available in Appendix Table [5
The results with metrics and
4 RESULTS
4.1 EXPERIMENTAL SETUP
The base model includes an audio encoder, audio projection, and language model (GPT2-base). It is trained end-to-end with the language model frozen, using captioning loss. This is referred to as the baseline or naive model. For ADIFF, we add separator and cross-projection layer to the architecture. ADIFF's training process has three stages to preserve audio information and prevent destructive gradient updates during unaligned full finetuning: unimodal pretraining, multimodal alignment training, and finetuning, as detailed in Section 3.3 For SoTA ALMs, we use Qwen-AC Chu et al. )2023) as it is the only ALM in literature that supports two audio inputs. We consider three versions of Qwen-AC: Zero-Shot (Z), LoRA (L), and full fine-tuning (F). All models except for the zero-shot version use the train set of ACD and CLD for training and are evaluated on the test set of ACD and CLD across three tiers.

---

Published as a conference paper at ICLR 2025
Studio
FSDS0K
GTZAN
Average
Model
I.M
COR
GRA
RDB
COR
GRA
RDB
COR
GRA
RDB
COR
GRA
RDB
OwenAC (Z)
2.64
2.76
2.20
2.95
3.33
2.56
3.23
Baseline
128M
2.99
3.26
3.21
3.43
3,36
3.47
3.37
3.28
3.49
3.26
3.35
3.39
QwenAC (L)
7B
3.05
3.41
3.25
3.35
3.73
3.45
3.31
3.29
3.63
3.24
3.48
3.44
QwenAC (F)
3.09
3.50
3.37
3.41
3.54
3.49
3.35
3.32
3.61
3.28
3.45
3.49
ADIFF
128M
3.12
3.73
3.46
3.75
3.53
3.57
Table 3: Human evaluation for generated explanations based on Correctness (COR); Granularity (GRN), and Readability (RDB). For QwenAC, indicates zero-shot, indicates LoRA finetune, and F indicates full- finetuning.
4.2 RESULTS
Objective evaluation. The performance of the naive model, QwenAC versions and ADIFF model is compared in Table 2 using objective metrics described in Section 2.2 The ADIFF model out- performs the naive baseline across all three tiers. Moreover, it also outperforms QwenAC (L) and QwenAC (F) across all Tiers of ACD and CLD except on Tier 2 CLD. For Tier 2, we observe sim- ilar trends as in ablation conducted for baseline architecture and language-only performance [5.] For Tier 2, we observe similar trends to those seen in the ablation studies conducted for the base- line architecture and language-only performance. Linguistically, Tier is the easiest to learn from, followed by Tier 3, and then Tier 1. This is because Tier has the fewest words, with most of them containing audio-related information. In contrast, about 15% of the words in Tier pertain to the linguistic structure of contrasting audios. This indicates that Tier 2's higher scores, even in subsequent experiments, are due to its linguistic simplicity for the model to match. Consequently, QwenAC, which uses a much stronger 7B parameter LLM, performs better than ADIFF, which has a 128M parameter LLM.
Subjective evaluation. The objective metries limited because they rely on linguistic biases, which can unfairly penalize diverse outputs (Morato & Mesaros (2021): Mei et al. (2024a)). There- fore, we also perform subjective evaluation, where human annotators rate the audio difference ex- planations across different dimensions of correctness (I-5), granularity (1-5), and readability (1-5). The dimension definitions and human evaluation setup and is explained in Appendix [.2] The sub- jective evaluation also ensures fair comparison with QwenAC (Z), which is not trained on the ACD and CLD datasets and thus do not align with the data distribution, leading to poorer objective metric scores. We evaluate the models across three scenarios. The first scenario, Studio, involves recordings from the same sources, such as different breeds and numbers of dog barks, all recorded in professional studio. The second scenario involves random samples from FSD50K (Fonseca eet al. (202T)). The third scenario uses random samples from GTZAN Genres (Tzanetakis & Cook (2002), featuring music from various genres and settings. The subjective evaluation results shown in Table 3 Across all scenarios, ADIFF outperforms the naive baseline and Qwen-AC (Z) across all metrics and domains. Specifically, ADIFF sees the largest improvement in the granularity metric, highlighting the model's 's ability to produce in-depth descriptions. ADIFF also beats the fine- tuned variants of Qwen-AC on average across all metrics. However, in two cases-readate readability on Studio and granularity on FSD50K-2 fine-tuned version of Qwen-AC outperforms ADIFF. This is likely because Qwen-AC's LLM, Qwen-7B, 7B, is significantly larger than the 128M LLM of ADIFF. Qualitative analysis shows it produces more coherent output, slightly improving readability. Addi- tionally, we highlight the difficulty of the task, as even ADIFF achieves average scores of 3.5 across the dimensions out of 5.
ABLATIONS
In this section, we examine various components of model architecture and the impact of training methods on audio difference task performance. It covers- baseline architecture and the contribution of linguistics (Section (.1). effect of cross- pross-projection layer (Section 5.2), scaling language model (Section 5.3) position captioning (Section 5.4), and impact of stage three fine-tuning (Section 5.5).
5.1 BASELINE ARCHITECTURE
To build the baseline architecture for the audio difference task, we draw on previous works on prefix tuning (Deshmukh et al. (2023a)). This architecture includes audio encoder extract features, an audio mapper to translate these features into the latent space of a language model, and a frozen language model (GPT2-base) to generate text. For the audio difference task, the audio
7

---

Published as a conference paper at ICLR 2025
encoder to extract features from both audio inputs, followed by mappers to convert these features into the language model's latent space. The frozen language model prompting and the cross-entropy loss setup remain unchanged. We refer this the baseline architecture for this task.
In question- answering and captioning tasks, models have been observed to learn linguistic infor- mation and answer questions without relying on perception modalities like vision audio. This phenomenon is noted both vision and audio literature (Lipping et al.) (2022). Similarly, for the audio difference task, models can game metrics by learning the language patterns instead of analyz- ing the audio, thereby inflating the metrics. To address this, we establish a baseline where the audio encoder is randomly initialized and kept frozen. This approach allows us to determine the max- imum performance achievable without analyzing audio, focusing solely on learning the language patterns for each tier of explanation. We refer to this as the language-only performance for the audio difference task.
We summarize the language-oni, and baseline performance in Table The full table is available in Appendix Table [6] Experiment A presents language-only performance, where the audio encoder randomly initialized and frozen. In contrast, Experiment B shows baseline performance with the audio encoder initialized from HTSAT pretrained weights and frozen. In Experiment A, we observe that Tier 2 is the casiest to learn from a linguistic standpoint, followed by Tier 3, and then Tier 1. This is because Tier I has the fewest words, with most containing audio-related information. Conversely, about 15% of the words in Tier 2 pertain to the linguistic structure of contrasting audios. This suggests that Tier 2's higher scores, even in subsequent experiments,
due to its linguistic simplicity for the model to match. In Experiment B, where the audio encoder is pretrained, the model performs better across all tiers, expected. This indicates that the model architecture effectively leverages audio information to improve explanations.
Tier
Tier 2
Tier 3
Task
Exp.
BLEU,
METEOR
SPIDEr
BLEU,
METEOR
SPIDEr
BLEU,
METEOR
SPIDEr
0,082
0.184
0.128
0.135
0.176
0.154
0.149
0.175
0.107
ACD
0.118
0,210
0,220
0.163
0.193
0.225
0.199
0.188
0.123
0.214
0.155
0.197
0.243
0.149
0.203
0.154
A
0.138
0.223
0.127
0.199
0.236
0.589
0.123
0.166
0, 127
CLD
0.128
0.237
0.212
0.233
0.234
0.641
0.157
0.199
0,166
0.156
0.257
0.195
0.280
0.267
0.904
0.127
0.187
0.196
Table 4: Architecture results. Experiment A is baseline architecture with random audio encoder weights. Ex- periment B is pretrained audio encoder weights. Experiment C is ADIFF which modifies baseline architecture with separator token and cross-projection. The results with all metrics and the average score available in Appendix Table
5.2 EFFECT OF CROSS-PROJECTION
We observe two limitations with the baseline model performance by qualitative analysis. First, when the audio imput includes perceptually similar sounds and the same audio events, the model tends to confuse the distinct information between audios. Second, the model struggles with Tier and Tier 3 explanations, where it needs to discern subtle audio differences such as scenes, acoustics, sources, and their composition rather than just audio events. To address the second limitation and partially the first, we introduce a cross- pross-proecction layer with latent separator token between the two audio inputs. We concatenate the first audio latent embedding, the separator embedding, and the second audio latent embedding to form prefix. The separator embedding is derived from the < lendoftext| > token embedding of GPT-2 base. This concatenated prefix is then passed through the proje-pro layer, which consists of transformer layers with learnable constant vector.
The results with the cross-projection layer are summarized in Table The full table is available in Appendix Table T6] Experiment B shows the performance of baseline architecture. Experiment C shows performance of baseline architecture with separator token and cross-projection layer. On average, we see improvements for ACD and CLD across the three tiers of explanations. Specifi- cally, in Appendix Table [6] the average metric score of Experiment C is consistently higher than Experiment B across all tiers of ACD and CLD.
5.3 SCALING LANGUAGE MODEL PARAMETERS
The performance lasguaelanguage models AAlabdulmohsin et al. (2024)) tends to improve as their scale increases. Empirical evidence suggests that this improvement often follows a predictable
8

---

Published as a conference paper at ICLR 2025
Tier
Tier
Tier
Task
Exp.
BLEU,
METEOR
SPIDEr
BLEU,
METEOR
SPIDEr
BLEU,
METEOR
SPIDEr
Base
0.131
0.214
0.287
0.155
0.197
0.243
0.203
0.154
ACD
Med
0.119
0.211
0.325
0.154
0.198
0.248
0.160
0.201
0.232
Large
0.124
0.221
0.240
0.156
0.197
0.147
0.162
0.207
0,165
XL
Base
0.156
0.257
0.195
0.280
0.267
0.904
U.127
0.187
0.196
Med.
0.149
0.270
0.317
0.264
0.287
0.641
0.143
0.203
0.204
CLD
Large
0.133
0.238
0.26.
0.180
0.223
0.574
0.177
0.191
0.227
0.112
0.256
0.275
0.234
0.255
0.527
0.212
Table 5:
Scaling language model results. The language model in ADIFF architecture (transformer decoder) ranges in size from 128 million (Base) to 1.5 billion parameters (XL), The results with metrics and the average score are available in Appendix Table
power law (Kaplan et al. (2020)). Similarly, recent studies have found correlation between the scale of language models and their performance on audio reasoning tasks (Deshmukh et al. (2024c).
However, these observations are typically made the compute-optimal frontier (Hoffmann et al. (2022) and are not often examined under limited compute conditions. In practical scenarios, where compute resources and time constrained, larger models may not be the best option. This has been illustrated in the case of language models, where the newer Chinchilla model (Hoffmann et al. (2022) outperformed its predecessor Gopher (Rae al. (202T), despite being four times smaller. Therefore, we study scaling language for audio-language models for prefix tuning.
To investigate the impact of scale, we modify the language model (transformer decoder) in the architecture depicted in Figure 3] We experiment with GPT2-base (128M), GPT2-medium (256M), GPT2-large (774M), and GPT2-XL (1,5B). Each model trained with the same compute budget, approximately equivalent to 30 epochs. The results, presented in Table indicate that with the same data (audio and text tokens) and limited compute, the base and medium models perform similarly average (Table (7). while the large and XL models perform worse. Morcover, the higher per- epoch computational cost of larger models widens the gap between small and large LM in ADIFF. It is important to note that do not scale the number of training tokens scale the language models, which affects performance.
To determine if increased computational power helps larger LMs, we train the models longer for an additional 20 epochs. The outcomes, shown in Figure 4 indicate that for the 128M and 256M models, performance peaks around 30 epochs. In contrast, for the 774M and 1.5B models, performance continues to improve with more epochs. This suggests that aligning and guiding larger models with prefix-tuning requires greater number of epochs. Given the computation budget and performance, we choose to use GPT2-base for ADIFF and subsequent experiments.
0.32
0.31
1289
Average sc0e
2509
0.30
7749
3.50
0.20
0.27
10
25
20
25
30
35
40
45
50
Ebochs
Figure 4: Change in average
tiers with increase in LM parameters.
5.4 AUDIO GROUNDING WITH POSITION CAPTIONING
To improve audio grounding, we instruct the model to caption either audio audio 2. This is accomplished by training the model on both audio captioning data and audio difference datasets. The audio captioning data includes {audiol, audio2, prompt). where the prompt is "caption the first audio" or "caption the second audio". By incorporating single or position-specife captioning data in the training process, we ensure the model does not get confused between audio and audio and accurately distinguishes between similar or acoustically alike audio events.
Table 6 presents the results after incorporating audio captioning data. Experiment uses baseline architecture with cross- projectroe,,,onn, while the bottom section shows the same architecture enhanced with captioning data. For reference, Experiment C in Table is the performance of baseline archi- tecture with cross-projection. On average, the ACD dataset shows improvements across all Tiers, whereas the CLD dataset yields mixed results. This discrepancy can be attributed to AudioCaps be- ing sourced from AudioSet, which mainly contains speech recordings. As a result, the ACD dataset includes more instances human speech being compared under various acoustic conditions. In contrast, the CLD dataset features wider variety of audio comparisons. Despite the mixed per-

---

Published as a conference paper at ICLR 2025
formance, continue to the model trained with audio captioning, as it demonstrates greater improvements during stage-3 finetuning.
Tier
Tier 2
Tier 3
Task
Exp.
BLBU,
METEOR
SPIDEr
BLEU,
METEOR
SPIDEr
BLBU,
METBOR
SPIDEr
ACD
0.129
0.208
0.300
0.162
0.195
0.296
0.164
0.199
0.150
0.135
0.221
0.303
0.180
0.197
0.345
0.171
0.208
0.183
CLD
0.242
0.689
0.136
0.188
0.124
E
0,203
0.302
0.652
0.213
0.235
0.692
0.191
0.220
0.417
Table 6: Audio grounding and finetuning results. Experiment D the ADIFF model with position captioning. Experiment E finetunes the language model of ADIFF along with position captioning. The results with all metrics and the average score are available in Appendix Table 18
5.5 STAGE-3 FINETUNING
In the final stage, following multimodal grounding, fine-tune the language model. This training is conducted over few epochs with a small learning rate of approximately le-6, using a cosine learning scheduler. The results of this fine-tuning are presented in Table 6 Experiment D is position captioming training. while Experiment E shows the performance post-finettuning. The final fine-tuning consistently improves performance across various tiers and datasets. We also find that the finetuned model outperforms the base model across three tiers, especially Tier 3. More details on qualitative analysis are available in Appendix K
6 HALLUCINATION
Language models often generate misleading or incorrect outputs, known as hallucinations. For Audio-Language Models (ALMs), this means producing responses not based on actual audio input, such as inventing audio events or misin- terpreting common sounds (Kuan et al. (2024); Nishimura et al. (2024)). This lack of ground- ing can impair deductive reasoning and com- parative skills. To address this, we au- dio grounding with position captioning to dif- ferentiate similar-sounding events from differ- ent sources. However, some hallucination is- sues persist. We aim to provide tools to detect these ballucinations by keeping the HTSAT au- dio encoder frozen. This encoder, trained on AudioSet, predicts 527 audio events and their presence probabilities over time, allowing users to verify the accuracy of generated descriptions. example of this is shown Figure 5]] The top pane displays the audio difference explanation generated by the model. For each audio, the log mel spectrogram and the top three audio event presence probabilities over time are plotted. From the figure, we can see that the model missed the whip sound in audio 2 in the difference explanation and provides way of analyzing explanations.
Figure 5: Audio event presence probabilitics from AD- IFF to detect hallucinations.
7 CONCLUSION
In this paper, we introduce the task of audio difference explanation, create relevant datasets, and propose an strong baseline. We present two new datasets from AudioCaps and Clotho. Using Large Language Models (LLMs), we generated three levels of explanations for audio differences: concise descriptions, brief sentences about events and scenes, and comprehensive explanations including semantics and emotions. We propose ADIFF, an audio prefix tuning- based language model with a cross-projee prosepproeection module and three-step training process address the limitation of the naive base- line. The objective and subjective evaluations show significant performance improvements over the naive baseline and SoTA ALM. We also conduct ablation studies to understand the effects of various components and find various insights relevant to training audio-language models. Our benchmarks and ablation studies show the effectiveness of our method, utilizing comparative reasoning gen- erate more human- explanations for audio differences.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Ibrahim M Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propo- sitional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp. 382-398. Springer, 2016.
Satanjeev Banerjee and Alon Lavie. METEOR: automatic metric for MT evaluation with im- proved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss (eds.), Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Mea- sures for Machine Translation and/or Summarizatiin, pp. 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL W05-
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024.
Hazim Bukhari, Soham Deshmukh, Hira Dhamyal, Bhiksha Raj, and Rita Singh. Selm: Enhancing speech emotion recognition for out-of-domain scenarios. In Interspeech 2024, pp. 2345-2349,
2024. doi: 10.2 2t05://mterspeech222222--22
Dermot Campbell, Edward Jones, and Martin Glavin. Audio quality assessment techniques review, and recent developments. Signal Processing, 1989)1148911500, 2009.
Ke Chen, Xingjian Du, Bilei 7hu, Zejun Ma, Taylor Berg-Kinkpatriee and Shlomo Dubnov. His- at: A hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 646-66 650. IEEE, 2022.
Yunfci Chu, Jin Xu, Xiaohuan Zhou, Qian Yang. Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.079/1 2023.
Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems, 36:1 1090--88088,
2023a.
Soham Deshmukh, Benjamin Elizalde, and Huaming Wang. Audio Retrieval with WavTexI5K and CLAP Training. In Proc. INTERSPEECH 2023, pp. 2948 2948-2952, 2023b. doi: 10.21437/ 2ttesspeech.2223-111
Soham Deshmukh, Dareen Alharthi, Benjamin Elizalde, Hannes Gamper, Mahmoud Al Ismail, Rita Singh, Bhiksha Raj, and Huaming Wang. Pam: Prompting audio-language models for audio quality assessment. In Interspeech 2024, pp. 3320-3324 2024a. doi: (0.21437/Interspecch. 2024-325.
Soham Deshmukh, Benjamin Elizalde, Dimitra Emmanouilidou, Bhiksha Raj. Rita Singh, and Huaming Wang. Training audio captioning models without audio. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 371-375, 2024b. doi: 10. N0.1109/CASSP944455 2024. 10448115411..
Soham Deshmukh, Shuo Han, Hazim Bukhari, Benjamin Elizalde, Hannes Gamper, Rita Singh, and Bhiksha Raj. Audio entailment: Assessing deductive reasoning for audio understanding. arXiv preprint 40Xiv:2407. 18062, 2024c.
Soham Deshmukh, Rita Singh, and Bhiksha Raj. Domain adaptation for contrastive audio-language models. In Interspeech 2024, pp. 1680-1684, 2024d. doi: 10.21 ht..1437/ntersppccch2222 2024-41..

---

Published as a conference paper at ICLR 2025
Hira Dhamyal, Benjamin Elizalde, Soham Deshmukh, Huaming Wang, Bhiksha Raj. and Rita Singh. Prompting audios using acoustic properties for emotion representation. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 11936-1 11936-11940, 2024. doi: 10.1 1091CCASP44445 2024. 10448213.
Konstantinos Drossos, Sharath Adavanne, and Tuomas Virtanen. Automated audio captioning with recurrent neural networks. In 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 374-378. IEEE, 2017.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: an audio captioning dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.
doi: 10. SAA.//AAASP447762220000955
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Benjamin Elizalde, Soham Deshmukh, and Huaming Wang. Natural language supervision for general-purpose audio representations. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 336-340. IEEE, 2024.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829-852, 2021.
Jort F. Gemmeke, Daniel P. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. 20/7 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776-780, 2017. doi: 10. 1011099CAASSP220777955566
Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sak- shi, Oriol Nicto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. arXiv preprint arXiv:2406. / 168,, 2024a.
Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi. Chandra Kiran Reddy Evuru, Ra- maneswaran S, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Compa: Addressing the gap in compositional reasoning in audio- Languala models. In The Twelfth In- ternational Conference on Learning Representations, 2024b. URL htt.ttps:: htt.ttss:///peeeeeiiitiee het/forun: id= 6NG08geWs
Yuan Gong, Alexander H Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Work- shop (ASRU), pp. 1-8. IEBE, 2023.
Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James R. Glass. Listen, think, and understand. In The Twelfth International Conference on Learning Representations, 2024. | https::t////.. /tttpss:/oppereviiii
n (ttorrmmmmd=nnBBPxaee
Zixin Guo, Tzu-Jui Wang, and Jorma Laaksonen. Clip4ide: Clip for image difference captioning. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computa- tional Linguistics and the 12th International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers), pp. 33-42, 2022.
Laurie M. Heller, Benjamin Elizalde. Bhiksha Raj. and Soham Deshmukh. Synergy between hu- man and machine approaches to sound/scene recognition and processing: An overview of icassp special session. arXiv preprini urXiv:2302.097/99 2023. URL wtt.hhss///..i....oooo wwt.xiii.orggbs////
2332.0977
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. In Proceedings of the 36th International Conference Newral Information Processing Systems, pp. 30016-30030, 2022.
12

---

Published as a conference paper at ICLR 2025
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations.
Harsh Jhamtani and Taylor Borg-Kirkpatrick. Learning to describe differences between pairs of similar images. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsuji (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 4024-4034, Brussels, Belgium, October-Noventle 2018. Association for Computational Lin- guisties. doi: 10.18653/11/D18- 1 URL WtRL/ttts::/// http:///caaahhhhho/m lothoology.org/D113
Keith Johnson and Keith Johnson. Acoustic and auditory phoneties. Phonetica, 61(1))56-58, 2004.
Philip N Johnson-Laird. Deductive reasoning. Annual review of 500))) 1999.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprini urXiv:2001.08360 2020.
Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. AudioCaps: Generating Captions for Audios in The Wild. In NAACL-HLT, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR (Poster), 2015. URLhttt::/aaxiv..oo org abs 1412. 6980
A Sophia Koepke, Andreea-Mainn Oncescu, João F Henriques, Zeynep Akata, and Samuel Albanic. Audio retrieval with natural language queries: A benchmark study. IEEE Transactions on Multi- media, 26:2655-2685, 2022.
Tatsuya Komatsu, Yusuke Fujita, Kazuya Takeda, and Tomoki Toda. Audio difference learning for audio captioning. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1456-1460, 2024. doi: 10.1109/ICASSP48485,
2024.10448085.
Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. In Forty- first International Conference on Machine Learning. 2024. URL
net httt//orumiiiiwY333 3WKZ jYe
Christian Kraetzer, Andrea Oermann, Jana Dittmann, and Andreas Lang. Digital audio forensies: first practical evaluation on microphone and environment classification. In Proceedings of the 9th Workshop on Multimedia and Security, MM See '07, pp. 63-74, New York, NY, USA, 2007. Association for Computing Machinery. ISBN 978 595938572. doi: 10.1 10.1145/1288869, 1288879.
URL httpt:://////ottt http///oooog////////// 11411114111111 2t5/122886999 01T.8886..1188888888 1288879
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations.
Chun-Y Yi Kuan, Wei-Ping Huang, and Hung yi Lee. Understanding sounds, missing the questions: The challenge of object hallucination in large audio-language models. In Interspeech 2024, pp.
4144-4148, 2024. doi: 00.21437//nccsspecch2222 2024 1076.
Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho- aqa: A crowdsoureed dataset for audio question answering. In 2022 30th European Signal Pro- cessing Conference (EUSIPCO), pp. 1140-1144, 2022. doi: 10.29119EUSIPPC050093.22222
9909680.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei. Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. In Interna- tional Conference on Machine Learning, pp. 11440-211477 PMLR, 2023.
Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image cap- tioning via policy gradient optimization of spider. In Proceedings of the IEEE international conference on computer vision, pp. 873-881, 2017.
13

---

Published as a conference paper at ICLR 2025
Robert C. Maher. Audio forensic examination. IEEE Signal Processing Magazine, 26(2):84-99)
2009. doi: 10.1109/MSP2200.991100
Xinhao Mei, Xubo Liu, Mark D Plumbley, and Wenwu Wang. Automated audio captioning: An overview of recent progress and new challenges. EURASIP journal on audio, speech, and music processing, 2022(1):26, 2022a.
Xinhao Mei, Xubo Liu, Jianyuan Sun, Mark D Plumbley, and Wenwu Wang. Diverse audio caption- ing via adversarial training. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8882-8886. IEEE, 2022b.
Xinhao Mei, Xubo Liu, Jianyuan Sun, and Mark D Plumbley. Towards generating diverse audio captions via adversarial training. IEEE/ACM transactions on audio, speech, and language pro- cessing, 2024a.
Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumb- ley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt- assisted weaell-labelled audio caption- ing dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:3339-3354, 2024b. doi: N0.10099ASS...224444199444
Ron Mokady, Amir Hertz, and Amit Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint 0tt::2111..973.. 2021.
Irene Martin Morato and Annamaria Mesaros. Diversity and bias in audio captioning datasets. In Detection and Classication of Acoustic Scenes and Events, pp. 90-94, 2021.
Taichi Nishimura, Shota Nakada, and Masayoshi Kondo. On the audio hallucinations in large audio- video language models. arXiv preprint htt::2401.097744 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318 2002.
Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning, pp. 28492-28518 28518. PMLR, 2023.
Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021.
Kenneth N Stevens. Acoustic phonetics, volume 30. MIT press, 2000.
Daiki Takeuchi, Yasunori Ohishi, Daisuke Niizumi, Noboru Harada, and Kunio Kashino. Audio difference captioning utilizing
disentanglement. In Proceedings of the 8th Detection and Classification of Acoustic Scenes and Events 2023 Workshop (DCASE2023), pp- 181-185, Tampere, Finland, September 2023.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Extending large language models for speech and audio captioning. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 11240, 2024a. doi: 10. 109/CASSSP444455 2222..00444.4.. 10446343.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024b. https: // /ooenreview www..wwwww....................ww htt.://oooouuoo/d1111777oropppp
G. Tzanetakis and P. Cook. Musical genre classification of audio signals. IEEE Transactions on Speech and Audio Processing, 10(5):293-302, 2002. doi: 10. 109/TSA 2002.800560.
14

---

Published as a conference paper at ICLR 2025
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575. 4575, 2015.
Shih-Lun Wu, Xuankai Chang, Gordon Wichern, Jee-weon Jung, François Germain, Jonathan Le Roux, and Shinji Watanabe. Beats-based audio captioning model with instructor embedding supervision and chatgpt mix-up. In Conf. Detection Classification Acoust. Scenes Events, Chal- lenge (pp. 1-5).
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Linli Yao, Weiying Wang, and Qin Jin. Image difference captioning with pre-training and contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 3108- 3116, 2022.
Mengxia Yu, Zhihan Zhang, Wenhao Yu, and Meng Jiang. Pre-training language models for com- parative reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12421-12433, 2023.
Ruijie Zhou, Soham Deshmukh, Jeremiah Greer, and Charles Lee. Narle: Natural language models using reinforcement learning with emotion feedback. arXiv preprint 1, 2021.
15

---

Published as a conference paper at ICLR 2025
A ETHICS STATEMENT
This paper adheres to the ethical guidelines set forth by the ICLR conference and aims to contribute to the research community in responsible and transparent manner. The study utilizes publicly available datasets, namely AudioCaps (Kim et al. (2019) and Clotho (Drossos et al. (2020), which have been used in accordance with their respective licenses and ethical use guidelines. No personally identifiable information (PII) or sensitive data was used in this research, ensuring the privacy and confidentiality of all subjects involved.
The research focuses on developing methods for explaining audio differences using natural lan- guage, and all experiments were conducted with the intent to advance scientific knowledge in positive and constructive way. We acknowledge the potential implications of our work in fields such as audio forensics and surveillance, and we emphasize that our models are intended solely for aca- demic and benign applications. We do not condone or support the misuse of our work in ways that could infringe upon individual privacy or be used for unethical purposes. All human evaluators and annotators participated voluntarily with their informed consent. There were no conflicts of interest or biases influencing the results of this study. Finally, we are committed to open science and will release datasets and model checkpoints, to the research community to promote transparency, reproducibility, and collaboration.
B REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we have taken several measures, which are thoroughly documented throughout the main paper, appendix, and supplemental materials. For our proposed model, ADIFF, and the associated benchmarks, we have provided downloadable source code anonymously in the supplementary materials to facilitate replication of our experiments. Detailed explanations of our experimental setup, hyperparameters, and model architecture can be found in Section 3 and Appendix G We have outlined the data processing steps, including the creation of the AudioCaps Difference (ACD) and Clotho Difference (CLD) datasets, in Section [.. and the Appendix E, ensuring transparency in data preparation and usage. All datasets used in this work publicly available, and have provided links to these resources along with data processing scripts in the supplementary materials. Our ablation studies in Section 5 further validate the robust- ness of model design, and comprehensive results are presented to demonstrate consistency and reliability of the findings.
C RELATED WORK
Audio-text learning. Audio-text learning focuses on developing multi-modal representations to address specific tasks. These tasks include text-to-audio retrieval (Deshmukh al.] (2023b),, au- dio captioning (Drossos et al. (2020); Mei et al. (002a))). and text-to-audio generation (Liu et al. (2023)) Among these, audio captioning is particularly relevant for the task of audio difference explanation, it involves generating natural language descriptions from audio recordings. The chitecture typically comprises an encoder to learn audio representations and a decoder to generate text tokens (Drossos et al. (2017) 7)) Research in this area has explored enhancing diversity (Morato & Mesaros] (2021) Mei et al. [2024a] (0225)) in outputs, improving audio encoder and training methods (Wu a and leveraging LLMS as decoder (Tang et al. (2024a)).
Audio-Language Models. Instead of building task-specific models, recent literature has focused on building general-purpose audio models by using language. These models are referred to as
Audio-Language Models. For example, CLAP (Elizalde et al.] (2023) 2024); Wu et al. (2023) ) and AudioFlamingo (Kong et al. (2024) is trained on millions of audio-text pairs learn multi- modal audio-text representations. and are used for tasks like zero-shot and few-shot classification and retrieval. To address open-ended tasks such as audio question answering and audio caption- ing, a decoder or LLM is added to the architecture. For example, Pengi (Deshmukh et al.) (2023a)) employed prefix-tuning to learn a mapper and prompt a frozen language model. The later Audio- Language Models like LTU (Gong al. (2024) 2023)), SALMONN (Tang al.] (2024b)). GAMA Ghosh et al. (2024a)) utilized LoRA (Hu et al.) to develop lightweight adapters on language mod- els. While most Audio Language Models (ALMs) concentrate non-speech audio, Qwen-audio and LTU-AS integrate both speech and audio understanding by utilizing the Whisper encoder (Rad-
ford et al. (2023)
16

---

Published as a conference paper at ICLR 2025
Difference Captioning. In the vision domain, the task of explaining the difference between two images is studied under Image Difference Captioning (IDC) (Jhamtani & BorggKirkpatrick (Jhamtani & Berg- Kirkpatrick (018)). attention-based (Park et al. (219)) clip-based decoder (Guo et (2022)) and cross-modal transformer (Yao et al. (2022)). In the audio literature, the first relevant work Komatsu iet al.] (2024) explored using difference learning to improve audio captioning. The data consists of mixture with audio and audio 2, and then the model predicts the caption for audio 1, indirectly highlighting the difference. This approach differs from directly comparing two audios and high- lighting differences; instead, it focuses on captioning one of the audios. The second work (Takeuchi et al. (2023) focuses on audio difference captioning. aiming to find the semantic difference between similar but slightly different audio clips. The difference captioning takes the form of "change," "re- move," or "add", for example, "change sound of footsteps." The task is concerned with adding or removing something from audio I, instead of explaining how the audios differ concerning audio events, sound sources, acoustics, and emotional response. Therefore, in the audio literature, the task of audio difference explanation with different tiers of explanation is yet to be explored.
(2018); Park et al. (2019)) The architectures explored include encoder-
D COMPARATIVE REASONING FOR AUDIO
Comparative reasoning Yuu et al. (2023) for audio involves analyzing and contrasting different audio sources, events, and signals to draw meaningful conclusions. This process typically starts with identifying the audio sources, which could range from live recordings, synthesized sounds, or pre-recorded tracks. Each source has unique characteristics such as frequency range, amplitude, and timbre. Audio events, which are specific occurrences within the audio signal like note being played or a word being spoken, are then identified and analyzed. Temporal characteristics, including the timing, duration, and sequence of these events, play crucial role in comparative reasoning. By examining these elements, one can discern patterns, anomalies, and relationships within the audio data. The signal processing aspect of comparative reasoning involves breaking down the audio signal into its fundamental components. For example, analysis of the spectral content, Signal-to cennal-o-noose to-noise ratio (SNR). harmonic distortion, and other metrics can be used to evaluate the quality and integrity of the audio signal. By comparing these metrics across different audio sources or events, one can make informed judgments about their similarities and differences. Moreover, Comparative reasoning in (1999)) where conclusions are drawn
audio is closely related to deductive reasoning (Johnson-Lain based on the analysis of specific data points. For instance, if two audio signals share similar temporal and spectral characteristics, been processed similarly. Conversely, significant differences in these characteristics could indicate different sources processing methods. By systematically comparing and contrasting audio data, one can make logical inferences and draw conclusions that are supported by empirical evidence. Therefore, enabling ALMs to comparative reasoning improves their overall logical reasoning ability.
might deduce that they originate from the same source or have
E USING LLMs TO GENERATE AUDIO DIFFERENCE EXPLANATION
Large Language Models (LLMs) have been effectively utilized to generate text descriptions for audio across various tasks, such as question-answering compositional reasoning, and deductive reasoning. These models leverage vast amounts of data and sophisticated algorithms to understand and generate human-like text. We adopt a similar approach for generating explanations for audio difference tasks, which involve identifying and describing differences between two audio recordings. The process involves three main steps: data sources, explanation generation, and explanation verification.
Data Sources: For our study, we limit the audio recordings and human- annotated descriptions to those from two audio captioning datasets: AudioCaps and ClothoV21. AudioCaps is a large- scale dataset that provides audio recordings along with detailed human-annotated descriptions. C2o is another comprehensive dataset that includes variety of audio recordings with cor- responding textual descriptions. By using these datasets, we ensure that our audio samples and descriptions are diverse and representative of real- worldw scenarios.
To generate explanations, we prompt an LLM to describe the differences between two audio recordings using the provided auman-annotated descriptions. To select captions for prompting LLM, we perform random sampling per audio to eliminate the edge case of multiple captions. This process involves three steps. First, we flatten the .csv files to include one audio file and single caption. For Clotho, which has 5 captions, this results in entries per audio. Second,
Explanation Generation:
17

---

Published as a conference paper at ICLR 2025
1. Data Sources
2. Explanation Generation
rocket flies by followed by/ loud explosion fire crackling as engine runs idie
The two audios differ in their acoustic properties, with the first audio featuring a constant, high-frequency rattling noise and sharp vibrations, indicating mechanical or physical source, whereas the second audio
You helpful assistant with expert knowledge about, audio, acoustics, and psychoacoustics. You study audio, which is the sludy of sound and its properties. You study acoustics, which
prompt builder
LLM
Water trickling followed by toilet flushing liquid draining through pipe
sampler
Explanation Verification
AudioCaps
Human verification
Clotho
Figure 6: Using LLM to generate audio difference explanation. The process involves three key steps: sampling from human-annotated data sources, generating explanations, and verifying the generated explanations. The verification step is conducted only for the test set.
Prompt for LLM
helpful assistant expert knowledge about audio, acoustics, psycheacoustics. study audio, which the study of sound and its prupertess You study acoustis, which revolve around the generation, propagation, and reception of sound waves You study Psychology which posits that sound complex stimulus that encompasses a vast range acoustic properties involving pyychomechamics. Your lask perform audio caplioning which consists of describing
uspects of cugnition, pyychuccustiiss andio content using natural language. To describe the acoustic content, you utilize words related to their acoustic properties, such their semantic relations, their spectro-temporal characteristics, frequency, londness, duration, materials, interactions, and sound sourcea.
Table 7: Base prompt for LLM before generating responses
we conduct sampling. For instance, the dataset contains N audios, we loop through the N audios and, for any 7th audio, randomly pick second audio (ith) not including the i+4th audio. This is achieved by constructing an index list that excludes the i, i+1, .... i+4 indices and sampling from this list. Third, the selected 7th and jth audios are then used to look up the captions for the 7th and randomly picked jth audio. These captions then become the seed for the LLM to generate the difference explanation. Our LLM prompting setup is similar the one used in previous studies Deshmukh et al. (2024c), with two key modifications. First, instruct the LLM to incorporate knowledge about sound sources, materials, acousties, and emotions when generating explanations. This ensures that the explanations are not only accurate but also rich in context and detail. Second, we define the tier of explanation by restricting the sources the LLM can use and the length of the explanation. This tiered approach allows detail. The base prompt is detailed in Table
generate explanations of varying complexity and
Finally, human annotators verify the difference explanations generated by the LLM. If an explanation is found to be inaccurate, the human annotators manually remove the hallucinated audio event and add necessary details correct it. Due to the cost and effort involved in this process, we restrict verification to the test set three tiers. This ensures that our testing dataset both accurate and reliable. The resulting dataset comprises audio recordings paired with three tiers of generated explanations.
Explanation Verification:
ANALYSIS OF GENERATED DESCRIPTION
The three tiers of audio difference explanation are explained in Section 2 while the dataset creation process is explained in Section 2.1 and Section E In creating the dataset, rely LIM generate
18

---

Published as a conference paper at ICLR 2025
30000
3900
0055
5090
4090
2091
Figure 7: Comparison of vocabulary differences between explanatory tiers: The left figure illustrates words that are present in the Tier explanation not in the Tier explanation, while the right figure highlights words unique to the Tier explanation that are absent the Tier 2 explanation.
the base difference explanation before it is verified for human annotators for the test-set. Therefore, it's important explore the types of audio differences described in different tiers to uncover insights and variations in the three tiers of difference explanations. For instance, by splitting text into words, converting them to lowercase, and focusing on words related to audio, we can identify words that appear in Tier 2 but not in Tier 1, showing unique audio characteristics used for the difference explanation in Tier 2. Similarly, we can use the same method to examine the distinctions between Tier 2 and Tier 3, where the differences often lie in more detailed and immersive sound characteristic descriptions. The results, presented in Figure 7 highlight two main points. First, the unique words between Tier and Tier 2 are significantly more than those between Tier 2 and Tier 1. This is expected, as Tier 3 explanations add considerably more detail than Tier 2. Second, the left plot of Figure shows that the additional characteristics mentioned in Tier over Tier I focus on audio events, sound sources, and the nature of the sounds, making it more analytical. In contrast, the right plot of Figure 7 illustrates that the additional characteristics in Tier 3 over Tier 2 focus on the sonic qualities of each audio, including potential sources, listener's experience, comparing the two audios in terms of the complexity and engagement, and therefore providing a more comprehensive difference description. Overall, we see the audio characteristics used in Tier 3 over Tier 2 (Figure right subplot) and Tier 2 over Tier (Figure 7 left subplot) are different, and focus on the different aspect and details between two audios. We further analyze the density of information across three Tiers in Appendix Section Q
G EXPERIMENTAL SETUP
Encoder and Decoder. We utilized the HTSAT (Chen et al. (2022)) audio transformer, trained on AudioSet (Gemmeke al. (2017)), as our audio encoder. The audio was sampled 32 kHz and converted into log Mel spectrograms with 64 Mel bins, a hop size of 320 ms. and window size of 1024 ms. For HTSAT. all audio files were randomly truncated To maintain computational efficiency, the maximum length of user text input was set to 40, The audio encoder remained frozen throughout the experiments, enabling hallucination output using audio event detection probabilities. For the decoder or Language model, we use the GPT2 series of models, specifically the GPT2-base (124M) variant. The model remained unchanged throughout all the experiments except for Section (533
10 seconds in length.
Projection. The audio projection uses an 8-layer transformer with a prefix length of 40. When combining two audio inputs, the total prefix length becomes 80. Adding separator token (1 token) and a text projection (40 tokens) results in total prefix length of 121. This combined input is then processed by the cross-projection layer, layer transformer, which maintains the prefix length of 121 in its output.
Training. We use Adam Optimiser Kingma & Ba 2015)) with warmup and step decay of the
learning rate. For stage 2, multimodal grounding, all models are trained for 30 epochs on A100 GPUs. Only for Section (53). we retrain the models for 50 epochs. In the final stage of fine-
19

---

Published as a conference paper at ICLR 2025
tuning, we limit the training to 10 epochs prevent the training loss from plateauing and to avoid catastrophic forgetting.
Inference. We top-k and top-p decoding because of its better performance in producing diverse explanations that accurately capture details. We explore different decoding strategies and compare the performance of top-k and top-p decoding against the greedy decoding in Section N.
H USER TEXT PROMPT
Audio-based prefix- tetiint models have been utilized for various tasks (Deshmukh et al. (2024b); Bukhari et al. (2024)) including as supervision to learn general-purpose audio representations (Deshmukh et al. (2023a)). The dataset for training these models consists of (audio, user input, output) triplets. During training, the user input remains fixed for a specific task and only varies across tasks. Consequently, the model performs the necessary task only when the exact text prompt or user input is provided. This limitation restricts the model's applicability. To address this, we explore supporting multiple input text prompts by randomly sampling prompts during training.
For the audio difference task, the ADIFF model must produce three tiers of explanation based on the user prompt. Additionally, needs to generate audio captions for the first or second audio to enhance grounding (5.4). We aim for the user to be able to request different levels of explanation using flexible prompting, and the model should still provide accurate outputs. To achieve this, we create a database of user prompts for each tier. During training, depending on the required output tier, we sample specific prompt from the database. our case, the database is simple dictionary containing about 100 sample user inputs per output tier. Randomly sampled examples of the user inputs are shown in Table g
Tier
Randomly sampled user prompts
"Summarize the differences between the two audios briefly.". "Highlight the main differences between the two audio tracks."
"Provide concise comparison the two audios."
"Explain the contrast between the audio pieces in long sentence.", ".
2 "In one elaborate sentence, summarize the differences between both audio files.", "Describe the variation between the two audio tracks in one extended sentence."
"Explain difference between both audios detail.",
3 "Could you elaborate the distinctions between the two audio tracks?",
"Please provide detailed comparison of both audio files."
Table 8: Randomly sampled three user prompts for each tier audio difference explanation.
I EVALUATION
1.1 OBJECTIVE EVALUATION
For objective evaluation, we leverage audio captioning metrics: BLEU, METFOR, SPICE, CIDEr, SPIDEr. The BLEU (Bilingual Evaluation Understudy) metric measures the precision of n-grams between the generated text and reference text, making simple and fast, though it doesn't account for recall and can be insensitive to word order and meaning. METEOR (Metric for Evaluation of Translation with Explicit ORdering) combines precision and recall, incorporating features like stemming and synonymy matching, which results in better correlation with human judgment at the sentence level, albeit with increased complexity and computational demands. ROUGE (Recall- Oriented Understudy for Gisting Evaluation) focuses on the recall of n-grams and longest common subsequences, making it popular for text summarization, though its recall-centric nature might not always reflect overall quality. CIDEr (Consensus-basse consensus between generated image descriptions and multiple reference descriptions, excelling in image captioning but requiring multiple reference captions for accuracy. SPICE (Semantic Propo- sitional Image Caption Evaluation) evaluates the semantic content of image captions by comparing scene graphs, capturing semantic meaning more effectively than surface-level metrics, though it is computationally intensive. Lastly, SPIDEr (SPICE and CIDEr) combines SPICE and CIDEr to leverage both semantic and consensus-basee evaluation, providing balanced assessment but inher- iting the computational complexity of both metrics. Therefore, we primarily use SPIDEr to compare models or calculate the average score across all the above metrics for model comparison.
Image Description Evaluation) measures
20

---

Published as a conference paper at ICLR 2025
1.2 SUBJECTIVE EVALUATION
While objective metrics are correlated with human evaluation, they have their limitations. There- fore, we also set up human evaluation task to assess the difference explanations generated by the model. this task, evaluators are asked to evaluate the differences between two audio samples. These differences could pertain to various aspects such audio events (e-g., specific sounds or oc- currences within the audio), acoustic scenes (e.g.. the overall environment or context of the audio), signal characteristics (c.g., frequency, amplitude, timbre), and listener emotions (c.g.. the emotional impact or response elicited by the audio). Evaluators then provide scores based on three dimensions: Correctness, Granularity, and Readability. These dimensions ensure that the evaluations compre- hensive, focusing on the accuracy, detail, clarity, and acoustic quality of the explanations provided. The evaluation dimensions and the corresponding score descriptions are: Correctness (1-5)
The explanation is mostly incorrect, with significant inaccuracies in identifying audio events, acoustic scenes, or signal characteristics.
2: The explanation has several inaccuracies but captures some correct audio events and acoustic scenes.
3: The explanation is partially correct, with mix of accurate and inaccurate details.
4: The explanation is mostly correct, with minor inaccuracies.
5: The explanation is entirely correct, accurately identifying all relevant audio events, acoustic scenes, and signal characteristics.
Granularity (1-5)
The explanation is very vague, lacking detail and specificity.
2: The explanation provides some detail but generally too broad.
3: The explanation is moderately detailed, covering key aspects but missing finer points.
4: The explanation is detailed, covering most aspects with good specificity.
5: The explanation is highly detailed, thoroughly covering all relevant aspects with precise specificity.
Readability (1-5)
The explanation is very difficult to read, with poor structure and clarity.
2: The explanation is somewhat difficult read, with several issues in structure and clarity.
3: The explanation is moderately readable, with some issues in structure and clarity.
4: The explanation is mostly readable, with minor issues in structure and clarity.
* 5: The explanation is very easy to read, with excellent structure and clarity.
We recruited five professionals to evaluate the generated audio difference explanations. Each anno- tator was asked use headphones listen both audio samples. After listening and analyzing the two audios, annotators provided scores based three criteria: correctness, granularity, and read- ability. Each criterion had a maximum score of 5. The exact task guidelines are shown in Table 9 an individual instance of the task is shown in Table TO and the results of the human evaluation are presented in Table
J CROSS-PROJECTION
We introduce cross- coos-prroection layer with a latent separator token between two audio inputs ad- dress two key challenges. First, it enhances the model's 's ability to differentiate between perceptually similar sounds and identical audio events. Second, improves Tier and Tier explanations, which involve discerning subtle audio differences such scenes, acoustic sources, and their composition, rather than merely identifying audio events. From an architectural standpoint, we concatenate the first audio latent embedding, the separator embedding, and the second audio latent embedding to form a prefix. The separator embedding is derived from the < -edoftext- > token embedding of GPT-2. This concatenated prefix is then processed through the cross cross-projecttonn layer, which consists of transformer layers with learnable constant vector. The output of the cross pross-proecctio layer is in the same latent space GPT-2, allowing to analyze the pross-projection outputs.
21

---

Published as a conference paper at ICLR 2025
Task Instructions
Your task to rate the difference explanation hetween the two audios. First, listen to each audio clip and analyze the difference between the two audios. Pay attention to differences in terms of audio events (dog barking vs cat meowing), sound sources (musical instrument, human voice), acoustic scene (park vs living room), signal characteristics (frequency, pitch, loudness, etc) emotion response (listening to the audio invokes sense of happiness, fear, you) and any other difference using world-knowledde or using deductive reasoning. Once you have analyzed the clip, asses the below audio difference explanations provided.
Important Notice:
Please note that certain sections this audio test may include unexpectedly loud sounds. To ensure comfortable experience, we suggest setting your volume to a suitable level before starting and adjusting it as necessary throughout the test.
Table 9: Task guidelines provided to annotators.
Please read the task guidelines before completing the task. First, listen to the two audio record- ings carcfully and analyze the difference between them. Second, read the descriptions which explains the difference between the two audios. Finally, provide your rating of the difference explanation across three dimensions correctness, granularity and readability
How accurate is the audio difference explanation? (Poor) The explanation is mostly incorrect, with significant inaccuracics in identifying audio events, acoustic scenes, or sigmal characteristics. (Fair) The explanation has several inaccuracies but captures some correct audio events and acous- tic scenes.
3 (Good) The explanation is partially correct, with a mix of accurate and inaccurate details. (Very Good) The explanation is mostly correct, with minor inaccuracics. (Excellent) The explanation entirely correct, accurately identifying all relevant audio events, acoustic scenes, and signal characteristics.
How detailed and granular is the audio difference explanation?
(Poor) The explanation is very vague, lacking detail and specificity. (Fair) The explanation provides some detail but generally too broad.
(Good) The explanation is moderately detailed, covering key aspects but missing finer points.
4 (Very Good) The explanation is detailed, covering most aspects with good specificity. 5 (Excellent) The explanation is highly detailed, thoroughly covering all relevant aspects with pre- cise spccificity.
How casy to read and understand is the audio difference explanation?
(Poor) The explanation is very difficult to read, with poor structure and clarity.
(Fair) The explanation is somewhat difficult read, with several issues in structure and clarity.
(Good) The explanation is moderately readable, with some issues in structure and clarity. (Very Good) The explanation is mostly readable, with minor issues in structure and clarity. 5 (Excellent) The explanation is very easy to read, with excellent structure and clarity.
Table 10: Individual labeling instance for audio difference explanation task.
Let the output of the cross projecproeetton layer be [n. d], where (n) represents the number of tokens, equal to the prefix length (121), and (d) is the hidden dimension, matching GPT-2's intermediate layer size of 768. The GPT-2 vocabulary embedding has dimensions [v, d]. where (v) is the vocab- ulary size and (d) is the hidden dimension. To find the closest match between the cross-projection tokens (n) and the vocabulary embeddings (v), we normalize the vectors and compute the dot product between them. The highest dot product indicates the most likely vocabulary token. These vocab- ulary tokens are numerical and tokenized using the BPE tokenizer. We then detokenize them to obtain English subwords. As expected, the interpreted prefix initially appears gibberish, similar to findings in prefix tuning for other tasks. However, these gibberish English words can be parsed to extract meaningful audio words subwords. Below, we present these subwords for readability.
22

---

Published as conference paper at ICLR 2025
1033
0.7
1.87
0.6
0.5
0.4
0.4
0.3
02
0.2
02
0.1
0.0
BLEU
BLEU
METEOR
ROUGE
SPICE
SPID
0.0
BLBU
SPICE
0.0
BLEUJ
BLEU 4
METEOR
ROWGRLL
SPICE
SPICE
0.8
0.
144 14
0.6
33
0.6
0.3
0.3
0.3
0.2
0.1
0.1
0.1
BLEU
BLEU
METEOR
ROUGE L
SPICE
SPIDER
BLEU
BLEU
METEON
SPICE
SPIG
SLEU
BLEU
METEOR
ROUGE_
SPICE
0.7
0.7
00
0.6
06
0.
0.4
0.4
0.2
0.2
02-
0.1
0.1
0.0
BLEU
BLEU
METEOR
ROUGEE
SPCEE
SPID
0.0
BUBU
SPCE
SPICET
0.0
BLGUJ
BLEU
METEOR
ROUGRR
SPICE
SPIDE
Figure 8: The plots for different ablation studies organized follows: The (1,1) subplot represents the naive baseline with random audio encoder weights, while the (1,2) subplot shows naive bascline with HTSAT pretrained audio encoder weights. The (1.3) subplot features ADIFF with cross- pross-projcttio and GPT2- base. Moving to the second row, the (2.1) subplot illustrates ADIFF with the GPT2-me med language model, the (2,2) subplot displays ADIFF with the GPT2-large language model, and the (2.3) subplot presents ADIFF with GPT2-XT.. In the third row, the (3,1) subplot replicates the ADIFF result from subplot (1,3), the (3,2) subplot shows ADIFF with audio position training, and (3,3) subplot combines ADIFF with audio position training and a finetuning stage.
prefix
prefix
prefix
explosion,
belicop.
high, froquency, pitch, dynamic, range
repetition, speokees
parsait, vintage
devilpment plus, dynamic rango, single
baby, cryy sprake,
speaker, checring,
prodict,
clap, speakies,
lnnd, surpise
Table 11: Parsed words and subwords from the interpreted prefix. The prefix here is the output of the cross- projection layer and the corresponding model outputs are shown in Table 12
We observe that even after the cross prose-proecction layer is supposed to mix information between au- dios, the prefix for audio 1 retains information about audio while the prefix for audio 2 retains information about audio 2. Instead, by using the cross- projection layer, it reuses the text prefix to store attributes and difference information, which is later used form the explanation. Therefore, the text prefix gets used for not only steering the language model but also for storing information about attributes to be used for comparison. For example, in the first row of Table II] the text pre- fix contains mid-range range, frequency, and pitch attributes, which differ between the two audios and appear in the ADIFF output (Table [2, row 1). Similarly, in the second example, we see loudness and dynamic range attributes in the prefix, with corresponding difference explanations using those
23

---

Published as a conference paper at ICLR 2025
for comparison (Table 12 row 2). The main caveat of these results is that we are approximately estimating the English token, assuming the output of the gross-projections is in the same space as GPT-2 and using dot product. However, this result still provides insight into the model's workings and how the pross-projection layer benefits and improves model performance for Tier 2 and Tier 3 explanations.
K EFFECT OF STAGE-3 FINETUNING
We also perform qualitative analysis and find that Model consistently outperformed Model 1 across three tiers. For example, one randomly picked example of which is shown in Figure [1] For Tier 1, Model 2 accurately described the ground truth for both audio samples, identifying the mix of human voices, animal sounds, and environmental noise in Audio I, and the presence of machine sounds and animal noises without human speech in Audio 2. In Tier Model captured the specific details of the dominant sound sources, correctly identifying the complexity and variety of sounds in Audio 1 and contrasting them with the mechanical sounds and horse's hooves in Audio 2. For Tier 3, Model 2 provided a more accurate and detailed comparison, describing the distinct sounds and their acoustic properties, as well as the overall ambiance and spatial characteristics of each audio, making it a more comprehensive and precise analysis compared to Model 1.
Ground Tier
Ground truth
Audio I foetures a mix of frumnn speech, anirel scu ds, and ensormmnnaal
As I isten to the tro audios, notoo ciestnt diferenoee in their accussic propertiss. Audic (1) featuress cataphery of sounds, with a dog's berking leking center stage, aczompor rid by is
ricken wifile Audio 2 focaseee machiee sourte and anamal mokeees with desind abserice hummen speach
man's voice. hirds chippng. and genti wind blowing into the micmphone The barking and balking have distest spaddm- pepprn charadennstes with the dog's tarking excibiingg a suitden shorp decay, whereas
GPT2
volce more gradual oncet decay, blrde high
cominsted iiss
Audio fewoures a mis of fruman voices, wind, and animal sounds, while Audio has single. gontie bird chilping with a man's vaice.
conniibutin
chatter contrast, Aud
humminn of engines, whic has a dissnntt troquonty and loudnoss ess profiic. followed by the neigh of horae, withc hask more abrupk atlack and decay. The bidds chinping in the background create sense continuity but the overal sound more focused on the mechanical and animal sounds, more prominee intiinn aertow of apeiial by desannoe.
GPT2
feetures
dog
has machite sounds, single gallep. distintt acoustic propertees.
GPT2
Ground fruth Tier
As Il lssen to the tro aud ol.os notio Audio 1, Icatu ing wind browing and dogs barking, has ng creatos suiden, highspiiched aound suddon anset and dosay, while dogs bankng suddon, high pliched, and hig loudress duration, contrent Audo feeturing bind
more cherdic and dynamis micic sounescape,
The offerenze betweeen theos two sudios les in the dominant sound source with
Audio featuring prominent humon amidet natural sou nds, whoreas chanaclofaed
and and passoalul stmosphiere The bindal chersing gantis partle, and garde sound with a gertte tampo, wnile the women's volce is pertle, gentie and genile sound with gentie tempo. The overall sound more calminn and pezcetul with a greater sensee of cotmmess and senerity
speekknng
horse's neigh: with birds chirping baceground serving common
throad botween the tio.
GPT2
The cifference bettwaren the toot audina that the first sucio leatunes mone Myramic and var soundscape with a mix huran voicess wind, and animall sounds odherness the secoard audio is more serure and serure se with gerzle lend chipping and a gentle human convertation
GPT2FT
As 11 lotwn to the ton a diaa notioe that Audio 1, lesturing dog be barking wind browing and man speaking mg. has more dy ami and varied acou siic profle compared to Audo 2. where birds chipp clopping. The dogss ba sarking characte attack decay, while the
a high frequenay hig pitches sound and mufted, with chirping horse's gondo, low floqu only rumble with slower atack and doday. while the ongine cieates more mullee mach or loil sound, with a slowar sttack and docas. Ovenall Aud o has more dyr ame and varied scoustic profilo, while Aucio 2 has a focus on mech anical and animal sounds, with greater sense siienily
GPPT2
cttack
fooping horse creating soothing backgroud with
speaking
The oiffrence botweon the two audios that the first audio features a more complex soundscace mutiill sound sources, inclucing human speech, pood clab acoualli prolle.
adte
sine gl consistet
Figure 9: We compare two models: first, GPT2, the model after stage-2 multimodal grounding, Second, GPT2 FT after stage-3 finetuning. The architecture for both models is the same and consist of audio encoder, audio projection, cross-projection and GPT2-base as the decoder. The top pane provides a concise explanation, the middle pane offers brief explanation. and the bottom pane presents detailed explanation.
L ABLATION EXTENSION
The ablations results are shown in Figure 8 for easy comparison. In the (1,1) subplot, the language- only model (with a randomly initialized and frozen audio encoder) demonstrates lower performance, particularly in Tier 1, which is the hardest due to its limited words and focus on audio-related content. In contrast, Tier is the easiest to learn due to its simpler linguistic structure, and Tier falls in between. The 12. subplot, which presents baseline model performance with a pretrained and frozen audio encoder, shows improved results across all tiers. However, qualitative analysis reveals two limitations with this baseline model: difficulty distinguishing between perceptually similar sounds and struggling to differentiate subtle audio differences in Tiers 2 and 3. To address these issues, a cross cross-projection layer with latent separator token is introduced, shown in the (3,1) subplot. This
24

---

Published as a conference paper at ICLR 2025
adjustment improves performance, particularly in distinguishing audios in ACD and CLD datasets across all tiers.
Further experiments to explore the impact of model scale are depicted in subplots (2.1). (2,2), and (2.3). Using GPT2-base, GPT2-medium, and GPT2-large (256M, 774M, and 1.5B parameters, respectively), the models were trained with the same compute budget. Results show that the base and medium models perform similarly, while the larger models require more epochs to achieve better performance. The (3.2) subplot presents results after incorporating audio captioning data to further improve audio grounding, where the model distinguishes between audios more accurately, especially in the ACD dataset, which includes more human speech comparisons. However, the CLD dataset shows mixed results due to its wider variety of audio comparisons. Despite this, the audio captioning model continues to be used for stage-3 finetuning. Finally, the (3,3) subplot presents the performance after fine-tuning the language model over few epochs with a small learning rate. This stage consistently improves performance across tiers and datasets, with the finetuned model outperforming the base model (subplot (3.1)) in all cases.
M QUALITATIVE ANALYSIS
We perform analysis of explanations generated by ADIFF. The different scenarios and explanation are shown in Table T2] In the first scenario, Audio (1) features sudden, sharp explosion followed by crackling sounds, while Audio (2) presents a continuous low-frequency hum of a helicopter. This comparison highlights differences in frequency content and dynamic range, with Audio (1) having more gradual onset and decay, and Audio (2) exhibiting sudden changes in loudness and frequency. The second scenario contrasts baby crying and woman speaking in Audio (1) with man speak- ing and a crowd cheering in Audio (2). Here, the focus is on the distinct acoustic signatures and dynamic ranges, with Audio (1) being more consistent and predictable, while Audio (2) has greater variability in loudness and pitch. Here, we compare different voice characteristics which involves understanding pitch, frequency, and intensity. In the third scenario. the model compares a brief, loud applause followed by a male voice in Audio (1) with sharp, percussive sounds in Audio (2). This scenario emphasizes differences in acoustic properties such as frequency. loudness, and duration. The fourth scenario involves a dog barking loudly in Audio (1) and several dogs barking with a man speaking in Audio (2), highlighting the chaotic and dissonant quality of the latter. In this sce- nario, we compare model performance on similar audio events to spot differences. Finally, the fifth scenario compares a group playing a piano in Audio (1) with a piano followed by a drum beat in Audio (2), focusing the timbre and frequency content. These scenarios were chosen to evaluate the model's ability to describe various audio events and scenes.
We also compare ADIFF generation against existing ALMs in the literature. As of yet, only Qwen- Audio Chat (Chu et al. (2023) supports two audios as input. The qualitative comparison is shown in Figure [o] We consider three scenarios: Contrasting Acoustic Qualities, Overlapping Sound Sources, and Overlapping Sound Events. For Contrasting Acoustic Qualities, Qwen Audio identified the first clip as someone cutting wood with chainsaw and the second as someone walking on snow, while ADIFF provided a detailed analysis, noting the harsh, high-frequency sound of the chainsaw and the mellow, low-frequency sound of walking on snow. In the Overlapping Sound Sources scenario, Qwen Audio described both clips as featuring dogs barking, with the first being playful and the second aggressive, whereas ADIFF detailed the first clip's 's consistent, sharp barks and the second clip's 's chaotic mix of dogs barking and man speaking. Lastly, in the Overlapping Sound Events scenario, Qwen Audio simply stated that both clips were the same song performed by different orchestras, while ADIFF offered a detailed comparison, highlighting the rich, resonant timbre of the first clip's piano and the structured sound profile of the second clip's piano and drum beat, noting the first as harmonious and calming and the second as energetic and attention-grabbe grabbing. Overall, we find ADIFF explanations to be more accurate (audio event coverage) and with finer granularity compared to Qwen-Audio.
N GENERATION DIVERSITY
If we utilize greedy decoding for both inference and evaluation, it results in outputs with repetitive starting phrases like "As listen to the two audios" and the same tokens being used to describe sound events and scenes, as shown in Table [2, This repetition is expected due to the nature of greedy decoding, which selects the most likely token each step. To enhance the diversity of the generated
25

---

Published as a conference paper at ICLR 2025
Scenario
Difference explanation from ADIFF
As listen to the two audios, I notice that Audio (1) features a sudden, sharp, and higu-frey explosion followed by series of crackling sounds, likely generated by a small object being hit. The explosion's frequency content is likely dominated by mid-to-high frequencies, with a prominent presence of low-frequency energy. In contrast, Audio presents a continuous, low-frequency hum of a helicopter flying overhead, with consistent pitch and amplitude. The helicopter's sound is likely generated by series of frequueyy transients, with prominent presence mid-to-high frequencies. The overall dynamic range Audio (2)
is much greater, with sudden changes in loudness and frequency content, whereas Audio (1) has a more gradual onset and decay.
As listen to the two audios, notice that the first audio, featuring a baby crying and woman speaking. has a distinct acoustic signature. The baby's 's cries are characterized by high-pitched, irregular patterns of sounds, with varying frequencies and durations, creating sense of urgency and distress. The woman's voice, the other hand, is softer and more gentle, with a consistent pitch and volume. In contrast, the second audio, featuring a man speaking and a crowd cheering. has a more structured sound profile. The man's voice is clear and intelligible, with consistent pitch and volume, while the crowd's s cheering adds a layer of complexity with varying frequencies, loudness, and durations. The overall dynamic range the second audio much greater, with sudden changes in loudness and pitch, whereas the first audio has a more consistent and predictable loudness.
2
The two audios differ significantly in terms their acoustic properties, with the first audio featuring a sudden, brief, and loud applause followed by clear and intelligible male voice, whereas the second audio consists of series of sharp. percussive, and percussive sounds from the tapping and hanging, with distinct differences in frequency, loudness, and duration.
3
As listen to the two audios, I notice that the first audio, featuring a dog barking loudly, has a distinct acoustic signature characterized by a consistent, repetitive pattern of barks with varying frequencics and durations. The sound is loud and has a sharp attack, with relatively short duration. In contrast, the second audio, featuring several dogs barking and man speaking, has loud and have a wide range of frequencies, from low rumbles yips. with varying durations and intensities. The man's 's voice is softer and more muffled, with a more gradual attack and decay. The overall texture of the second audio is more chaotic and dissonant, with greater emphasis on the dog's vocalizations and the man's speech.
chaotic and dissonant quality. The dog's barks are high-pitched
notice that the first audio, fcaturing group of people playing a piano, has distinct acoustic signature characterized by rich, resonant timbre with a wide range of frequencies, from low rumbles to high- nigh-pitched notes. The sound is rich low-frequency energy, with a prominent presence of mid-range frequencies, giving it a bright and piercing quality. In contrast, the second audio, featuring a piano playing followed by a drum beat, has more structured and structured sound profile. The piano's 's timbre is characterized by consistent pitch, with a prominent presence of figh-frequency energy, creating a sense of rhythm and melody. The drum beat's 's frequency content is more balanced, with a prominent presence of low- low-frequency energy. creating a sense of tension and release. Overall, the first audio is more harmonious and calming, while the second audio is more energetic and attention -raabbin....
5
Table 12: Example outputs generated from ADIFF for different scenarios
26

---

Published as a conference paper at ICLR 2025
Contrasting Acoustic Qualities
Qwen Audio
Yes, there difference between audio clips. The first audio clip of someone cutting wood with chainsaw, while the second audio dlip of someone walking on snow.
ADIFF
As listen to the two audios, notice that the first audio, featuring a person cutting wood, has a distinctively harsh abrasive quality it. sound is characterized by high-frequency
content, likely in the range of 2-5 kHz, which gives it harsh and abrasive quality. The sound also quite loud, with duration that is quite long. suggesting a sustained and intense sound. In contrast, the second audio, featuring someone walking through snow, has a more mellow and gentle quality. The sound marked by a lower frequency content, likely in the range of 100-200 Hz, which gives softer and gentie quality. The sound also quite loud, with duration that is quite long, suggesting sustained and intense sound. Overall, the two audios have distinct acoustic properties that reflect their respective sound sources and environments.
Overlapping Sound Sources
Qwen Audio
Both audio clips feature dogs barking. However, the first audio clip features dogs barking in a playful and excited manner, while the second audio clip features dogs barking more aggressive and anxious manner.
ADIFF
audio, featuring dog barking loudly, distinct acoustic signature characterized by consistent, repetitive pattern of barks with varying frequencies and durations. The sound is loud and has sharp attack. with relatively short duration. contrast, the second audio, featuring several dogs barking and a man speaking, has chaotic dissonant quality. dogs' barks frequencies, from low rumbles high-pitched yips, with varying durations and intensities. The man's voice is softer and more muffled, with a more gradual attack and decay, The overall texture of the second audio is more chactic and dissonant, with a greater emphasis on the dog's vocalizations and the man's speech.
listen two audios,
wide range
Overlapping Sound Events
Qwen Audio
The two audio clips of the same song performed different orchestras.
ADIFF
notice that the first audio, featuring group of people playing a piano, has distinct acoustic signature characterized by a rich, resonant timbre with wide range frequencies. from low rumbles to high-pitched notes. The sound rich in low-frequency energy, with prominent presence of mid-range frequencies, giving it bright and piercing quality. In contrast, the second audio, featuring a piano playing followed by a drum beat, has a more structured and structured sound profile. The piano's timbre is characterized by a consistent pitch, with a prominent presence high-frequency energy, creating sense of rhythm and melody. The drum beat's frequency content is more balanced, with prominent presence of low-frequency energy, creating a sense of tension and release. Overall, the first audio is more harmonious and calming, while the second audio is more energetic and attention-grabbing.
Figure 10: Comparing ADIFF generation against current ALMs across three scenarios: Contrasting Acoustic Qualities, Overlapping Sound Sourees, and Overlapping Sound Events. Notably, Qwen-Audio Chat the only ALM the literature that supports two audio inputs.
content, we explore top-k and top-p decoding. We set top-p to a high value (0.8) to limit the sampling of low-probability tokens. Additionally, we choose top-k value of 3, allowing other high-scoring tokens chance to be selected, thereby introducing randomness and improving the diversity of the output. The results are shown in Table [3] Using greedy decoding, the temporal order of events for audio I is incorrectly identified. However, with top-k and top-p decoding, the temporal order of events for audio 1 is correctly identified. Overall, top-k and top-p decoding increases the diversity
27

---

Published as a conference paper at ICLR 2025
Asi
ancompanied
andscape. The characterized choracteriste,
rumbles high pianned varying duration. barking cominated
marked
borking Audio The engine's dominated low-fisuuand rumbre, predizzable enviromment whereas Audio
lack
dynamic
Log Spectrogram
Hell bins
0.55
0.50
0.45
0.40
Log Hel Spectrogram Audio 2
1023
bms
Events across
preserose
Veride
0.-
Propeler, AYSCIRA
randomly selected example showcases the model's detailed explanation of audio differences, accompanied by log mel spectrograms and audio presence plots. The top pane displays the model-generated explanation. For each audio clip. it includes the log mel spectrogram and the top three audio event presence probabilities time.
Figure 11:
of the output, reducing monotonicity and helps to improve performance. The results with top-k and top-p decoding are compared against greedy decoding in Table 25
O ADIFF WITHOUT PROSS-PROJECCE
LAYER
In Table 5.2
examine the impact of the cross-projection layer. The middle section of the ble presents the performance of the baseline architecture, while the bottom section illustrates the performance with the inclusion of the separator token and cross-projection layer. On average. notice improvements for ACD and CLD across all three tiers of explanations. However, in the final ADIFF model, the LLM fine-tuned, potentially learning what the cross- pross-projection layer achieves, thereby rendering it redundant. To address this, we conducted another ablation study comparing the ADIFF model with three-step training without with the cross-projection layer. The results of this experiment are shown in Table 14 We found that the difference between ADIFF with and without the cross-pres pross-projection layer is reduced. Nevertheless, ADIFF with the cross projection layer still outperforms ADIFF without the cross-prn cross-projection layer across all tiers for the ACD and CLD datasets. We hypothesize that when the LM is fine-tuned, the cross-projection layer acts as block attention over the audio and text prefixes, which has been shown to improve performance in vision LLMs [Beee et al. (2024). Thus, retaining the cross cross-projection layer still leads to better performance even when the LLM is fine-tuned.
cross- puosjpproectionn layer against the ADIFF model
P USING WAVCAPS FOR TRAINING
AudioCaps Kim et al. 2019) and Clotho Drossos et al. (2020) are used for audio captioning evalua- tion and used in every Audio-LLaguaan Model paper Deshmukh al. (2023a); Ghosh et al. (2024a):
28

---

Published as a conference paper at ICLR 2025
Decoding
Generated difference explanation
As listen the two audios, notice that the first audio is characterized by sudden onset of sounds, starting with the clear and crisp voice of a young female, followed by a sharp, high-pitched sound of spraying, and then a piercing scream. In contrast, the second audio features a steady, gentle flow of water, accompanied by a soft and distant voice of a woman speaking. The most striking difference is the dynamic range and temporal structure of the two audios. The first audio has a sudden and dramatic change in sound levels, with clear distinction between the soft voice and the loud spraying and screaming, whereas the second audio has more consistent and gradual sound level, with the water flow and distant voice blending together seamlessly.
Ground truth
As listen to the two audios, notice that Audio 1. featuring a girl speaking and spraying, has a distinctively different acoustic profile compared to Audio 2. which consists of water running and distant woman speaking. The first audio is characterized by a sudden, sharp attack of spray sound, followed by woman's ' voice speaking in calm tone, and then sudden, sharp attack of a spray sound. The frequency content of the two audios is also distinct, with Audio I featuring a wider range of frequencies, from low rumbles to high-pitched squeals, whereas Audio has a more even distribution of frequencies across the audio spectrum. Overall, the two audios have distinct acoustic properties, with Audio featuring sudden, attention- attention-grahbing event and Audio 2 featuring a calming, serene atmosphere.
Greedy
As listen the two audios, notice that Audio 1, featuring woman talking followed by spraying and spraying noise, has distinct sonic landscape. The woman's 's voice is characterized by high-frequeney range, likely in the 100-500 Hz range, with rapid tempo and a steady loudness level. The spraying noise adds sudden, high-energy event, followed by sharp, transient event, and then brief pause before woman speaks again. In contrast, Audio 2, featuring water running, has more gradual and continuous sound, with a gentle, mid-free hid-frequency range (500-2000 Hz) that provides a sense of texture and continuity. The sound of the water is characterized by a consistent, gentle flow, with a consistent loudness level and a steady tempo. The difference lies in the type of sound sources. materials, and interactions involved, with Audio featuring sudden, attention-grabbing events and Audio 2 presenting more continuous, soothing background.
Top-k and Top-p (1)
As listen the two audios, notice a stark contrast between them. Audio (1) features a sudden, high- frequeeny burst of a female voice. followed by a sudden, sharp. and high-frequency spraying sound, and then another female voice. The audio is characterized by a sharp attack and decay, with a clear and intelligible voice that is likely in the mid-to high frequency range. In contrast, Audio (2) features a gentle, low-frequency gurgling sound from water source, accompanied by distant speech. The audio is dominated hy mid-to-high frequency range, with smooth and continuous quality that is characteristic of water sounds. The acoustic properties of these two audios fundamentally different, with Audio (1) evoking sense of excitement and excitement, while Audio (2) evokes sense of serenity and relaxation.
Top-k and Top-p (2)
Table 13: Generated outputs from different decoding methods. (1) and (2) indicates run and run respectively.
29

---

Published as a conference paper at ICLR 2025
Task
Tier
BLEU,
BLEU,
BLEUs
BLEU,
METEOR
ROUGE,
CIDE
SPICE
SPIDEr
AVG
ACD
0.384
0.256
0.182
0.129
0.208
0.352
0.300
0.268
ACD
0.444
0.304
0.223
0.162
0.150
0.377
0.407
0.185
0.296
0,288
ACD
0.477
0.305
0.164
0.199
0.323
0.152
0.150
0.237
CLD
0.446
0.338
0.258
0.191
0.304
0.478
0,832
0.190
0.511
0.394
CLD
0,517
0.354
0,261
0,194
0,231
0,458
0.971
0.251
0.611
0,427
CLD
0.496
0.325
0.228
0.166
0.203
0.357
0,232
0.184
0.208
0.266
ACD
0.405
0,276
0.192
0.135
0.381
0.303
0.280
ACD
0.440
0.301
0.232
0,180
0.197
0.379
0.518
0.173
0.345
0.307
ACD
0,486
0.317
0,225
0,171
0,208
0,331
0,204
0.162
0.183
0,254
CLD
0.433
0.338
0.265
0.203
0.302
0.4888
1,093
0.211
0.652
U.443
CLD 0.519
0.362
0.274
0,213
0,235
0.464
1.124
0.260
0.692
0,460
CLD
0.522
0.355
0.255
0.191
0.220
0.360
0.634
0.200
0.417
0.350
Table 14: ADIFF model performance without cross poss-projection layer. The top half of the table shows the performance the ADIFF model without cross projectton layer. The bottom half of table shows performance of the ADIFF model with the cross-projection layer. Both models are trained on ACD and CLD train split. The Tier 1, 2, and 3 classifications correspond to the explanation tiers detailed in Section
Task
Tier
BLEU,
BLEU,
BLEU3
BLEU,
METEOR
ROUGE,
CIDEs
SPICE
SPIDEr
AVG
ACD
0.351
0.247
0.174
0.118
0.210
0.368
0.220
0.236
ACD
0.425
0.298
0.221
0.163
0.193
0.375
0,284
0,166
0.225
0.261
ACD
0.442
0,282
0.199
0,153
0,188
0.318
0.117
0.129
0.123
0,217
CLD
0.368
0.266
0.192
0.128
0.237
0.431
0,270
0.156
0.212
0.251
CLD
0.517
0.376
0.294
0,233
0,234
0.476
0.990
0.292
0.641
0,450
CLD
0.448
0.298
0.211
0.157
0.199
0.327
0.127
0.205
0,166
0,238
ACD
0.382
0.257
0.185
0.132
0.214
0.374
0.235
0,250
ACD
0.426
0.296
0.221
0.166
0.204
0.377
0,234
0.191
0.212
0.259
ACD
0,480
0,306
0,216
0.165
0,202
0.329
0.196
0.151
0.173
0,246
CLD
0.387
0.288
0.217
0.140
0.285
0.456
0,287
0.174
0.230
0.274
CLD
0,514
0,376
0,292
0,233
0,236
0.473
1,272
0.239
0.756
0.488
0.485
0.214
0,200
ACD
0,359
0.224
0.156
0.110
0,183
0,331
0.411
0.104
0.258
0,237
ACD
U.442
0.302
0.219
0.163
0.199
0.376
0,311
0,170
0.241
0.269
ACD
D.462
0.290
0,200
0.151
0.194
0.309
0.030
0.134
0.082
0,206
CLD
0.350
0.258
0.195
0.126
0,231
0,401
0,290
0.117
0.204
0,241
CLD
0.558
0.421
0.336
0,273
0.254
0.517
1.620
0.301
0.958
0,581
0.468
0,282
0.130
0,207
0.172
0.235
ACD
0.405
0.276
0.192
0.135
0
0.381
0.469
0.137
0.303
0,280
ACD
0.440
0.301
0.232
0,180
0.197
0.379
0.518
0.173
0.345
0.307
ACD
0.486
0.317
0.225
0.171
0.208
0.331
0.204
0.162
0.183
0.254
CLD
0.433
0.338
0.265
0.203
0.302
0.488
1,093
0.652
0.443
CLD
0.519
0.362
0,274
0,213
0,235
0.464
1.124
0.260
0.692
0.460
0.522
0.355
0.255
0.191
0,220
0.360
0.417
0.350
Table 15: Comparing baseline models against ADIFF model performance. The top half the table shows the
performance of the naive model. The second part shows the performance of QwenAC finctuned with LoRA. The third part shows the performance of QwenAC finetuned fully. While the bottom half of the table shows the performance of the ADIFF model. Both models are trained on ACD and CLD train split. The Tier 1, 2, and 3 classifications correspond to the explanation tiers detailed in Section 2
Gong et al. (2024 2023);
: Chu et al. (2023): Tang et al. (2024b) in literature. Therefore, AudioCaps Kim et al. (2019) and Clotho Drossos et al. (2020) for creating the ACD and CLD dataset. The human annotations in AudioCaps and Clotho may have problems, therefore also perform human verification on the test difference explanations generated to remove any discrepan- cies. To improve performance on diverse audio recordings and scaling training, audio captioning and Audio-Language literature has used WavCaps Mei et al. (2024b) dataset. The WavCaps dataset large-scale, weakly-labelled audio captioning dataset designed for audio-language multimodal research. It comprises approximately 400k audio clips with paired captions. The audio clips are sourced from various web sources, including FreeSound, BBC Sound Effects, SoundBible, and the AudioSet Strongly-labelled Subset. The dataset uses ChatGPT to filter and transform raw descrip- tions into high-quality captions. As there is no human verification in the process, we only use this data for training and not for evaluation. To generate the audio difference explanation for WavCaps, we follow the same process listed Section 2.1 and Section E We call the resulting dataset Way- Caps Difference (WCD) and its statistics shown in Table 19 For example, in Tier 1, the WCD
30

---

Published as a conference paper at ICLR 2025
Task
Tier
BLEU;
BLEU,
BLEUs
BLEU,
METEOR
ROLGE,
CIDE
SPICE
SPIDEr
AVG
0.303
0.199
0.130
0.082
0.184
0.319
0.128
0.178
ACD
0.396
0.263
0.188
0.135
0.176
0.349
0.156
0.152
0.154
0.219
0.175
0.107
0.207
CLD
0.350
0.257
0.195
0.138
0.223
0.409
0.1
0,138
0.127
0.217
CLD
0.507
0.343
0.258
0.199
0.236
0.459
0.922
0,256
0.589
0.419
CLD
0.411
0.249
0.170
0.123
0.166
0.300
0.111
0,143
0.127
0.200
0,210
0.368
0.236
ACD
0,425
0.298
0.221
0,163
0.193
0.375
0.284
0,166
0.225
0.261
ACD
0.442
0.282
0.199
0.153
0.188
0.318
0.117
0.129
0.123
0.217
CLD
0.368
0.266
0.192
0.128
0.237
0.431
0,270
0.156
0.212
0.251
CLD
0.517
0.376
0.294
0.233
0.234
0.476
0.990
0.292
0.641
0.450
CLDD
0.448
0.298
0.211
0.157
0.199
0.327
0.127
0.205
0,166
0.238
0.258
0.214
0.364
0.287
0.265
ACD
0.427
0.287
0.209
0.155
0.197
0.373
0.305
0.181
0.243
0.264
ACD 0.471
0.293
0.200
0.149
0.203
0.316
0.161
0.147
0,154
0.233
CLDD
0.395
0.296
0.224
0.156
0.257
0.431
0.214
0.176
0.195
0.260
CLD 0.585
0.427
0.340
0.280
0.267
0.525
1.485
0.322
0.904
0.571
0.266
0.127
0.187
0.305
0.233
Table 16:
Architecture results. The top half of the table baseline architecture with random audio encoder weights. The middle half the table uses pretrained audio encoder weights. The bottom half table ADIFF which modifies baseline architecture with separator token and cross- coss-prooection
LM
BLEUU
BLEU
BL.UUs
BLEU
ROUGE
Base
ACD-1
0.376
0.258
0.183
0.131
0.214
0.364
0.445
0.129
0.287
0.265
Base
ACD-2
0.427
0.287
0.209
0.155
0.197
0.373
0.305
0.181
0.243
0,264
Base
ACD-3
0.471
0,293
0,200
0.149
0,203
0.316
0.161
0.147
0.154
0,233
Base
0.176
Base
CLD-2
0.585
0.427
0.340
0,280
0,267
0.525
1.485
0.322
0.904
0,571
Base
CLD-3
0.444
0.266
0.177
0.127
0.187
0.305
0.221
0.171
0.196
0,233
Med.
ACD-I
0.401
0.264
0.178
0.1
0.211
0.373
0.520
0.131
0.325
0.280
Med.
ACD-2
0.427
0.287
0.210
0.154
0.370
0.324
0.171
0.266
Med.
ACD-3
0.468
0,300
0.210
0,160
0,201
0.337
0.304
0.160
0,232
0.264
Med.
Med.
CLD-2
0,560
0.410
0.324
0,264
0,287
0,510
0,941
0.340
0.641
0,475
Med.
CLD-3
0.474
0.296
0,200
0.143
0,203
0.326
0,262
D.146
0.204
0,250
Large
ACD-1
0.362
0,250
0.175
0,124
0,221
0,371
0.356
0.124
0,240
0,247
Large
D.141
Large
ACD-3
0.479
0.311
0.216
0.162
0,207
0.328
0,182
0.149
0.165
0,244
Large
CLD-1
0.198
0.133
0.419
0,405
0.121
Large
CLD-2
0,502
0.329
0.239
0,180
0,223
0,457
0.930
0.219
0.574
0,406
Large
CLD-3
0,483
0.318
0.230
0.177
0.191
0.331
0.263
0.192
0.227
0,268
XL
ACD-1
0.373
0.259
0,181
0.129
0,216
0,376
0,391
0.153
0.272
0,261
XI.
ACD.2
0.173
XL
ACD-3
0.498
0.310
0.209
0,154
0,208
0.338
0.072
0.141
0.107
0,226
XL
CLD-1
0.373
0.271
0.195
0.112
0.256
0,420
0.402
0.149
0.275
0.273
XL
CLD.2
0.546
0.386
0.296
0.234
0,255
0.486
0.788
0.266
0.527
0.420
XI.
CL.D33
0.490
0.300
0.205
0.146
0.208
0.344
0.243
0.180
0.212
0.259
Table 17:
Scaling language model results. The language model in ADIFF architecture (transformer decoder) ranges in size from 128 million (Base) to 1.5 billion parameters (XI.).
Train split has 366k examples with explanations having a median of 27, maximum of 58, and vocabulary size of 38425, while Tier has 366k examples and the explanations have median of 155, maximum of 234, and a vocabulary size of 51028.
We trained the ADIFF model using a three-stage training process on the WavCaps, ACD, and CLD datasets, and evaluated it on the ACD and CLD test sets. A summary of the results is shown in Table 21 with detailed metrics in Table 20 Although incorporating WCD into the training data improves performance some metrics, it generally results in performance decline on the ACD and CLD test sets. This performance drop after adding WavCaps has been observed in existing literature Mei et al. (2024b); Deshmukh al. (2023a) 2024b), primarily because adding WayCaps changes the training distribution, shifting it further away from the testing distribution of ACD and CLD. This shift highlights limitation of objective metrics that focus on linguistics. However, integrating WCD
31

---

Published as a conference paper at ICLR 2025
Task
Tier
BLEU,
BLEU,
BLEU
BLBU,
METEOR
ROUGE,,
CIDE
SPICE
SPIDEr
AVG
ACD
0.384
0.256
0.182
0.129
0.208
0.352
0.300
0.268
ACD
0.444
0.304
0.223
0.162
0.195
0.377
0.407
0.185
0.296
0,288
ACD
0.305
0.150
0.237
CLD
0.446
0.338
0.258
0.191
0.304
0.478
0.832
0.190
0.511
0.394
CLD
0,520
0,380
0,299
0,236
0,242
0.489
1.108
0.270
0.689
0,470
CLD
0.439
0.277
0.190
0.136
0.188
0,340
0.069
0.179
0.124
0.216
0.405
0,275
0.192
0.135
0.303
0,280
ACD
0.440
0.301
0.232
0,180
0.197
0.379
0.518
0.173
0.345
0.307
ACD
0.485
0.317
0,225
0,171
0,208
0,331
0,204
0.162
0.185
0,254
CLD
0.433
0.338
0.265
0.203
0.302
0.4888
1,083
0.211
0.652
U.443
CLD 0.519
0.362
0.274
0,213
0,235
0.464
1.124
0.260
0.692
0,460
CLD
0.522
0.355
0.255
0.191
0.220
0.360
0.634
0.200
0.417
0.350
Table 18: Audio grounding and finetuning results. The top half of the table shows ADIFF performance with position captioning. The bottom half of Table finetunes the language model of ADIFF
Tier
Examples per tier
Split
Med.
Vocab.
Med.
Max
Vocab.
Med.
Max
Vocab.
CLD CLD
Train
19195
27
49
6528
51
92
6462
135
221
10818
Val
7026
Test
5225
27
28
5225
52
86
40.99
156
219
7152
ACD
Train
48660
29
3287
52
104
8483
155
235
12891
ACD
Val
2456
28
2350
52
87
2563
154
227
4566
ACD
Test
4680
29
3287
53
95
3329
154
220
5489
Table 19: Comparing WavCaps Difference (WCD) dataset statistics with AudioCaps Difference (ACD) and
Clotho Difference (CLD) dataset
into ADIFF significantly enhances the model's vocabulary at inference, increases the coverage of sounds and audio sources, and provides greater granularity compared ADIFF without WCD.
Tier
3
Task
Models
BLEU,
METEOR
SPIDEr
BLEU,
METEOR
SPIDEr
BLEU,
METEOR
SPIDEr
ADIFF
0.135
0,221
0.303
0.180
0.197
0.345
0.171
0.208
0.183
ACD
ADIFF WCD
0.105
0.189
0.198
0.197
0.205
0.259
0.152
0.197
0.152
CLD
ADIFF
0.203
0.302
0.652
0.213
0.235
0.692
0.191
0.220
0.417
ADIFF WCD
0.126
0.246
0.278
0.322
0.290
1.022
0.180
0.211
0.367
Table 20: ADIFF performance with WavCaps difference explanation (WCD) added to training data. The top half of the table shows the performance models on the task ACD, while The bottom half of the table shows the performance of models on the task of CLD, The results with all metrics and the average score are available in Appendix Table 21
Task
Hier
BLEU,
BLEU2
BLEU,
BLEU,
METEOR
ROUGE,
CIDEr
SPICE
SPIDEr
AVG
ACD
0.405
0.276
0.192
0.135
0,221
0.381
0.469
0.137
0.303
0,280
ACD
0.232
0.180
0.197
0.379
0.307
ACD
0.486
0.317
0.225
0.171
0,208
0.331
0.204
0.162
0.183
0.254
CLD
0.433
0,338
0,265
0,203
0,302
0.488
1,093
0.211
0.652
0,443
CLD
0.519
0.362
0.274
0,213
0.235
0.464
1.124
0.260
0.692
0.460
CLD
0.522
0.355
0.255
0.191
0.220
0.360
0.634
0.200
0.417
0.350
ACD
D.346
0.226
0.155
0.105
0.189
D.323
0.292
0.104
0.198
0.215
ACD
0.257
0.197
0.399
0.259
0.294
ACD
0.463
0.284
0.198
0.152
0.197
0.304
0,161
0.142
0.152
0.228
CLD
0.368
0,269
0.196
0.126
0,246
0,401
0.461
0.096
0.278
0,271
CLD
0.599
0.467
0.387
0.322
0.290
0.546
1.73
0.318
1.022
0.631
CLD
0.505
0.328
0.235
0.180
0.211
0.348
0.564
0.170
0.367
0.323
Table 21: ADIFF performance with WavCaps difference explanation (WCD) added to training data. The top
half of the table shows the ADIFF model performance. The bottom part shows the performance of ADIFF with WCD added the training data. The Tier 1, 2. and classifications correspond the explanation tiers detailed Section
32

---

Published as a conference paper at ICLR 2025
LLM prompt
You are a helpful assisiani with expert knowledge ahout audia, acoustis, and psychusacusttics You study audio, which is the study revolve around generation, propagation, and reception
sound and propertes. study acoustics, You study Psychology which posits that sound as complex stimulus that encompasses vast range of acoustic properties involving aspects of cogaition. psychoacousties, and psychomechanics. Your task is to rate the difference explanation between the two audios and provide score how detailed and granular the audio diflerence explanation. Pay attention differences lerms andio events (day barking vs cal meowing), sound sources (musical instrument, hurman voice), acoustic scene (park vs living room), signal characteristics (frequency, pitch, loadness, etc) emotion response (listening the audio invokes sense of happiness, fear, you) and any other difference using world-knowlodge or using deductive reasoning. The scere can take the value of 1,2.3.4, and 5. The score (poor) indicates provides some detail but is gencrally too broad. The score of 3 (good) indicates the explanation is moderately detailed, covering key aspects but missing finer points. The score of (Very Good) indicates the explanation detailed, covering most aspects with good specificity. 5 (Excelleni) the explanation is highly detailed, thoroughly covering all relevani aspects with precise specificity. Return your rating as JSON with the key "szore".
waves.
explanation is very vague, lacking detail specificity. The score (fair) indicates explanation
Table 22: Prompting LLM to score information density in the generated audio difference explanation
Q INFORMATION DENSITY IN DIFFERENCE EXPLANATION
We create the three Tiers in Section 2 with each tier containing more detailed and granular informa- tion than the previous Tier. We explain the data statistics in 2.1 and further analysis on difference explanations in F In this section, we demonstrate the quality of the three tiers, showing that Tier-3 offers greater granularity in descriptions compared to Tier- 1. We propose three ways to verify and prove this.
Q.1 QUALITATIVE ANALYSIS AND INFORMATION DENSITY SCORE
Qualitatively, one can review the data files and compare them. Each entry contains audio filepath 1, audio filepath2, and an answer. This format is consistent across the data, making the difference ex- planations directly comparable across the ACD test set Tier- Tier-2, and Tier-3. Manual inspection will reveal that Tier-3 contains more information or is denser than Tier-2 compared to Tier-
Quantitatively, we can construct metries that measure the density of information per difference ex- planation and compare these metrics across tiers. One proxy metric is the unique vocabulary used across the three tiers, shown in data statistics Table [] We Tier-3 is higher than in Tier-2 and Tier-1, indicating more vocabulary and concepts introduced in Tier-3. However, more vocabulary does necessarily imply more information, especially con- cerning audio characteristics. Therefore, we propose an information density score, asking human annotators to rate each explanation between 1 and 5. Due to budget constraints, we use an LLM to simulate human evaluation and provide scores between and 5. The I.MM prompt used is shown in Table 22] We use this prompt to get scores for each difference explanation across the ACD and CLD datasets. Then we average the score across each Tier of difference explanation. The results are shown in Table 23
that the unique vocabulary in
Tier-I
Tier-2
Ticr-3
Avg. score
2.91
3.64
4.17
Table 23: Information density score across the three Tiers. The Avg. density score is obtained by averaging the density score per difference explanation averaged across the ACD and CLD explanations for each Tier.
The results demonstrate that Tier-3 information density scores are the highest, while Tier-1 scores the lowest. Although this metric serves as proxy for human evaluation, the difference of at least 0.5 between the scores, averaged over 85k examples per tier (ACD + CLD), indicates that Tier-3 explanations offer greater granularity and information in descriptions compared to Tier-1 explanations.
Q.2 ENTROPY OF RACH TIER
Each Tier of the ACD and CLD dataset contains different levels of information. To measure this information one use entropy. Entropy is measure of uncertainty or randomness in dataset,
33

---

Published as a conference paper at ICLR 2025
and computing it for corpus at the character and word levels will provide insights into the structure and predictability of the language. Character-level entropy quantifies the variability in how individual characters used and combined, reflecting language complexity, encoding efficiency, and text predictability. Word-level entropy, on the other hand, captures the diversity and distribution of words, shedding light on lexical richness and the underlying linguistic patterns. To compute entropy, we use the Shannon's 's formula:
H -
Σ p(x) log, p(x)
where p(z) is the probability of a character or word occurring in the corpus. One note is assume simple unigram model, where each token's 's probability is independent of others. For character-level entropy, we count the frequency of each character. Normalize these frequencies to convert to valid probabilities and compute entropy by equation 4 For word-level entropy, we tokenize the corpus into words, compute their frequencies, and normalize as before obtain probabilities. Finally, we sum the entropy contributions for each unique word and compute entropy by equation
Entropy
Tier-1
Tier-2
Tier-3
Character
4,28
4.35
4.54
Word
7.89
8.01
8.50
Table 24: Character and word entropy across the three Tiers. For the entropy calculation. we construct the corpus for cach Tier. This involves combining ACD and CLD data for cach Tier across the three splits of dev, val, and test.
The results shown in Table 24 A higher entropy at the character or word level suggests greater diversity or unpredictability in the corpus. A higher character entropy would mean the text has a wide variety of characters with relatively even distribution (e.g., many unique letters, numbers, punctuation marks, etc). That is the text is less redundant, meaning characters appear less pre- dictably. Similarly, higher word entropy would mean the corpus has rich vocabulary with words appearing in a more evenly distributed manner. That is the text is less predictable at the word level, indicating diverse topics or language use. Across datasets, the character-leve entropy is relatively stable, implying the character set and its usage (letters, punctuation, spaces, etc.) don't 't vary sig- nificantly between corpora. The word-level entropy is higher for Tier-3, indicating more complex and varied vocabulary. Lower entropy in Tier-1 suggests more uniform and constrained language, which is expected for shorter texts or structured responses. We also computed the numbers per data source and found the CLD datasets tend to have slightly higher word-level entropy compared to the ACD datasets (c.g., detail answers have ~8.52 vs. ~~8.43) which reflect differences in diversity of text content ACD is soureed from AudioSet which mainly contains speech. From model devel- opment perspective, higher word-level entropy datasets (e.g.. Tier-3) will be more challenging for models, requiring greater capacity to handle vocabulary diversity. Though entropy allows to mea- sure uncertainty in a dataset, it does not measure semantic information, which would be measured in the density score proposed in Section Q.1
Task
Tier
BLEU;
BLEU2
BLEUs
BLEU.
METBOR
ROUGE,
CIDEr
SPICE
SPIDEr
AVG
ACD
0.405
0,276
0.192
0.135
0.221
0.381
0.303
0,280
ACD
0.440
0.301
0.232
0.180
0.197
0.379
0.518
0.173
0.345
0.307
ACD
0.486
0.317
0.225
0.171
0,208
0.331
0.204
0.183
0.254
CLD
0.433
0.338
0.265
0.203
0.302
0.488
1,093
0.211
0.652
0,443
CLD
0,519
0.362
0,274
0,213
0,235
0.464
1.124
0.260
0.692
0,460
CLD
0.522
0.355
0.255
0.191
0.220
0.360
0.634
0.200
0.417
0.350
ACD
0,287
0.207
0.148
0.217
0.388
0.174
0.242
ACD
0.497
0.365
0.289
0,234
0,234
0.437
0,200
0.222
0.211
0,299
ACD
0,500
0,321
0,226
0.169
0,211
0,340
0.117
0.171
0.144
0,244
CLD
0.412
0.295
0.215
0.152
0.219
0.402
0.163
0.119
0.141
0.235
CLD
0.491
0.368
0.296
0.245
0.230
0.445
0.1.6
0.231
0.194
0.295
CLD
0.495
0.320
0.227
0.171
0.203
0.337
0.088
0.167
0.128
0.237
Table 25: ADIFF performance with top-k and top decoding vs greedy decoding. The top half of the table shows the ADIFF model performance with top-k and top-p decoding. The bottom part shows the performance of ADIFF model with greedy decoding. We observe overall improvement performance with top-p and top-k decoding. The Tier 1, 2, and 3 classifications correspond to the explanation tiers detailed in Scction 2
34


Reviews:
[{"review_id": "0qCVX0CEfS", "reviewer": "Reviewer_EdMr", "summary": "This paper proposes a new task  - Audio Difference Explanation, which involves generating natural language description of differences between two audio recordings. They motivate this new task by pointing its significance in fields like audio forensics, audio quality assessment, and synthetic audio generation. They point out that  the current landscape of Audio-Language models like SALMONN and GAMA are geared towards applications like audio question answering, audio captioning, etc but havent addressed audio difference explanation. \n\nThe authors introduce new datasets Audio Caps Difference (ACD) and Clotho Difference (CLD). These contain pair of audios with text representing difference between the audios generated by an LLM. The text descriptions are at three levels of detail: concise, brief, detailed. These descriptions are human verified and hallucination corrected for the test subset. \n\nThe authors propose a model ADIFF to perform this task. This model encodes the two audios, followed by a projection layer to project these embeddings in LLM space. inserts a separator token in the latent space and concatenates the BPE tokenized representations from a question of the form \"Explain the difference between the two audios\". They pass this through a cross-projection layer, the output of which serves as an prompt to the LLM backbone (GPT2 in their work), which in turn produces a textual description of the difference between the audios. The authors indicate a 3 stage training of their model - 1. unimodal training of audio encoder, LLM (using pretrained models suffice), 2. audio grounding in language and 3. finetuning all components.\n\nThe authors provide ablations on how the cross projection layer is important. As a baseline, authors propose a \"naive\" baseline - which is similar to existing work in literature. The authors also perform both objective and subjective evaluations of their proposed model, and perform various ablations to understand it better.", "questions": "- Its unclear from the paper why  Qwen-Audio Chat is a valid baseline for subjective evaluation in Table 3. The authors mention that its the only ALM that supports two audio inputs, but thats not a reason enough to use this as a baseline for subjective evals if it is not designed to do audio difference explanation task? \n\n- Insufficient information on human evaluation - its unclear how many evaluations were done here, how many reviewers were involved and how it was ensured that the evaluation is fair. Its unclear what the numbers in Table 3 mean, and hence deriving insights out of them is hard.\n\n- Authors mention \"ADIFF sees the largest improvement in granularity metric\", but this is not very clear from the table 3 itself, a more detailed table, maybe in appendix that highlights this could be more readable.\n\n- Its unclear why the datasets used for Table 3 different from ACD and CLD datasets?\n\n- Its unclear how the LLM is finetuned. Is a full finetune performed? How about using PEFT  instead and utilizing larger models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Introduction of a new task - The paper addresses a previously underexplored area by defining the audio exploration task and the paper motivates this task well.\n\n- Dataset creation and availability -  The authors took great care in creating new datasets ACD and CLD for this task - which is a significant contribution for motivating further research on this task. \n\n- Novelty in Model Architecture - the authors build upon existing literature to introduce a new cross projection layer that helps compare the two audios properly - a component which is absent in existing literature on audios.\n\n- Comprehensive Evaluation - The authors do both subjective and objective evaluations to establish the models performance. The authors also include an ablation with random weights for Audio encoder - thus establishing a lower bound on the objective evaluations.\n\n- Hallucination detection tools - Authors provide means to detect hallucinations in the models responses using audio event predictions via a HTSAT encoder.", "weaknesses": "- Absence of a concrete baseline - the baseline the authors compare ADIFF against is a clearly inferior version of the same model, and thus guaranteeing that ADIFF would perform better than this baseline. This baseline is more of an ablation of the components of the model.\n\n\n- A major contribution of this work is \"cross projection\" layer that distinguishes ADIFF from baseline and existing literature. However there is Insufficient evidence of importance of the cross projection layer due to two factors:\n\n1. The authors motivate that cross projection layer is very important by utilizing the bottom two sections of Table 4 where middle section represents  baseline model with frozen LLM, and lower section indicate baseline model + cross projection layer and sep token. But in the main model ADIFF, they finetune the LLM too, which could also learn what cross-projection learns, hence making having a cross projection layer redundant. If the comparison was between  baseline + LLM finetuned vs baseline + cross projection layer + LLM finetuned, it would indicate more conclusively that cross posting is useful.\n\n2. Authors use GPT2 as the LLM model which ranges from a few hundred million params to 1.5 billion params. Its unclear which LLM was used for the cross projection ablation. Arguably, a larger LLM + finetuning could learn the task of cross projection layer. Its been shown that prompt tuning is very powerful for models like Llama3 8B models, and this ablation might not yield any difference there.\n\n\n- The paper’s structure and presentation can be improved. There is redundancy and lack of detail in tables and captions —such as the overlapping results in Tables 2 and 4, missing key information about the LLM model used, and insufficient explanations of scores presented in Table 3, which collectively make it challenging to interpret the results accurately.\n\n- The paper inaccurately equates compute across different-sized models in section 5.3 by training them for the same number of epochs, failing to account for the higher per-epoch computational cost of larger models. This undermines the  validity of their scaling comparisons in this section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j2kx5BM8cj", "reviewer": "Reviewer_uxzd", "summary": "This paper introduces a novel task focused on understanding and explaining the differences between audio recordings through various human perception primitives. It proposes a language model-based (LM-based) approach to generate explanations for the differences between two audio samples, establishing a pipeline that utilizes a large language model and existing audio metadata to create different tiers of audio differences. The experiments evaluate various configurations of the proposed audio difference model, considering factors such as text-encoder size, model components, and multi-stage training paradigms. Additionally, the paper discusses the challenges associated with creating annotations and explores potential real-world applications for this task.", "questions": "1.Regarding the structure of the sequence sent to the cross-projection layer in Figure 3, what is the length of the audio prefixes (i.e., [prefix 1] and [prefix 2])? From my understanding, HTS-AT provides only one audio embedding for the entire segment of audio samples. Do [prefix 1] and [prefix 2] consist of just a single token representing the latent embedding from HTS-AT?\n\nI suggest including a more detailed explanation of this aspect, rather than solely focusing on the right side of Figure 3.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper contains three strengths:\n\n1.  The paper introduces a considerable novelty by defining a new audio task inspired by the development of multimodal language models. The approach to explaining audio differences effectively enhances the LLM's ability to understand audio inputs at both the semantic and acoustic levels. Additionally, the method incorporates primitives of human perception into the dataset creation process, progressively decomposing audio elements from audio events to acoustic scenes, signal-level analysis, emotions, and other properties. This enriches the data quality and practical usability.\n\n2. The evaluation is thorough, primarily focusing on ablation studies. The experimental framework is well-structured, starting with a baseline model and a reference state-of-the-art model (Qwen-Audio Chat), followed by ablation studies that investigate different design choices within the proposed model.\n\n3. The paper includes thoughtful discussions regarding the hallucination tendencies of the LLM when generating annotations. It provides insights into how the authors filter these data using existing methods to enhance dataset quality.\n\nOverall, I find this paper valuable and believe it makes significant contributions to the audio community.", "weaknesses": "There is one weakness of this paper:\n\n1. The presentation of the experimental section requires further refinement. The experimental tables are somewhat difficult to follow, as the model names are not consistently listed in most of them. Additionally, highlighting or bolding the best values for each metric would enhance clarity and emphasize the differences more effectively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OV63ZkM6ZX", "reviewer": "Reviewer_syVc", "summary": "The authors introduces a new task, to explain differences between audio recordings using natural language. The authors show that current audio language models cannot complete this task, and propose AudioCaps and Clotho \"Difference\" sets, generated by LLM and verified by human in test set. These sets are trained using ADIFF, the proposed model, to bring performance boost.", "questions": "I suggest the authors complete the ablation experiments and additional finetuning experiments suggested.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The paper proposes a valid methodology in generating \"audio difference\" captions through LLMs using human captions; the methodology is verified through experiments.", "weaknesses": "1. The three-tier captioning process seems arbitary; no ablation study is conducted for this (i.e. if trained only on tier-3, can the models have good performance on tier-1? If they are purely hierarchical, then training on tier-3 should in theory yield good results on lower tiers.)\n2. Both AudioCaps and Clotho's captioning quality is questionable; WavCaps and other larger caption datasets contain more diversity, yet no experiment is conducted on them.\n3. The SOTA model, Qwen-Audio, seems to not be finetuned for this task as well. It prompts the question that finetuning a stronger model with out ADIFF may solve the problem entirely just through the proposed dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E0affe9cSf", "reviewer": "Reviewer_6ejz", "summary": "The authors propose a new derived dataset from AudioCaps and Clotho for describing differences between audios in three different tiers corresponding to the complexities of outputted descriptions. They also propose ADIFF, an architecture improving upon baseline by adding a separator token and cross-projection layer for various captioning metrics.", "questions": "- What are the types of differences described in different tiers? Is there a distribution around differences in sounding objects, differences in sounding characteristics within the same category of sound? It would be really helpful to provide these analysis from LLM generated captions.\n- For table 3, is the human evaluation also based on the difference between two audios? If so, other than the three datasets utilized (Studio, FSD50K, GTZAN), is there a reason why proposed ACD and CLD are not included in the human evaluation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed task in audio difference is novel and interesting. This is a good contribution to the audio reasoning research community. The three-tier approach makes sense and these generated descriptions can be utilized in various research topics.\n- The authors provide thorough ablation studies in proposed architectures including architectural choices, language model sizes, and training scenarios, etc.", "weaknesses": "- The choices of captioning metrics seem insufficient. As also observed by the authors in section 5.1, these metrics can be biased to the words irrelevant to the differences in audio. To complement the metrics, it might worth adding other benchmark such that the differences in sounding objects can be captured from the SED output from each audio, or the characteristics of audio can be derived from other signal processing techniques?\n- Showing the complete captioning metrics in all the tables (2, 4, 5, 6) does not seem to help in the narratives, especially not many of them are referred in the text. It might worth consolidate and only show a couple, for example, in the audio captioning research, the community is starting to converge to several more representative metrics that are more relevant to the task 1. Also it might be helpful to bulk the high performant numbers that are also referred to in the text.\n- In table 1, is the number of examples from the number of audio in each dataset? It seems that the proposed task requires two audio, therefore the dataset should be consisted of {a_i, a_j, e_i}? If this is the case, how many pairs are included in each of the split? Also, in the original dataset, each audio can include more than 1 caption (except the train split in AudioCaps), how are these captions selected as the seed for LLMs to generate descriptions, these are not clearly explained in the main paper nor in the appendix.\n\n1 https://dcase.community/challenge2024/task-automated-audio-captioning#evaluation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)