CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models
Authors: Sreyan Ghosh
Ashish Seth
Sonal Kumar
Utkarsh Tyagi
Chandra Kiran Reddy Evuru
Ramaneswaran S
S Sakshi
Oriol Nieto
Ramani Duraiswami
Dinesh Manocha
Venue: ICLR 2024 poster (2024)
Keywords: audio
audio-language
compositional reasoning
Primary area: datasets and benchmarks

Abstract: A fundamental characteristic of audio is its compositional nature. Audio-language models (ALMs) trained using a contrastive approach (e.g., CLAP) that learns a shared representation between audio and language modalities have improved performance in many downstream applications, including zero-shot audio classification, audio retrieval, etc. However, the ability of these models to effectively perform compositional reasoning remains largely unexplored and necessitates additional research. In this paper, we propose CompA, a collection of two expert-annotated benchmarks with a majority of real-world audio samples, to evaluate compositional reasoning in ALMs. Our proposed CompA-order evaluates how well an ALM understands the order or occurrence of acoustic events in audio, and CompA-attribute evaluates attribute-binding of acoustic events. An instance from either benchmark consists of two audio-caption pairs, where both audios have the same acoustic events but with different compositions. An ALM is evaluated on how well it matches the right audio to the right caption. Using this benchmark, we first show that current ALMs perform only marginally better than random chance, thereby struggling with compositional reasoning. Next, we propose CompA-CLAP, where we fine-tune CLAP using a novel learning method to improve its compositional reasoning abilities. To train CompA-CLAP, we first propose improvements to contrastive training with composition-aware hard negatives, allowing for more focused training. Next, we propose a novel modular contrastive loss that helps the model learn fine-grained compositional understanding and overcomes the acute scarcity of openly available compositional audios. CompA-CLAP significantly improves over all our baseline models on the CompA benchmark, indicating its superior compositional reasoning capabilities.

Published as a conference paper at ICLR 2024
COMPA: ADDRESSING THE GAP IN COMPOSITIONAL REASONING IN AUDIO-LANGUAGE MODELS
Sreyan Ghosh
Ashish Seth
Sonal Kumar" ** Utkarsh Tyagi"
Chandra Kiran Evuru
S. Ramaneswaran
Sakshi
Oriol Nieto
Ramani Duraiswami Dinesh Manocha
University of Maryland, College Park, USA *dobe,, USA
NVIDIA, Bangalore, India
(sreyang, sonalkum, utkarsht, ckevuru, ramanid, dmanocha @und. edu
ABSTRACT
A fundamental characteristic of audio is its compositional nature. Audio-language models (ALMs) trained using a contrastive approach (e.g., CLAP) that learns a shared representation between audio and language modalities have improved per- formance many downstream applications, including zero-shot audio classifi- cation, audio retrieval, etc. However, the ability of these models to effectively perform compositional reasoning remains largely unexplored and necessitates ad- ditional research. In this paper, we propose CompA, a collection of two expert- annotated benchmarks with a majority of real-world audio samples, to evaluate compositional reasoning in ALMs. Our proposed CompA-order evaluates how well an ALM understands the order or occurrence of acoustic events in audio, and CatpA-attribute evaluates attribute-binding of acoustic events. An instance from either benchmark consists of two audio-caption pairs, where both audios have the same acoustic events but with different compositions. An ALM eval- uated on how well it matches the right audio to the right caption. Using this benchmark, we first show that current ALMs perform only marginally better than random chance, thereby struggling with compositional reasoning. Next, pro- pose CompA-CLAP, where we fine-tune CLAP using a novel learning method improve its compositional reasoning abilities. To train CompA-CLAN we first propose improvements to contrastive training with composition-awe aware hard neg- atives, allowing for more focused training. Next, propose novel modular contrastive loss that helps the model learn fine-grained compositional understand- ing and overcomes the acute scarcity of openly available compositional audios. significantly improves over all our baseline models on the CompA benchmark, indicating its superior compositional reasoning capabilities.
INTRODUCTION
In recent years, multi-modal contrastive pre-training has emerged active area of research due to its great success in various downstream applications. CLIP (Radford et al., (021). a pioneer- ing vision-language model (VLM) trained using a multi-modal contrastive approach between vi- sion and language representations, achieves state-of-the-art (SoTA) results in a variety of vision understanding tasks and enables crucial auxiliary tasks like image captioning (Mokady al. [221]. text-to- -iagg generation (Rombach et al., 2022). etc. Inspired by CLIP, Elizalde et al. (2023) in- troduce CLAP (Contrastive Language Audio Pre-Training), an audio-languas model (ALM) that employs contrastive pre-training between audio and language representations and achieves SoTA in 16 downstream audio understanding tasks like zero-shot audio classification, text-to-audio retrieval, etc. CLAP imposes shared representation space for audios and their text descriptions (captions), making it superior for audio-language perception than other models.
Despite much success. Wu et al. (2023) show that ALMs often act as bag of words and lack natural language comprehension. Understanding the relationship between text in captions and the corre-
Equal technical contribution." Work done intern Adohe Research.

---

Published as a conference paper at ICLR 2024
sponding content of the audio is fundamental goal of audio processing, and the fact that different word orders correspond to differently perceived audio should be reflected in the capabilities of the ALMs. This phenomenon, also known compositional reasoning, may be characterized as the ALM's capacity to understand the interrelationship among multiple discrete acoustic events in au- dio, such as order of occurence and attribute-indding, as conveyed through the words in the caption. Work in natural language has shown that transformers are often remarkably insensitive to word or- der (Sinha et al. 2021). Prior research has shown that models like CLIP, despite being trained on abundant data, exhibit a deficiency in compositional reasoning (Thrush et al. 2022 Ma et al.) [023] Yuksekgonul et al. 2023). Yuksekgonul et al. (2023) argue that one of the primary causes of this is that contrastive pre-training optimizes for retrieval, and it is easy for these models to perform well on retrieval even without having a good compositional understanding. In the past, researchers have proposed benchmarks (Thrush et al., 2022) Yuksekgonul et al. (203)) and tried inducing compo- sitional reasoning in vision-lange Iision-language models (VLMs) Yuksekkonul et al., 2023 Ma et al. 2023
Jiang et al. 2.22 but such attempt has been made in the audio space yet.
Main Contributions. In this paper, we perform the first systematic study for compositional reason- ing in ALMs. To this end, propose two novel expert-annotated benchmarks and novel learning paradigm to teach ALMs compositional reasoning. Our main contributions are two-fold:
1. We develop two expert-annotated benchmarks, CompA-order and with majority real-world audios, that serve as a test-bed to evaluate the compositional reasoning of ALMs. CompA-order and CompA-attribute have 400 and 200 test instances, respectively, where each instance has two or three audio-caption pairs. Each audio is dif- ferent in composition, and each caption contains the same words but in a different order to account for the different compositions. The task of an ALM is to correctly match the audios with their captions. While CompA-order used evaluate the models' ability to under- stand the order of occurrence between two acoustic events in an audio, CompA-atrribute is used to evaluate the models' ability to link attributes to specific acoustic events (attribute- binding). More than 90% of audio snippets in CompA are sourced from real-world audio samples from AudioSet (Gemmeke et al., 2017 by expert annotators experienced in audio and language research. We discuss in Section why current benchmarks do not evaluate compositional reasoning and how CompA serves as an essential step to fill this gap.
2.
We propose a robust learning solution for improving compositional reasoning in audio-language models. First, we propose several improvements to contrastive learning with composition-awarc hard negatives (Yuksckgonul et al. 2023). Employing hard neg- atives for each audio in the batch has proven to be an effective solution and we propose the following improvements:
(1) We formulate the objective such that the hard negative captions for a particular audio are ignored by other audios in the batch. (2) We use an LLM generate semantically viable negatives. Additionally, due to the lack of compositional audios in current training datasets, we propose a novel dataset with 110k+ audio-caption pairs based on the AudioSet strong subset (Hershey et al. 2021). Next, we propose a mod- ular contrastive learning objective that does not require any existing compositional audio- caption pairs. Our proposed solution leverages a template- based audio-caption creation approach and improves CLAP's fine-grained pabilities. More specifically, we aim at aligning captions of various granularities to an audio, each of which represents a decomposed version of the audio scene. Additionally, we mine multiple negatives from the positives by interchanging orders and attributes. CompA- CLAP outperforms all our baselines on the CompA benchmark by 10%-28% while retain- ing performance on existing retrieval and zero-shot classification benchmarks. We open- source code and data: https: /tttps://szeyan8// qithub. comoa iclr/.
serribute-binding and order- underssadddttnii ca-
2 COMPA BENCHMARK
Overview. We propose two expert-annotated benchmarks, CompA-order and CompA-attribuu structured in the Winograd twin sentence format, to evaluate an ALMs' capabilities to understand various types of compositional relationships. Each instance in each benchmark has two three audio-caption pairs, where each audio has the same acoustic events but with a different composi- tion, and each caption has the same words (specifies detailed in next Sections). The task of an ALM
2

---

Published as a conference paper at ICLR 2024
is to match the right caption with the right audio and vere-versa. We are particularly inspired by the Winoground dataset proposed by Thrush et al. (2022) built for evaluating visio- asio-linguistic com- positional reasoning. Our work is also inspired from Winograd twin sentence format, originally proposed in the Winograd schema challenge (Levesque et al. 2012). detailed Section S
Annotation. Both benchmarks were annotated by four subject-matter experts specializing in au- dio and language. To build CompA-order, we used natural audio samples sourced from AudioSet Strong (Hershey et al.) 2021). AudioSet Strong has temporal labels, and annotators were asked to annotate continuous stretches of audio with the desired acoustic events. We list down annotation rules, illustrate the snapshot of the annotation tool, and provide more information on the annotator backgrounds in Appendix B.3] We opted for natural audio over synthetic alternatives to capture inherent elements such as background noise and interventions, thereby offering a test-bed for ALS that closely mimics real-world conditions. Finally, a continuous stretch of audio with the annotated events sliced (with gold timestamps), and caption is written for it.
2.1 WHY ARE CURRENT BENCHMARKS INSUFFICIENT FOR EVALUATING COMPOSITIONAL REASONING IN ALMs?
Fig. [ iillustrates the results of an experiment where we show that it is easy for ALMs to perform well on the most commonly used audio-retrieval bench- marks, Clotho (Drossos et al., (2020) and Audio- Caps (Kim et al., 2019). even without proper word ordering. Additionally, we analyze the distribution of unique nouns per caption and see that most of the audio samples in the test set of both bench- marks have only a single acoustic event and are non-compositional. Wu et al. (2023) also report that CLAP achieves similar benchmark performance when trained with captions stripped of all but nouns and verbs. This suggests that even ALMs without compositional reasoning abilities can perform well on these benchmarks.
Cooho Validitioo
AudioCaps Validonion
130
Gotho Test
AudioCaps Test
137
Figure 1: Comparison of CLAP performance af- shuffling the word order in captions. CLAP undergoes an average degradation of 0.04 in top- I(R 1) and 0.03 in top- 10 (R 10) retrieval.
2.2 COMPA-ORDE -ORDER
Fig. illustrates an example from CompA-order. We build CompA-order to evaluate an ALM's abil- ity to understand the order of occurrence between multiple acoustic events. At its core, an acous- tic event in an audio can either succeed another event, precede another event, occur simultane- ously with it. CompA-order has 400 test instances, with each instance comprising a minimum of two audio-caption pairs (Co,Ag;C),AA)) dios have the same two acoustic events, but the order of occurrence of the event in the audio differs. Out of the 400, 100 of these instances have an extra pair A2). where the two acoustic events occur simul- taneously. Expanding CompA with audios comprising more than two acoustic events remains part of future work. The captions for the audios in an instance with two pairs have the exact same words but in a different order, except for the instances with three pairs, where only a single word that defines the preposition between the events is changed. More details are in Appendix B.1
CompA-Order
Order
* Dveete
Captions
growt lger sucseeded
"Hunion converationn succooded by grout of ligae"
"The growt of tgar amiddit human conversde anrdo.
where the au-
Figure 2: CompA-order evaluates an ALMs' capability to understand the order of occurrence between multiple acoustic events in audio.
2.3 COMPA-ATTRIBUTE
We build CompA-attribut to evaluate an ALM's ability to reason attribute-bndding for acous- tic events. CompA-attributt has 200 test instances, with each instance comprising two audio-caption pairs A2) where each audio has the same two acoustic events
3

---

Published as a conference paper at ICLR 2024
but as if associated with different attribute. The term attribute is broad, and for our case, we consider multiple attributes like source (example in Fig. B qualitative ("Static hiss joins random notes played by synthesizer." "Static notes joins random hiss played by synthesizer.),, etc. The captions have the exact same words but in different order. For CompA-atie -atpA-attribute, we used synthetically generated audios from WavJourney Liu al.) (2023d) that are carefully validated by experts who also supervise the generation process. More details are Appendix
CompA-Attribute
Captions
X
daby woman laugha"
"Ai woman cries unie baby laughs"
Figure 3: CompA-attribute evaluates an ALM's capability understand attribute-binding for multiple acoustic events an audio.
and B.3
2.4 EVALUATION
Performance on both CompA-order and CompA-attribute is measured on three different met- rics to measure different aspects of compositional reasoning in ALM. We inspired by Winoground (Thrush et al., 2022) for our evaluation metrics. Given two audios Ag and A₁ and their corresponding captions Co and C1, from instance in either benchmark, we define the text score that measures whether an ALM can select the correct caption, given an audio. Thus, our metric (Co, Ao, C1, A1) is defined follows:
if (Co, An) (C1, and & (C1, A,) > 8 (Co, A1)
f f(C. Ao. C1. A1)
otherwise
where s(.) is the cosine similarity between audio-caption pair. Intuitively, the model has higher similarity for its ground-truth caption than the alternative caption describing composition- ally different audio, performs better at compositional audio-to-text retrieval. The second metric is the audio score, which measures whether ALM select the correct audio, given caption. We define our metric g (Co, Ag. C1, A1) as follows:
s (Co, Ao) > s (Co,Ai) and (C1, A1) > (C1,AA) otherwise
(Co, C1, A
Intuitively, if the model has a higher similarity for its ground-truth audio than the alternative audio that is compositionally different, it performs better compositional text-to-audio retrieval. Finally, we define group score combining the audio and text scores defined above as follows:
if f (C,, Ao, A1 and (Co, Ag, C1 A1)
h (Co, An. C1,
otherwise
The group score helps evaluate performance for an entire benchmark instance or set of sentences. This is motivated by prior work by Elazar al. (202T), who show that the individual evaluation metrics tend to overestimate model performance by computing scores for the twin sentences indi- vidually, Finally, we average across all instances in the benchmark dataset for all three scores.
COMPA-CLAP! IMPROVING COMPOSITIONAL REASONING IN ALMS
Fig. illustrates the training methodology of CompA-CLAP, CompA-CLAP is initialized with CLAP weights and is fine-tuned twice to learn compositionality. In the next sub-sections, we first discuss why currently available training datasets insufficient to make ALM learn composi- tional reasoning. Second, we detail the two stages CompA-CLAP training, which prove to be superior to CLAP in compositional reasoning, when tested CompA benchmarks.

---

Published as a conference paper at ICLR 2024
3.1 WHY CURRENT TRAINING DATASETS ARE INSUFFICIENT TO LEARN COMPOSITIONAL REASONING? CAN SYNTHETIC DATA HELP?
Problem. Unlike vision-language models, which benefit from being trained on internet-scale data (Schuhmann et al. (02T), datasets for training ALMs are generally orders of magnitude smaller.
While the presence of compositional audio in the
LAION Audio-620K
CompA Audiodet
training dataset does not guarantee the model will
acquire compositional reasoning skills (Thrush et al.
10%
10%
18%
2356
2022), most proposed composition learning tech- niques from literature operate under the assump- tion that compositional audio is available within the training dataset (Yuksekgonul et al., (023). The largest ALM to date is trained only 630k audio- text pairs (Wu et al.] 2023). The authors propose LAION-audio-630K, pooled from several existing openly available datasets. However, most these audios do have more than one acoustic event. Fig. 4 illustrates this phenomenon by show- ing the distribution of unique nouns per caption in LAION-audio-6300 To expand the pre-training dataset, the authors further propose netword-toccaptionn augmentation on the large-scale AudioSet dataset (Gemmeke et al., 2017). Precisely, they feed the audio labels to Pre- Pre-trained Language Model (PLM), which generates coherent caption from the labels. However, we argue that such an augmentation scheme would generate captions that do not describe the correct ordering of events and would adversely affect the models' compositional reasoning abilities. Table 1 from Wu et al. (2023) also shows that augmenting LAIONAudio-66 630K with the entire 2M AudioSet for CLAP training almost never improves performance on any benchmark. We hypothesize that this is due to their Seworr-to-Caption generation algorithm, which does not guarantee temporally aligned captions.
Number of Unique Acoustio Events:
189
Figure 4: Comparison of unique acoustic events per audio between LAION Audio-630K and CompA-AudioSet.
simple yet not very effective solution. A simple solution to the above- -ooee-met problem is to generate compositional audio using recent advancements in -ot-tto-audio generation. We prompted SoTa text-to-audio generation models (Liu et al. 2023b Ghosal et al. 2023), including WavJour- ney (Liu et al. 2023d) generate audio for a caption generated using GPT-4. From our experiments, we conclude that current text-to-audio generation models struggle with diverse and compositional audio generation. Most of the time, the models struggled generate cohesive audio with more than two acoustic events, and the generated audio did not constitute of events mentioned in the caption.
3.2 VANILLA CONTRASTIVE LADIO-LANGUAGE PRE-TRAINING
Methodology. We train our own version of CLAP with contrastive audio-language pre-training but with minor modifications. For the audio encoder, employ HTSAT-large(Chen et al. 2022). as also employed by Wu et al. (2023). For the text encoder, we replace the RoBERTa encoder with an instruction-tuned Flan-T5-large encoder (Chung al. 2022) Ghosal et al. (2223 showed the effectiveness of employing an instruction-tur -ttnetd encoder for multi-modal contrastive pre- treiiiinnng.. Using audio and text encoders, we solve the contrastive objective similar to CLAP (Elizalde) jet al. 2023), which solves the InfoNCE loss (Oord et al.) 2018) between a batch of audio and text embeddings. Each audio in the batch of size B has positive caption and 2 negative captions. The same holds for text representations.
Experimental Protocol. For pre-training, we make minor modifications the LAION-audio-630k pre-training dataset proposed by Wu et al. (2023). We introduce 16mpA-661k, with =661k kk unique audio-capill audio-caption pairs. We list down all the sources of CompA-661k
Appendix B.2 Our version of CLAP outperforms (Wu et al., 2023) on all existing retrieval benchmarks for both text-to-audie audioo and audio-to-text retrieval by 4.67%% and CompA-order and CompA-attribur by 1 11.85%-23.854
3.3
CONTRASTIVE PRE- TRAINING WITH COMPOSITIONALLY AWARE HARD NEGATIVES
Overview & Background. In this subsection, we describe our methodology to fine-tune ALM using a modified contrastive learning (CL) formulation with compositionally-awaat
hard negatives. Following prior work in VLM fine-tuning (Yuksekgonul et al.) 2023), we modify the vanilla CL objective and introduce compositionally aware hard negative captions for each audio in the batch to teach the model compositional reasoning. However, compositionally aware hard negative captions
5

---

Published as a conference paper at ICLR 2024
Compositonally Alsare Hard Negalivss
Modular Contussive Leaming
X-X
X-X
XXXX
100/0/ila
Hoblojd
Illustaaioon of contrastive learning techniques for improving compositional reasoning in ALMs. Left: Contrastive training with compositionally-ave aware hard negatives where each audio has K hard negative captions generated using LLM, and each audio in the batch ignores negatives of other audios in the batch for more focused training. Right: Our proposed Modular Contrastive training employs multiple positives and negatives for each audio the batch generated using template-basce algorithm. Each positive describes compositional relationships of various granularities in the audio, and this helps the model learn fine-grained order and crrrbbutt-biddng.. An audio in the batch ignores the positives and negatives of other audios.
Figure 5:
can only be formed for existing compositional audios, and the absence of it in open-source datasets a considerable challenge to this approach.
A new training dataset with compositionally rich audio-text pairs. Building on the hypothesis that training compositionally rich audio dataset can lead to better compositional reasoning, we propose PaadioSet-Comp... a new paired dataset, with = 110k audio-text pairs for complex composi- tional audios. To build this dataset, we generated captions for the AudioSet strong by prompting an I.M and then asked two human annotators to evaluate and correct the captions. We experimented with both GPT-4 (OpenAT 2023) and LLaMa-2 (Touvron et al., 2023). and we found GPT-4 to be better at generating captions. The AudioSet strong subset has temporally strong labels, i.e., it includes timestamps for custom prompt to generate captions that have the correct ordering of sound events. Fig. 4 compares the average number of acoustic events per audio in existing and our dataset.
Methodology. mine compositionally aware hard negative captions from existing captions, we employ LLM. Prior work in VLM fine-tuning employs traditional NLP techniques YYukeegonuul ct al. 2023). we found an LLM to better deal with anomalies and generate linguistically and seman- tically viable sentences. While most open- and closed-source LLMS qualitatively competitive on this task, we adhere to GPT-4 as it performed better at generating structured outputs. A com- parison of negatives generated using various LLMs and details on prompts used can be found in Appendix B.4 To generate hard negatives, we ask the LLM to swap acoustic event ordering, replace the preposition, or swap noun- verb associations. This helps generate negatives with similar contexts but different compositions. Additionally, we constrain the LLM to perform these operations only when the resultant swap resembles plausible real-worke scenario. This helps our model learn more meaningful compositional relationships in the feature space. An example can be seen in Fig. 5) and more examples can be found in Appendix B.4
Given a dataset DD with N audio-caption pairs (A,T), for every audio on E A and its correspond- ing caption L,, T, we generate K compositionally-awaaa
negative captions hard The generated negative captions serve as hard negatives in our modified contrastive fine-tuning objective. Unlike prior work, we formulate the objective such that hard negative captions for a particular audio are ignored by other audios in the batch. This helps in more composition-focused training. Thus, our final objective defined as follows:
d-2- = -1, aaao log Σ exp (4 (5/))
(4)
8
2.2 -0, tijo log Σ exp (op(attz/o) Σ exp tai
hand
10)
(5)
where 2-22-a is the contrastive loss for text that treats audio as negatives and e0-2t is the con- trastive loss for audio that treats text as negatives. B is the batch size indexed by i and j.

---

Published as a conference paper at ICLR 2024
is the temperature parameter and (th, ] [ is the th negative caption for audio sample a:- Finally, we combine both the losses with appropriate scaling factors, aj and az to optimize: L = 1 - (ale? + taz??")
2B
i=1
Experimental Protocol. For training, we use only the compositional audios from Clotho and Au- dioCaps in addition our AudioSet-CompA dataset. This results in = =10k training pairs, which is significantly low-resource compared with prior-art teaching model compositional reasoning. We use K=3 negatives and provide values for other values of K Appendix C..] We initialize the training with vanilla CLAP weights and only train the last few layers to overcome the prob- lem where fine-tuning distorts pre-trained features when the fine-tuning and pre-training algorithms don't 't match (Wortsman et al. 2022; Goyal et al. 2023). Syper--arameters details in Appendix B.7
3.4 MODULAR CONTRASTIVE LEARNING FOR UNDERSTANDING
COMPOSITION
Overview & Background. This Subsection describes modular contrastive learning, a novel con- trastive learning formulation backed with a novel synthetic data creation methodology that over- comes problems in compositionally-avar hard negative contrastive training, like scarcity of existing compositional audio-caption pairs and learning fine- grained compositional reasoning.
As discussed earlier, ALM training suffers from an acute lack of compositional audio. Thus, scal- ing contrastive pre-training with hard negatives poses a significant challenge. Additionally, a single hard negative for an audio with multiple difficult-to- -o-isttiioii acoustic events can be too compli- cated for the model to understand effectively. This is because real-world audio often complex and unpredictable. This hinders the model from learning fine-grained attribute- binding and order information. Thus, we propose a new learning methodology that can be easily scaled, does not require any existing compositional audio-caption pairs, and helps the model learn fine-grained com- positional reasoning. More specifically, we employ a modular template-based approach to create compositional audios and their captions from single acoustic events and their labels (which are abundantly and easily available). Then, we align each audio in the batch with positive captions of various granularity and generate hard negatives with different compositions, which allows the model to focus on fine-grained compositional relationships in the audio.
Template-based synthetic creation of audio-caption pairs. Fig. 6 illustrates the proposed algo- rithm. We propose a simple and scalable template-based approach to create compositional audio, their caption, and hard negatives for training. We first create pool of several unique audio snippets and their labels, each comprising a sin- gle acoustic event. Each label might have sev- eral unique audio snippets corresponding to it. Using the snippet and the label corresponding to the acoustic event of the audio, we either con- audio over
(Al Alie
DII
Tragor MILI ScNGccooity Bay COT
catenate these audios or overlay the other (More details in B...I). For this pa- per, we slice longer audios from the AudioSet strong subset (with the provided temporal la- bels) to create the pool of unique audio snippets and employ a maximum of 4 snippets corre- sponding to 4 unique events for synthetic audio creation. Additionally, to assure high quality, we don't 't concatenate or overlay random acous- tic events but ask an LLM to create unique au- dio scenes based on the available labels (examples in Table D). To create its corresponding caption, we employ pre-defined templates with diverse prepositions. Additionally, since we create compo- sitional audio in a modular fashion, it allows us to easily obtain captions of various granularity and multiple compositional relationships in the audio. To generate
Cla terakur
TOCONII TORY
Converaaioe
Figure 6: Illustration of our template-based audio- caption creation process. An LLM first generates a scene from pool of available acoustic events. We then perform simple operations audio and their captions. This process also enables us to generate positives (and their corresponding negatives) that define fine-grained relations between events.
generate compositional
complexities, each defining a hard negative that defines an alternate composition, we either switch attributes among events or word order. Examples can be found in Table
7

---

Published as a conference paper at ICLR 2024
Table I: Performance comparison of CompA-CLAP with baselines on benchmark datasets. Left: Text-to- Audio and Audio-to- to- Texi retrieval performance on AudioCap / Clotho (in the same format). Right: Zero-shot classification results on four benchmark datasets. While our CLAP achieves SoTA performance in almost all cases, CompA-CI.AP retains its performance even after fine-tuning for compositionality.
Model
T-A Ratrival Res
A-T Retiesal Res
R
R0:10
R
RP:00
FSC-50
USIK
FS220K
CLAPLAAOO
CLAP
MI-ACT
MMT
36.2/172
34.67 16.7
33.9/144 144
56.1667
60.77 96.6
72.022. 21.6
702/4.11
42.942.9
820054.1
84.5333
83.5/55.
4199/20
45.0/24.
994/132
7.6/7.
781/446
76.7/511
220/
768/227
86.71
839;
84.6.50 58.7
88.06669
36.6
50.2
War2CLIP
Audo Up
CLAP CLAR-AIDD--ad--ll
694
41.4
82.6
83.0
65.3
13.2
84.6
58.6
43.1
35.5/17.0
8067567
37.87 253
332/315
36.7767.8
Compy
CoMPA CLAP Gears
36./116 168
78.6/43.5
98.2/56.1
47.8/259
90.2/67.6
Methodology. Given pool of acoustic events and their labels P, we first generate a dataset D with N training instances, where each instance consists of an audio, its fine-graine fine-grained positive captions, and their corresponding negatives denoted as (A,T ( Every instance t. E T is a set of Kmad fine-grained positives, denoted by poos
'kekpps + and similarly ney E T is a set of K neg fine-
neg grained negatives, denoted by (iakene Other audios in the batch ignore the generated fine-grained
positives and negatives for a particular audio. Thus, we define our objective as follows:
a:/o
log 1=1
Σ
exp (f/az/o)
(6)
R=1
B
n--2-1
K Kioss
ag
pos
1o
log
Σ
exp
(at tyyo))
Σ
exp
T 1. //)
where 2-2 is the contrastive loss for text that treats audio negatives and gn-2-t is the contrastive loss for audio that treats text as negatives. B is the batch size indexed by and j. the temper- ature parameter. (1""") ) )e[ and (1 >e[[,, are th generated fine-grained positive and negative caption for audio sample ag. Finally we combine both the losses with appropriate scaling Σ.= (3 + 326
factors 31 and B3, to optimize: _ 2
Experimental Protocol. We generate 251k audios for training. Our pool P has =500 unique audio snippets. The maximum number of acoustic events we use to make an audio is 4. Since the number of positives and negatives for each audio can grow combinatorially, we restrict the maximum to 7 for each. We initialize model from CLAP, fine-tuned with hard negatives. Training hyper- parameters are detailed in Appendix B.7 We call the resultant model CompA-CLAP.
RESULTS
Table [ (left) compares CompA-CLAP with four other baselines text-to-audio and audio-to-text retrieval tasks on Clotho and AudioCaps. These four baselines are MMT (Oncescu et al.) (02T). MI-
Table 2: Performance comparison of CompA-CLAN CAMPA-CLAP with baselines on CompA-order and CompA-attribute. attribute.
CompA-onkrr Audia
Compt-atehuic
Model
Test
Group
Text
Audio
Group
Hurna
ACT (Mei al.) (022), CLAP (Elizalde ct al. 2023) and CLAP-LAION (Wu et al.
MMT
15.90....
6.85.2
3.50
20.50..
4.88ga.e
2 2.12...5
ML-ACT
21.8kasm
8.00..
4.5b
$1.6..
22.50ga1
8:50g1 4s
4.10g m
58
0.54g
<
2023
While our CLAP trained on
m
CompA CLAP
48.76...
35.6
3865
46.28.mm
22.58.00
1513
CompA-6611 outperforms all baselines in most cases, CompA-CLAP, fine-tuned for better compositional reasoning, performs on par with CLAP with minimal performance degradation. Table (right) compares CompA-CLAP with four other basclines on zero-shot classification datasets on ESC-50 (Piczak), US8K (Salamon (ct al.) 2014), VGGSound (Chen ct al.] (2020) and FSD50k (Fonseca et al., 2022).
Had Segative
36.56.045
20.20
17.31am
Medier Contantive
35.0
21.35
45.68
10.57gmm
1304
CLAP yoars)
33.75 m0
15.7hwus
11.00
20.10
15.2
) The four base- lines are Wav2CLIP (Wu et al. (022), AudioCLIP (Wu et al. 2022), CLAP, and CLAP-LAION. All scores have been averaged for runs on random seeds.
Table 2 compares the results of CompA- -LAP with our baselines CompA-order and CompA- attribute benchmarks. First, our vanilla CLAP performs better than all other baselines from litera- ture, outperforming CLAP--LAION by x6%-33% over both benchmarks. CompA-CLAP, which is
8

---

Published as a conference paper at ICLR 2024
CLAP trained consecutively with hard negatives and modular contrastive learning, improves per- formance on both benchmarks by = 10%6-28% over CLAP. We also notice -4%-13% performance degradation when compositionally-wwar hard negative training is skipped before Modular con- trastive training. Modular contrastive training after hard negative training improves performance on benchmarks by =2%-11% However, all models, including CompA-CIAP, perform worse than our random baseline on CompA-attribuct which leaves plenty of room for improvement.
We also discuss some common mistakes observed upon result analysis. 1) CompA-CLAP per- forms better in cases when the acoustic events in an audio sound more distinct (eg., tiger growling and human speaking) and underperforms where they sound very similar (eg., human sounds). 2) CompA CLAP also suffers from the long-tailed problem in AudioSet, wherein it underperforms in audios with acoustic events seen less during training. 3) In CompA-order, the model underperforms on prepositions not seen during training.
5 RELATED WORK
Compositional Reasoning. Early work in linguistics has tried to understand what models know about word order (Sinha et al. 2021), syntax (Gauthier et al. 2020; Gulordava et al. 2018: Hu
al. 2020: Linzen et al. 2 or the complex interaction between syntactic and semantic cate- gories (Kann et al., 2019; Truss et al. 2020 Warstadt et al. 2019 2020). Learning visio-linguistic compositionality has been extensively studied in prior art ( (Yuksekgonul et al. 2023 et a 2023). Winground (Thrush et al. 2022). the work closest to ours, proposes a benchmark with 400 test cases the Winograd twin sentence format, with pairs of compositionally different images and captions with the same words but in a different order. The task, similar to the one proposed in this paper, is to match the right image with the right caption. They additionally show that current VLMs per- form no better than random chance. The Winograd twin sentence format
originally proposed
in the Winograd schema challenge (Levesque et al., 2012) and has been earlier used for a variety of aanguage-related tasks (Rudinger al. 2018 (Sakauuch al., [20T Zhao et al. 2018). Following this, (Yuksekgonul et al.) 2023) propose a large-scale benchmark with over 50,000 test cases and compositionally different captions by swapping relational tokens within sentences. Lack of com- positional reasoning in VLMs has affected multiple downstream tasks like text-to-image generation (Conwell & Ullman, 2022) and Visual Question Answering (Bogin et al. 2021). A similar prob- lem was observed in AudioLDM (Liu et al., 2023a) which employs CLAP as text encoder for text-to-audio generation and fails to generate compositional audios (Ghosal al. 2023).
Audio and Language. Recent developments indicate an increasing trend in leveraging language as modality for interaction with audio systems. Downstream tasks like text-to-audio generation (Ghosal et al. 2023; Liu al.) [2023a] Huang et al.) (023) and text-to-music generation (Agostinelli et al. 2023 have gained much popularity. Other tasks include text-guided audio source separation (Liu et al. 2023c) audio captioning (Ghosh et al., 2023), etc. DDshmuuh al. (2023) integrate language models with audio encoders and frame all audio tasks as text gentreeeertiion tasks. Their model, Pengi, achieves So'T performance 22 downstream tasks, which shows promises of effective language modality integration for enhanced audio system interactions. Most of these models employ a text encoder audio encoder accomplish their task. CLAP, which learns shared representation between audio and language modalities and achieves impressive performance on zero-shot tasks, proving be a compelling model for cross-modal understanding and interaction.
6 CONCLUSION AND FUTURE WORK
In this paper, we first discuss several problems with existing audio-retrieval benchmarks and show that it is easy for ALMs to perform well on these benchmarks without any compositional under- standing. To address this gap, we propose CompA, a suite of two benchmarks to evaluate an ALM's compositional reasoning capabilities. Using this benchmark, we first show that current ALMs strug- gle with compositional reasoning. Next, we propose CompA-CLAP, an audio-language model that addresses the lack of compositional reasoning in ALMs. To train CompA-CLAR, we first propose improvements to contrastive pre- teetrriiinng with compositionally-nne aware hard negatives. Then, we pro- pose novel modular contrastive training approach that helps the model learn ine-grained order information and attribute binding. As part of future work, we would like to expand the size of CompA and find better and novel solutions to teach an ALM compositional reasoning.

---

Published as a conference paper at ICLR 2024
REFERENCES
dump of BBC's sound effects library, 12 2018. URL. http://bbcst
acropolis org. This dump was created using the script found at plete.
Identifier: BBCSSonnEEffetsCCom
SoundBible - Free Sound Clips, Sound Bites, and Sound Effects, 2023. Goouddbible wbl.. comm Accessed: 25 September 2023.
URL https://
Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. Musiclm: Generating music from text, 2023.
Ben Bogin, Shivanshu Gupta, Matt Gardner, and Jonathan Berant. Covr: A test-bed for visually grounded compositional generalization with real images. arXiv preprint arXiv:2109.10611 2021.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020.
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Kang-Kirpattrikk, and Shlomo Dubnov. Hts- at: hierarchical token-semante audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 646-650. IEEE, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling cestruction-iinttrne language mod- els. arXiv preprint arXiv:2210. 141:1, 2022.
Colin Conwell and Tomer Ullman. Testing relational understanding in text-guided image generation. arXiv preprint arXiv:2208.000 2022.
Soham Deshmukh, Benjamin Elizalde, and Huaming Wang. Audio retrieval with wavtext5k and CLAP training. CoRR, abs/2209.14222 2022a.
Soham Deshmukh, Benjamin Elizalde, and Huaming Wang. Audio retrieval with wavtext5k and clap training. arXiv preprint arXiv:2209. 14275, 2022b.
Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks, 2023.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP). pp. 736-740. IEEE, 2020.
Yanai Elazar, Hongming Zhang, Yoav Goldberg, and Dan Roth. Back to square one: Artifact de- tection, training and commonsense disentanglement in the Winograd schema. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, November 2021.
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Eduardo Fonseca, Xavier Favory, Jordi Pons. Frederic Font, and Xavier Serra. FSD50K: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829 8528 2022.
Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. SyntaxGym: An online
platform for targeted evaluation of language models. In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 70-76, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.ce acl- demos. 10. URL
https: //acl antho log.lllo......o.. acl.e....lc---- del.a.deeo.....m...
10

---

Published as a conference paper at ICLR 2024
Jort F Gemmeke. Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference acoustics, speech and signal processing (ICASSP), pp. 776-780. IEFE, 2017.
Deepanway Ghosal, Navonil Majumder, Ambuj Mchrish, and Soujanya Poria. Text-to-audio gen- cration using instruction tuned Ilm and latent diffusion model. arXiv preprint arXiv:2304.13333 2023.
Sreyan Ghosh, Sonal Kumar, Chandra Kiran Reddy Evuru, Ramani Duraiswami, and Dinesh Manocha. Recap: Retrieval-augmenned audio captioning, 2023.
Sachin Goyal, Ananya Kumar, Sankalp Garg. Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2023.
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. Color- less green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long Papers), Pp. 1195-1205, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/11--11 1108. https::
//aclantho ogy. /wgg//88-111100
Shawn Hershey, Daniel PW Ellis, Eduardo Fonseca, Aren Jansen, Caroline Liu, R Channing Moore, and Manoj Plakal. The benefit of temporally-strong labels in audio event classification. In ICASSP 2021-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 366-370. IEEE, 2021.
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom em- beddings, convolutional neural networks and incremental parsing. To appear, 2017.
Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. A systematic assess- ment of syntactic generalization in neural language models. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pp. 1725-1744, Online, July 158.. URL
2020. Association for Computational Linguistics. doi:
Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. arXiv preprint arXiv:2301. /2661, 2023.
Kenan Jiang, Xuchai He, Ruize Xu, and Xin Eric Wang. Comclip: Training-free compositional image and text matching. arXiv preprint arXiv:2211. 13854, 2022.
Katharina Kann, Alex Warstadt, Adina Williams, and Samuel R. Bowman. Verb argument struc- ture alternations word and sentence embeddings. In Proceedings of the Society for Com- putation in Linguistics (SCiL) 2019, pp. 287-297, 2019. doi: 10.7275445ss-yy86 URL
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Papers), pp. 119-132, 2019.
Hector Levesque, Ernest Davis, and Leora Morgenstem. The winograd schema challenge. Thir- teenth international conference the principles of knowledge representation and reasoning, 2012.
Tal Linzen, Emmanuel Dupoux. and Yoav Goldberg. Assessing the ability of LSTMs to learn syntax- sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521 535, 2016. doi: 10.1 1621tall 00115. URL https: ///a:://c/ http:://eee.aaaaaoooooror///////

---

Published as a conference paper at ICLR 2024
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.125000 2023a.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei. Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. In An- dreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 21450-21474. PMLR, 23-29 Jul 2023b.
Xubo Liu, Qiuqiang Kong. Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang, Mark D. Plumbley, and Wenwu Wang. Separate anything you describe, 2023c.
Xubo Liu, Zhongkai Zhu, Haohe Liu, Yi Yuan, Meng Cui, Qiushi Huang, Jinhua Liang, Yin Cao, Qiugiang Kong, Mark D Plumbley, al. Wavjourney: Compositional audio creation with large language models. arXiv preprint arXiv:2307. 14335, 2023d.
Vincent Lostanlen, Carmine-Emann Emanuele Cella, Rachel Bittner, and Slim Essid. Medley-solos-De DB: cross-collection dataset for musical instrument recognition, September 2019. URL https:
Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna.
Crepe: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10910-10921, 2023.
Xinhao Mei, Xubo Liu, Jianyuan Sun, Mark D Plumbley, and Wenwu Wang. On metric learning for audio-text cross- modal retrieval. arXiv preprint arXiv:2203.15537, 2022.
Ron Mokady, Amir Hertz, and Amit Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:21 11.1111..0974... 2021.
Irene Martin Morato and Annamaria Mesaros. Macs multi-annotator captioned soundscapes, July 2021. URLLsttps://dbl.orggg/// wtt../5.212222a2.ee 5114771
Andreea-Maris Oncescu, A Koepke, Joao F Henriques, Zeynep Akata, and Samuel Albanie. Audio retrieval with natural language queries. arXiv preprint 1rXiv:2105.02/99 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint /00.:v3/0 2018
OpenAl. Gpt-4 technical report, 2023.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306. 0166, 2023.
Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Anmual ACM Conference Multimedia, pp. 1018-1018. ACM Press. ISBN 978- 1-7---4-5033359 4. doi: 10.1 0..114//27333332886399 URI. http:: //d1: acm, o www.cr///iiattte cttatittaitoooooo on. wtt.aatiiio....oo.ooooomom... cfw.coiimiooo.oo 2733373 2806390
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
8748-8763. PMLR, 2021.
Nils Reimers.
Sentence transformers.
htttps://iithb. http:////gttuuub.o//te....
sentence tti-ttraaaa trans 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
12

---

Published as a conference paper at ICLR 2024
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. arXiv preprint arxiv:184.09330/ 2018.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM International Conference Multimedia, MM '14, pp. 1041-1044, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 9781450330633. doi: 10.1145/2647868.26665 2647868 2655045
URL hitps: /ttp://///o..rr/11...22.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2 1.02/ 14, 2021.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2888-20 2913, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi:
URL
https: //acl antho www.oogg..oo..ooo...............
wwo.or///ooo 2021. emn emmlp-ma p- in. 230
Sonniss Limited. gameaudi September 2023.
Sonniss Game Audio, 2022.
URL https://sonti: com/
Registered England, UK. Company number: 09377364. Accessed: 25
Tristan Thrush, Ethan Wilcox, and Roger Levy. Investigating novel verb learning in BERT: Se-
lectional preference classes and alternation-based syntactic generalization, In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 265- 275, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/22 1/0200. blackboxnlp-1.22 URL httpss:://acaaaaholloooooooo wrg/ htt.://www.....oo..o.oo////aaaoa 1.22
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 5238 5248, 2022.
Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Learning from between-class examples for deep sound recognition. In International Conference on Learning Representations, 2018. URL
net wtt.:e/wwwwweeeeeeeeeoorruddddaa
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.009888 2023.
George Tzanetakis, Georg Essl, and Perry Cook. Automatic musical genre classification of audio signals, 2001. URL http::/ simiiiiii r2001. smi wwtp:ettt/dffee http://ttettpptttttttetteee
pdd
Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia. Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, and Samuel R. Bowman. Investigating bert's knowledge of language: Five analysis methods with npis. CoRR, abs/1909.02500 2019. URL
1909.02533
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The benchmark of linguistic minimal pairs for English. Trans- actions of the Association for Computational Linguistics, 8:377-392, 2020. doi: 10.1162/
tac3__000321
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi. Ali Farhadi. Hongseok Namkoong, et al. Robust fine-tuning of shoo models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7959-7971, 2022.
13

---

Published as a conference paper at ICLR 2024
Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. 2022.
Ho-Hsiang Wu, Oriol Nieto, Juan Pablo Bello, and Justin Salomon. Audio-text models do not yet leverage natural language. In ICASSP 2023-2023 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui Taylor Berg- wig-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword- to-caption augmentation. In JEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2023. URL https: http:////peeee...eeem/o
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech, 2019.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint 06876, 2018.
A COMPOSITIONALITY
Compositionality refers to the property that the meaning of a complex expression is determined by the meanings of its constituent parts and the rules used to combine them. In the context of audio, compositionality can manifest in various ways. Below are different types of compositionalities observed in audio:
1. Temporal Compositionality: Temporal compositionality refers to the organization and combination of audio elements over time. It involves how individual sounds or musical notes are arranged and structured to form cohesive and meaningful audio sequence. Ex- amples include rhythms, melodies, and the timing of sound events within a composition.
2. A Attribute Compositionality: Attribute compositionality in audio refers to the manner in which different perceptual or structural attributes of sound elements combine or interact to create complex auditory experience.
3. Spectral Compositionallity Spectral compositionality pertains the combination of fre- quency components within an audio signal. It involves the arrangement of different fre- quencies and how they interact to create the overall timbre or color of the sound. In music, this can relate to the orchestration of different instruments or the layering of various sonic elements.
4. Harmonic Compositionality: Harmonic compositionality focuses on the relationships be- tween different pitches or tones an audio composition. involves the creation of har- monies, chords, and the interplay between different musical notes to produce a harmoni- cally rich and structured auditory experience.
5. Spatial Compositionality: Spatial compositionality refers to the arrangement and organi- zation of sound sources in space. In the context of audio production immersive audio experiences, involves how sounds are positioned, panned, or spatially distributed to create sense of depth, width, and localization within the auditory space.
14

---

Published as a conference paper at ICLR 2024
6. Timbral Compositionality: Timbral compositionality relates to the manipulation and combination of different timbres or sonic qualities. It involves how variations in tone color, texture, and sound processing contribute the overall expressive and aesthetic qualities the audio. Timbral compositionality is often prominent electronic music and sound design.
7. Semantic Compositionality: Semantic compositionality extends beyond the perceptual aspects of sound and encompasses the symbolic or semantic meaning associated with audio elements. involves how sounds convey meaning, emotions, or convey specific messages within a larger audio context.
B ADDITIONAL DETAILS
B. COMPA BENCIMARK
Table 3: Left: Frequency of different kinds of sounds in the CompA-order benchmarks. The grouping was done manually by the human annotators (top). Right: Frequency of unique parts-of-speech audio cap- tions.
Deseription
Comple
@Compt-sttrilttte
Human Speech
106
Rackgrwind Sound
10
Land Animal Sounds
144
Backgrond Sound Distinct
142
CompA-Order
CompA-Allribule
Homan Sound
168
bmm Simn
Unique Nouns
795
296
Hewsshald
Seands
116
Unique Verbs
250
163
Machanica Flectiaal Sonnks
Unique Prepositions
21
27
Soand
Unique Adjectives
Equipencni Sound
Sound
Sound
and Telephone Sound
Constraction
Table (left) shows the frequency distribution of various kinds of sounds among all instances in the CompA-order and CompA-attribute benchmarks. Both benchmarks benefit from acoustic diversity. For CompA-order, the highest occurrence includes human speech and background sounds, and the lowest occurrence of wind and construction sounds. For CompA-attribute, the highest occurrence includes human speech and human sound, and the lowest occurrence mobile and telephone sounds and construction sounds. Human Speech consists of audios of female speech, woman speaking, child speech, etc, Background Sound consists of audios breaking, rumbling, etc, Bird Sound consists of audios of chirping, quacking, etc, Land Animal Sound consists of audios of roar moo, etc, Background Sound Distinct consists of audios of revving, explosion, etc, Human Sound consists of audios of sneezing. cheering, etc, Alarm and Siren consists of audios of steam Whistle, alarm clock, etc, Household and Indoor Sound consists of audios of chopping, scraping, etc, Mechanical and Electrical Sound consists of audios of rypewriter, clicking, etc, Bell Sound consists of audios of door Bell, church bell, etc, Tools and Equipment Sound consists of audios of gunshot, electric razor, etc, Vehicle Sound consists of audios of train horn, motorcycle, etc, Water Sound consists of audios of rain, waves, etc, Mobile and Telephone Sound consists of audios of telephone ringing, dial tone, etc, Wind Sound consists audios of wind chime, wind howl, etc, Construction Sound consists of audios of drilling, hammering, etc.
Tab (right) shows the frequency of various parts speech in CompA-order and CompA-attribute. Both benchmarks benefit from rich linguistic diversity. This emphasizes more robust evaluation of ALMs with our benchmark.
Table provides examples from both benchmarks.
B.2 COMPA-661K
Table lists all sources from which CompA-661k is pooled. We propose several changes over AION-audio-6633
15

---

Published as a conference paper at ICLR 2024
Table 4: List of sources from where CompA-661K is pooled.
Dataset
#Sents
MACS
Morato & Mesaros 2021
14400
ESC-50
2000
SONISS Nomnis Limiter
2022
1631
Musical Instrmmett Aggotnielr
2023
9774
SoundBible Eoo 2123
1232
LibriTIS Zor 2019
93772
Free Sound
202
259020
Medley-solos [Loseisen cl 2019
732
WavText5K [Dssmmoon
4347
MusicCaps Aoverriee
2645
GTZAN TTanoaaki 21 2001
6014
FSD55K
40966
BBC (BBC
31201
Urbansourkk (Salamoo al. 2014
17410
CompA-AudioSSe
108311
AudioCaps Kim al. 2019
48649
Clotho Draassos
19195
Table 5: List of various datasets used Pre-training and Evaluation
Dataset
Source
Experiment
Pre-training
CompA-661K
refer to Table
Vanilla Contrastive Learning
CompA-AudioSet
AudioSet Genineee et al. 2017
Contrastive Learning with Hard Negutives
Modular Contrastive Dataset
AudioSet Strong Hershey et 2021
Modular Contrustive Learning
Evaluation
Compe-order
AudioSer Gemmeke et ml.
2017
Evaluation Dataset
CorpA-attribute
AudinSet Gemmeke
Evaluation Dataset
WavJourney Liu et al.
Table 6: Examples from the CompA-orden (left) and CompA- (right) benchmark.
Description
CompA order Captions
Description
CompA-attrbuee Captions
Captioa
The growl of tiges succooded by huuns comvrnation.
Caytion
A croud speaks and mas cheers.
Rev, Caption
Humin comvenation exccoeded by the groa of tiger.
Rev. Capioo
A croud cheers and man speaks.
Triplet
The growl of figer amide human conversttion
Caytion
As a woman whislles, a young kid mukes doclaration and
Capiion
harking dog preceding man's comersation.
pensa speaks and young male shouts out excitelly
Rev. Caption
man's convenatioo preceding barking dog.
Rev. Capiio
As a woman speaks. a young hid make a deciamiion and
Triples
mai comeesing with barking dog the background.
penon whistles and a young male shouts our excitedly
Capioo
howling dog, soen after joincl by speaking woman.
Caption
Police noise blares amid general town sizen
Rev. Caption
spenking woman, soon after joinel by howling dog
Rev. Caption
Police sires blares unid general town noise
Triplet
huw intermised with speaking woman.
Caption
Chirping with light wind and distent motiling
Caption
bleating followed by people speaking.
Rev. Caption
Reatling with light
and diston chirping
People speaking.
bleating goat.
Caption
Light music buzzing followed
elecuiiitty
Triplet
People speaking interminllad
bleating gost
hrietly playing as
on surface.
Captios
Caption
telephane chime
occurring poior telephone
Rev. Capiion
cloctiiity brislly buzzing as
Light music playing followed by suufice
Triplet
conversation interleaved
telephone
Caption
music bux
and plays
Caption
speaking peior usage of register.
Rev. Caption
music box plays tune
Cuption
register used prict to speaking.
Caption
over radio
crying
Triplet
mas spesking while cash registe being used.
Caption
crying
man tilknng
16

---

Published as a conference paper at ICLR 2024
land artimal sounds human spocch
Reverse
Mechinlema must repeter as whrniing and
sek linel tolooe ersoing sound
bo cuturlyy futowed lagu
Soleet Dewcttetee
Time
Vide (siech VM
2403
1734
C
1300
2000
2
201
Made
4383
5201
1449
Figure 7: Snapshot of the illustration tool used for annotating CompA-orden
B.3 ANNOTATION
Fig. iillustrates a snapshot of the annotation tool used for CompA-order. The audio and the corre- sponding timestamps are obtained from AudioSet strong subset (Hershey al., (02I]. We first per- form pre-processing to bring down the search space of paired audios. Details about pre-processing can be found later in the Section. Next, two expert annotators find all possible pairs of audio where two acoustic events occur in the opposite order using annotation tool illustrated in Fig. 7 Natu- ral interventions or overlayed with natural disturbances were allowed. Additionally, each annotator writes creative captions for the identified audio segment. Captions for both audios instructed to have the same words but different word ordering to account for the compositional change. Post the annotators found 400 such pairs of audio-caption pairs, they were instructed to find a triplet audio segment for any 100 such pairs from a pool of available audio segments, where the audio events in the pair occurred simultaneously with each other.
For CompA-attributte the entire process was done manually. In the first stage, only the audios where an attribute swap resulted in genuine real-worke audio were selected. Then, for each audio, an audio with swapped attributes was found manually. Finally, if no such audio was found, Wavjourney (Liu jt al. 2023d) was employed to generate the audio. Post this step, a manual inspection was done to validate the clarity of the events in the audio.
Annotator Demographies. Our team of annotators comprises highly skilled domain experts in au- dio, each bringing a wealth of research experience. They possess a deep understanding of sound analysis and are adept at discerning intricate details audio recordings. Their expertise is not just technical but also theoretical, allowing them to approach the annotation process with nuanced per- spective. This background enables them to handle complex audio data with precision and insight, ensuring high-quality annotations that are both accurate and meaningful. Their combined experi- ence in audio research is a valuable asset to our project, contributing significantly to the depth and reliability of our annotated audio corpus.
Annotator Process. Out of the total 600 pairs in both benchmarks, all four annotators annotated 150 pairs each (100 and 50 for CompA-order and CompA-attribute, respectively). After the initial
17

---

Published as a conference paper at ICLR 2024
round of annotations, each annotator reviewed and annotated set previously handled by fellow annotator in second round (assigned randomly).
Annotation Rules.
1. Primary Event Focus: Annotators must prioritize the identification and labeling of the two main events in each audio. While acknowledging other background sounds, the focus should be on clearly distinguishing and documenting these primary events.
2. Background Sound Limitation: Background or incidental sounds can be present, but they should not overpower or obscure the primary events. Annotators should ensure that these main events remain distinct and easily identifiable despite the presence of additional sounds.
3. Consistent Event Ordering: Each audio contains the same two events, but reverse or- der. Annotators should carefully document the sequence of these events, ensuring accuracy in reflecting their reversed order across paired audios.
4. Clarity and Distinctiveness: In instances where natural sounds or other auditory elements are present, annotators should verify that these do not interfere with the clarity and dis- tinctiveness of the main events. The two primary events should be identifiable without ambiguity.
5. Contextual Relevance: Any additional sounds included in the recordings should be con- textually relevant and not misleading. These sounds should support the primary events' setting or scenario without leading to confusion about the main events themselves.
Annotation Pre- Pre-processing, To decrease the search space of finding pairs of audios with the desired acoustic events and opposite order of occurrence, we filter the search space using rules and logic. We provide the code in our Supplementary.
B.4 COMPARISON OF VARIOUS TECHNIQUES FOR NEGATIVE MINING
Table 7 compares various LLMs and traditional NLP techniques for mining compositionally aware hard negatives for contrastive training. While LLaMa (Touvron al., 2023) and LLaMa-2 (Tou- vron et al. 2023) are competitive in quality to GPT-4, Falcon (Penedo et al. 2023) almost always produces the same output as the input caption. On the other hand, traditional NLP techniques using spacy (Honnibal & Montani 2017) almost always generate sentences that are not linguistically and
semantically viable.
B.5 PROMPTS
We use prompts generate the negatives of given caption fist to generate the negatives by changing the temporal order of events in the caption, and second to generate the negatives by swapping the verbs associated with the nouns until the the sentence made sense after swapping. The prompt used for both are listed below respectively:
1. will provide you with a list. Each item in the list is a caption of an audio file. The cap- tion may have temporal sequence of multiple audio events taking place. The temporal sequence of events can be of any order, which may describe sounds occurring together, followed by each other, etc. You have return 3 modified captions that change the audio composition by changing the temporal order of events through modifying phrases in the caption. Except the words that contribute the ordering of the events, the wording of the actual events should be kept as unchanged as possible. You are only allowed to make minor modifications to words that not contribute the temporal sequence, so that the mod- ified caption is linguistically correct. Return only a list for each caption in the input list, with compositionaly modified caption. Here are some examples: Original Caption: "A man speaks twice before an electric toothbrush starts, followed by clicking sounds.', Modified Captions: ['Clicking sounds occur after a man speaks twice, followed by the start of an electric toothbrush.'. 'A man speaks twice with an electric toothbrush starting and click- ing sounds in background', 'A man speaks twice after an electric toothbrush starts amidst clicking sounds in background", "A man speaks twice clicking sounds occur, followed
18

---

Published as a conference paper at ICLR 2024
Table 7: Comparison different LLMs and Traditional NLP technique to generate negative of the given Original caption
Model
Caption
Original
Somcone sncezes, crowd laughs
GPT-4
Someone laughs, a crowd sneezes creaming is heard, followed by a large gun being fired. A crowd laughs, followed by someone sneezing. Someone laughs, crowd sneezes
LLLMMa
LLaMa-2
Falcon
Traditional NLP
crowd sneezes, Someone laughs
Original
An infant crying as man is talking followed by a toy train horn honking An infant talking as man crying followed by a toy train horn honking A man talking and an infant cries, amidst the sound of toy train horn honking.
GPT-4
LLaMa
LLaMa-2
A man is talking and infant cries, amidst the sound of toy train horn honking.
Falcon
An infant talking a man is crying followed by a toy train horn honking An infant train as crying is talking followed by honking man toy horn
Traditional NLP
Original
A woman speaks and then a kid yells loudly as soft voices speak softly in the background
GPT-4
A woman yells and then a kid speaks loudly as soft voices speak softly in the background
L.LaMa
Soft voices speak softly in the background, then a woman speaks, followed by kid yelling loudly.
LLaMa-2
Soft voices speak softly in the background. then a woman speaks, followed by kid yelling loudly.
Falcon
A woman speaks and then a kid yells loudly as soft voices speak softly in the background
Traditional NLP
A background speaks and then voices yells loudly as soft woman speak softly in the kid
Original
Wood thumping then clanking followed by metal snupping
GPT-4
The sound of metal snapping after wood thumping and clanking
LLaMa
Metal snapping is heard, followed by wood thumping and clanking.
LLaMa-2
Metal snapping is heard, followed by wood thumping and clanking.
Falcon
The sound of metal snapping after wood thumping and clanking
Traditional NLP
thumping Wood then clanking followed by metal snapping
Original
A large gun is fired then several heeps goes off and a woman speaks A large beep is fired then several guns go off and woman speaks
GPT-4
LLaMa
Screaming heard, followed by large gun being fired.
LLaMa-2
Screaming is heard, followed by large gun being fired.
Falcon
A large gun is fired then several beeps goes off and a woman speaks
Traditional NLP
A several gun is fired then large beeps goes off and a woman speaks
by the start of an electric toothbrush'' 'An electric toothbrush starts, followed by a man speaking twice, and then clicking sounds.']: If you see, all the modified captions change the temporal order of the 3 events in the original caption. Here is another example: Original Caption: "A man speaks, followed by whack and whoosh sound effect, then cheering and shouting from a crowd', Modified Captions: ['A man speaks amidst a whack and whoosh sound effect and cheering and shouting from a crowd', 'A whack and whoosh sound effect is heard, followed by a man speaking, then cheering and shouting from a crowd.', 'A man speaks while whack and whoosh sound effect heard, followed by cheering and shout- ing from a crowd.'" 'A whack and whoosh sound effect, along with cheering and shouting from a crowd, is heard before a man speaks.", A man speaks and then there is cheering and shouting from crowd, followed by whack and whoosh sound effeett']. Similar to the previous example, the temporal ordering of all events in the original caption were changed in the modified captions. For some captions, you will have modifications, like 'Birds and fowl are heard in the background.' because they have just one event. So in this case just return empty list. Please don't 't return captions that translate to the same audio com- position. Please only return a json with these lists of 3 modified captions and nothing else. Here is the list of captions:
2. I will provide you with a list. Each item in the list is a caption of an audio file. The caption may have multiple noun-verb combinations, i.e., a noun performing a verb. You need to swap the verbs and associate them with different noun until the swap and the resulting sentence make sense. For example: you cannot swap an animal sound with a noun describing human. Here is an example: Original Caption: "A woman speaks with a baby crying in the background as occasional breathing is heard.' Modified captions: I'A woman crying with baby speaking in the background as occasional breathing is heard.']. Only one swap was possible with this example. Original Caption: 'A baby cries while male singing and music play in the background.' Modified captions: ['A baby sings while
19

---

Published as a conference paper at ICLR 2024
male cries and music play in the background."] ; Original Caption: 'Human sounds and environmental noise interrupted by occasional taps.' Modified captions: ['Human noise and environmental sounds interrupted by occasional taps.'.. J. If you think more than one swap is possible, return multiple. For some captions, no swap will be possible. This includes captions with just one noun-verb pair or where swaps don't make sense. Original Caption: "A man speaks, and a dog barks in background noise followed by shout.' Modified captions: []: Original Caption: "A cat meows as music plays and female sings.' Modified captions: []: Original Caption: 'A baby cries.' Modified captions: []: Original Caption: 'A woman talks nearby as water pours.' Modified captions: []. In all these cases, swapping the noun and verbs do not make sense, for example water cant talk and women cant pour. In such cases, return an empty list. Please only return a JSON with these lists for each caption in the list given to you and nothing else. Here is the list of captions:
We generate AudioSet-CompA by prompting GPT-4 for time-aligned compositional captions for audios in the AudioSet strong dataset. The AudioSet strong dataset has time-aligned labels (the start and the end time for each event, c.g.) for each acoustic event in an audio. Thus, we prompt GPT-4 with these time-aligned labels and ask it generate a coherent caption from this, and indeed, GPT- performs accurately at this task. The prompt used for this task is listed below:
1. will give you list of of lists. Each list in the list describes 10 second audio file in the form of multiple tuples. Each tuple describes a sound event with their starting and ending times in the audio. For example, '(Wind-0.0-10.0) signifies that Wind was heard through the 10 second audio and (TTick6.899-7.011 signifies a sound of clock tick was heard from 6.899 to 7.01 span of the 10s audio. The events in the list are tem- porally aligned. You need to create one liner caption out of this list of multiple tu- ples that 1) Follows the temporal sequence of events 2) The caption in detail describes the compositionality of the audio, i.e. for eg.., what occurs before what and with what and 3) Make sure you are using grammatical subject- subject-ver-oojeet sentences. Directly de- scribe the sounds and avoid using the word "heard". You should also use your reason- ing skills to not blindly follow the temporal events in the list but output a caption of a 10 second audio that explains plausible real life 10 second event for which you may also ignore some events to achieve. Here are some examples: Input: ['(Laughter-4.88 (rfck-8.603-8.694))
6. 6.144), '(Female speech, woman speaking- (.t.--.7..66.555) ground 0.00se-0.0010.0)) "(Speech-1.366 1.35--3.56) "BBreathing-A 8.8.8899.8877777 (11...6687)"" A fe- male laughter is heard followed her talking over a constant background noise in the back- ground. Input: ['(Female speech, woman speaking-2.734-3.775 2.73-3.78373 (Music-0.0-8.350) (0cean-44373-9) 9,399', (Rain-0.0-4.086)" its raininng and music is playing in the background. Input: ('(Female speech, woman speaking-7.354-99 ..054-9..39)' "(Dishes, pots, and pans- 9.0699990-10.0);. (Meecanisms--000-00) "Generic impact sounds-9.55 99551-9.8443) Output: Generic mechanism sounds are heard throughout while later on a female starts speaking followed by dishes and pots clanging. Input: 11Appllause000-6.70
'(Back-
Output: woman is speaking while while
'(Male speech, man speaking-3.692 4.481)', "(Male speech, man speaking-5.091. 5.091-5.781)) '(Male speech, man speaking-6. 6.778 7.601), (Tick-9.25-934471 Output: A man is heard talking while the crowd is applauding throughout, and following this towards the end a ticking sound is observed. Just return a single list of one liner captions that are (less than 30 words). The list format should be ['caption for audio 1', caption for audio 2' 2'.... Here is the list of items:
(Crowd-000-10.0))
We used the following prompt generate scenes for modular contrastive learning using GPT-4:
1. will provide you with a list of unique sound events. have taken these events from labels in audio dataset. Generate acoustic scenes that are highly plausible in the real-world. Here is the list of events:
B.6 MODULAR CONTRASTIVE LEARNING
B.6.1 TEMPLATE-BASED SYNTHETIC CREATION OF AUDIO-CAPTION PAIRS
Algorithm | demonstrates the algorithm for our template-bart based synthetic audio-caption creation process. For concatenating, we use the append(.) function in Soundfile library with crossfade
20

---

Published as a conference paper at ICLR 2024
Algorithm Template Based Audio- Audio-Caption Creation
Data: Acoustic Events Dataset {A (Audio), (Lobel)): // Generate List of Possible Acoustic Scenes
= (roompt L)
// Generate Compositions tional Audio and rine-grained Positive and Negative
Captions
initialize(A, T (
for iori=1la |8| do
initialize t Tpliat T for j = [6, do if ViiAAcoustieBven(E,, then (A*")t) append[getAAuioo(., ".ppen")))pp((....... append(&s.) end
// Generate fine-grained positive and negatives by changing the order or operation(t/+) else iisspperatio((E.) then
01. a2 (A4 pop(),
for k= =o
(T "ppprdlcouutteeeee(("""""M"A,a," (Ta appardsssspreerrhhchhgggggnnnnnn (T [ E(g-1)
end
// Generate Composi tioal Audio isEquals(" " then (A"")) a (concatenateAAudiiaa....
if isEqulll))
else (E, isEquals( " *) then (A append Avdoola end end
(A), uppend((A ((*"), pop())
1/ Template based conversion to captions (T000) aappnn((onverr//aaaptonn
) appeealcoooeerrttttttiiiia
end end
of 10% of the duration of the shorter audio file. For overlay, we employ the algorithm proposed by Tokozume et al. 2018) mix two sounds. Additionally, Table shows examples of positives and negatives generated for our modular constrastive learning approach, and Table shows examples of acoustic scenes output by GPT to create the final audio.
Table Template- based fine- grained positive and negative captions for various acoustic scenes,
Harmer and ouenn Liszee
Elaminer and Lraatt and this sagaz
engane then
Hurrmer Jet engine and
Unaile and thn
Hamter and thes lei ctaine
Austuze zan then pncohmmentt auabily EApEr cneckin
Vraman and than Macina anndi Engre krccing
Engine troding
Maclins and thes Jncokannne
Maciiss ansce hakhaama
Machiss pas undda by larsse lookin
baliammee and lasa Madias son
Wilgpeiing curboraa tomale eniing coocede semee spoct surron quaking
lomal oiging mostpal by hemolo spach wimes geating
Wis Whsgerins exciloed Fanale snging sacceede by Fersie speocs. geaking
Wiigening succosded by Fonule spoods spealins saccolol by Senie boon
Whoppiia (Renale sinpig Fetale
Whiperins exciloyd by Ponak aeging ssscedde by Seeis boxsi
speads vounn gesking Soak tosns)
Whiperiea exellored
semas speakin vinceddeily by Scuis bexts
wood, vemas speakins aaccedded by iisiss
Fonale sprck spokkng vaoceddellty Sontc booss
Wligerins escllyad by Fanale aaging warlyyol Fanls
soech venuo speking
21

---

Published as a conference paper at ICLR 2024
Table 9: Examples acoustic scenes created for various numbers of audio snippets for final synthetic audio creation our proposed template-based synthetic audio-caption creation process.
Number of Acoustic Events
Acoustic Scenes
Vehicle Hammer
Female speech, woman speaking Tap dance
Crow * Rain
Mochanical bell Engine starting
Bird * Whimper (dog)
Crow * Roaring cats (lions, tigers)
Doorbell Engine
Bell Engine starting
Train whistle Car
Buzzer Drawer open or close
Keys jangling (Howl (wind) Male singing)
(Tap dunce Applause) Female speech, woman speaking
Drill (Computer keyboard Emergency vehicle) Chainsaw (Power windows, electric windows Train)
(Light engine (high frequeney) + Train horn) * Children shouting (Engine starting Bell) Human group actions
3
(Propeller, airscrew + Siren) * Booing
Train whistle Diul tone Children shouting
(Truck- Fusillade) Howl (wind)
(Cart Rumblee * Wind noise (microphone)
Coo (Background noise Fly. housefly Speech synthesizer)
Bird vocalization, bird call, bird song (Stream, river + Roaring cats (lions, tigers) Speech)
Chicken, rooster (Environmental noise + Mosquito + Child speech, kid speuking) (Train Drill) * (Booing Wind) Booing * (Child speech, kid speaking + Male singing Environmental noise)
B.7 HYPER-PARAMETER SETTINGS
Vanilla CLAP Training. For vanilla contrastive pre-training with CompA-661k, we use batch size of 24, and Adam optimizer with a learning rate of Ic-4, and warm-up of 3200 steps, and train for 45 epochs. For training with compositionally aware hard negatives, we start with vanilla CLAP weights and train for 20 epochs with no warm-up. We follow a similar setup for modular contrastive training.
B.8 BASELINES
For retrieval- based evaluation (text-to-audio and audio-to-text), we compare CompA-CLAP with six baselines:
MMT (Oncescu al. 2021) introduces the task of retrieving audio using free-form natural language queries. The authors argue that this is a more intuitive and flexible way to search for audio than traditional methods, which rely text annotations, and also demonstrate the benefits of pre-training on diverse audio tasks.
ML-ACT (Mei al., 2022) studies the impact of different metric learning objectives on the audio- text retrieval task and linds that NT-Xent loss is a promising approach that achieves stable per- formance across different datasets and training settings, and outper-forms the popular triplet-based losses. Metric learning objectives are commonly used to train cross-modal retrieval models by map- ping data to an embedding space where similar data are close together and dissimilar data are far apart.
CLAP (Deshmukh et al. 2022a) introduces a framework for audio retrieval that uses a contrastive learning objective and two audio encoders to connect language and audio content.
CLAP-LAION (Wu* et al. 2023) proposes a pipeline for contrastive language-audio pre-training to develop an audio representation by combining audio data with natural language descriptions. They construct a contrastive language-audio pre-training model by considering different audio encoders and text encoders. They incorporate the feature fusion mechanism and wowoodtto-aaptinn augmen- tation into the model design to further enable the model to process audio inputs of variable lengths and enhance performance.
22

---

Published as a conference paper at ICLR 2024
Table 10: Performance comparison CompA-CLAP with baselines on Real- seal-world Synthetic distribution CompA-attribnn benchmark.
CompA-attribute
Model
Text
Audio
Group
Human
80.30 84.10
82.40 / 86.20
79.80 / 83.50 16.67 / 15.33
Random
21.0/210
25.0 22.0
CLAP
33.27 /37.21
6.14 8.12
4.66 6.23
CLAP-LAION
34.78 /38.43
6.52 8.94
5.07 / 7.82
CompA- CLAP (ours)
44.2820.02 46.3420.05
22.5220.04 20.00
15.13 27.110
Wav2CLIP (Wu al. 2022 proposes pre-trained audio representation learning method that dis- tills knowledge from Contrastive Language-Imaa Pre-training (CLIP). Wav2CLIP projects audio into shared embedding space with images and text, which enables multimodal applications such as zero-shot classification and cross- moddl retrieval.
AudioClip (Wu al. 2022) is an extension of the CLIP model that can handle audio addition to text and images by incorporating the ES audio-model in the CLIP framework. It was trained on the AudioSet dataset, which contains millions of audio clips with corresponding labels.
C ADDITIONAL RESULTS
C.1 SELECTING K
CompA-antie
Completttibute
Compaeander
CompA-alt bute
Naclin Negitios Foatta dl
Marlis fie hapaova Postuss
Compostiorallyfwwee Hard Negatives
Modular Contrastive Leaming
Figure 8: Ablation on varying K on CompA-order/attriuut benchmark
Figure 8 provides insight regarding how to select K which is the number of hard negatives for Compositional Hard Negative training, and also the max limit for selecting generated fine- grained positives as well as negatives for Modular Contrastive Learning.
C.2 MEASURING QUALITY OF NEGATIVE SAMPLES
In this section, we measure the quality of negative samples generated in AudioSet-CompA. For each caption f4 E T, we randomly sample 1000 captions f, from the same dataset, which act as negatives. To compare the quality of the generated negatives hhard vs !e we encode these captions with sentence transformer (Reimers) (223)) and calculate relative difference between the cosine similarity of sim(t,; chard) and sim((,, 4, neg which comes to be 70.7%. From this, conclude that the overall quality of generated hard negatives is superior to the negatives sampled from the dataset.
C.3 COMPARING SYNTHETIC Vs ORIGINAL COUNTERPARTS DURING EVALUATION
In this section, compare the performance of CompA-CLAP on the original subset Vs the syn- thetic subset of the CompA-Attribute benchmark. While the original subset, also known as real- world distribution, is derived from the AudioSet (Gemmeke et al. 2017), the synthetic distribution is prepared using WayJourney (Liu et al. 2023d) and constitutes about 25% of the CompA- benchmark. Table 10 shows the result across both the distribution. Compared to the baselines,
23

---

Published as a conference paper at ICLR 2024
Table 11: Performance CompA-CLL CLAP across various acoustic scenes CompA
Codpe-Order Order benchmark.
CompA-order Audio
Acoustic Scenes
Text
Group
Human Speech, Land Animal Sounds
71.22
68.41
63.80
Human Speech, Bell Sounds
77.11
72.53
68.
Human Speech, Human Sound
21.23
11.41
Tools and Equipment Sound, Construction Sound
20.12
10.42
8.10
Table 12: Analyzing CompA- CompA-CLAP Performance: Exploring variations across selected acoustic events based on it's frequency AudioSet (training set)
CompA-order Audio
Acoustic Scenes
Text
Group
Male speech, Generic Impact Sound
61.32
58.42
52.13
Woman speaking, Music
65.21
60.14
58.23
Video game sound, Plop
16.23
8.21
6.24
Eruption, Stream
12.32
7.13
5.21
observe consistent boost in the performance across both distributions in the CompA-attribute benchmark.
C.4 ERROR ANALYSIS
The analysis, shown in Table evaluates CompA-CLAP performance across combina- tions of acoustic events, considering acoustic similarities between audios. Notably, the model excels in scenes with acoustically dissimilar audios (e.g.. Human Speech and Land Animal Sounds), but its performance dips in scenes with similar audio types (e.g.. Human Speech and Human Sound).
Table 12 also highlights the long-tailed problem in AudioSet, where the CompA-CLAP underperforms in audio with acoustic events seen less during training, for instance Video game sound, Stream, Plop. Eruption, etc.
Table 13 illustrates the qualitative analysis, showcasing instances of significant improve- ment in CompA-CLAP compared to other baseline models within the acoustic composi- tions of the CompA-order and CotpA-attribute benchmarks.
Table 13: Some examples of acoustic compositions in the CompA-order and CompA-attribute benchmarks where we observed significant improvement in CompA-CLAP compared the other baselines
CompA-order (Acoustic Scene)
CompA-attribute (Acoustic Scene)
Frog croaking followed by music playing.
speaks woman panics
Man speaking before hell ringing.
A bohy laughs while woman make sounds
man speaking followed by gunshot
A sueezes and an adult lnughs
Tiger growling followed by people speaking.
Adult female speaking and young child crying
24

---

Published as a conference paper at ICLR 2024
Table 14: Examples of negatives generated for contrastive training with compositionally-awaal hard negatives. Sentences marked with gray show negatives where the order of occurrence of the acoustic events in the audio differs, and sentences marked with orange show negatives where the attribute between multiple acoustic events is switched.
Label
Caption
Original Caption:
Men speaking then sneezing.
Negative:
Men sneezing then spcaking.
Original Caption:
A child screams, an adult male is talking and vehicles are revving.
Negative:
Vchieles are revving preceding
screams and an adult male talking.
Negative:
adult male talks preceding vehicles and child
Negative:
child talks, adult male screaming and vehicles revving.
Original Caption:
crowd cheering and shouting. thumping occurs, adult female speaks, and adult male speaks.
Negative:
An adult female speaks following the checring and shouting crowd, followed by thumping and
adult male spcaking
Negative:
Thumping occurs, an adult male speaks, then a crowd is heard choering and shouting and
adult female spcaks.
Negative:
A crowd speaking and shouting. thumping occurs, adult female cheers, and adult male speaks
Original Caption:
High pitched drilling followed by rustling.
Negative:
Rustling sounds followed by high pitched drilling
Negative:
Rustling and high pitched drilling happening together
Negative:
High pitched rustling followed by drilling
Original Caption:
The entire time a loud static sound is joined with a constant clicking
Negative:
entire time
clicking accompanies loud sound
Negative:
Simultancously. loud static sound and constant clicking are heard The entire time a loud clicking is joined with a constant static sound
Negative:
def (sin no_positive_list, no_negative_list)
=**
Build Mask for generated positive and negative Captions
Args:
sim_shape (List): shape of Simi Ammilarity matrix
no_positive_list (torch.Tensor) Number of positive captions per audio.
no_negative_list (torch.Tense) Number negative captions per audio.
Returns:
mask_tenson (torch. Tensor)
13
14
forcate index_map and index_range to identify Ithe specific ranges for each texts generated cations (pos/neg) index_map I] no_gen_captiin torch.cat noe_psiitiv_liitt no_negative_listt
17
18
for i in range (1en (no_gen_capticn))
it not index_maa _app:
21
index map. append
append (no gen___e_______ gen_captionnnncceeeeeeeeeeeooooe [i]+1)
22
else:
23
index_map. append (index_map[i] + no_gen_caption [i] + 1)
index_range [[x for x, in zip (index_map, no_gen_caption);
26
lintialize the mask
27
= float (i-inf') + torch, ones (sim_shaape paptien_size = sim_shape[0]
30
for range (caption_size)
31
gen_index 0 #Get index w.r.t the generated pos/neg cations index = #Get index w.r.t the original captions
32
for k, (start_index, end_index) in enumerate (index_range)
25

---

Published as a conference paper at ICLR 2024
if start start_indes index <= and k end index: gen_index
37
batch_indox = start_index break
38
if == batch_index: paak_tensoor(,33
41
42
else:
43
mask_tensor i1 Igen index] inde_index] 1
return
Listing I: Source Code To Build Mask
26


Reviews:
[{"review_id": "hg271fpt48", "reviewer": "Reviewer_x6Uk", "summary": "Objects in an image have spatial relationships (which occur where in the image) where as events in an audio clip have temporal and attribute relationships (which sound occur when, who made which sound). The authors claim that the current audio-language models (ALMs) poorly model such temporal relationships. They propose a novel benchmark, \"CompA\" to test compositional reasoning (temporal as well as attribute relationships) abilities of ALMs. CompA is comprised of two parts. CompA-order to evaluate temporal relationship and CompA-attribute to evaluate attribute binding. Finally, authors introduce a model (CompA-CLAP) to improve ALM compositional reasoning abilities. CompA-CLAP is a fine-tuned CLAP model, where in the authors include compositionally aware hard-negatives for contrastive learning.These hard-negatives are derived using text-LLMs for the test set and template-based methods for the training set.", "questions": "* Have the author's considered rich transcription approach? Since authors are using strongly annotated data, can the model be trained to output a sequence like ..  <event1_start> <event2_start><event2_end><event1_end> .. here event2 occurs in the midst of event1. This would avoid the process of carefully designing negatives and the model would still learn temporal relationships among audio events. Attributes can also be modeled in a similar fashion. I would imagine this format could be more flexible in terms of attribute modeling. (e.g., personA-talking vs personA-singing and so on.. ). An LLM should have no problem taking such an output sequence and convert it to human-readable format (either implicitly or explicitly). \n* Can the author's clarify if the order of hard negative training and modular contrastive training matter. (section 4: 2nd paragraph).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The submission identifies a gap in current audio-language model evaluation i.e., temporal and attribute relationships of audio events, and proposed the novel CompA benchmark to evaluate the same. \n* Introduced a novel CompA-CLAP model which learns audio event relationships through contrastive learning and goal motivated negative sample design.\n* Well furnished details in appendices for reproducibility.", "weaknesses": "* The core technical contribution of submission boils down to - \"selecting appropriate negatives for the task at hand\", an established idea in contrastive learning, thus raising the question of novelty in terms of technical contribution.\n* The structuring of manuscript could be better. e.g.,\n  - Transition from section 3.3 to 3.4 is not clear in the first reading.\n  - Every section reads like a mini-paper with it's own background, methodology. May be add a paragraph at the beginning of section to prep the reader of upcoming sections and their differences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u3JlKBAktn", "reviewer": "Reviewer_jJm8", "summary": "This paper simultaneously propose new datasets (CompA) and a novel training method (CompA-CLAP) for the task of compositional reasoning in Audio Language Models (ALMs). The dataset contribution is separated between two subsets, CompA-order, which is targeted at the temporal ordering of audio events, and CompA-attribute, which targets attribute reasoning. Additionnally, the authors show that most ALM fail at grasping compositonal reasoning and propose large improvements on this problem with a new method, CompA-CLAP which both introduce a specific loss and a methodology for modular contrastive learning.\n\nOverall, the paper is very well written and the methodology is very soundly presented by both exhibiting a profound issue of ALM, proposing novel datasets and even a large improvement in terms of learning methodology. My major (and almost only) criticism comes from the fact that the concept of « compositionality » presented on this paper is rather largely focused on « temporal » compositionality (even in the attribute case), and the paper would gain from a broader presentation of different possible types of compositionality. However, I still strongly believe that this paper would be a good addition to ICLR and recommend for acceptance.", "questions": "With regards to generating examples, although I understand that using prompts and GPT can perform a large array of tasks, your temporal task could be quite easily simulated (collating examples together, such as done with the Mixup augmentations). Could you quantify gains and differences with those more straightforward approaches ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper is very rich in new proposals, but all of these remain soundly theoretically grounded. Also, the proposal of new highly curated datasets are also an always-welcome addition for the community.", "weaknesses": "As previously stated, my major (and almost only) criticism comes from the fact that the concept of « compositionality » presented on this paper is rather largely focused on « temporal » compositionality. Although I understand that it is mandatory to make choices as this is still a very young research topic, I think the paper would gain in strength if a more clear definition of various types of compositionality would be presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Caq3E3G9fQ", "reviewer": "Reviewer_Xast", "summary": "Summary: This paper presents a new benchmark, CompA, for measuring two forms of compositionality in audio-language models. These are ordering (sequencing of acoustic events within a clip) and attribute binding (determining which objects perform acoustic sources make which sounds in a clip). They also propose a novel pipeline to train a new contrastive model, CompA-CLAP, by augmenting existing datasets with (a) hard negatives in terms of both forms of compositionality (ordering and attribute binding) and (b) data augmentation by concatenating and overlaying audio clips with known labels, where all text is generated via LLM. Using their proposed benchmark, they show improvements from the novel data pipeline.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I am posting my full review here, as I believe it easier to discuss both strengths and weaknesses together.\n\n# Overall\n\nOverall, the paper is an ambitious effort to propose a new benchmark measuring a known limitation of existing audio-language models, and also to achieve improved performance on this benchmark. However, reading the paper, it also feels that important details are sometimes lacking both on the benchmarking and the modeling/empirical sides of the paper, in part due to this decision to cover so much ground. It does not appear that the missing information is available in the supplementary material either. The proposed method also performs below random baseline on 2 of the 6 proposed benchmark metrics, but this does not appear to be discussed in the main text. I recommend adding some important clarifying details about the benchmark, clarifying experimental decisions and details, and reducing the \"sales pitch\" framing of the paper. I am enthusiastic about the direction and would encourage the authors to continue to refine the benchmark and their work. However, I feel that major revisions to the paper, and perhaps the benchmark, are currently in order, and that it is probably not ready for acceptance in its current form.\n\n# Major Comments\n\n* The tone of the paper often leans too strongly toward \"sales pitch\" for my taste. I would recommend to tone down this language, and focus on making quantifiable and verifiable claims based on the data and methods in the paper. Just a sampling of the phrases the authors use to dismiss other work, minimize problems, or promote theirs:\n\n  - \"only a word changed\" (a few sentences after the claim \"the audios have the exact same words\")\n  - \"it is trivial for ALMs to perform well on these benchmarks\"\n  - \"CLAP undergoes only minor degradation in retrieval performance\"\n  - \"25% synthetic...but highly curated by expers\" (what is *highly* curated?)\n  - \"a trivial but ineffective solution\"\n  - \"a highly custom prompt\"\n  - \"Our version outperforms ... by significant margins\" (no measures of significance used)\n\n* Since the authors are not only introducing a novel model, but actually proposing their dataset to serve as a benchmark, more justification and detail are needed. For example:\n  - There is no mention in the paper of where the benchmark can be accessed, how it can be used by other works, what format it is provided in, etc. This is essential information for a public benchmark (it is ok to maintain anonymity by describing where the benchmark *will* be available upon acceptance, if the description is sufficiently clear).\n  - The two components of the benchmark consist of only 400 and 200 samples respectively. Is this sufficiently large for a reliable benchmark for the field? Even existing benchmarks in the audio-text domain consist of thousands of examples (AudioSet, AudioCaps, MusicCaps). If 200-400 samples is considered sufficient, please provide empirical evidence. For reference, a Clopper-Pearson confidence interval for even 400 samples would be quite wide: a width of roughly 0.1 for a 95% CI depending on the approximation used, when the success rate is 50%.\n  - It is critical that the paper provide more detail on the annotation process in the main text. The current version provide almost no information, except that four annotators participated. Please describe, for example: number of annotators *per example*; the exact annotation procedure performed. Rater agreement metrics (in supplementary) would be useful but are not required. More detail is also needd regarding the screenshot in Figure 6; it is not clear at all what task the raters are even performing. Much of Section A.3 is in the passive tense, so it is also hard to tell who performed what tasks (\"a manual inspection was done\",\"Wavjourney was employed\"). Many of these details should also appear in the main text.\n  - The abstract states that the benchmark consists of \"a majority of real-world samples\", and elsewhere it is mentioned that 90% of audio are from real-world samples. (1) Can you provide results separated by real-world vs. not real-world? (2) What is the nature of the \"non real-world\" data?\n  - The choice of metrics seems not entirely clear. It would be helpful to either (1) cite works which already use similar metrics, to motivate their used based on wide preexisting adoption in the ALM community, or (2) provide more detailed motivation for why the metrics are appropriate. For example, why is \"group score\" the conjunction of audio score and text score and not, e.g., their harmonic mean in the style of an F-score?\n  - There are no quantifiable measurements of incertainty or variability in the comparisons presented in the paper. The authors say that their results are averaged over 3 random seeds; can they provide the (variance, stddev) over these seeds? What about Clopper-Pearson or other appropriate confidence intervals for the metrics reported? As is, it is impossible to assess the significance of the differences between metrics in Table 2.\n\n* Some important experimental design decisions in the paper are not clearly motivated and not validated with empirical studies. This includes:\n  - Why did the authors only fine-tune CLAP (when they have already shown the CLAP weights to be ineffective) instead of training it from scratch, when the authors apparently have access to the full original pretraining datasets?\n  - Why do the authors fine-tune two separate times, instead of performing a single phase of joint fine-tuning?\n  - Why does the fine-tuning occur (hard negatives (3.3) --> synthetic pairs (3.4)) instead of the reverse ordering?\n\nAll of these decisions could be validated empirically, but I do not see these results in the paper.\n\n* Both examples in Figure 4 both seem to be flawed training example. (1- Left) The negatives both appear to be correct. All of these examples (the true caption, and the negatives) describe three sound events occurring simultaneously. (2 - right) \"Tiger roars amidst thunker\" and \"Tiger roar followed by\" cannot both be positive labels for the same sample - only one can be correct. Both of these issues seem to raise concerns about the quality of the data used here; particularly since the \"highly custom prompt\" described does not appear to be shared in the paper.\n\n* Having the \"CompA-CLAP\" row in Table 2 be bold is misleading. In particular: \"CLAP (ours)\" outperforms ComA-CPA in attribute audio score. Additionally, CompA-CLAP performs below random baseline on audio and group scores in CompA-attribute. This should be highlighted clearly in the paper and abstract (which currently says that \"CompA-CLAP significantly improves over all our baseline models on the CompA benchmark\" but does not mention its below-chance performance on 2 of the 6 benchmark task metrics).\n\n* The paper makes several additional changes to the CLAP model\n\n# Minor Comments\n\n\n* \"Sequence\" and \"attribute binding\" should be defined clearly and early in the paper (ideally before these phrases are used).\n\n* Caption in Figure 2 should describe the metrics being presented in the Figure clearly. The Figure is also missing clear y-axis labels. The phrase \"minor\" is not quantifiable. This figure could also be improved by (a) measures of variability such as error bars or Clopper-Pearson confidence intervals or (b) baselines to show that the CLAP models actually degrade to \"trivial\" performance (is R@1 of 0.42 on AudioCaps val, or R@1 of 0.35 on AudioCaps test, \"trivial\"?).\n\n* A lot could be done to make the paper more self-contained. As-is, it is missing a great deal of relevant background information, which is exacerbated by the deferral of the related work to Section 5.\n\n* \"to assure high quality, we don’t concatenate or overlay random acoustic events but ask an LLM to create unique audio scenes based on the available labels\" - please clarify both the overall procedure here, and how this assures high quality.\n\n* \"where augmenting LAION-audio-630K with 2 million audios from AudioSet improved performance on benchmarks only marginally.\" Please clarify what is meant in this sentence.\n\n* The \"experimental protocol\" sections in 3.2 and 2.4 here do not describe protocols; please clarify or remove these sections. I would recommend a single \"experimental protocol\" section, since these subsections only describe individual components of the overall experimental setup.\n\n# Typos etc.\n\n* Section 1 should reference that prior work is deferred to, and discussed in, Section 5.\n* \"attribute-binding\" vs \"attribute binding\" both used in the paper.\n* \"The group score evaluates of the model performed well for the evaluation instance in the benchmark\" - this is not a meaningful description.\n* 3.4: \"due the the inherenly complex and non-linear nature of audios in the wild\" - please clarify what is meant here, and how nonlinearity relates to the hardness of an example.\n\n# Edit after discussion\n\nI have increased my scores for soundness (2->3) and presentation (2->3) along with my overall rating (3->6) after the author responses, which made significant changes to the paper which both improved it and addressed several of my concerns.", "weaknesses": "See above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ka5yXGSqpl", "reviewer": "Reviewer_Cw3v", "summary": "The authors propose and curate a new benchmark for evaluating the compositional reasoning capability in audio-language models. And further show that to train with curated data and modular contrastive with hard positives and negatives help to improve the compositional reasoning capability.", "questions": "- Regarding the data curated for training, there are CompA-661k, and AudioSet-compA, another 110k audio-text pairs for complex compositional audios? How are these different subsets used in different training scenario? It might worth adding a table to show clearly what combination of datasets, training strategies, whether there is hard negative and modular contrastive utilized, and link them directly to your abbreviation in the result table (CLAP, CompA-CLAP, CLAP-CompA-661k, etc.) Currently it is not easy to associate from the narratives to exactly how each variation in the result tables comes from.\n- In Table 1, what is the difference between CLAP (ours) and CLAP-CompA-661k (ours)?\n- For the common mistakes mentioned in the discussion, is it possible to perform some error analysis? Or confusion matrices to show these more quantitatively?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The curation of CompA-order and CompA-attribute benchmarks are relevant and needed contributions to the community. This helps to provide another perspective to examine the properties and limitations of current SOTA methods for audio-language models.\n- To provide both quantitative analysis of the current limitations in 2.1 and qualitative analysis in the last paragraph of 3.1 both provide solid motivations of this work.", "weaknesses": "- It is a bit difficult to get an entire picture of number of different datasets curated and how each of them are curated, a lot of the information are buried in the appendix. Consider in the main paper including a table describing different datasets curated, how many of them for each subset used for training, and evaluation, their source, and methods (leverage ChatGPT, replace the sounding object, swap the orders, etc.), and then refer to the table in the narratives.\n- The data curation is one of the main contribution for this work, however currently a lot of details are left in the appendix. Consider integrating more information such as the source datasets, and some manipulated examples to the main narratives, this can help the reader to understand this work early on.\n- A minor suggestion: The figures and tables are far from the actual text in the appendix, consider reorganize them to make it easier to read.\n- A minor comment: A lot of the texts in the tables and the figures are very small and difficult to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)