CoLLAT: On Adding Fine-grained Audio Understanding to Language Models using Token-Level Locked-Language Tuning
Authors: Amila Silva
Spencer Whitehead
Chris Lengerich
Hugh James Leather
Venue: NeurIPS 2023 poster (2023)
Keywords: Audio Understanding
Contrastive Learning
Audio-Language Grounding
TL;DR: CoLLAT is a framework that enhances audio understanding by fine-grained audio-text grounding in a locked language embedding space, achieving superior performance in various tasks while unlocking audio guidance for text-guided downstream applications.

Abstract: Humans can easily understand various audio concepts, but conventional audio classification models fail due to their inability to predict unseen classes during training. To address this challenge, recent literature has explored contrastive language-audio pretraining to learn an audio understanding model using natural language supervision from a pretrained language model. However, despite their reasonable zero-shot performance in audio understanding, these models typically fail to achieve optimal performance while preserving the text understanding capabilities of the pretrained language model. They also perform poorly when comprehending audio clips with multiple audio concepts. To bridge these gaps, we propose $CoLLAT$: $Co$ntrastive $L$ocked $L$anguage and $A$udio $T$uning. This is a framework to effectively learn an audio understanding model with a locked language model, which is learned using a novel pretraining objective for audio-to-text grounding to yield fine-grained audio understanding. Our extensive experiments, which include several downstream applications such as audio classification, cross-modal retrieval, and audio-guided image generation, demonstrate that $CoLLAT$ yields state-of-the-art performance for audio understanding. Additionally, it unlocks audio guidance to applications built on top of pretrained language models.

CoLLAT: On Adding Fine-grained Audio Understanding to Language Models using Token-Level Locked-Language Tuning
Amila Silva
Spencer Whitehead'
Chris Lengerich'
Hugh Leather
'Meta AI (FAIR)
The University of Melbourne
amilasQunimelb. edu. ed,
{srw5, clengerich, hhwtcherreehrrrr @meta. com
ZS Audio Classification (Aceuracy for ESC50)
CoLLAT
duck quacking followed by dog whimpering while birds chirp the background"
0.8
CLAP
CLAP duck, quack, fireworks, birds
CoLLAT
Top 5 zero-shot predictions order
duck dog, quack,
birds, whimper (dog)
0.7
AudioCLIP
Cons-CLAP
L77- CLAP
0.6
audio-guided image generation results
0.3
0.32
0.34
0.36
0.38
ZS Text Classification
(Accuracy for AG News)
(b)
Figure 1: CoLLAT yields fine-grained audio-to-text grounding by token-level alignment of audio-text with locked pretrained text embedding space, which (a) enables fine-grained audio understanding and novel multimodal capabilities (e.g., audio-guided image generation); and (b) achieves superior performance across both unimodal (audio, text) and multimodal (e.g., audio and text) settings.
Abstract
Humans easily understand various audio concepts, but conventional audio clas- sification models fail due to their inability to predict unseen classes during training. To address this challenge, recent literature has explored contrastive language-audio pretraining to learn an audio understanding model using natural language supervi- sion from pretrained language model. However, despite their reasonable zero-shot performance in audio understanding, these models typically fail to achieve optimal performance while preserving the text understanding capabilities of the pretrained language model. They also perform poorly when comprehending audio clips with multiple audio concepts. To bridge these gaps, propose CoLLAT: Contrastive Locked Language and Audio Tuning. This is a framework to effectively learn an audio understanding model with locked language model, which is learned using a novel pretraining objective for audio-to-text grounding to yield fine-grained audio understanding. Our extensive experiments, which include several downstream ap- plications such as audio classification, cross-modal retrieval, audio captioning and audio-guided image generation, demonstrate that CoLLAT yields state-of-the-art performance for audio understanding. Additionally, it unlocks audio guidance to applications built on top of pretrained language models.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Motivation. The sound perception system in humans is capable of understanding complex audio concepts and interpreting them in way that allows us to interact with our environment effectively [21] For example, when humans listen to sound clip of a duck quacking followed by dog whimpering while birds chirp the background (see example in Fig. [T), they can easily discern various audio concepts (e.g., a duck quacking, a dog whimpering, birds chirp) in that sound clip, enabling them to construct a holistic understanding of the sound clip. However, conventional audio classification models [22) (24 [30] To 25 that aim to associate audio recordings with one or more categories from a set of predefined categories fail to be competitive with the human auditory system due to their inability to predict unseen classes during training. To address this issue, recent literature has explored contrastive language-aude pretraining [3] 7 to learn an audio understanding model using natural language supervision. These methods aim to learn audio encoder and text encoder simultaneously. which produce joint embedding space for audio and text. The joint embedding space is learned to preserve the correspondence of audio-text pairs present in the pretraining dataset. Such joint embedding space creates an open vocabulary for audio-language correspondence, enabling new audio-language capabilities such as zero-shot audio classification and audio+language guided image generation. However, existing works on contrastive language-audio pretraining [3 35 lack the following strengths.
typically initialize their text encoder using CLIP 27 a strong pre- trained text encoder. However, these techniques [3] often fail to achieve optimal performance for audio-language understanding without fine-tuning the text encoder (i.e., by keeping the text encoder frozen). In Fig. [b] we report the performance of set of baselines, where CLAP, AudioCLIP, and Cons-CLAP models fine-tune the pre-trained CLIPP text encoder, while the rest keep the text encoder frozen. As can be seen, fine-tuning the text encoder improves cross-modal (e.g., zero-shot audio classification) performance, but it comes at the cost of losing its language understanding capabilities. This could be attributed to the lack of sufficient textual information in publicly available audio-text datasets (e.g.. AudioSet [8). which are relatively smaller in size compared to the datasets used to originally pretrain CLIF In this work, we study the possibility of achieving SOTA performance for audio-text understanding with a pretrained text encoder that remains locked. Incorporating audio understanding without tuning the CLIP text encoder is not only beneficial for capitalizing the pre- trained text encoder's 's existing capabilities, but it also enables audio or audio+text guidance to any AI application that uses text-guidance from the CLIP text encoder (e.g.. CIIP--uided image generation) without needing to retrain the inplication-specine model (e.g., diffusion model of CT.IP-guides image generation pipeline). Recent studies on language- visual pretraining 3 have demonstrated that matching the sizes of the encoders is vital for achieving strong cross-modal performance while keeping one encoder locked. Nevertheless, most existing audio-text pretraining methods use audio encoders that are substantially smaller in size compared to the text encoder. Building upon this observation, we propose an audio-text pretraining architecture that jointly scales both the audio and language encoders, allowing for strong cross modal performance with locked language tuning.
First, previous works 13
35
Second, existing contrastive audio-text pretraining techniques 355 13 have limited efficacy when understanding complex audio clips with multiple audio concepts (see results in Table 2}. This shortcoming could be attributed to the absence of explicit fine-grained cross-modal grounding in these techniques, as they solely depend on global embeddings i.e., a single vector summarizing the semantic content of given text/audio input, to maintain the correspondence between audio and text. Although such global embedding alignment methods achieve reasonable performance in audio understanding, encoding audio and text into global embeddings will lose much fine-grained information, which is critical to distinguish hard audio-text pairs. In contrast, the token-level embeddings of the intermediate layers of transformer-base better understanding of fine-graie -raiira concepts they have one more token-level embeddings dedicated to each fine- ane-grained concept in an audio/text input. Despite the knowledge in such token- level embeddings are explicitly exploited in other domains for fine- ine-grained cross-modal grounding (e.g., text-image [34, 37 and text-video (39)). to the best of our knowledge, none of the previous audio-text pretraining techniques have explored this. To bridge this research gap, this work studies the importance of exploiting token-level audio-text alignment along with the global-level alignment yield better fine-grainee understanding of audio-lext correspondence.
encoders for text and audio possess
CLIP's original pretraining dataset almost 200 times larger than AudioSet.
2

---

Contributions. The contributions of this work are to propose:
A neural architecture for effective contrastive audio-language pretraining that yields SOTA performance for audio-text understanding while keeping the text encoder locked. An improved audio-text pretraining objective function that explicitly encourages the model to learn fine-grained audio-text grounding to unlock complex audio comprehension.
We verify the superiority of our proposed framework using a diverse set of experiments zero-shot audio classification, cross-modal retrieval, audio captioning and audio-guided image generation, which shows that our framework yields SOTA performance for both unimodal (audio text) and multimodal (audio and text) understanding, while capitalising on the existing capabilities of the text encoder (e.g., text-guided image generation) and alleviating its substantial cost of re- te-triining.
2 Related Work
2.1 Audio-to- Text Grounding
Audio-to-text grounding refers to establishing connection between textual concepts and audio concepts. In earliest research efforts [22] 24 30 TO. 25), the audio-to-text grounding problem was approached as classification task that utilized machine learning models to link audio recordings to pre-defined class labels. These approaches explored various machine learning models, spanning from
traditional machine learning models such Support Vector Machine 19
31 32), to advanced neural [4] and transformers [10] [8].
architectures like Convolutional Neural Networks (CNN) 126
23
These models were designed to operate either static 22
24
30 trainable [ time- free-freueeccy
transformations of raw audio. While these approaches were successful in predicting the class labels used during training, they often struggled to predict novel audio concepts.
To address this challenge, recent works 35 B 7 5
have explored that how to learn a joint embedding space to preserve the correspondence between audio and text with the language supervision from pretrained text encoders. This approach enables an open vocabulary for audio concepts to unlock applications such as zero-shot audio classification. These works typically employ two separate encoders: one for audio and one for text. Each encoder maps the given input, such as an audio clip or text prompt, to an embedding vector that preserves the knowledge of the input. These models learn the parameters of the encoders using a contrastive loss function [27], which aims to ensure that the embeddings of corresponding text-audio pairs are similar each other while pushing the embeddings of dissimilar pairs further apart.
To exploit the language comprehension abilities of large pretrained text encoders 27 28 these studies often initialize the text encoder with a pretrained model like CLIP 277. These text encoders typically have billions of parameters and are trained using large-scale datasets such as LAION. For
the audio encoder, different works adopt various architectures Wav2CLIP adopts ResNet-based architecture M. AudioCLIP employs ESResNeXt [2], and CLAP uses CNN14 14 [17] Then, these works jointly train the audio and text encoder to preserve audio-text correspndence. In addition to the differences in audio encoder architecture, these works vary in how they construct text prompts and their use of auxiliary objectives. For example, AudioCLII 13] creates text prompts by concatenating the discrete class labels of an audio clip from datasets like AudioSet (9). On the other hand, CLAP [7] utilizes a set of audio-text datasets that provide semantically and syntactically meaningful captions for each audio clip to train their model. Additionally, AudioCLIP [3] incorporates auxiliary objectives to preserve audio-image and image-text correspondences along with the audio-text correspondance.
Despite these differences, all these works fail to yield optimal performance for audio-text understand- ing without fine-tuning the text encoder - i.c., keeping the weights of the pretrained text encoder. Previous studies [4] suggest that this may be due to the size mismatch of the audio and text encoders observed in previous works 13 (see Figure 2ap. As shown in Fig. 2b simply scaling the audio encoder to match the size of the text encoder does not result in optimal performance too, especially given the relatively small dataset available with audio-text pairs. To address this issue, recent studies B 4 in other multimodal domains propose the use of neural architectures [4] that enable joint scaling of the sizes of multimodal encoders. Nevertheless, to the best of our knowledge, such architectures have not been explored in the context of audio-text understanding.
To bridge this gap, our work proposes novel neural architecture for audio-text pretraining that solves the size mismatch problem in existing works, enabling model to achieve optimal performance for
3

---

CoLLAT
15
A.curage 0
CLAP
0.25
AudioCLIP
0.24
CLAP
0.6
CoLLAT
Wav2CLIP
9
10
0.5
0.5
0.5
Audio Eucoder Size Tezt Encoder Size
Audio Encoder Size
Tezt Encoder Size
(8)
(b)
Figure (a) Size ratio of audio and text encoders across different methods; and (b) zero-shot audio classification performance of CLAP (using FSD50K) with different audio encoder sizes, while keeping the size of the text encoder fixed to that of COLLAT.
text embedding
text (x*) fulbing firensekers poppiin"
Frozen
"Thie
Text
Forward
Encoder
(A2)))
shared parameiers
Cross Attention
Audio Encuder
Token Interaction Module
Frozen
Text
Encoder
Self Attention
audio (x*)
audio embodding (x*)
(a)
(6)
Figure 3: (a) Overview of CoLLAT, consisting of the Token Interaction Module for mapping audio tokens to text tokens by adopting series of blocks as detailed in (b).
audio-to-text understanding without the need for fine-tuning the text encoder. The strenghts of our architecture three-fold: (1) it preserves the language understanding capabilities of the pretrained text encoder, resulting in better audio-to-text grounding even with relatively small audio-text dataset; (2) it is compatible with text encoders of any scale, making it more viable for learning audio understanding using evolving language models; and (3) it maps audio embeddings into the same representation space as the pretrained text encoder, introducing audio-guided controllability to any downstream application that relies on text guidance from the pretrained text encoder.
2.2 Token-level Grounding
Existing audio-text contrastive pretraining techniques 335 13 do not explicitly consider the fine- grained alignment of audio concepts and textual concepts. These methods are only trained to align the global embeddings (i.e., last layer output from the encoders which summarizes all the concepts in the corresponding input) from the encoders. Such global embeddings may be able to capture the individual concepts in complex audio clips containing multiple audio concepts. In contrast, token-level hidden representations of audio/text encoders can effectively represent fine-grained concepts. Therefore, it is crucial to have teken-level alignment between audio and text to achieve more fine- saaeeegranedd understanding audio.
This research challenge has been deeply studied in text-visual domain. For example, the works in [3 2 34 37 have shown the importance of aligning the token-level hidden representation of the visual and text encoders to yield fine-graing visual understanding. The same concept has also been shown be important for similar other learning tasks such knowledge distilliation [38] 33). However, token-level contrast between audio and text modalities has not been well-studied in previous audio-text pretraining methods. Some works [16] I1] have focused on explicit token-level attention for audio-related downstream applications like audio captioning, but they are limited to specific tasks. Therefore, our work introduces token-leve level alignment as an auxiliary task for conventional contrastive audio-text pretraining yield fine-graines audio understanding.

---

3 CoLLAT
Our model aims to learn fine-graine audio understanding with the help of a pretrained text encoder, without fine-tuning the text encoder. Unlike existing techniques that use separate audio and text encoders, our model shares the pretrained text encoder, which remains frozen during training, for both audio and text encoding. The main strengths of this architecture are twofold: (1) it allows us to use large pretrained language encoders (with billions of parameters) for the text encoder without introducing a significant mismatch between the encoders used for audio and text, as this architecture scales the encoder size for each modality jointly; and (2) it enables knowledge (i.e., intermediate layers) in the text encoder to encode audio. By using a frozen shared text encoder for both audio and text encoding. the corresponding audio-text pairs are forced to implicitly align their token-level embeddings to produce similar global embeddings. Figure 3 (a) shows a high-level schematic of the proposed model architecture.
leverage the implicit
We reuse the pretrained CLIP text encoder [27 as our text encoder, but our model is compatible with any other pretrained text encoder any scale. This text encoder expects the static token embeddings of the given text prompt as input. To encode audio in a way that compatible with the pretrained text encoder, we adopt a transformer-based audio encoder [I0] to produce token-level embeddings for the audio tokens (i.e., patches in the log mel spectrogram) and a cross-attention-base
token interaction module produce corresponding text token embeddings from the audio token embeddings. As shown in Fig. (a), our model takes either audio and/or text as inputs and produces corresponding embeddings while preserving audio-text correspondence. We formally define our architecture and training objectives below.
Let ID {Xa, XX' be set of audio-text pairs, where each pair is represented as {x=, x²). Here, x² E FXT is the processed audio clip represented as a log mel spectrogram with F spectral components (e.g. Mel bins) and T time bins; and xf is the tokenized text prompt explaining the audio concepts in x.
3.1 Model Architecture
For a given {z", n'), we preprocess x" as a sequences of patches of its 2D audio spectrogram and x² as sequence of tokens by tokenizing the text prompt via the pretrained BPE-based tokenizer in CLIP. We denote these preprocessing functions for audio and text as preprocess, and preprocess; (), respectively, and the output from these functions as {
M and [ respectively.
{ 1 Ji=1 = preprocess (z"), {z})}=1 = preprocess(("")
Here, .M and N denote the maximum lengths of the audio and text sequences, respectively. The audio tokens {xt ) are embedded using a transformer-basee audio encoder [10] fa O,,, which returns latent representations of the audio tokens (h J The text tokens { are initially embedded using the static embedding matrix E, of the pretrained text encoder.
(1 = fa(x"), (1g) = E, x
The audio token embeddings are then passed through audio-text token interaction module fa-+O. which aims to predict the corresponding text token embeddings { using the audio token embeddings {h M through fine-grained token alignment. The output from this module is denoted J==1
as (h. -t N J==1 since it has the same number of tokens as the text token embeddings.
[h]- Our token interaction module consists of multiple pooos--tteenion-baae based blocks 37 (see Fig 3 (b)) in series. These blocks take the currently predicted text token embeddings from the previous block as input and attempt to denoise them using the audio token embeddings of the corresponding audio clip. For the first block in the interaction module, pass a set of random vectors as input, which are randomly initialized following a normal distribution with a mean zero and a standard deviation of
fa-
1. This series of token interaction blocks is analogous to a denoising pipeline [15] that is guided by the audio token embeddings to produce the corresponding text token embeddings.
Finally, the global audio and text embeddings are produced by encoding { and (h) 1 using the pretrained text encoder f(().
z" = =e(((h) -1 D1 2' = =((((()))
5

---

Our framework is trained using the following objective functions:
3.2 Cross-modal Token-Level Alignment Loss
To achieve the fine-grained grounding between audio and text while making CoLLAT agnostic to the order of the classes presented in the text, this loss function aims to find one-to-one mapping between text token embeddings (h, ر from text and the produced text token embeddings (hg-")}- using audio and make those token-level embeddings close to each other. We formulate the cross-modal token-level loss function as follows:
Leross = 1lo(th) (اليي
where denotes the function that reorder the token-level embeddings an instance (h- such that the cross-modal token-level loss function is minimized, which is pre-computed before each weight update. Given the time complexity of finding the optimal solution to g in Eq. we greedily compute g as follows: given the token embeddings of {h) }= and {hξ " Jy=1+ initiate the process by starting from the right-moost token embedding in {hg])=r- This token then paired with the J)=11 closest token embedding in { Next, proceed to the second token embedding from the " from the possible
right in (h)) excluding the already mapped token embedding set to be paired with the selected text token embedding. We iterate this process until we obtain the greedy one-to-one mapping between all the tokens in } and { ز We observed that this greedy approach could produce optimal re-ordering unless there are very similar classes in the
same audio clip. Consequently, we not anticipate a significant variation in the performance if the
optimal reordering can be achieved, which we leave future work given the time complexity of training the model with the optimal reordering.
3.3 Cross-modal Global Contrastive Loss
Motivated by (27), this objective function is introduced to preserve the global correspondence of the audio-text pairs the training dataset. Given the global embeddings of a batch B of audio and text pairs/ (2g, Z J. we compute the similarity metric between audio and text pairs in B as:
S = * (Zg Z ") Sh
* (ZB Zz
where temperature parameter to scale the range of logits. The similarity matrix S [| has [B] correct pairs in the diagonal and |B - B incorrect pairs in the off-diagonal, where [B] refers to the batch size. We formulate our cross-modal global contrastive loss for the batch as follows:
B
Leross- global
+ log]softmaa St-a) (u.a)
(2)
B
1=
3.4 Unimodal Token-Level Alignment Loss
Despite the reasonable performance of CoLLAT with the aforementioned two objective functions, we observed that the learned representations are not robust against weak perturbations of audio such jitter (see ablation study in Section 5 for more details). To address this challenge, previous works [6] 40 adopts self-supervised objectives that aim to maximize the similarity of the embeddings of two different yet correlated views produced by perturbing the same audio input. Motivated by these works, we formulate our unimodal token-level loss for a given audio-text pair {x", xf) to improve the robustness our framework follows:
Luni-token = 1
where [ yy=1 is the token-level embeddings of the perturbed version of x² by adding Gaussian noise. This loss forces the model to understand all the token-level embeddings (i.c., fine-grained concepts) of z using its perturbed version.
Overall Objective. The final objective function of CoLLAT as follows:
L X1 Tecross-toke >2 Lugross - global + X3 Truii
where X1, Ag and Ag control the importance of each loss term.

---

Table 1: Dataset Statistics
lask
Classification
Cross-modal Recrivaal
Andho-guided
Captioning
Image Gen.
Dataset
ESC:50
UrbanSoundKK
TUT
FSDSOK
AudioSet
VGGSoumJ
Apd
AudhoCaps
AudhoCopp
instances
2K
BK
6.3K
5IK
20.4K
15K
46K
46K
46K
50
527
N/A
Metric
Acc.
Acc.
Acc.
MAP
MAP
MRR
MAP@ 10
COCO Metries
N/A
4 Experimental Setup
4.1 Datasets
Training. We adopt the AudioSet dataset as the training dataset, which consists of 2,041,792 audio and text label pairs collected using 10s long YouTube videos. Each audio clip can have multiple text labels to represent different audio concepts in different granular levels.
Evaluation. We evaluate our model using four main downstream applications: (1) audio classification; (2) cross-modal retrieval; (3) audio captioning: and (4) audio-guided image generation. We adopt widely used real-world datasets to evaluate our model, which are shown in Table
Pre-processing. We preprocess audio clips by representing them 128- 28-dimensional log Mel filterbank (fbank) features, computed with a 25ms Hamming window every 10ms. This results in a 128x100/ spectrogram as input
audio encoder, where is the length of the audio clip in seconds. Next, we split the spectrogram into a sequence of N 16 x 16 patches with an overlap of 6 in both time and frequency dimensions, where N is the number of patches and the effective input sequence length for the Transformer. We flatten each 16x16 patch to a 1D patch embedding of size 768 using a linear projection layer, which we refer to as the patch embedding layer. Since the Transformer architecture does not capture the input order information, and the patch sequence is also not in temporal order, we add trainable positional embedding (also of size 768) to each patch embedding to allow the model to capture the spatial structure of the 2D audio spectrogram.
Following the findings in 77). we adopt the template of "This is sound of [class label 1]. fclass label 2J, and [class label CJ" to generate a text prompt for an audio clip with C' number of class labels. Such templates have been shown be more effective than simple concatenation 77). Our decision to not utilize natural language descriptions or keyword-to-captin
augmentation (36 in favor of simple templates is due the following reasons: (1) the cross- modal token-level loss function in CoLLAT aims to explicitly map each token in the given text prompt with its corresponding counterpart in the audio tokens (i.e., the patches in the Mel-Spectrogram of the audio). It may not be feasible to find such a mapping for certain tokens in natural language descriptions (e.g., stop words, adjectives) with the audio tokens. Consequently, having complex text prompts could adversely impact the training of CoLLAT; and (2) CoLLAT maintains the text encoder frozen during training. as it was pre-trained using text labels from the LAION dataset rather than natural language descriptions. Thus, attempting to produce text embeddings for natural language descriptions using the CLIP text encoder to train CoLLAT could introduce a data shift problem to negatively impact the training of CoLLAT
Training Details. After performing a grid search, we set {A1, A2, As} as (1,0.1, 0.01). We utilize GPU cluster comprised of 8 V100 GPU cards for training CoLLAT. With this hardware configuration, it takes approximately 320 GPU hours to train the model on AudioSet using the optimal hyperparameter setting.
5
Results
5.1 Audio Classification
Table presents a comparison of CoLLAT with five differrnt baselines in audio classification. The first baseline, called Supervise, trains a transformer from scratch without relying on features from a pretrained joint embedding space, while the other four baselines learn a joint embedding space between audio and text via contrastive pretraining. To conduct our experiments, we used five widely used datasets: ESC-50, US8K, TUT, FSD50K, and AudioSet, under two settings: zero-shot (ZS) and linear probe (LP). For zero-shot (ZS) setting, we first extract the embeddings for the audio clips and the possible set of labels using the audio and text encoder in each baseline. Then, we compute cosine distance between these text and audio embeddings to rank different class labels for given audio clip.
7

---

Table 2: Results for Audio Classification. Here, ZS and LP denote zero-shot and linear-probe experimental setups for audio classification. For uni-label datasets ESC-50, US8K and TUT,
Accuracy is reported as the metric. multi-label datasets (FSD50K and AudioSet) adopts MAP (Mean Average Precision). Higher the value is better for both metrics.
ESC (Ace)
USSK (Aec)
TUT (Ace)
ESDSOK (MAP)
AudioSet (MAP)
Z8
LP
ZS
LP
ZS
LP
ZS
LF
ZS
LP
Supervise
N/A
0.53
N/A
0.60
0.32
NIA
YamNet
N/A
0.85
N/A
0.78
N/A
0.63
NIA
0.50
N/A
0.27
Wav2CLIP
D.1
0.86
0.40
0.81
0.24
0.63
0.03
0.43
0.02
0.30
AndioCLIP
0.69
0.97
0.65
0.90
0.27
0.70
0.13
0.54
0.03
0.28
CLAP
0.77
0.96
0.73
0.88
0.30
0.72
0.14
0.59
0.06
0.32
0.29
FSD-50K (MAP)
ESC 50 (Acc)
0.9
0.2
Backbone Audio Encoder
ESC 50
FSD. 50K
(Acc)
ResNet R51
0.80
0.14
0.8
0.1
CNN14 17
0.82
0.16
AST (Random Initialization)
10
0.82
0.18
AST (Pretrained Initialization)
0.84
0.19
0.7
0
10
(b)
(8)
Figure 4: (a) Zero-shot audio classification performance of CoLLAT for ESC-50 and FSD-50K: solid line when different intermediate layers are used to compute the token alignment loss (x axis denotes the half of the index of the layer used to compute the loss); and dashed line when the number of cross cross-attention blocks in the token interaction module varies (x axis denotes the number of cross attention blocks). (b) Ablation study with different backbone audio networks using ZS audio classification. The pre-trained AST was pre-trained for audio classification using AudioSet.
For linear probe (LP) setting, we freeze our audio encoder as feature extractor and only train a 1-layer transformer decoder predict the class labels.
Our results show that the Supervise baseline does not perform well across all datasets, which suggests that audio-text pre-training generally improves performance, particularly for tasks with limited labeled data (such as ESC-50). Among the other baselines, CoLLAT outperforms them mostly across all datasets. In particular, performance improvements from CoLLAT are significant for datasets that consist of complex audio clips with multiple sound clips such as AudioSet and FSD50K. For instance, CoLLAT outperforms the strongest baseline for the AudioSet dataset by 50% under zero-shot setting and 22% under linear probe setting. This observation highlights the superiority of CoLLAT in comprehending complex audio clips with multiple audio concepts.
Ablation Study using Zero-shot Audio Classification. In Fig. 4a we provide ablations with different layer selections for token-level loss computation and the performance of CoLLAT with different number of cross-attention blocks. This ablation study guided adopt the initial token-level layer in the text encoder to compute the token-level loss functions and 8 cross-attention blocks in the final architecture of the CoLLAT'S token interaction module. Table 44b presents CoLLAT's performance with different audio encoder choices, which shows pre-trained AST as the most suitable backbone architecture. Additionally, we noticed that the AST backbone architecture converges faster compared
Table 3: (a) Ablation loss functions using Zero-shot Audio Classification with clean and noisy audio samples; (b) Results for Cross-modal Retrieval, following the experimental setups in B5 and (36 for VGGSound and AudioCaps respectively. Higher is better for all the figures.
ESC-50
FSD-50K (MAP)
VGGSound (MRR)
AupioCaps (MA 10)
clean
moisy
clean
noisy
I-A
A-TT
T-
LAL
0.84
0,59
0.19
0.17
Way2CLIP
0.057
0.068
0.52
0.38
Lerose- token
0.81
0.50
0.13
0.11
AndioCLIP
0.060
0.073
0.58
0.52
Luni coken
Lerose globat
0.82
0.54
0.17
0.16
CoLLAT
0.093
0.112
0.62
0.59
(a)
(b)
8

---

Table Results for Audio Captioning using the AudioCaps dataset. The experimental setup freezes each model as feature extractor and only train 1-layer transformer decoder to predict text sequences. follow the DCASE challenge setup (20]
and report standard COCO caption evaluation metrics.
BLEU,
BLEU,
BLEU3
BLEU+
CIDEr
METEOR
ROUGE
SPICE
AndioCLIP
36.1
CLAP
48.4
31.7
20.4
13.4
13.7
30.2
35.1
8,9
CoLLAT
55.1
37.2
24.3
15.1
16.3
43.0
39.6
11.3
Human
65.4
48.9
37.3
29.1
28.8
91.3
49.6
21.6
AUDIO
CoLLAT
CLAP
AUDIO +TEXT
CoLLAT
CLAP
sound of thauder and geatle rain
sound of thander and gentle rain
"with sea in Ale background"
A sound of blowing horn of train
A sound blowing horn of irain
"with mountain
h
the background"
A sound of severul birds chirping
A sound of several hirds chirping
wwith waterfall the background"
+
Figure 5: Audio-guided Image Generation Results
to the other backbones. This could be attributed to the stronger inductive biases present in the other encoders as compared to AST [0].
We conducted an ablation study in Table 3a to demonstrate the significance of the loss functions used in CoLLAT. The proposed token-level moosl loss function, Leross-token: is shown to play a crucial role in enhancing audio- unddio-understadding performance. This is especially evident in FSD-50K, which consists of audio clips with multiple labels, where the performance for zero-shot audio classification is improved by 46.2% in MAP. We also found that the robustness of CoLLAT substantially improves with the use of the unimodal-toke zero-shot classification performance for noisy audio samples that are generated by adding Gaussian noise with zero mean and 0.1 standard deviation. Touni-token improves the performance for such samples by 11.3% and 30.8% for ESC-50 and FSD-50K, respectively.
level loss, Luni-token: as it enhances the
5.2 Cross-modal Retrieval
Since pretrained CLIP text encoder maps text a joint embedding space of images and text, being able to map audio into the same CLIP embedding space (without fine-tuning the text encoder) enables cross-modal retrieving capabilties between audio, text and images. Tbblee3]sshow the results collected for cross- modal retrieval task using the VGGSound and AudioCaps datasets. We adopt the experimental setups proposed in [35] for the audio-image and image-audio retrieval tasks and [36] for the audio-text and text-audio retrieval tasks. For fair comparison, train all the baselines with a frozen text encoder.
As can be seen, CoLLAT outperforms all the baselines by as much as 47.6% and 51.4% in Mean Reciprocal Rank (MRR) for the Audio-to-Image and image-to-Audit retrieval tasks, and 13.4% and 6.7% in MAP for Audio-to-Tess retrieval, and Text-to-Audio retrieval, respectively. We observed that the performance improvements are particularly significant due to the ability of our approach to understand fine-graines concepts to differentiate hard image-text pairs in the VGGSound.
5.3 Audio Captioning
We report the results for Audio Captioning using the AudioCaps dataset Table, Following 35 we only trained 1-layer transformer on top of the CoLLAT's output, while keeping CoLLAT's parameters fixed to collect the results for audio captioning. We adopt the same experimental setting for the basclines too. As can be seen, CoLLAT outperforms CLAP and AudioCLIP by 9.7%, 13.2%,

---

42.4%, 9.7% and 26.9% in BLEU1 CIDEr, METEOR, ROUGE and SPICE respectively. These results further show the ability of CoLLAT fine-graing ane-grained details the sound clip as it needed to generate a holistic caption for given audio clip.
5.4 Audio-guided Image Generation
Given CoLLAT's ability to produce corresponding text token-level embeddings for a given audio clip, it can be naively used to provide guidance for any downstream application built on top of text token embeddings from pretrained text encoders. We adopt text- text-guided image generation as one such example. Most existing text- exxtgguidd image generation models adopt token-level embeddings from pretrained text encoder to guide image generation using a given text prompt. Nevertheless, most existing audio-to-text grounding techniques cannot be naively extended to introduce audio guidance for such applications, they are unable to generate token-level text embeddings for given audio prompt.
To evaluate this task, we adopt pretrained text-guided image generation model following [29] which requires text guidance from a pretrained CLIP text encoder. As a baseline, we adopt the same architecture as CoLLAT, but trained only using Leross- gilibal We denote this baseline as CLAP in Figure As shown in Figure CoLLAT is capable of generating images that cover all the concepts present in the given audio clip. This observation further confirms the potential of our approach to comprehend complex audio clips with multiple concepts.
Broader Impact
Our work has a significant impact as it addresses the limitations of conventional audio understanding models by enhancing their ability to predict finer-graines classes within complex audio clips. We demonstrated that CoLLAT achieves offte-of-thearr performance in audio understanding, as validated by its success in various downstream applications. Furthermore, unlocks the potential for audio guidance in applications built upon pretrained language models. Thus, work opens up possibilities for improved audio-to-text grounding, advancing multimodal applications. Additionally, considering the capability of CoLLAT to represent audio clips using either text or image, it holds promise in enhancing accessibility and promoting inclusivity for individuals with diverse disabilities.
It is important to note that the model developed in this research does not possess the ability to uniquely identify individuals in audio recordings for tasks such speaker recognition and speaker identification. CoLLAT does not possess the ability to comprehend information conveyed through speech in audio clips. Given the privacy concerns associated with the generation of human faces using audio-gun audio-guided image generation models, it is advised to exercise caution and refrain from utilizing them for such purposes. crucial to prioritize privacy and ethical considerations when applying these technologies to avoid any potential misuse or infringement on individuals' rights.
7 Conclusion
This work proposes CoLLAT, novel framework for learning an audio understanding model using natural language supervision from a locked pretrained language model. The proposed framework effectively exploits the implicit knowledge large language models instead of just relying on their final global embeddings. We introduce novel pretraining objective for CoLLAT that enforces token- level alignment between audio and text modalities to yield fine- aeee-grrined audio understanding. Our results demonstrate that CoLLAT achieves -fate-of-the-art performance for downstream applications such as audio classification, cross-modia retrieval and audio captioning. Since CoLLAT keeps the pre-trained language model locked during training, our experiments also show that it unlocks new applications such as -uiidd image generation.
Since CoLLAT can be considered a -ggnossic framework for learning knowledge under- standing models for new modalities with natural language supervision, extending CoLLAT to new modalities (video) is an interesting future direction to explore. Another notable limitation of CoLLAT its inability to understand speech signals in the audio clips. Therefore, it could be worthwhile to explore how to introduce speech signals to CoLLAT while preserving its existing capabilities.
10

---

References
[1] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In Proc. of ICASSP, pages 721-725, 2020.
[2] Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. Fine-grained video-text retrieval with hicrarchical graph reasoning. In Proc. of CVPR, pages 10638-10647, 2020
[3] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794 2022.
[4] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0223 2022.
[5] Soham Deshmukh, Benjamin Elizalde, and Huaming Wang. Audio retrieval with wavtext5k and clap training. arXiv preprint arXiv:2209. 14275, 2022.
[6] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting, arXiv preprint arXiv:2 106. 141 12, 2021.
[7] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap: Learning audio concepts from natural language supervision. arXiv preprint arXiv:2206.04769, 2022.
[8] Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Chan- ning Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In Proc. of ICASSP, 2017.
[9] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labelas dataset for audio events. In Proc. of ICASSP, pages 776-780, 2017.
[10] Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. arXiv preprint 2rXiv:2104.0177 2021.
[11] Félix Gontier, Romain Serizel, and Christophe Cerisara. Automated audio captioning by fine-tuning bart with audioset tags. In Proc. of DCASE, 2021.
[12) Andrey Guzhov, Federico Raue, Jömn Hees, and Andreas Dengel. Esresne (x) t-fbsp: Learning robust fime-freyuency transformation of audio. In Proc. of IJCNN, pages 1-8, 2021.
[13] Andrey Guzhov, Federico Raue, Jömn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In Proc. of ICASSP, 2022.
[14] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Chan- ning Moore, Manoj Plakal, Devin Platt, Rif Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In Proc. of ICASSP, 2017.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proc. of NewrlPS, 2020.
[16] Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. AudioCaps: Generat- ing Captions for Audios in The Wild. In Proc. of NAACL-HLT, 2019.
[17] Qiuqiang Kong, Yin Cao, Turab lqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley, Panns: Lcrge-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions Audio, Speech, and Language Processing, 28:2880-2899 2020.
[18) Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer. Efficient training of audio transformers with patchout. arXiv preprint 17X.022110.05069, 2021.
11

---

[19] Chien- Chang Lin, Shi-Huang Chen. Trieu-Kien Truong, and Yukon Chang. Audio classification and categorization based on wavelets and support vector machine. IEEE Transactions on Speech and Audio Processing, 13(5):644-6) 651, 2005.
[20] Samuel Lipping, Konstantinos Drossos, and Tuoams Virtanen. Crowdsourcing a dataset of audio captions. In Proc. of DCASE, 2019.
[21] Richard F Lyon. Machine hearing: An emerging field [exploratory dsp]. IEEE signal processing magazine, 27(5):131-139, 2010.
[22] Annamaria Mesaros, Aleksandr Diment, Benjamin Elizalde, Toni Heittola, Emmanuel Vincent, Bhiksha Raj. and Tuomas Virtanen. Sound event detection in the dcase 2017 challenge. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(6):992-1006, 2019.
[23] Kamalesh Palanisamy, Dipika Singhania, and Angela Yao. Rethinking cnn models for audio classification. arXiv preprint arXiv:2007.11154 2020.
[24] Karol J Piczak. Environmental sound classification with convolutional neural networks. In Proc. of MLSP, 2015.
[25] Karol Piczak. Esc: Dataset for environmental sound classification. In Proc. of ACM MM, 2015.
[26] Jordi Pons and Xavier Serra. Randomly weighted enns for (music) audio classification. In Proc. of ICASSP, 2019.
[27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. of ICML, 2021.
[28] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21()):588--551, 2020.
[29] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proc. of CVPR, 2022.
[30] Justin Salamon and Juan Pablo Bello. Deep convolutional neural networks and data augmen- tation for environmental sound classification. IEEE Signal processing letters, 24(3):279-22) 283, 2017.
[31] Nicolas Scaringella and Daniel Mlynek. A mixture of support vector machines for audio classification. IEEE MIREX, London, 2005.
[32] Sameh Souli and Zied Lachiri. Audio sounds classification using scattering features and support vectors machines for medical surveillance. Applied Acoustics, 130:270- 2018.
[33] Frederick Tung and Greg Mori. Aimilarity-preservvi knowledge distillation. In Proc. of CVPR, pages 1374, 2019.
[34] Michael Wray, Diane Larlus, Gabriela Csurka, and Dima Damen. Fine-grained action retrieval through multiple parts-of-speech embeddings. In Proc. of CVPR, pages 450-459, 2019.
[35] Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. In Proc. of ICASSP, 2022.
[36] Yusong Wu, Ke Chen. Tianyu Zhang, Yuchen Hui, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword- to-caption augmentation. In Proc. of ICASSP, 2023.
[37] Chen-War Xie, Jianmin Wu, Yun Zheng, Pan Pan, and Xian-Sheng Hua. Token embeddings alignment for cross-modal retrieval. In Proc. of ACM-MM, 2022.
12

---

[38] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self-supervision. In Computer Vision-ECC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX, pages 588 604. Springer, 2020.
[39] Jianwei Yang. Yonatan Bisk, and Jianfeng Gao. Taco: Token-aware cascade contrastive learning for video-text alignment. In Proc. of CVPR, pages 11562-11572 2021.
[40] Zhihan Yue, Yujing Wang. Juanyong Duan, Tianmeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proc. of AAAI, 2022.
13


Reviews:
[{"review_id": "cf6cugvVfI", "reviewer": "Reviewer_swwz", "summary": "The paper introduces CoLLAT, an audio-language framework that makes use of a pre-trained language model. The framework is trained with a contrastive objective which encourages learning of fine-grained audio-text grounding. The paper presents very strong results for diverse downstream tasks, such as zero-shot audio classification, audio-image retrieval, and audio-guided image generation.", "questions": "* Is CLAP shown in Figure 5 referring to [6]? If not, different naming should be used. \n\n* I believe that some relevant prior works should be referenced, e.g. [C,D,E].\n\n* What is the computational cost to train the framework and what kind of hardware was used for training?\n\n[C]: Wu et al: LARGE-SCALE CONTRASTIVE LANGUAGE-AUDIO PRETRAINING WITH FEATURE FUSION AND KEYWORD-TO-CAPTION AUGMENTATION, ICASSP 2022\n\n​​[D]: Deshmukh et al.: Audio retrieval with wavtext5k and CLAP training, 2022\n\n[E]: Wu et a.: Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation, ICASSP 2023\n\n\n\n* Typos:\nl.62: lost -> lose\n\nl.136 alignemnt -> alignment\n\nl.181: space missing after “CLIP.”\n\nl.232: We adopt AudioSet dataset -> We adopt the AudioSet dataset\n\nFigure 4 is a table and should be labelled accordingly.\n\nWhat does t in $128x100 t$ in l.240 refer to?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed framework leverages a locked pre-trained language model (CLIP text encoder) as audio and text encoder. It learns to map the audio input to input tokens for the frozen text encoder, which retains the strong capabilities of the language model. The model is trained to encourage very fine-grained understanding of audio. It achieves very convincing results on various downstream tasks.", "weaknesses": "Given that this paper claims to yield fine-grained audio-to-text grounding, I believe that it would be good to show results for audio-text retrieval on CLOTHO, AudioCaps, and SoundDescs [A,B]. \n\nOne of the main model contributions, the token interaction module, should be explained better and in more detail. For instance, it is not mentioned how many blocks are used (l.192: “multiple cross-attention-based blocks”). In addition, it would be good to add a detailed explanation on the denoising pipeline in this module. Furthermore, different components of this block should be ablated in the model ablation study.\nSimilarly the paper does currently not contain a model ablation for 1) the use of the same encoder for audio and text instead of using a different audio model, 2) the effect of using pre-trained encoders. These should be added to validate the architectural choices.\n\nOverall, I think the paper is interesting, and I would be happy to consider raising my score if the missing evaluations and ablations are added and convincing.\n\n\n\n[A]: Oncescu et al.: Audio Retrieval with Natural Language Queries, INTERSPEECH 2021\n\n[B] Koepke et al.: Audio Retrieval with Natural Language Queries: A benchmark study, Transactions on Multimedia 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable comments and feedback. Please refer to the attached PDF in the global response for supplementary results. \n\n__1. Results for several additional tasks.__ Please find the results for text-to-audio retrieval, audio-to-text retrieval, and audio captioning using the AudioCaps dataset in Figure 1.b and Table 1 in the supplementary PDF in the global response. In the latest results, CoLLAT outperforms the baselines (by approximately 20% in SPICE for audio captioning, 13.4% in MAP for audio-to-text retrieval, and 6.7% in MAP for text-to-audio retrieval), demonstrating a similar performance trend as the previously reported results in the main paper. Due to time constraints, we will consider collecting results with CLOTHO and SoundDescs as future work.\n\nPlease take note that the experimental setup for the zero-shot column in the audio classification results table in the main paper is similar to the audio-to-text retrieval, as it solely depends on the similarity of embeddings between text and audio to identify corresponding pairs, without utilizing any downstream classifier as in the LP setting. Therefore, the zero-shot audio classification results can also be considered as supplementary results for audio-to-text retrieval.\n\n__2. More ablations.__ In the supplementary PDF of the rebuttal, we provide additional ablations with different audio encoder choices (refer to Fig. 1.b), varying numbers of cross-attention blocks (refer to Fig. 2.b), and distinct layer selections for token-level loss computation (refer to Fig. 2.a). These ablations were conducted to guide different design decisions in CoLLAT. In our final architecture, we have incorporated 8 cross-attention blocks within the token-interaction module. We will clearly mention all these hyperparameters in the final version.\n\n__2.1. Impact of using the same encoder for audio and text instead of using a different audio model__ CoLLAT's architecture adopts the same frozen text encoder in the audio encoding pipeline (refer to Fig. 3 in the main paper). Considering the nature of the token-level loss functions, if we employ completely different models (without any parameter sharing) for audio and text encoding, it needs to train an audio encoder that matches the scale of CLIP (with over 1B parameters) for audio encoding. This learning process is resource-intensive, especially given the added complexity of $g$ in the cross-modal token-level loss function. To tackle this challenge, CoLLAT proposes a cost-effective alternative by efficiently sharing the parameters of a pre-trained text encoder within its audio encoding pipeline. With this architecture, our audio encoder can effectively utilize all the language understanding capabilities of the pre-trained text encoder when processing audio. This may explain why CLIP achieves significantly superior results in tasks like audio captioning compared to the baseline, which employs different models for audio and text encoding. \n\n__2.2. Impact of using pre-trained encoders__ Adopting pre-trained encoding heads to initialize audio encoders has been shown to be useful in previous similar works (e.g., AudioCLIP). We observe a similar trend for the ESC-50 and FSD-50K datasets with CoLLAT, which is reported in Fig. 1.b of the supplementary PDF. We also noticed that pre-trained initialization improves the convergence speed.\n\n__3. Differences between CLAP and the model used in Figure 5__ To effectively utilize the denoising diffusion pipeline for image generation as a readily available solution without re-training, two key requirements need to be met: (1) the text encoder must remain frozen during the audio encoder training, and (2) the audio encoder should accurately predict token embeddings of the corresponding text prompt for a given audio clip. This is mainly because the selected denoising diffusion model accepts token-level embeddings of either the text prompt or audio clip to guide the image generation process. To fulfill these requirements, we adopt CoLLAT's architecture and CLAP's training objective to train the baseline model denoted as CLAP in Fig. 5. For further details, please refer to lines 297 to 305 in the main paper. In the final version, we will ensure that different naming is used for this baseline model.\n\n__4. About the missing citations__ Thank you for pointing out these missing citations. It seems that the same work was mistakenly pasted for both [C] and [E]. We would greatly appreciate it if you could let us know if you have found two different works that should be cited for [C] and [E]. We will ensure that all these works are properly cited in the final version.\n\n__5. Computational cost and hardware__ We utilize a GPU cluster comprised of 8 V100 GPU cards for training CoLLAT. With this hardware configuration, it takes approximately 320 GPU hours to train the model on AudioSet using the optimal hyperparameter setting. We will include these details in the main paper.\n\n__6. Formatting issues__ Thank you for pointing the formatting issues and typos. We will correct all of these issues in the final version.\n\n__7. Meaning of $t$ in $128x100t$__ Here, t is the time duration of the audio clip in seconds. We will define this notation clearly in the final version. \n\n\n\n\n\n\n\n"}, {"review_id": "cYyx4nhPO7", "reviewer": "Reviewer_NoKh", "summary": "This paper introduces a method to train an audio encoder to produce audio embeddings that match text embeddings. Contrary to other methods, the authors suggest to keep the text encoder fixed so as to enable its use in other contexts. More specifically, a Transformer audio encoder produces audio embeddings. A second model produces audio-to-text embeddings with the same dimension as the reference text, which are re-ordered and matched with the text embeddings. This module allows the original audio embeddings to have a fine-grained correspondance with the text. Finally the outputs of the frozen text encoder applied to the reference text embeddings and the embeddings derived from the audio-to-text model are matched with a contrastive loss. The experiments show good results for audio classification tasks as well as cross-modal retrieval and audio-guided image generation.", "questions": "3.1:  how are the initial input defined in the denoising pipeline?\n\n3.2: greedy reordering: how much better would the method be if the optimal reordering was found? could the reordering be learned by the token interaction module, or why isn't it? How the order of the classes in the text and in the audio relate in the training set?\n\n4.1: In the template, does the ordering of the classes matter? (e.g. if the classes appear one after the other in the audio)\n\nSec. 5.1: Zero Shot: is the possible set of labels in the text encoder also matching the template presented in 4.1?\n\n", "limitations": "yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "  - The paper is clear and convincing. The introduction and related work section are especially well written and clear, containing a good presentation of the challenges and contributions\n  - The motivation to keep the pre-trained text encoder untouched is clear and interesting. The results are a good proof of the interest of this approach. The examples illustrate well the effectiveness of the method.\n  - The idea to produce fine-grained correspondance between the audio embeddings and the text is reasonable and pretty good.\n  - The experiments look sound and are convincing", "weaknesses": "  - The presentation and notation could be more explicit at some places, for example: \n    - Intro, 2nd paragraph: \"as shown in fig 1b\" : not clear what to look at in 1b. Maybe explain which one is the fine-tuned CLIP?\n    - it looks like there is a formatting issue in 3.1\n    - in 3.3, the notation requires more explanation. It looks like the two S are transpose of each other.\n    - Eqn. 2 requires more explanation. what is the (i, i) subscript? softmax along which dimension of S?\n  - It would be nice to see some examples of classes/text in different datasets (maybe in supplementary material) to illustrate the complexity of datasets\n  - Some parts could benefit from more explanations, for example:\n    - How is the reordering $g$ computed?\n    - How are the initial input defined in the denoising pipeline?\n  - It would be interesting to have a deeper study and analysis of the results in Fig. 2. While reading, I expected to see the corresponding experiments and analysis in the experiment section.\n\n\nMinor:\n  - l.107: \"correspndence.\"\n  - formatting issue in the text in Sec. 3.1 making the section hard to read and understand\n  - Missing reference for AudioSet\n  - Evaluation: missing \"(3)\" for the third task  \n  - Sec. \"4.1 Datasets\" does not only deal with datasets. There is not Sec. 4.2", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable comments and feedback. Please refer to the attached PDF in the global response for supplementary results.\n\n__1. Unclarity about the fine-tuned CLIP models in Fig. 1b__ In Fig. 1b, CLAP, AudioCLIP, and Cons-CLAP models fine-tune the pre-trained CLIP text encoder. In contrast to CLAP and AudioCLIP, Cons-CLAP incorporates a constraint to alleviate catastrophic forgetting of CLIP's existing knowledge during training. This constraint penalizes the model if it significantly forgets existing knowledge. In the final version, we will clearly specify and highlight these points.\n\n__2. Formatting issue in Section 3__ Thank you so much for pointing out these issues. In Sec. 3.1, the notations should be corrected as bellow. We will correct these issues in the final version.\n\n${x^a_i}{i=1}^M \\rightarrow$ {$x^a_i$}$_{i=1}^M$\n\n${x^t_j}{j=1}^N \\rightarrow$ {$x^t_j$}$_{j=1}^N$\n\n${h^a_i}{i=1}^M \\rightarrow$ {$h^a_i$}$_{i=1}^M$\n\n${h^t_j}{j=1}^N \\rightarrow$ {$h^t_j$}$_{j=1}^N$\n\n__3. Explanation on $S$ in Section 3.3__ In Section 3.3, two $S$ matrices are $|B|\\times |B|$ matrices that are transposes of each other, as you mentioned. Here, $|B|$ refers to the batch size. The $(i,i)$ subscript of the $S$ matrices represents the elements along the main diagonal. The elements along the main diagonal in $S$ represent the dot product similarity of the embeddings between corresponding audio and text pairs. We will make these facts clear in the final version of the paper.\n\n__4. Few examples from different datasets__ We will include a few examples from each dataset and detailed statistics related to complexity (such as annotator agreement) in the final supplementary material of the paper.\n\n__5. About how $g$ is computed__ $g$ aims to find the permutation of the token-level embeddings in {$h^{a-t}_j$}$_1^N$ that minimizes the L1-norm distance between the permuted audio token embeddings of an audio clip and the corresponding text token embeddings. $g$ is computed greedily given the time complexity of finding the optimal solution for $g$. Our greedy algorithm operates as follows: Given the token embeddings of {$h^{a-t}_j$}$_1^N$ and {$h^{t}_j$}$_1^N$, we initiate the process by starting from the right-most token embedding in {$h^{t}_j$}$_1^N$. This token is then paired with the closest token embedding in {$h^{a-t}_j$}$_1^N$. Next, we proceed to the second token embedding from the right in {$h^{t}_j$}$_1^N$, excluding the already mapped token embedding in {$h^{a-t}_j$}$_1^N$ from the possible set to be paired with the selected text token embedding. We iterate this process until we obtain the greedy one-to-one mapping between all the tokens in {$h^{a-t}_j$}$_1^N$ and {$h^{t}_j$}$_1^N$. We observed that this greedy approach could produce optimal re-ordering unless there are very similar classes in the same audio clip. Consequently, we do not anticipate a significant variation in the performance if the optimal reordering can be achieved, which we leave as future work given the time complexity of training the model with the optimal reordering.\n\n__6. Initialization of the input to the denoising pipeline in the token interaction module__ All the elements in the input matrix were randomly initialized by following a normal distribution with a mean of zero and a standard deviation of 1.\n\n__7. Impact of the order of classes in text prompts__ CoLLAT is agnostic to the order of the classes presented in the text. This is the primary motivation behind the introduction of $g$, which reorders the token embeddings in {$h^{a-t}_j$}$_1^N$ prior to computing the token alignment loss. With the aid of $g$, CoLLAT's objective is solely focused on producing all the textual concepts from a given text prompt using the corresponding audio clip, but incorrect ordering is not penalized. Consequently, the ordering of class in template in 4.1 does not matter. \n\nFor a given audio clip with multiple labels, there are multiple ways to concatenate these labels together and generate the text prompt using the template described in Section 4.1. However, since we do not provide any information about the specific order utilized to create the text prompt to our token interaction module (providing such information to the token interaction module causes problems during inference), it is not possible for the module to learn the reordering process for a given (audio, text) pair by itself. This is the main motivation to make CoLLAT agnostic to the order of the classes in the text prompt.\n\n__8. Impact of the oder of sound events in audio clips__ Due to cross-attention mechanism in CoLLAT's token interaction module, CoLLAT remains agnostic to the order of classes present in the audio. Even in cases where classes overlap within the same time period, CoLLAT can accurately map them to their corresponding text counterparts if their frequencies differ. This capability is possible because CoLLAT's audio encoder treats each patch in the Mel-spectrogram of an audio clip as a distinct audio token.\n\n__9. About Zero Shot setting__ Yes, the possible set of labels is also processed using the same template as described in Section 4.1 (_This is a sound of [class label]_) before being inputted into the text encoder to obtain the corresponding text embeddings. \n\n\n\n"}, {"review_id": "Jq8tRoa88J", "reviewer": "Reviewer_Ujf9", "summary": "This paper presents a novel approach to train audio embeddings grounded in text embeddings for audio classification. They propose a trainable audio embedding layer while keeping the pre-trained text embedding module frozen. Apart from the contrastive loss term that maximises similarity between corresponding audio and text embeddings, authors propose a loss term that exploits audio-text alignment at token level. They also propose a loss term that minimizes the distance between embedding obtained for clean vs noisy audio sample. The authors show that their model outperforms other SOTA models for audio classification and audio retrieval.", "questions": "1. For token interaction module, was the token sequence of length N provided as input (query) to first layer initialized randomly? \n2. Learning rate of 0.1 for Adam seems high. Clap uses 1e-3 while AST used 5e-5. Did the authors see unstable training with such high learning rate?\n3. Did authors try with loss terms minimizing the distance between token embedding from audio and actual text embeddings obtained from intermediate layers of the pre-trained text encoder?\n4. Why did authors choose to use learned position embeddings? Previous research has shown that using fixed sinusoidal position embedding o rotary position embeddings can also work well with transformer architectures. Did authors try using fixed position embeddings?", "limitations": "Authors have addressed the concerns about identifying personal information from audio recording. they have also considered the possibility that one can potentially use audio guided image generation to generate human faces which can lead to privacy and ethical issues. In my opinion they have adequately addressed the ethical concerns.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "One of the key strengths of the paper is that they freeze the pre-trained text embedding module and pull the audio embedding closer to the text embeddings obtained from the frozen module. This enables their audio embedding module to convert audio waveforms to text embeddings which can be recognized by other models using the pre-trained text module. This enabled them to showcase the efficacy of their method for audio guided image generation using pre-trained diffusion models. The proposed loss terms are well motivated. They do extensive evaluations comparing their model with other baselines and show it beats them for the two tasks of audio classification and cross-modal retrieval. The ablation studies show case the benefit of each of the proposed modules. The audio guided image generation further shows the benefit of cross-modal token level alignment loss.", "weaknesses": "1. Not enough details are provided to reproduce the experiments e.g. detail of their AST backbone for audio encoder, architecture details of token interaction module.\n2. Functionality of the permutation module “g” in cross-modal token level alignment loss is not very clear (equation 1). Is the permutation “g” different for different training examples? Is it different for the same example across different epochs? What is the permutation used in inference time? Or do the authors not permute the token embedding generated from audio during inference?\n3. For cross-modal token-level alignment loss authors do a permutation of the token level embeddings obtained from audio before matching them with text tokens. For a length N sequence, there would be N! ways the sequence can be arranged. It would seem that would lead to a significant increase in training time trying to find the permutation that leads to the lowest loss especially when done for all the examples in a batch for all epochs.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable comments and feedback. Please refer to the attached PDF in the global response for supplementary results.\n\n__1. Initialization of the input to the token interaction module__ All elements in the input token embedding matrix were randomly initialized, following a normal distribution with a mean of zero and a standard deviation of 1. We will provide a clearer explanation of this point in Line 195 of the main paper.\n\n__2. Learning rate__ We did not observe significant instabilities with the selected learning rate of 0.1, combined with the other optimal hyperparameters. During the initial iterations with a small learning rate, it was observed that the absolute values of parameter updates could be relatively small. This behavior was more noticeable when utilizing a later layer of the text encoder to compute the token-level loss functions. This behavior can be attributed to two factors: (1) the strong dependence of the optimal learning rate on the objective functions and the weights assigned to them (lambda values), rather than the architecture; and (2) the better pre-trained initialization of most components in CoLLAT.\n\nPlease also note that the chosen learning rate serves as an upper bound. The learning rate is adaptively adjusted for each parameter when using the Adam optimizer. However, we observed instances of instabilities and encountered challenges related to reaching trivial solutions (e.g., averaged embedding as the token embeddings) when training CoLLAT solely with the cross-modal token alignment loss without applying $g$.\n\n__3. Results when different intermediate layers are used to compute the token-level alignment loss__ We present the results for this study in Figure 2.a. As can be see, we have identified that the first layer is the optimal layer for performing token-level alignment. This finding may be attributed to the following reasons:\n* As CoLLAT utilizes a frozen text encoder in both the text and audio encoding pipelines, performing cross-modal token-level alignment at an earlier layer of the frozen text encoder inherently aligns the token embeddings in subsequent layers as well.\n* The token embeddings from the intermediate layers in CLIP are more contextualized, resulting in less distinct embeddings for different tokens due to the information-sharing with their contextual surroundings. CoLLAT effectively leverage this by employing the first layer of CLIP (i.e., static embeddings of CLIP) to compute the cross-modal token-level loss function. This approach enhances the likelihood of achieving optimal reordering with our greedy reordering method to compute $g$, as the token embeddings show greater differences at the earliest layers.\n\n__4. Regarding positional embeddings__ The decision to adopt learnable positional embedding in CoLLAT was primarily motivated by AST [1], as we employed the pre-trained checkpoint of AST during our experiments. In our audio encoder, the patches of the audio Mel spectrogram are treated as tokens. Given the existence of long-distance and somewhat arbitrary spatial dependencies among such tokens, the adoption of learnable positional embedding is widely accepted in similar domains [1,2]. For our text encoder, we follow the same positional embedding technique as CLIP, as we keep our text encoder fixed during training.\n\n[1] Gong, Yuan, Yu-An Chung, and James Glass. \"Ast: Audio spectrogram transformer.\" arXiv preprint arXiv:2104.01778 (2021).\n\n[2] Dosovitskiy, Alexey, et al. \"An image is worth 16x16 words: Transformers for image recognition at scale.\" arXiv preprint arXiv:2010.11929 (2020).\n\n__5 More details about the architecture__ We adopt the same architecture as AST to define our audio encoder. The cross-attention block in the token-interaction module is architecturally similar to the cross-attention block proposed in [3]. Our CLIP text encoder is taken from the CLIP encoder in the Stable Diffusion V2 model. To ensure the paper is self-contained, we will include these specific details in the supplementary material of the final version. In the supplementary material of the rebuttal, we present the ablations (see Fig. 2b) that we conducted to determine the optimal number of cross-attention blocks in the token-interaction module. Following this study, we set the number of cross-attention blocks to 8. Additionally, in Figure 1.b of the supplementary material, we provide the results obtained using different audio encoders, which led us to select AST as our backbone architecture.\n\n[3] Chen-Wei Xie, Jianmin Wu, Yun Zheng, Pan Pan, and Xian-Sheng Hua. Token embeddings alignment for cross-modal retrieval. In Proc. of ACM-MM, 2022.\n\n__6 More details about the functionality of $g$__ The introduction of $g$ in the cross-modal token alignment loss allows CoLLAT to be agnostic to the order of classes within the text. As a result, CoLLAT's objective solely focuses on generating all the textual concepts from a given text prompt using the associated audio clip, without penalizing incorrect ordering. Since there are multiple ways to concatenate the text labels of an audio clip to form a text prompt, CoLLAT's token interaction module cannot predict the corresponding text tokens in the correct order. This is because the module does not have access to information regarding the specific order used to create the text prompt. Providing such information to the token interaction module can lead to inference-related problems such as which order should be used during inference. The permutation $g$ is different for different training examples and even for the same example across different epochs, especially during the early stages of training. This permutation is necessary only for computing the cross-model token alignment loss function and is not required for inference (obtaining the audio embedding). We agree with you regarding the complexity of $g$, which is N! for each instance, and it acts as the bottleneck in the training loop of CoLLAT."}, {"review_id": "1CkWom0IMn", "reviewer": "Reviewer_GZR1", "summary": "The authors propose a novel ways to train audio-text model by adding a token interaction module followed by the same frozen text encoder from the text modality after audio encoder to learn audio representation. And they show that proposed approach yield state-of-the-art results in couple audio tagging and classification tasks, cross-modal retrieval, and qualitative analysis on audio guided image generation.", "questions": "- For the audio and text encoder selections, have you tried other options, for example ResNet/CNN based for audio and other text encoders such as BERT or RoBERTa? For the other systems to compare as shown in Table 2, they all utilize different encoder architectures, such ablation study can help improve and disentangle whether the improvement comes from proposed architecture versus encoder choices.", "limitations": "- This work is slightly limited in then evaluation with only audio tagging/classification tasks, it can benefit from adding more audio-text tasks involving longer form natural languages to provide more holistic views.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach of appending token interaction module followed by text encoder is a very interesting and novel idea to leverage language model.\n- Downstream evaluation are provided with both zero-shot and linear probe settings, this provides more thorough understanding of the behavior for proposed systems.", "weaknesses": "- Proposed architecture is really interesting, however, the training with AudioSet labels in constructing prompts as described in section 4.1 line 248-250 seem arbitrary. The claim that \"such templates have been shown to be more effective than simple concatenation\" needs more in depth evidence or reference. Also given the works in increasing audio captioning data and applied to audio-text model pre-training (e.g. [1], [2]), this work can benefit from pre-training utilizing these audio and natural language descriptions besides pre-training with AudioSet with arbitrary templating on labels.\n- On the downstream tasks, other than audio tagging/classification tasks, language-based audio retrieval is another highly relevant task which provide different perspective to the audio-text pre-training, consider adding those audio-text benchmarks [2] to the evaluation to expand the coverage in this work.\n\n[1] Elizalde, Benjamin, et al. \"Clap learning audio concepts from natural language supervision.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.\n\n[2] Wu, Yusong, et al. \"Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable comments and feedback. Please refer to the attached PDF in the global response for supplementary results.\n\n__1. About the selected template__ Following the zero-shot performance of the strongest baselines in our experiments [1], we employ the template in Section 4.1 rather than simple concatenation. Please refer to Table 3 in [1] for a comparison of performance using different templates. Adopting this approach enables us to demonstrate that the performance improvement achieved by CoLLAT over CLAP is not attributed to the template selection. We will clearly cite [1] in the relevant section of the paper.\n\nWe made the decision to not utilize natural language descriptions or a keyword-to-caption augmentation [2] in favor of simple templates for the following reasons: \n* The cross-modal token-level loss function in CoLLAT aims to explicitly map each token in the given text prompt with its corresponding counterpart in the audio tokens (i.e., the patches in the Mel-Spectrogram of the audio). It may not be feasible to find such a mapping for certain tokens in natural language descriptions (e.g., stop words, adjectives) with the audio tokens. Consequently, having complex text prompts could adversely impact the training of CoLLAT, particularly given the greedy algorithm we utilized to compute $g$ for mapping audio tokens with text tokens.\n*  CoLLAT maintains the text encoder frozen during training, as it was pre-trained using text labels from the LAION dataset rather than natural language descriptions. Thus, attempting to produce text embeddings for natural language descriptions using the CLIP text encoder to train CoLLAT could introduce a data shift problem, which could also negatively impact the training of CoLLAT.\n* The scale of the selected training dataset, AudioSet, which consists of approximately 2 million audio and text label pairs, is significantly larger compared to datasets with audio and text captions. In fact, it is more than three times larger than the largest publicly available dataset of this kind, LAION-630K.\n\nBut we acknowledge that these points should be validated with sufficient evidence; however, due to time constraints, we leave this validation as future work.\n\n[1] Elizalde, Benjamin, et al. \"Clap learning audio concepts from natural language supervision.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.\n\n[2] Wu, Yusong, et al. \"Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.\n\n__2. Results for several additional tasks__ Please find the results for text-to-audio retrieval, audio-to-text retrieval, and audio captioning using the AudioCaps dataset in Figure 1.b and Table 1 in the supplementary PDF in the global response. In the latest results, CoLLAT outperforms the baselines (by approximately 20% in SPICE for audio captioning, 13.4% in MAP for audio-to-text retrieval, and 6.7% in MAP for text-to-audio retrieval), demonstrating a similar performance trend as the previously reported results in the main paper. \n\n__3. Ablations with different encoder choices__  We present the results with different choices of audio encoders in Figure 1.b of the supplementary PDF in the rebuttal. Our observations show a slight performance improvement with the AST backbone when using a pretrained initialization. Additionally, we noticed that the AST backbone architecture converges faster compared to the other backbones. This could be attributed to the stronger inductive biases present in the other encoders as compared to AST [3].\n\nDue to time constraints, we leave the performance evaluation with text encoders for future work. We anticipate the results to be consistent across different transformer-based architectures such as BERT, ROBERTA, and CLIP, provided they are pre-trained on the same contrastive objective as CLIP. However, we acknowledge that additional experiments are necessary to evaluate performance across different text encoders trained on various pre-training objectives (e.g., masked language modeling). Considering that most downstream applications, such as text-to-image generation, rely on the text conditioning from CLIP, we have chosen it as our main text encoder. This decision aligns with one of the primary motivations of this work, which is to cost-effectively introduce audio guidance to downstream applications that are built on text guidance from pre-trained text encoders. Please also note that the majority of baselines in our study adopt the same text encoder.\n\n[3] Gong, Yuan, Yu-An Chung, and James Glass. \"Ast: Audio spectrogram transformer.\" arXiv preprint arXiv:2104.01778 (2021).\n\n\n\n   \n\n"}]

Author rebuttal:
We thank all the reviewers for their valuable comments and feedback. Kindly find the attached PDF in this global response for additional results that support our rebuttal.


Meta-review: All reviewers have agreed that this paper has sufficient novelty and is worthy of acceptance.

This paper proposes an audio-language framework using pre-trained models and encourages fine-grained audio-text alignment. The authors conducted diverse experiments to justify the effectiveness. After all concerns were addressed by the authors (e.g. adding more ablation and more tasks for better justification), all reviewers voted to accept this paper. The authors also promised the reviewers to include more experiments in the camera-ready revision to make it more comprehensive.
Decision: Accept (poster)