SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\{\tilde{h}_j^{a - t}\}_{j = 1}^N$  is the token-level embeddings of the perturbed version of  $x^a$  by adding a Gaussian noise. This loss forces the model to understand all the token-level embeddings (i.e., fine-grained concepts) of  $x^a$  using its perturbed version.
Overall Objective. The final objective function of CoLLAT is as follows:
$$
L = \lambda_ {1} \cdot L _ {\text {c r o s s - t o k e n}} + \lambda_ {2} \cdot L _ {\text {c r o s s - g l o b a l}} + \lambda_ {3} \cdot L _ {\text {u n i - t o k e n}} \tag {4}
$$
where  $\lambda_1, \lambda_2$  and  $\lambda_3$  control the importance of each loss term.
Table 1: Dataset Statistics
Task	Audio Classification	Cross-modal Retrieval	Audio Captioning	Audio-guided Image Gen.
Dataset	ESC-50	UrbanSound8K	TUT	FSD50K	AudioSet	VGGSound	AudioCaps	AudioCaps	AudioCaps
# instances	2K	8K	6.3K	51K	20.4K	15K	46K	46K	46K
# classes	50	10	15	50	527	309	N/A	N/A	N/A
Metric	Acc.	Acc.	Acc.	MAP	MAP	MRR	MAP@10	COCO Metrics	N/A
Training. We adopt the AudioSet dataset as the training dataset, which consists of 2,041,792 audio and text label pairs collected using 10s long YouTube videos. Each audio clip can have multiple text labels to represent different audio concepts in different granular levels.
Evaluation. We evaluate our model using four main downstream applications: (1) audio classification; (2) cross-modal retrieval; (3) audio captioning; and (4) audio-guided image generation. We adopt 6 widely used real-world datasets to evaluate our model, which are shown in Table 1.
Pre-processing. We preprocess audio clips by representing them as 128-dimensional log Mel filterbank (fbank) features, computed with a 25ms Hamming window every 10ms. This results in a  $128\times 100t$  spectrogram as input to our audio encoder, where  $t$  is the length of the audio clip in seconds. Next, we split the spectrogram into a sequence of N  $16\times 16$  patches with an overlap of 6 in both time and frequency dimensions, where  $N$  is the number of patches and the effective input sequence length for the Transformer. We flatten each  $16\times 16$  patch to a 1D patch embedding of size 768 using a linear projection layer, which we refer to as the patch embedding layer. Since the Transformer architecture does not capture the input order information, and the patch sequence is also not in temporal order, we add a trainable positional embedding (also of size 768) to each patch embedding to allow the model to capture the spatial structure of the 2D audio spectrogram.
Following the findings in [7], we adopt the template of "This is a sound of [class label 1], [class label 2], ... and [class label  $C$ " to generate a text prompt for an audio clip with  $C$  number of class labels. Such templates have been shown to be more effective than simple concatenation [7]. Our decision to not utilize natural language descriptions or a keyword-to-caption augmentation [36] in favor of simple templates is due to the following reasons: (1) the cross-modal token-level loss function in CoLLAT aims to explicitly map each token in the given text prompt with its corresponding counterpart in the audio tokens (i.e., the patches in the Mel-Spectrogram of the audio). It may not be feasible to find such a mapping for certain tokens in natural language descriptions (e.g., stop words, adjectives) with the audio tokens. Consequently, having complex text prompts could adversely impact the training of CoLLAT; and (2) CoLLAT maintains the text encoder frozen during training, as it was pre-trained using text labels from the LAION dataset rather than natural language descriptions. Thus, attempting to produce text embeddings for natural language descriptions using the CLIP text encoder to train CoLLAT could introduce a data shift problem to negatively impact the training of CoLLAT.
Training Details. After performing a grid search, we set  $\{\lambda_1, \lambda_2, \lambda_3\}$  as  $\{1, 0.1, 0.01\}$ . We utilize a GPU cluster comprised of 8 V100 GPU cards for training CoLLAT. With this hardware configuration, it takes approximately 320 GPU hours to train the model on AudioSet using the optimal hyperparameter setting.
5.1 Audio Classification
Table 2 presents a comparison of CoLLAT with five different baselines in audio classification. The first baseline, called Supervise, trains a transformer from scratch without relying on features from a pretrained joint embedding space, while the other four baselines learn a joint embedding space between audio and text via contrastive pretraining. To conduct our experiments, we used five widely used datasets: ESC-50, US8K, TUT, FSD50K, and AudioSet, under two settings: zero-shot (ZS) and linear probe (LP). For zero-shot (ZS) setting, we first extract the embeddings for the audio clips and the possible set of labels using the audio and text encoder in each baseline. Then, we compute cosine distance between these text and audio embeddings to rank different class labels for a given audio clip.
Table 2: Results for Audio Classification. Here, ZS and LP denote zero-shot and linear-probe experimental setups for audio classification. For uni-label datasets - ESC-50, US8K and TUT, Accuracy is reported as the metric. multi-label datasets (FSD50K and AudioSet) adopts MAP (Mean Average Precision). Higher the value is better for both metrics.
ESC-50 (Acc)	US8K (Acc)	TUT (Acc)	FSD50K (MAP)	AudioSet (MAP)
ZS	LP	ZS	LP	ZS	LP	ZS	LP	ZS	LP
Supervise	N/A	0.53	N/A	0.63	N/A	0.60	N/A	0.32	N/A	0.24
YamNet	N/A	0.85	N/A	0.78	N/A	0.63	N/A	0.50	N/A	0.27
Wav2CLIP	0.41	0.86	0.40	0.81	0.24	0.63	0.03	0.43	0.02	0.30
AudioCLIP	0.69	0.97	0.65	0.90	0.27	0.70	0.13	0.54	0.03	0.28
CLAP	0.77	0.96	0.73	0.88	0.30	0.72	0.14	0.59	0.06	0.32
CoLLAT	0.84	0.97	0.77	0.89	0.29	0.74	0.19	0.64	0.09	0.39
(a)
Figure 4: (a) Zero-shot audio classification performance of CoLLAT for ESC-50 and FSD-50K: solid line - when different intermediate layers are used to compute the token alignment loss (x axis denotes the half of the index of the layer used to compute the loss); and dashed line - when the number of cross-attention blocks in the token interaction module varies (x axis denotes the number of cross attention blocks). (b) Ablation study with different backbone audio networks using ZS audio classification. The pre-trained AST was pre-trained for audio classification using AudioSet.
Backbone Audio Encoder	ESC-50 (Acc)	FSD-50K (MAP)
ResNet [35]	0.80	0.14
CNN14 [17]	0.82	0.16
AST (Random Initialization) [10]	0.82	0.18
AST (Pretrained Initialization) [10]	0.84	0.19
For linear probe (LP) setting, we freeze our audio encoder as feature extractor and only train a 1-layer transformer decoder to predict the class labels.
Our results show that the Supervise baseline does not perform well across all datasets, which suggests that audio-text pre-training generally improves performance, particularly for tasks with limited labeled data (such as ESC-50). Among the other baselines, CoLLAT outperforms them mostly across all datasets. In particular, performance improvements from CoLLAT are significant for datasets that consist of complex audio clips with multiple sound clips such as AudioSet and FSD50K. For instance, CoLLAT outperforms the strongest baseline for the AudioSet dataset by  $50\%$  under zero-shot setting and  $22\%$  under linear probe setting. This observation highlights the superiority of CoLLAT in comprehending complex audio clips with multiple audio concepts.
Ablation Study using Zero-shot Audio Classification. In Fig. 4a, we provide ablations with different layer selections for token-level loss computation and the performance of CoLLAT with different number of cross-attention blocks. This ablation study guided to adopt the initial token-level layer in the text encoder to compute the token-level loss functions and 8 cross-attention blocks in the final architecture of the CoLLAT's token interaction module. Table 4b presents CoLLAT's performance with different audio encoder choices, which shows pre-trained AST as the most suitable backbone architecture. Additionally, we noticed that the AST backbone architecture converges faster compared
Table 3: (a) Ablation of loss functions using Zero-shot Audio Classification with clean and noisy audio samples; (b) Results for Cross-modal Retrieval, following the experimental setups in [35] and [36] for VGGSound and AudioCaps respectively. Higher is better for all the figures.
ESC-50 (Acc)	FSD-50K (MAP)
clean	noisy	clean	noisy
CoLLAT	0.84	0.59	0.19	0.17
(-)Lcross-token	0.81	0.50	0.13	0.11
(-)Luni-token	0.85	0.53	0.18	0.13
(-)Lcross-global	0.82	0.54	0.17	0.16
(a)
VGGSound (MRR)	AudioCaps (MAP@10)
A→I	I→A	A→T	T→A
Wav2CLIP	0.057	0.068	0.52	0.38
AudioCLIP	0.060	0.073	0.58	0.52
CLAP	0.063	0.074	0.57	0.47
CoLLAT	0.093	0.112	0.62	0.59
Table 4: Results for Audio Captioning using the AudioCaps dataset. The experimental setup freezes each model as a feature extractor and only train a 1-layer transformer decoder to predict text sequences. We follow the DCASE challenge setup [20] and report standard COCO caption evaluation metrics.
BLEU1	BLEU2	BLEU3	BLEU4	CIDEr	METEOR	ROUGE	SPICE
AudioCLIP	50.2	34.9	22.1	14.4	14.4	28.7	36.1	8.7
CLAP	48.4	31.7	20.4	13.4	13.7	30.2	35.1	8.9
CoLLAT	55.1	37.2	24.3	15.1	16.3	43.0	39.6	11.3
Human	65.4	48.9	37.3	29.1	28.8	91.3	49.6	21.6
Figure 5: Audio-guided Image Generation Results
to the other backbones. This could be attributed to the stronger inductive biases present in the other encoders as compared to AST [10].
We conducted an ablation study in Table 3a to demonstrate the significance of the loss functions used in CoLLAT. The proposed token-level cross-modal loss function,  $L_{cross - token}$ , is shown to play a crucial role in enhancing audio-understanding performance. This is especially evident in FSD-50K, which consists of audio clips with multiple labels, where the performance for zero-shot audio classification is improved by  $46.2\%$  in MAP. We also found that the robustness of CoLLAT substantially improves with the use of the unimodal-token level loss,  $L_{uni - token}$ , as it enhances the zero-shot classification performance for noisy audio samples that are generated by adding Gaussian noise with zero mean and 0.1 standard deviation.  $L_{uni - token}$  improves the performance for such samples by  $11.3\%$  and  $30.8\%$  for ESC-50 and FSD-50K, respectively.
5.2 Cross-modal Retrieval
Since pretrained CLIP text encoder maps text to a joint embedding space of images and text, being able to map audio into the same CLIP embedding space (without fine-tuning the text encoder) enables cross-modal retrieving capabilities between audio, text and images. Table 3b shows the results collected for cross-modal retrieval task using the VGGSound and AudioCaps datasets. We adopt the experimental setups proposed in [35] for the audio-image and image-audio retrieval tasks and [36] for the audio-text and text-audio retrieval tasks. For a fair comparison, we train all the baselines with a frozen text encoder.
As can be seen, CoLLAT outperforms all the baselines by as much as  $47.6\%$  and  $51.4\%$  in Mean Reciprocal Rank (MRR) for the Audio-to-Image and Image-to-Audio retrieval tasks, and  $13.4\%$  and  $6.7\%$  in MAP for Audio-to-Text retrieval, and Text-to-Audio retrieval, respectively. We observed that the performance improvements are particularly significant due to the ability of our approach to understand fine-grained concepts to differentiate hard image-text pairs in the VGGSound.
5.3 Audio Captioning
We report the results for Audio Captioning using the AudioCaps dataset in Table. 4. Following [35], we only trained a 1-layer transformer on top of the CoLLAT's output, while keeping CoLLAT's parameters fixed to collect the results for audio captioning. We adopt the same experimental setting for the baselines too. As can be seen, CoLLAT outperforms CLAP and AudioCLIP by  $9.7\%$ ,  $13.2\%$ ,
$42.4\%$ $9.7\%$  and  $26.9\%$  in  $BLEU_{1}$ ,  $CIDEr$ ,  $METEOR$ ,  $ROUGE$  and  $SPICE$  respectively. These results further show the ability of CoLLAT fine-grained details in the sound clip as it is needed to generate a holistic caption for a given audio clip.
5.4 Audio-guided Image Generation
Given CoLLAT's ability to produce corresponding text token-level embeddings for a given audio clip, it can be naively used to provide guidance for any downstream application built on top of text token embeddings from pretrained text encoders. We adopt text-guided image generation as one such example. Most existing text-guided image generation models adopt token-level embeddings from a pretrained text encoder to guide image generation using a given text prompt. Nevertheless, most existing audio-to-text grounding techniques cannot be naively extended to introduce audio guidance for such applications, as they are unable to generate token-level text embeddings for a given audio prompt.
To evaluate this task, we adopt a pretrained text-guided image generation model following [29], which requires text guidance from a pretrained CLIP text encoder. As a baseline, we adopt the same architecture as CoLLAT, but trained only using  $L_{cross - global}$ . We denote this baseline as CLAP in Figure 5. As shown in Figure 5, CoLLAT is capable of generating images that cover all the concepts present in the given audio clip. This observation further confirms the potential of our approach to comprehend complex audio clips with multiple concepts.
Our work has a significant impact as it addresses the limitations of conventional audio understanding models by enhancing their ability to predict finer-grained classes within complex audio clips. We demonstrated that CoLLAT achieves state-of-the-art performance in audio understanding, as validated by its success in various downstream applications. Furthermore, it unlocks the potential for audio guidance in applications built upon pretrained language models. Thus, our work opens up possibilities for improved audio-to-text grounding, advancing multimodal applications. Additionally, considering the capability of CoLLAT to represent audio clips using either text or image, it holds promise in enhancing accessibility and promoting inclusivity for individuals with diverse disabilities.
It is important to note that the model developed in this research does not possess the ability to uniquely identify individuals in audio recordings for tasks such as speaker recognition and speaker identification. CoLLAT does not possess the ability to comprehend information conveyed through speech in audio clips. Given the privacy concerns associated with the generation of human faces using audio-guided image generation models, it is advised to exercise caution and refrain from utilizing them for such purposes. It is crucial to prioritize privacy and ethical considerations when applying these technologies to avoid any potential misuse or infringement on individuals' rights.
This work proposes CoLLAT, a novel framework for learning an audio understanding model using natural language supervision from a locked pretrained language model. The proposed framework effectively exploits the implicit knowledge in large language models instead of just relying on their final global embeddings. We introduce a novel pretraining objective for CoLLAT that enforces token-level alignment between audio and text modalities to yield fine-grained audio understanding. Our results demonstrate that CoLLAT achieves state-of-the-art performance for downstream applications such as audio classification, cross-modal retrieval and audio captioning. Since CoLLAT keeps the pre-trained language model locked during training, our experiments also show that it unlocks new applications such as audio-guided image generation.
Since CoLLAT can be considered a modality-agnostic framework for learning knowledge understanding models for new modalities with natural language supervision, extending CoLLAT to new modalities (video) is an interesting future direction to explore. Another notable limitation of CoLLAT is its inability to understand speech signals in the audio clips. Therefore, it could be worthwhile to explore how to introduce speech signals to CoLLAT while preserving its existing capabilities.
[38] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self-supervision. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX, pages 588-604. Springer, 2020.
[39] Jianwei Yang, Yonatan Bisk, and Jianfeng Gao. Taco: Token-aware cascade contrastive learning for video-text alignment. In Proc. of CVPR, pages 11562-11572, 2021.
[40] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proc. of AAAI, 2022.