SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Amila Silva $^{1,2}$
Spencer Whitehead<sup>1</sup>
Chris Lengerich<sup>1</sup>
Hugh Leather<sup>1</sup>
Meta AI (FAIR)
2The University of Melbourne
amilas@unimelb.edu.au, {srw5,clengerich,hleather}@meta.com
CLAP
"A duck quacking followed by a dog whimpering while birds chirp in the background"
Top 5 zero-shot predictions in order
duck, wind, quack, fireworks, birds
duck, dog, quack, birds, whimper (dog)
audio-guided image generation results
Figure 1: CoLLAT yields fine-grained audio-to-text grounding by token-level alignment of audio-text with a locked pretrained text embedding space, which (a) enables fine-grained audio understanding and novel multimodal capabilities (e.g., audio-guided image generation); and (b) achieves superior performance across both unimodal (audio, text) and multimodal (e.g., audio and text) settings.
CoLLAT
ZS Audio Classification (Accuracy for ESC50)
(b)
Humans can easily understand various audio concepts, but conventional audio classification models fail due to their inability to predict unseen classes during training. To address this challenge, recent literature has explored contrastive language-audio pretraining to learn an audio understanding model using natural language supervision from a pretrained language model. However, despite their reasonable zero-shot performance in audio understanding, these models typically fail to achieve optimal performance while preserving the text understanding capabilities of the pretrained language model. They also perform poorly when comprehending audio clips with multiple audio concepts. To bridge these gaps, we propose CoLLAT: Contrastive Locked Language and Audio Tuning. This is a framework to effectively learn an audio understanding model with a locked language model, which is learned using a novel pretraining objective for audio-to-text grounding to yield fine-grained audio understanding. Our extensive experiments, which include several downstream applications such as audio classification, cross-modal retrieval, audio captioning and audio-guided image generation, demonstrate that CoLLAT yields state-of-the-art performance for audio understanding. Additionally, it unlocks audio guidance to applications built on top of pretrained language models.
Motivation. The sound perception system in humans is capable of understanding complex audio concepts and interpreting them in a way that allows us to interact with our environment effectively [21]. For example, when humans listen to a sound clip of a duck quacking followed by a dog whimpering while birds chirp in the background (see example in Fig. 1a), they can easily discern various audio concepts (e.g., a duck quacking, a dog whimpering, birds chirp) in that sound clip, enabling them to construct a holistic understanding of the sound clip. However, conventional audio classification models [22, 24, 30, 10, 25] that aim to associate audio recordings with one or more categories from a set of predefined categories fail to be competitive with the human auditory system due to their inability to predict unseen classes during training. To address this issue, recent literature has explored contrastive language-audio pretraining [13, 7] to learn an audio understanding model using natural language supervision. These methods aim to learn an audio encoder and a text encoder simultaneously, which produce a joint embedding space for audio and text. The joint embedding space is learned to preserve the correspondence of audio-text pairs present in the pretraining dataset. Such a joint embedding space creates an open vocabulary for audio-language correspondence, enabling new audio-language capabilities such as zero-shot audio classification and audio+language guided image generation. However, existing works on contrastive language-audio pretraining [13, 35, 7] lack the following strengths.
First, previous works [13, 35, 7] typically initialize their text encoder using CLIP [27] – a strong pretrained text encoder. However, these techniques [7, 13] often fail to achieve optimal performance for audio-language understanding without fine-tuning the text encoder (i.e., by keeping the text encoder frozen). In Fig. 1b, we report the performance of a set of baselines, where CLAP, AudioCLIP, and Cons-CLAP models fine-tune the pre-trained CLIP text encoder, while the rest keep the text encoder frozen. As can be seen, fine-tuning the text encoder improves cross-modal (e.g., zero-shot audio classification) performance, but it comes at the cost of losing its language understanding capabilities. This could be attributed to the lack of sufficient textual information in publicly available audio-text datasets (e.g., AudioSet [8]), which are relatively smaller in size compared to the datasets used to originally pretrain  $\mathrm{CLIP}^1$ . In this work, we study the possibility of achieving SOTA performance for audio-text understanding with a pretrained text encoder that remains locked. Incorporating audio understanding without tuning the CLIP text encoder is not only beneficial for capitalizing the pretrained text encoder's existing capabilities, but it also enables audio or audio+text guidance to any AI application that uses text-guidance from the CLIP text encoder (e.g., CLIP-guided image generation) without needing to retrain the application-specific model (e.g., diffusion model of a CLIP-guided image generation pipeline). Recent studies on language-visual pretraining [3, 4] have demonstrated that matching the sizes of the encoders is vital for achieving strong cross-modal performance while keeping one encoder locked. Nevertheless, most existing audio-text pretraining methods use audio encoders that are substantially smaller in size compared to the text encoder. Building upon this observation, we propose an audio-text pretraining architecture that jointly scales both the audio and language encoders, allowing for strong cross-modal performance with locked language tuning.
Second, existing contrastive audio-text pretraining techniques [35, 7, 13] have limited efficacy when understanding complex audio clips with multiple audio concepts (see results in Table 2). This shortcoming could be attributed to the absence of explicit fine-grained cross-modal grounding in these techniques, as they solely depend on global embeddings – i.e., a single vector summarizing the semantic content of a given text/animal input, to maintain the correspondence between audio and text. Although such global embedding alignment methods achieve reasonable performance in audio understanding, encoding audio and text into global embeddings will lose much fine-grained information, which is critical to distinguish hard audio-text pairs. In contrast, the token-level embeddings of the intermediate layers of transformer-based encoders for text and audio possess better understanding of fine-grained concepts as they have one or more token-level embeddings dedicated to each fine-grained concept in an audio/text input. Despite the knowledge in such token-level embeddings are explicitly exploited in other domains for fine-grained cross-modal grounding (e.g., text-image [34, 37] and text-video [39]), to the best of our knowledge, none of the previous audio-text pretraining techniques have explored this. To bridge this research gap, this work studies the importance of exploiting token-level audio-text alignment along with the global-level alignment to yield better fine-grained understanding of audio-text correspondence.
<sup>1</sup>CLIP's original pretraining dataset is almost 200 times larger than AudioSet.
Contributions. The contributions of this work are to propose:
- A neural architecture for effective contrastive audio-language pretraining that yields SOTA performance for audio-text understanding while keeping the text encoder locked.
- An improved audio-text pretraining objective function that explicitly encourages the model to learn fine-grained audio-text grounding to unlock complex audio comprehension.
We verify the superiority of our proposed framework using a diverse set of experiments - zero-shot audio classification, cross-modal retrieval, audio captioning and audio-guided image generation, which shows that our framework yields SOTA performance for both unimodal (audio or text) and multimodal (audio and text) understanding, while capitalising on the existing capabilities of the text encoder (e.g., text-guided image generation) and alleviating its substantial cost of re-training.
2.1 Audio-to-Text Grounding
Audio-to-text grounding refers to establishing a connection between textual concepts and audio concepts. In earliest research efforts [22, 24, 30, 10, 25], the audio-to-text grounding problem was approached as a classification task that utilized machine learning models to link audio recordings to pre-defined class labels. These approaches explored various machine learning models, spanning from traditional machine learning models such as Support Vector Machine [19, 31, 32], to advanced neural architectures like Convolutional Neural Networks (CNN) [26, 23, 14] and transformers [10, 18]. These models were designed to operate on either static [22, 24, 30] or trainable [12] time-frequency transformations of raw audio. While these approaches were successful in predicting the class labels used during training, they often struggled to predict novel audio concepts.
To address this challenge, recent works [35, 13, 7, 5] have explored that how to learn a joint embedding space to preserve the correspondence between audio and text with the language supervision from pretrained text encoders. This approach enables an open vocabulary for audio concepts to unlock applications such as zero-shot audio classification. These works typically employ two separate encoders: one for audio and one for text. Each encoder maps the given input, such as an audio clip or text prompt, to an embedding vector that preserves the knowledge of the input. These models learn the parameters of the encoders using a contrastive loss function [27], which aims to ensure that the embeddings of corresponding text-audio pairs are similar to each other while pushing the embeddings of dissimilar pairs further apart.
To exploit the language comprehension abilities of large pretrained text encoders [27, 28], these studies often initialize the text encoder with a pretrained model like CLIP [27]. These text encoders typically have billions of parameters and are trained using large-scale datasets such as LAION. For the audio encoder, different works adopt various architectures - Wav2CLIP adopts ResNet-based architecture [1], AudioCLIP employs ESResNeXt [12], and CLAP uses CNN14 [17]. Then, these works jointly train the audio and text encoder to preserve audio-text correspondence. In addition to the differences in audio encoder architecture, these works vary in how they construct text prompts and their use of auxiliary objectives. For example, AudioCLIP [13] creates text prompts by concatenating the discrete class labels of an audio clip from datasets like AudioSet [9]. On the other hand, CLAP [7] utilizes a set of audio-text datasets that provide semantically and syntactically meaningful captions for each audio clip to train their model. Additionally, AudioCLIP [13] incorporates auxiliary objectives to preserve audio-image and image-text correspondences along with the audio-text correspondence.
Despite these differences, all these works fail to yield optimal performance for audio-text understanding without fine-tuning the text encoder – i.e., keeping the weights of the pretrained text encoder. Previous studies [4, 3] suggest that this may be due to the size mismatch of the audio and text encoders observed in previous works [7, 13] (see Figure 2a). As shown in Fig. 2b, simply scaling the audio encoder to match the size of the text encoder does not result in optimal performance too, especially given the relatively small dataset available with audio-text pairs. To address this issue, recent studies [3, 4] in other multimodal domains propose the use of neural architectures [4], that enable joint scaling of the sizes of multimodal encoders. Nevertheless, to the best of our knowledge, such architectures have not been explored in the context of audio-text understanding.
To bridge this gap, our work proposes a novel neural architecture for audio-text pretraining that solves the size mismatch problem in existing works, enabling our model to achieve optimal performance for
Figure 3: (a) Overview of CoLLAT, consisting of the Token Interaction Module for mapping audio tokens to text tokens by adopting a series of blocks as detailed in (b).
Figure 2: (a) Size ratio of audio and text encoders across different methods; and (b) zero-shot audio classification performance of CLAP (using FSD50K) with different audio encoder sizes, while keeping the size of the text encoder fixed to that of CoLLAT.
audio-to-text understanding without the need for fine-tuning the text encoder. The strengths of our architecture is three-fold: (1) it preserves the language understanding capabilities of the pretrained text encoder, resulting in better audio-to-text grounding even with a relatively small audio-text dataset; (2) it is compatible with text encoders of any scale, making it more viable for learning audio understanding using evolving language models; and (3) it maps audio embeddings into the same representation space as the pretrained text encoder, introducing audio-guided controllability to any downstream application that relies on text guidance from the pretrained text encoder.
2.2 Token-level Grounding
Existing audio-text contrastive pretraining techniques [35, 7, 13] do not explicitly consider the fine-grained alignment of audio concepts and textual concepts. These methods are only trained to align the global embeddings (i.e., last layer output from the encoders which summarizes all the concepts in the corresponding input) from the encoders. Such global embeddings may not be able to capture the individual concepts in complex audio clips containing multiple audio concepts. In contrast, the token-level hidden representations of audio/text encoders can effectively represent fine-grained concepts. Therefore, it is crucial to have token-level alignment between audio and text to achieve a more fine-grained understanding of audio.
This research challenge has been deeply studied in text-visual domain. For example, the works in [39, 2, 34, 37] have shown the importance of aligning the token-level hidden representation of the visual and text encoders to yield fine-grained visual understanding. The same concept has also been shown to be important for similar other learning tasks such as knowledge distillation [38, 33]. However, token-level contrast between audio and text modalities has not been well-studied in previous audio-text pretraining methods. Some works [16, 11] have focused on explicit token-level attention for audio-related downstream applications like audio captioning, but they are limited to specific tasks. Therefore, our work introduces token-level alignment as an auxiliary task for conventional contrastive audio-text pretraining to yield fine-grained audio understanding.
3 CoLLAT
Our model aims to learn fine-grained audio understanding with the help of a pretrained text encoder, without fine-tuning the text encoder. Unlike existing techniques that use separate audio and text encoders, our model shares the pretrained text encoder, which remains frozen during training, for both audio and text encoding. The main strengths of this architecture are twofold: (1) it allows us to use large pretrained language encoders (with billions of parameters) for the text encoder without introducing a significant mismatch between the encoders used for audio and text, as this architecture scales the encoder size for each modality jointly; and (2) it enables us to leverage the implicit knowledge (i.e., intermediate layers) in the text encoder to encode audio. By using a frozen shared text encoder for both audio and text encoding, the corresponding audio-text pairs are forced to implicitly align their token-level embeddings to produce similar global embeddings. Figure 3 (a) shows a high-level schematic of the proposed model architecture.
We reuse the pretrained CLIP text encoder [27] as our text encoder, but our model is compatible with any other pretrained text encoder at any scale. This text encoder expects the static token embeddings of the given text prompt as input. To encode audio in a way that is compatible with the pretrained text encoder, we adopt a transformer-based audio encoder [10] to produce token-level embeddings for the audio tokens (i.e., patches in the log mel spectrogram) and a cross-attention-based token interaction module to produce corresponding text token embeddings from the audio token embeddings. As shown in Fig. 3 (a), our model takes either audio and/or text as inputs and produces corresponding embeddings while preserving audio-text correspondence. We formally define our architecture and training objectives below.
Let  $\mathbb{D} = \{X^a, X^t\}$  be a set of audio-text pairs, where each pair is represented as  $\{x^a, x^t\}$ . Here,  $x^a \in \mathbb{R}^{F \times T}$  is the processed audio clip represented as a log mel spectrogram with  $F$  spectral components (e.g. Mel bins) and  $T$  time bins; and  $x^t$  is the tokenized text prompt explaining the audio concepts in  $x_i^a$ .
3.1 Model Architecture
For a given  $\{x^a,x^t\}$ , we preprocess  $x^{a}$  as a sequence of patches of its 2D audio spectrogram and  $x^{t}$  as a sequence of tokens by tokenizing the text prompt via the pretrained BPE-based tokenizer in CLIP. We denote these preprocessing functions for audio and text as preprocessa() and preprocesst(), respectively, and the output from these functions as  $\{x_i^a\}_{i = 1}^M$  and  $\{x_{j}^{t}\}_{j = 1}^{N}$ , respectively.
$$
\left\{x _ {i} ^ {a} \right\} _ {i = 1} ^ {M} = p r e p r o c e s s _ {a} \left(x ^ {a}\right), \left\{x _ {j} ^ {t} \right\} _ {j = 1} ^ {N} = p r e p r o c e s s _ {t} \left(x ^ {t}\right)
$$
Here,  $M$  and  $N$  denote the maximum lengths of the audio and text sequences, respectively. The audio tokens  $\{x_i^a\}_{i = 1}^M$  are embedded using a transformer-based audio encoder [10]  $f_{a}(.)$  , which returns latent representations of the audio tokens  $\{h_i^a\}_{i = 1}^M$  . The text tokens  $\{x_j^t\}_{j = 1}^N$  are initially embedded using the static embedding matrix  $E_{t}$  of the pretrained text encoder.
$$
\{h _ {i} ^ {a} \} _ {i = 1} ^ {M} = f _ {a} (x ^ {a}), \{h _ {j} ^ {t} \} _ {j = 1} ^ {N} = E _ {t} \cdot x ^ {t}
$$
The audio token embeddings are then passed through an audio-text token interaction module  $f_{a - t}()$ , which aims to predict the corresponding text token embeddings  $\{h_j^t\}_{j = 1}^N$  using the audio token embeddings  $\{h_i^a\}_{i = 1}^M$  through fine-grained token alignment. The output from this module is denoted as  $\{h_j^{a - t}\}_{j = 1}^N$  since it has the same number of tokens as the text token embeddings.
$$
\left\{h _ {j} ^ {a - t} \right\} _ {j = 1} ^ {N} = f _ {a - t} \left(\left\{h _ {i} ^ {a} \right\} _ {i = 1} ^ {M}\right)
$$
Our token interaction module consists of multiple cross-attention-based blocks [37] (see Fig.3 (b)) in series. These blocks take the currently predicted text token embeddings from the previous block as input and attempt to denoise them using the audio token embeddings of the corresponding audio clip. For the first block in the interaction module, we pass a set of random vectors as input, which are randomly initialized following a normal distribution with a mean of zero and a standard deviation of 1. This series of token interaction blocks is analogous to a denoising pipeline[15] that is guided by the audio token embeddings to produce the corresponding text token embeddings.
Finally, the global audio and text embeddings are produced by encoding  $\{h_j^t\}_{j=1}^N$  and  $\{h_j^{a-t}\}_{j=1}^N$  using the pretrained text encoder  $f_t(\cdot)$ .
$$
z ^ {a} = f _ {t} \left(\left\{h _ {j} ^ {a - t} \right\} _ {j = 1} ^ {N}\right), z ^ {t} = f _ {t} \left(\left\{h _ {j} ^ {t} \right\} _ {j = 1} ^ {N}\right)
$$
Our framework is trained using the following objective functions:
3.2 Cross-modal Token-Level Alignment Loss
To achieve the fine-grained grounding between audio and text while making CoLLAT agnostic to the order of the classes presented in the text, this loss function aims to find one-to-one mapping between text token embeddings  $\{h_j^t\}_{j = 1}^N$  from text and the produced text token embeddings  $\{h_j^{a - t}\}_{j = 1}^N$  using audio and make those token-level embeddings close to each other. We formulate the cross-modal token-level loss function as follows:
$$
L _ {c r o s s - t o k e n} = \left\| g \left(\left\{h _ {j} ^ {a - t} \right\} _ {j = 1} ^ {N}\right) - \left\{h _ {j} ^ {t} \right\} _ {j = 1} ^ {N} \right\| _ {1} \tag {1}
$$
where  $g$  denotes the function that reorder the token-level embeddings of an instance  $\{h_j^{a - t}\}_{j = 1}^N$  such that the cross-modal token-level loss function is minimized, which is pre-computed before each weight update. Given the time complexity of finding the optimal solution to  $g$  in Eq. 1, we greedily compute  $g$  as follows: given the token embeddings of  $\{h_j^t\}_{j = 1}^N$  and  $\{h_j^{a - t}\}_{j = 1}^N$ , we initiate the process by starting from the right-most token embedding in  $\{h_j^t\}_{j = 1}^N$ . This token is then paired with the closest token embedding in  $\{h_j^{a - t}\}_{j = 1}^N$ . Next, we proceed to the second token embedding from the right in  $\{h_j^t\}_{j = 1}^N$ , excluding the already mapped token embedding in  $\{h_j^{a - t}\}_{j = 1}^N$  from the possible set to be paired with the selected text token embedding. We iterate this process until we obtain the greedy one-to-one mapping between all the tokens in  $\{h_j^t\}_{j = 1}^N$  and  $\{h_j^{a - t}\}_{j = 1}^N$ . We observed that this greedy approach could produce optimal re-ordering unless there are very similar classes in the same audio clip. Consequently, we do not anticipate a significant variation in the performance if the optimal reordering can be achieved, which we leave as future work given the time complexity of training the model with the optimal reordering.
3.3 Cross-modal Global Contrautive Loss
Motivated by [27], this objective function is introduced to preserve the global correspondence of the audio-text pairs in the training dataset. Given the global embeddings of a batch  $B$  of audio and text pairs  $\{Z_B^a, Z_B^t\}$ , we compute the similarity metric between audio and text pairs in  $B$  as:
$$
S _ {B} ^ {a - t} = \tau * (Z _ {B} ^ {a} \cdot Z _ {B} ^ {t} ^ {T}), S _ {B} ^ {t - a} = \tau * (Z _ {B} ^ {t} \cdot Z _ {B} ^ {a} ^ {T})
$$
where  $\tau$  is a temperature parameter to scale the range of logits. The similarity matrix  $S\in \mathbb{R}^{|B|\times |B|}$  has  $|B|$  correct pairs in the diagonal and  $|B|^2 -|B|$  incorrect pairs in the off-diagonal, where  $|B|$  refers to the batch size. We formulate our cross-modal global contrastive loss for the batch  $B$  as follows:
$$
L _ {c r o s s - g l o b a l} = \frac {1}{2 | B |} \sum_ {i = 0} ^ {| B |} \log \left[ \operatorname {s o f t m a x} \left(S _ {B} ^ {a - t}\right) _ {(i, i)} \right] + \log \left[ \operatorname {s o f t m a x} \left(S _ {B} ^ {t - a}\right) _ {(i, i)} \right] \tag {2}
$$
3.4 Unimodal Token-Level Alignment Loss
Despite the reasonable performance of CoLLAT with the aforementioned two objective functions, we observed that the learned representations are not robust against weak perturbations of audio such as jitter (see ablation study in Section 5 for more details). To address this challenge, previous works [6, 40] adopts self-supervised objectives that aim to maximize the similarity of the embeddings of two different yet correlated views produced by perturbing the same audio input. Motivated by these works, we formulate our unimodal token-level loss for a given audio-text pair  $\{x^a, x^t\}$  to improve the robustness our framework as follows:
$$
L _ {u n i - t o k e n} = \left\| \left\{h _ {j} ^ {a - t} \right\} _ {j = 1} ^ {N} - \left\{\tilde {h} _ {j} ^ {a - t} \right\} _ {j = 1} ^ {N} \right\| _ {1} \tag {3}
$$