SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rakshith S Srinivasa Jaejin Cho Chouchang Yang
Yashas Malur Saidutta Ching-Hua Lee Yilin Shen Hongxia Jin
Samsung Research America, Mountain View, CA
{r.srinivasa, jaejin.cho, c.yang1}@samsung.com {ym.saidutta, chinghua.l, yilin.shen, hongxia.jin}@samsung.com
This paper considers contrastive training for cross-modal zero-shot transfer wherein a pre-trained model in one modality is used for representation learning in another domain using pairwise data. The learnt models in the latter domain can then be used for a diverse set of tasks in a zero-shot way, similar to "Contrastive Language-Image Pre-training (CLIP)" [1] and "Locked-image Tuning (LiT)" [2] that have recently gained considerable attention. Most existing works for cross-modal representation alignment (including CLIP and LiT) use the standard contrastive training objective, which employs sets of positive and negative examples to align similar and repel dissimilar training data samples. However, similarity amongst training examples has a more continuous nature, thus calling for a more non-binary treatment. To address this, we propose a novel loss function called Continuously Weighted Contrastive Loss (CWCL) that employs a continuous measure of similarity. With CWCL, we seek to align the embedding space of one modality with another. Owing to the continuous nature of similarity in the proposed loss function, these models outperform existing methods for zero-shot transfer across multiple models, datasets and modalities. Particularly, we consider the modality pairs of image-text and speech-text and our models achieve  $5 - 8\%$  (absolute) improvement over previous state-of-the-art methods in 0-shot image classification and  $20 - 30\%$  (absolute) improvement in 0-shot speech-to-intent classification and keyword classification.
1 Cross-modal alignment and transfer
Learning visual representations using natural language supervision has proven to be a powerful way to unlock impressive zero-shot performance on a number of downstream tasks [1; 2; 3; 4; 5; 6]. In this paper, we draw inspiration from these works and study the task of cross-modal alignment for zero-shot transfer for pairs of modalities. Let  $\mathcal{U}$  and  $\mathcal{V}$  denote a pair of modalities. For example,  $\mathcal{U}$  may be text modality, and  $\mathcal{V}$  maybe image modality. We are interested in the following: given a pre-trained model  $f_{\theta}:\mathcal{V}\to \mathcal{Q}$  for data in  $\mathcal{V}$  (where  $\mathcal{Q}$  denotes the embedding space), how can we use a paired dataset of the form  $\{u,v\}, u\in \mathcal{U}, v\in \mathcal{V}$ , to best learn a model  $g_{\phi}:\mathcal{U}\rightarrow \mathcal{P}$  (where  $\mathcal{P}$  is the embedding space corresponding to  $\mathcal{U}$ ) such that the learnt structure in the embedding space  $\mathcal{Q}$  can be aligned with that of  $\mathcal{P}$ ? Once trained, the models  $g_{\phi}$  and  $f_{\theta}$  can be used on a diverse set of downstream tasks including interfacing with Large Language Models (LLMs) (see [7]) in a zero-shot way, thus avoiding the need for costly, task-specific, labeled datasets.
Our motivation in studying the above problem lies in the fact that powerful pre-trained models existing in certain modalities, but are lacking in other modalities. For example, the recent advances
Figure 1: Comparison of zero-shot transfer performance between baseline CL and proposed CWCL. (Left): zero-shot image classification accuracy measured across training epochs for the image-text modality pair. (Right): zero-shot speech-to-intent classification measured across training epochs for the speech-text modality pair. CWCL consistently performs better than CL.
in language models have resulted in very powerful models to process text data, while no such models exist for speech and audio data. Unlike text based models that can now generalize to new tasks in a zero-shot way, speech and audio models are still trained in a task-specific way (for example, automatic speech recognition (ASR)). Further, collecting labeled datasets in speech domain offers its own set of challenges including quality control, noise, removing silence to name a few [8; 9]. Similarly, even when pre-trained models are available for certain modalities such as images, there might be challenging sub-modalities (or domains) like medical imaging on which pre-trained models may not be trained on [10]. However, large scale paired datasets maybe available, which connect the above modalities. For example, large datasets of speech and the associated (possibly noisy) transcripts are available easily on the internet. Similar, pairs of text and images, pairs of medical and raw text [10] maybe more easily available. Based on this observation, methods have been proposed to train image and text encoders by aligning features corresponding to paired image and text data [1; 3]. Upon training, these models demonstrate impressive zero-shot performance on a number of downstream tasks such as image classification and image-text retrieval. While in these works both encoders are trained from scratch, authors in [2] showed that using a frozen pre-trained image classification model as the image encoder and only training the text encoder significantly boosts downstream zero-shot performance. We observe that this abstract concept of using a pre-trained model in one modality to supervise models in another modality using pairwise data can then be applied to any pair of modalities.
Our main focus in this paper is on how best to train such cross-modal models that leverage pretrained models in one modality. We find that standard contrastive loss used in training such models is inefficient at maximizing the amount of supervision that can be extracted from the pre-trained models. In particular, to learn the embeddings in the "unfrozen" modality, existing methods only use the embedding of the corresponding paired data from the other modality for supervision. However, there maybe many samples from the supervising modality that are similar, and to various degrees of similarity. To address this inefficiency, we propose a new loss function called continuously weighted contrastive loss (CWCL) for contrastive training of multi-modal models. The proposed loss function leads to better supervision and hence better alignment between the two modalities.
We study our proposed loss function using two pairs of modalities, image-text and speech-text. For image-text pair, we find that the proposed loss function leads to an improvement of  $6 - 8\%$  (absolute) compared to the best baseline on zero-shot image classification tasks. For speech-text, it leads to a  $20 - 30\%$  (absolute) improvement on zero-shot speech-to-intent classification and zero-shot keyword spotting tasks. Further, our models achieve performance comparable to models trained with supervision using task-specific datasets. As shown in Figure 1, we find that models trained using the proposed loss function are data and compute-efficient. They achieve higher accuracy with fewer pairs of data samples during training. Further, embeddings of downstream test datasets generated using our models show strong alignment among data that belong to the same class, even though the models have never been exposed to these datasets. We show an example in Figure 2, where embeddings extracted from speech signals in the SLURP test dataset show significantly improved sense of similarity for data from the same class, even though no label information was provided to the model.
2 Continuously weighted contrastive loss
2.1 Existing frameworks for contrastive training
Various forms of contrastive learning has been successfully employed in both self-supervised learning [11; 12; 1; 2; 13; 14] and in supervised learning [15].
Contrastive loss for self-supervised and multi-modal learning: The traditional contrastive loss
Figure 2: The similarity matrix between embeddings of the two modalities that are aligned via (Left): baseline CL and (Right): proposed CWCL. The axis labels correspond to the intent of utterances (for example, "news_query" represents utterances with news-related questions). CWCL results in a more "block" diagonal pattern than CL, indicating that speech and text samples with the same intent are more aligned while samples with different intents are more separated. This can be attributed to the continuous weighting mechanism of CWCL. Note that these embeddings are from a downstream test dataset which was never exposed to the model during training. The visualization confirms that CWCL leads to a higher degree of alignment between similar data samples.
function is used in both single-modality self-supervised learning as well as multi-modal alignment. We explain the formulation used in multi-modal alignment and briefly explain how the same function is used in the single-modality setting. Let  $\mathcal{B}$  denote a batch of training data consisting of pairs of data samples from two modalities of size  $N$ :  $\mathcal{B} = \{(u_i, v_i)\}_{i=1,\dots,N}$ , where  $u_i$  is from modality  $\mathcal{U}$  and  $v_i$  is from modality  $\mathcal{V}$ . Let  $u_i$  and  $v_i$  be encoded into embeddings denoted as  $p_i$ ,  $q_i$  respectively. This can be done by separate, modality-specific encoders or by shared encoder. Then, the traditional contrastive loss function (CL) (to align  $\mathcal{U}$  with  $\mathcal{V}$ ) is defined over  $\mathcal{B}$  as
$$
\mathcal {L} _ {C L, \mathcal {U} \rightarrow \mathcal {V}} = \frac {- 1}{N} \sum_ {i = 1} ^ {N} \log \frac {\exp \left(\left\langle p _ {i} , q _ {i} \right\rangle / \tau\right)}{\sum_ {j \in [ N ]} \exp \left(\left\langle p _ {i} , q _ {j} \right\rangle / \tau\right)}, \tag {1}
$$
where  $[N]$  denotes the set  $\{1,2,\dots ,N\}$ . Note that a similar loss function  $\mathcal{L}_{CL,\mathcal{V}\to \mathcal{U}}$  maybe defined and the total loss function is given as  $\mathcal{L}_{CL,\mathcal{U}\to \mathcal{V}} + \mathcal{L}_{CL,\mathcal{V}\to \mathcal{U}}$ . By minimizing (1), the encoders learn to align pairs of data. Note that in doing so, for each  $u_{i},v_{i}$  is considered as a positive example and all other samples  $\{v_i\}_{j\in [N],j\neq i}$  are considered to be negative examples. This is also illustrated in Figure 4, where the diagonal matrix indicates the set of positive examples chosen (for each row and column). As an example, in [1; 2], for each image, only the corresponding text is used as a positive example and all other text samples are used as negative examples (and vice-versa).
Contrastive loss for supervised learning: It is conceivable that in a given training batch, there is more than one "positive" sample. However the information about which samples are related to each other may be missing in self-supervised learning. However, this information is available in a supervised learning setup. Let  $\mathcal{T}$  denote a batch of training data of size  $M$  consisting of samples and labels:  $\mathcal{T} = \{(x_i,y_i)\}$ . Further, let  $z_{i}$  be the embedding generated by the model. Then, it is clear that the set  $\mathcal{P}_i = \{x_j,j\neq i|y_j = y_i\}$  forms a set of positive examples. This idea was explored in [15], where the following loss function was proposed to leverage the label information:
$$
\mathcal {L} _ {\text {s u p c o n}} = \frac {- 1}{M} \sum_ {i = 1} ^ {M} \frac {1}{| P (i) |} \sum_ {j \in P (i)} \log \frac {\exp \left(\langle z _ {i} , z _ {j} \rangle / \tau\right)}{\sum_ {k \in [ N ] , k \neq i} \exp \left(\langle z _ {i} , z _ {k} \rangle / \tau\right)}. \tag {2}
$$
Note that the above loss function can be interpreted as taking the average of pair-wise  $\mathcal{L}_{CL}$  over the positive set. The authors show that a combination of the above loss and the task loss yields better performance than using the task loss alone. However, this method requires labeled datasets.
<sup>1</sup>The authors also propose another variant of the supervised constrastive loss function. Although we do not discuss it here, it is similar in spirit to (2).
Figure 3: Existing contrastive learning methods treat samples in a batch as either strictly positive or negative. However, similarity between data samples has a more continuous and non-binary nature. In this figure, we provide an example of the nature of similarity in the context of paired image-text data. Note that the 'weight' terms in the figure are contrived for illustration purposes. The proposed CWCL loss function attracts all other data samples to a degree proportional to their similarity. Similarity itself is measured using intra-modal inner product between samples' embeddings.
In the above two loss functions and other similar variants studied in the literature, we find two shortcomings. Firstly, other similar examples that may be present in the training batch are not considered. In the self-supervised setting, all the other similar samples are considered as negative examples. In the supervised setting, some classes might be similar to each other (for example, multiple breeds of dogs), but are considered to be negative examples to each other. Secondly, similarity is considered to be binary. As a result, all "positive examples" are attracted equally, and all "negative examples" are repelled equally. However, we observe that samples in a training batch maybe similar to each other to varying degrees. Some samples might be more similar to each other, a few others less so many others may be dissimilar. For a more detailed explanation, see Figure 3.
2.2 Can we account for non-binary similarity?
To address the above shortcomings, we propose a novel loss function called Continuously Weighted Contrastive Loss (CWCL). We use the same setup as that in multi-modal training used to define (1). The loss function (to align  $p_i$  with other  $q_j$ 's) is defined as
$$
\mathcal {L} _ {\mathrm {C W C L}, u \rightarrow v} = \frac {- 1}{N} \sum_ {i = 1} ^ {N} \frac {1}{\sum_ {j \in [ N ]} w _ {i j} ^ {\nu}} \sum_ {j \in [ N ]} w _ {i j} ^ {\nu} \cdot \log \frac {\exp \left(\left\langle p _ {i} , q _ {j} \right\rangle / \tau\right)}{\sum_ {k \in [ N ]} \exp \left(\left\langle p _ {i} , q _ {k} \right\rangle / \tau\right)}, \tag {3}
$$
where  $w_{ij}^{\nu}$ 's denote the intra-modal similarity weights between  $v_i$  and  $v_j$  in modality  $\mathcal{V}$ . Note that a similar loss function to align modality  $\mathcal{V}$  with modality  $\mathcal{U}$  may be defined, with the intra-modal similarity weights computed between  $u_i$  and  $u_j$ . We will refer to the intra-modal similarity weights simply as weights for ease of usage and we will drop the superscript, unless the modality needs to be specified. Note that the weights are computed pair-wise, within each training batch.
Before we describe how these weights may be computed, we highlight the properties that they need to have. Firstly, we normalize the weights to be between 0 and 1:  $w_{ij} \in [0,1]$ . Secondly, "similar" samples from within a given domain should have higher weights and dissimilar samples should have lower weights. With these properties, note that  $\mathcal{L}_{\mathrm{CWCL}}$  provides a way to interpolate between the self-supervised and fully-supervised variants described earlier. When the weights are given as  $w_{ij} = \mathbb{1}_{\{i\}}(j)$  where  $\mathbb{1}_S$  denotes the indicator function w.r.t set  $S$ , it is equivalent to  $\mathcal{L}_{\mathcal{CL}}$ . On the other hand, in the supervised setting, if  $w_{ij}$  is defined as  $w_{ij} = 1$  for all pairs  $i,j$  belonging to the same class, but 0 otherwise, it is equivalent to  $\mathcal{L}_{\mathrm{supcon}}$ . More importantly,  $\mathcal{L}_{\mathrm{CWCL}}$  allows the model to use a continuous sense of similarity, by i) computing a softmax function for all pairs in the
Figure 4: The classical CL-based methods (e.g., CLIP [1], LiT [2], etc) can be interpreted as using a binary weight matrix for choosing the positive examples. The proposed CWCL utilizes a continuous weight matrix to account for the non-binary nature of similarity for improved alignment.
training batch (inner summation in Equation. (3)) and ii) weighting these softmax terms by the similarity weights. Further, note that all pair-wise inner products are already computed even in (1), (2). Therefore, computing  $\mathcal{L}_{\mathrm{CWCL}}$  is similar in computational complexity to  $\mathcal{L}_{\mathrm{CL}}$  and  $\mathcal{L}_{\mathrm{supcon}}$ .
2.3 How can we obtain intra-modal similarity weights?
In the traditional self-supervised setting, no information about similarity between training data points may be available. This might also be the case in multi-modal learning such as in[1], where the modality encoders are initialized randomly. However, authors in [2] explored the idea of using pretrained models as initialization in multi-modal models. Further, they find that freezing the pre-trained model (except maybe for a final linear layer) yields the best performance. This setup offers a natural way to obtain the similarity weights. We can measure the similarity between the embeddings from the pre-trained model. We focus on this setup, where we use frozen, pre-trained models for one modality to train models in another modality. Note that even though the model encoding the first modality is frozen, follow-up layers maybe added and trained. Let  $\mathcal{V}$  be the "frozen" modality with a pre-trained initialization. Then to align modality  $\mathcal{U}$  with  $\mathcal{V}$  using (3),  $w_{ij}^{\mathcal{V}}$  maybe computed as  $w_{ij}^{\mathcal{V}} = \langle q_i,q_j\rangle /2 + 0.5$  in order for  $w_{ij}\in [0,1]$ . We do not explore other formulations in this paper. A natural question is about how such weights can be computed for the modality  $\mathcal{U}$ . If the model in modality  $\mathcal{U}$  is also initialized using a pre-trained model, the weights may be computed in a similar way. However, in this paper, we only focus on the cross-modal transfer, with similarity weights being computed only in the frozen modality initialized with a pre-trained model. Assuming the modality  $\mathcal{V}$  is the frozen one, our loss function is given as
$$
\mathcal {L} _ {\text {c r o s s - m o d a l t r a n s f e r}} = \mathcal {L} _ {\mathrm {C W C L}, \mathcal {U} \rightarrow \mathcal {V}} + \mathcal {L} _ {C L, \mathcal {V} \rightarrow \mathcal {U}}. \tag {4}
$$
Note that the exact configuration and choice of which modality to freeze will depend on the pairs of modalities being considered, the quality of pre-trained models and paired datasets available.
CLIP like models: CLIP [1] and ALIGN [3] introduced a set of foundational vision-language models where the encoders, one for the image, another for the text modality output embeddings in a shared space. In this set of works the encoders for all the modalities are randomly initialized and trained from scratch. Other works have looked at extending the concept to other modalities like image-audio [16], others have explored richer embedding [17], adaptive prompt learning [5], architectural advancements like Mixture-of-Experts [6]. Some works also considered the problem setting where embeddings of both image and text are processed together so that specialized text query relevant image embeddings can be obtained [18; 19]. Another notable work, [4], the authors obtained impressive performance by improving individual encoders by processing image and text embeddings together to minimize caption-loss. Additionally, [20] proposed an extension to the cross-modal contrastive loss that can leverage labeled training data by combining it with SupCon [15]. Recent works such as [21; 22; 23] consider the alignment between images and text that do not belong to the same pair, similar to our proposed method. In [21; 23], both encoders are trained from scratch
by using a self-distillation process. Such a training process requires careful parameter tuning and generally has lower performance ([21] achieves about  $42.4\%$  0-shot on ImageNet) compared to using pre-trained models, as demonstrated by the metrics. Another difference between our work and the above works is that we consider intra-modal similarity to attract image-text pairs. Owing to the availability of strong pre-trained uni-modal models, intra-modal offers a clear way to identify similar data samples. In [22], the authors consider using a third, object detection model to obtain similarity between images. However, their method is specific to image-text modality pair. It may also lead to performance degradation, as seen in the zero-shot metrics reported.
LiT like models: Alternatively, LiT [2] proposed the idea of leveraging strong pretrained models in one domain and aligning the embeddings of another domain to the pretrained model's embedding space. Works in this line have looked at extending to multiple domains like image-audio-text [24], music-audio [25], speech-text for speech translation [26]; fine-grained query specific image embeddings [27] and benefits of cross-modal alignment for regularizing unimodal classifiers [28]. Along with building a model capable of conversation, [29] proposed the use of cross-modal attention layers to improve image-text cross-modal alignment. However, none of these works consider the problem of similarity across samples within the same modality that is explored in our work. Further, all these works are complementary to CWCL and can be improved by it. Handful of works explore similarity amongst samples [30; 13; 30] propose removing certain samples from the negative set used to compute contrastive loss (1) if their average similarity to the other samples in the batch is greater than a certain threshold [30]; [13] propose using a threshold on similarity to decide which samples are positive pairs and negative pairs and combining it with [15]. However, the above works still consider similarity as a binary entity, where as CWCL uses a continuous model.
Incorrect negatives in contrastive learning: Contrastive learning incorrectly assumes that for a given sample, every other sample in the dataset is dissimilar [31]. In the self-supervised learning one of the remedies proposed is to re-weight the negative part of the contrastive loss' denominator to account for the presence of similar (or positive) samples [32]. However, in the case of cross-modal alignment with pretrained models, the pretrained model is a better indicator of the similarity [30; 13].
In this section, we provide experimental results that demonstrate that CWCL leads to better zero-shot transfer performance. We study two pairs of domains, namely image-text and speech-text. For image-text pair, we demonstrate zero-shot transfer to image classification and image/ text retrieval. On both tasks, CWCL shows improved performance over existing methods for zero-shot transfer. Next, we report results for speech-text modality pair, where we consider the tasks of speech-to-intent classification and keyword spotting. Given the difficulties in collecting task-specific speech datasets, we expect CWCL-based zero-shot transfer to have a large impact in this domain. Note that our main goal is to study the effect of using CWCL. We use open source, publicly available and easily accessible datasets for our study and leave the task of training with larger datasets to future work.
4.1 Cross-modal transfer between image and text modalities
Model architecture: Our model architecture follows that in [2] and has a vision encoder and a text encoder. For the vision encoder, we use the ViT-L/16 model architecture [33] pre-trained on ImageNet. We compute the similarity weights using the embeddings from before the final linear layer that is not frozen during training. For the text encoder, we consider two architectures: transformer encoder architecture with 12 layers, output dimension 768, and number of heads set to 12 and we also consider the BERT-large architecture.