SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets for contrastive training: All our experiments are based on the combination of two publicly available datasets, CC12M and YFCC15M. The CC12M dataset is a subset of the Conceptual Captions dataset [34] defined in [35]. We use a set of 10 million images that are still available in the set of URLs (since the rest of them have been taken down). The YFCC15M dataset is a subset of the Yahoo Flicker Creative Commons dataset [36] defined by [1] by filtering for high quality English text. It contains a set of 15 million image-text pairs. Model training details are provided in A.1.
4.1.1 Zero-shot image classification
For zero-shot image classification, we experiment on 5 datasets: ImageNet [37] validation, ImageNet-V2 [38], ImageNet-R [39; 40], ImageNet-A[41] and ObjNet [42], similar to [2]. We provide our
experimental results in Tables. 1, 2. The results for SimCon [13], and LiT [2] are obtained from our own experimentation. For [13], we use their loss function in our set up. For [2], we use their recommended experimental settings from their paper. Note that the metrics are obtained by using the same model architecture and dataset for all the methods being compared. CWCL yields a significant boost over the other methods in zero-shot performance. Further, as shown in Figure1, CWCL achieves higher accuracy with fewer image-text training pairs. Owing to the CWCL formulation, the text embeddings generated by our model are designed to be similar to a larger set of similar images than the baseline methods, hence leading to better generalization.
Table 1: Zero-shot image classification performance using the ViT-L/16 + 12-layer transformer configuration. CWCL achieves a significant improvement in zero-shot image classification across multiple datasets, including out-of-domain datasets such as ObjectNet.
Method	ImageNet (%)	ImageNet-V2(%)	ImageNet-R(%)	ImageNet-A(%)	ObjNet(%)
CLIP	31.3	-	-	-	-
OpenCLIP	34.8	30	-	-	-
SimCon	67.9	58.57	59.32	37.16	44.9
LiT	66.84	58.82	61.28	37.31	45.08
CWCL (Ours)	74.41	66.25	67.37	45.58	50.5
Table 2: Zero-shot image classification using the ViT-L/16 +BERT-large configuration. CWCL-based training achieves state-of-the-art (when trained on publicly available datasets) performance on all of zero-shot experiments.
Method	ImageNet (%)	ImageNet-V2(%)	ImageNet-R(%)	ImageNet-A(%)	ObjNet(%)
LiT	71.2	62.98	63.8	40.28	48.1
CWCL (Ours)	76.48	67.86	68.7	47.27	52.38
4.1.2 Zero-shot Image-text retrieval
We also examine the zero-shot image-text retrieval capabilities of our proposed method. Note that our experiments are only towards comparing standard contrastive loss with CWCL. We leave the task of training with larger datasets [1; 2; 3] and using multi-objective training (which maybe used along with contrastive tuning to obtain better retrieval performance) [34; 29; 19] for future exploration. In our experiment, we simply compare the performance of models trained with contrastive loss (as done in [2]) to that of models trained using CWCL. We use the MS-COCO validation dataset [43] to study zero-shot retrieval performance of these models. We report our results in Table 3. Retrieval metrics for the ViT-L/16+12 layer transformer configuration model are provided in Table 6 in the Appendix. Models trained with CWCL outperform those trained using the standard contrastive loss function.
Table 3: Zero-shot retrieval results on MS-COCO dataset by using ViT-L/16+BERT-large configuration as the image and text encoders respectively.
Method	I →T retrieval	T→I retrieval
R@1	R@5	R@10	R@1	R@5	R@10
LiT	34.58	59.78	70.68	28.49	54.04	65.87
CWCL (Ours)	40.36	66.62	77.76	30.04	54.84	66.06
4.1.3 Robustness to templates for zero-shot classification
An added benefit of the proposed CWCL formulation is that our model is robust to the templates/prompts used in zero-shot tasks. In zero-shot image classification, the labels are converted to text prompts in order to adapt the task of classification to that of alignment. In particular, both [1; 2] use a set of 80 "template" sentences to convert each label into 80 sentences, extract the text embeddings for all the sentences and use their mean embedding as the representation of the corresponding class. We expect that CWCL leads to robustness w.r.t the choice of such templates or prompts. We study this
Figure 5: Comparison of robustness to templates. (Left): the baseline CL method of LiT [2]. (Right): the proposed CWCL approach. The number displayed at each bar reflects the decrease in accuracy due to using only a subset of templates compared to using the full set.
by changing the number of template sentences used to build the classifier embeddings. In particular, we design simple templates such as "this is a photo of ", and "this is an image of " and experiment over  $k = 1,5,10$  templates. We provide further details on the templates in Section A.1.2. We report the results for our model and that of [2] in Figure 5. As can be seen, models trained using CWCL are able to obtain peak performance with fewer number of templates, whereas models trained using standard contrastive loss require a higher number of templates to build better classifier embeddings. We believe that robustness to the choice and the number of template sentences/prompts used is crucial to improve the ease of usage of such models.
4.2 Cross-modal transfer between speech and text modalities
The proposed method can be applied to speech-text cross-modal learning to transfer semantic knowledge from text embeddings to speech embeddings. Speech models with language understanding are desired for tasks in the field of spoken language understanding (SLU) [44; 45]. SLU differs from automatic speech recognition (ASR), which simply generates a transcription, but does not have language understanding. SLU models, unlike ASR models can then be used on a wide variety of downstream tasks such as intent classification (in multiple domains) [8], keyword spotting [46].
In general, speech pre-training schemes usually include information about the phonemes or paralinguistic information (e.g. speaker, emotion, pathology, etc.), but they do not include semantics in language. While some works have explored the usage of contrastive learning to train SLU models, they use the standard contrastive training method [47]. However, similar to the image-text case, this may not be efficient. For instance, "Turn on the volume", is closer in meaning to "Increase the sound" than "Set an alarm for 7:00 AM tomorrow morning". In this case, the standard cross-modal contrastive loss is unable to learn the cross-modal relationship between the text of the first sentence and the speech of the second sentence since they are considered to be a "negative pair". This is precisely what is addressed by CWCL. As we demonstrate later, CWCL achieves a significant boost in performance on downstream tasks.
We train a speech-text multi-modal model with a dataset where speech and its corresponding transcript are available. Note that this is a generic dataset that is not specific to any SLU task. We use a pretrained, frozen text encoder, owing to the availability of strong pre-trained language models. A trainable linear layer is added on top of the frozen text encoder to match the dimensionality of the speech and text embedding. We also use a pre-trained speech encoder that is robust to diverse acoustic condition and further train it using the proposed loss function.
Model architecture: For the speech model, we used the encoder of the pre-trained Whisper ASR [48], which is expected to be robust to different acoustic conditions. For the text models, we found 49 publicly available hugging face models by searching with filters as, task: Zero-shot classification, libraries: Transformers, and languages: English. We manually added one RoBERTa-based model fine-tuned on MASSIVE [49] data. All 50 models were compared on zero-shot text-to-intent classification using the SLURP dataset [8] and the top 2 models were selected. The best model (we call it RoBERTa+S) was the RoBERTa-based model fine-tuned on MASSIVE data since the data includes SLURP (only the text data)  $^{2}$ . The second model was a BART-based model fine-tuned on Yahoo Answers topic classification (we call it BART+Y)  $^{3}$ .
$^{2}$ https://huggingface.co/qanastek/XLMRoberta-Alexa-Intents-Classification
<sup>3</sup>https://huggingface.co/joeddav/bart-large-mnli-yahoo-answers
Datasets for contrastive training For cross-modal training, we used the Common Voice Corpus 13.0 [50]. This dataset consists of roughly 2400 hours of speech data and the corresponding transcripts obtained using crowd-sourcing and includes speech from a diverse set of demographics across age and gender. We use the English subset. Model training details are provided in A.2.
4.2.1 Zero-shot speech-to-intent classification
After the cross-modal embedding alignment stage, we evaluated the models on the zero-shot speech-to-intent classification task. The task is to classify a given speech sequence into one of the intent classes. The main difference between the zero-shot and supervised intent classification is the zero-shot classification can be done without training a classifier.
Class embedding generation: Similar to the image-text case, we compute the embeddings of a given speech signal and compute its similarity with the text embeddings for all the intent classes. These class embeddings are obtained as averaged embedding of text sentences' embeddings of the corresponding classes. During inference, the class embedding that has the highest similarity score with the input speech embedding is chosen as the predicted class.
Dataset: We used the SLURP [8] and STOP [51] datasets for evaluation. In the SLURP dataset, we used all the text sentences in the train subset to generate the class embeddings for 60 intent classes where intent is defined as the concatenation of scenario and action labels, following ESPnet-SLU [52]. We did not use the train_synthetic subset since more than half of the text sentences overlap with the devel and test subsets. On average, 191 text sentences were used per class. We compare the systems by evaluating them on the devel and test subsets. In the STOP dataset, we used the 8 unique domain labels as intent labels. Although not intents in a strict sense, the domain labels can be considered a simpler version of the intent labels. Since significantly more sentences are available in STOP, we randomly extracted 200 sentences per domain from the training set to generate the class embeddings. The evaluation was done on the validation and test sets.
Results: In previous works, speech-to-intent classification has been done with an ASR-NLU pipeline system where the speech is first transcribed by ASR (speech-to-text), after which the transcription is classified into an intent using NLU (text-to-intent) [8]. We refer to the text-to-intent performance achieved by the pre-trained text encoders as the "reference" performance. This provides an estimate of the performance that can be expected from the speech encoder on the speech-to-intent task.
The first results of speech-to-intent classification are shown in the SLURP and STOP (without the superscript  $\#$  ) columns in Table 4. In all cases, multi-modal training with the CWCL loss outperformed the CL loss. On the SLURP dataset, RoBERTa+S has a higher reference performance compared to BART+Y because the fine-tuning data for RoBERTa+S included the SLURP text data. This also leads to a better performance compared to using the BART+Y model as the text encoder.
On the STOP dataset, RoBERTa+S has a lower reference compared to BART+Y, implying that the RoBERTa+S' text model overfits the SLURP data. However, the RoBERTa+S-based speech intent classification was still better than the BART+Y-based one. This implies that the text model architecture could be another factor that contributes to transferring performance to the speech model. To be specific, the RoBERTa+S was RoBERTa which consists of only encoder layers while the BART+Y was the encoder-decoder-based BART model. Another thing to note is that CWCL with RoBERTa+S outperforms the text-to-intent reference performance on the STOP (87.87 vs. 84.78) dataset. This is because, during the cross-modal alignment stage using CWCL, the speech tower might have learned how to utilize acoustic cues in addition to linguistic information from a given speech utterance, to align its embedding to the semantic embedding from the text tower. However, this did not happen in the case of SLURP, because the SLURP dataset includes more intent classes than STOP (60 vs 8 classes), thus being more challenging in transferring knowledge from text to speech during the cross-modal alignment stage.
Experimenting with different templates to generate text embeddings: So far, each class embedding used in zero-shot intent classification was generated by averaging all the corresponding text sentences' embeddings from the class in the training subset. Although collecting the text data with the intent labels can be less expensive than collecting speech data with the intent labels, the former may not always be possible. To address this, we manually devised a fixed set of general templates that were applied to every class. For example, templates are of the form "This audio is about [class]", and "The utterance is related to [class]", and the text embeddings are averaged to obtain the class embedding. For the exact templates we used, readers may refer to Appendix A.2.5. The results are
Table 4: Top-1 accuracy for zero-shot speech-to-intent classification (SLURP and STOP) and keyword spotting (GSCV2) after thick vertical line. Superscript  $\#$  is used to indicate use of general templates for class embedding extraction. Supervised results are provided in gray after the double-horizontal line: [52] is for speech-to-intent and [53; 54; 55] are for keyword spotting.
Method	Text model	SLURP	SLURP#	STOP	STOP#	GSCV2	GSCV2#
CL (baseline)	RoBERTa+S	40.35	23.68	70.13	50.56	64.74	59.65
BART+Y	22.73	8.06	55.67	42.07	56.33	45.54
CWCL (Ours)	RoBERTa+S	63.80	40.75	87.87	67.77	81.02	82.77
BART+Y	53.12	30.51	80.99	73.08	88.81	89.43
Text-to-intent (reference)	RoBERTa+S	88.19	59.86	84.78	69.10	100	98.20
BART+Y	77.03	45.93	92.93	79.11	100	100
ESPnet [52]	-	77.00	-	-	-	-	-
Att. RNN [53]	-	-	-	-	-	93.9	-
Wav2Vec2 [54]	-	-	-	-	-	96.6	-
M2D [55]	-	-	-	-	-	98.5	-
shown in the SLURP# and STOP# columns in Table 4. We again observe that the proposed CWCL loss outperforms the CL loss.
Comparison to supervised training: We also present results of a supervised SLU model on SLURP, based on ESPnet-SLU [52]. Considering our system is zero-shot, the result is noteworthy. For STOP, we could not find previous supervised works that evaluated systems the same way.
Due to lack of space, we present the following results in A.2.2. In Table 7, we found that leveraging pre-trained models was more beneficial than training from scratch for speech-text embedding alignment. As seen in Table 8, locking the text encoder and fine-tuning the speech encoder gave the best performance. We found that batch size is not a critical factor, as shown in Table 9.
4.2.2 Zero-shot keyword spotting (KWS)
We also tested our model for KWS using the Google Speech Command Dataset V2 (GSCV2) [9] where we classified among the 35 keywords in the Google Speech Command. The result is shown in the columns after the thick vertical line in Table 4. For the results in the first column (without  $\#$ ), we used each keyword as is to extract the class embedding from the text model. For the second column (with  $\#$ ), we used the general template used in the speech-to-intent experiments. The results show that the proposed method outperforms the baseline. With CWCL, the KWS\# outperformed KWS. This could be because the text models that generate the class embeddings are usually trained with sentence-level samples, not word-level ones whereas the keywords are words, i.e., the KWS class embeddings are extracted from words whereas KWS\# are extracted from sentences constructed using templates, thus resulting in better class embedding.
Comparison to supervised training: Interestingly, results achieved with the BART-based text model are comparable to the supervised learning mechanisms of [53; 54; 55]. Note that the self-supervised mechanisms use training data to train the final linear classifier [54; 55]. However, our models without any training data still achieve close to  $90\%$  accuracy. This will be useful when defining new keywords as collecting large datasets for keyword classification becomes difficult [9]. Additional results are provided in Table 10 and in Table 11, respectively in Appendix A.2.
In this paper, we make the observation that existing contrastive learning based methods for cross-modal alignment using pre-trained models are not efficient in extracting supervision from the pretrained embeddings. In particular, many similar examples that do not form pairs in the training data are ignored. We address this by developing a novel loss function that accounts for the continuous nature of similarity and uses information from all similar examples in a training batch. We train models for two pairs of modalities using this loss function, namely image-text and speech-text. In both cases, we observe a significant increase in zero-shot performance on downstream tasks. We believe that the proposed loss function will be impactful in leveraging powerful pre-trained models and transferring the learnt knowledge to other modalities and domains.
[49] Jack G. M. FitzGerald, Christopher Leo Hench, Charith S. Peris, Scott Mackie, Kay Rottmann, A. Patricia Dominguez Sánchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, L. Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and P. Natarajan, "Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages," ArXiv, vol. abs/2204.08582, 2022.
[50] Rosana Ardila, Megan Branson, Kelly Davis, Michael Henrtty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber, "Common voice: A massively-multilingual speech corpus," arXiv preprint arXiv:1912.06670, 2019.
[51] Paden Tomasello, Akshit Shrivastava, Daniel Lazar, Po-Chun Hsu, Duc Le, Adithya Sagar, Ali Elkahky, Jade Copet, Wei-Ning Hsu, Yossi Adi, et al., "Stop: A dataset for spoken task oriented semantic parsing," in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 991-998.
[52] Siddhant Arora, Siddharth Dalmia, Pavel Denisov, Xuankai Chang, Yushi Ueda, Yifan Peng, Yuekai Zhang, Sujay Kumar, Karthik Ganesan, Brian Yan, et al., "Espnet-slu: Advancing spoken language understanding through espnet," in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7167-7171.
[53] Douglas Coimbra De Andrade, Sabato Leo, Martin Loesener Da Silva Viana, and Christoph Bernkopf, “A neural attention model for speech command recognition,” arXiv preprint arXiv:1808.08929, 2018.
[54] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12449–12460, 2020.
[55] Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino, "Masked modeling duo: Learning representations by encouraging both networks to model the input," in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1-5.
[56] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt, "Openclip," July 2021, If you use this software, please cite it as below.
[57] Ilya Loshchilov and Frank Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations, 2019.
A Appendix A
A.1 Experimental details for aligning image and text modalities
A.1.1 Model training details
We build upon the code repository in [56]. We train our models for a total of 70 epochs, where each epoch uses a subset of 6 million images. The batch size is set to 16000. Note the number of training steps in this case is equal to 26,250. We train on 4 A100 GPUs. Note that we experimented with different sizes for the subset used in each epoch (ranging from 2 million to the full dataset) and we obtained the best performance when the size was 6 million (for our method and the baseline methods that we train). We use a learning rate of 0.001, AdamW optimizer with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$  and a weight decay of 0.0001 [57].
A.1.2 Simple templates to test model robustness
One of the advantages of cross-modal 0-shot transfer is the ability of the trained models to be used on downstream tasks without any further training. However, the downstream task still needs to be adapted to the task of modality alignment. We discuss this adaptation in the context of image classification and provide details about our experiments reported in Section 4.1.3.
In [1; 2], the downstream task of image classification task is solved by first changing the class labels to sentences. The sentences are then converted to embeddings using the text encoder. Given a test image, the text embedding that it aligns the most with determines its class. In particular, both works use a set of 80 "template sentences" to convert each label to 80 sentences. The text embedding representing a given label is then computed as the average of the embeddings of these 80 sentences.
We observe that the classification accuracy depends on the choice of these template sentences, as also seen in [5]. To illustrate this, we formulate  $k = 1, 5, 10$  simple template sentences and use them to generate the classifier embeddings. We list these sentence in Table 5. Note that for  $k = 1$ , we use the first sentence only and for  $k = 5$ , we use the first 5 sentences. Our motivation in choosing simple sentences is to mimic the process of an end user who may not have the resources to carefully design the template sentences. Our goal is to test our model's robustness under such a scenario. As shown in Figure 5, a model trained using standard contrastive tuning shows poor performance as the number of template sentences is reduced. This shows that to achieve high accuracy, an end user must design template sentences that are complex enough. However, a model trained using CWCL maintains its performance across varying number of template sentences, even when only simple templates are used. Our hypothesis is that owing to the continuous nature of the similarity used during training, the model has learnt better cross-modal associations.
Table 5: Simple template sentences that we use to generate classifier embeddings.
a photo of a {}
an image of a {}
a picture of a {}
this is a {}
a snap of {}
a shot of {}
an illustration of {}
an example of {}
a {} is pictured here
In this picture, we can see a {}
A.1.3 Cross-modal retrieval
We also examine the zero-shot image-text retrieval capabilities of our proposed method. Note that our experiments are only towards comparing standard contrastive loss with CWCL. We leave the task of training with larger datasets [1; 2; 3] and using multi-objective training (which maybe used in conjunction with contrastive tuning to obtain better retrieval performance) [34; 29; 19] for future exploration.
Table 6: CWCL improves upon the CL-based alignment method for image-text retrieval.
Method	I →T retrieval	T→I retrieval
R@1	R@5	R@10	R@1	R@5	R@10
CL	30.42	54.32	65.82	24.17	49.04	61.05
CWCL (Ours)	35.10	61.52	73	25.69	50.04	61.59
In our experiment, we simply compare the performance of models trained with contrastive loss (as done in [2]) to that of models trained using CWCL. We use the MS-COCO validation dataset [43] to study zero-shot retrieval performance of these models. We report our results in Table 6. Models trained with CWCL outperform those trained using the standard contrastive loss function.
A.2 Experimental details for aligning speech and text modalities
In this section, we provide additional details about training the speech-text alignment models.
A.2.1 Model training details