CWCL: Cross-Modal Transfer with Continuously Weighted Contrastive Loss
Authors: Rakshith Sharma Srinivasa
Jaejin Cho
Chouchang Yang
Yashas Malur Saidutta
Ching-Hua Lee
Yilin Shen
Hongxia Jin
Venue: NeurIPS 2023 poster (2023)
Keywords: contrastive loss
multimodal representation learning
zero-shot learning
intent classification
pre-trained models
modality alignment
cross-modal transfer
TL;DR: This paper considers contrastive training for cross-modal 0-shot transfer wherein a pre-trained model in one modality is used for representation learning in another domain using pairwise data.

Abstract: This paper considers contrastive training for cross-modal 0-shot transfer wherein a pre-trained model in one modality is used for representation learning in another domain using pairwise data. The learnt models in the latter domain can then be used for a diverse set of tasks in a 0-shot way, similar to Contrastive Language-Image Pre-training (CLIP) and Locked-image Tuning (LiT) that have recently gained considerable attention. Classical contrastive training employs sets of positive and negative examples to align similar and repel dissimilar training data samples. However, similarity amongst training examples has a more continuous nature, thus calling for a more `non-binary' treatment. To address this, we propose a new contrastive loss function called Continuously Weighted Contrastive Loss (CWCL) that employs a continuous measure of similarity. With CWCL, we seek to transfer the structure of the embedding space from one modality to another. Owing to the continuous nature of similarity in the proposed loss function, these models outperform existing methods for 0-shot transfer across multiple models, datasets and modalities. By using publicly available datasets, we achieve 5-8% (absolute) improvement over previous state-of-the-art methods in 0-shot image classification and 20-30% (absolute) improvement in 0-shot speech-to-intent classification and keyword classification.

CWCL: Cross-Modal Transfer with Continuously Weighted Contrastive Loss
Rakshith Srinivasa
Jaejin Cho
Chouchang Yang
Yashas Malur Saidutta
Ching-Hua Lee
Yilin Shen
Hongxia Jin
Samsung Research America, Mountain View, CA
fr.siinivasa, jaejin.che cho, G.yangg//sansung com Cym. saidutta, chinghua.l, shen, hongxia, jaaaaajinnnaae. jin)suuuuung com
Abstract
This paper considers contrastive training for cross-modal zero-shot transfer wherein pre-trained model in one modality is used for representation learning in another domain using pairwise data. The learnt models the latter domain can then be used for a diverse set of tasks in zero-shot way, similar to "Contrastive Language-Imas Pre-training (CLIP)" M and "Locked-image Tuning (LiT)" that have recently gained considerable attention. Most existing works for cross-modal representation alignment (including CLIP and LiT) use the standard contrastive training objective, which employs sets of positive and negative examples to align similar and repel dissimilar training data samples. However, similarity amongst training examples has more continuous nature, thus calling for more non-binary treatment. To address this, we propose novel loss function called Continuously Weighted Contrastive Loss (CWCL) that employs continuous measure of similarity. With CWCL, we seek to align the embedding space of one modality with another. Owing the continuous nature similarity in the proposed loss function, these models outperform existing methods for zero-shot transfer across multiple models, datasets and modalities. Particularly, we consider the modality pairs of image-text and speech-text and our models achieve 5-8% (absolute) improvement over previous sfate-of-the-ar art methods in O-shot image classification and 20-30% (absolute) improvement in O-shot speech-to- -o-teentt classification and keyword classification.
Cross-modal alignment and transfer
Learning visual representations using natural language supervision has proven to be powerful way unlock impressive zero-shot performance on number of downstream tasks II this paper, we draw inspiration from these works and study the task of cross-modal alignment for zero-shot transfer for pairs of modalities. Let U and V denote a pair of modalities. For example, Ii may be text modality, and V maybe image modality. We are interested in the following: given pre-trained model fo - Q for data in (where Q denotes the embedding space), how can we use paired dataset of the form {u, v), U1 v E V. to best learn a model Je 4 - P (where P the embedding space corresponding 24) such that the learnt structure in the embedding space Q can be aligned with that of P? Once trained, the models do and fa can be used on diverse set of downstream tasks including interfacing with Large Language Models (LLMs) (see way, thus avoiding the need for costly, task-specific, labeled datasets.
6. In
a zero-shot
Our motivation in studying the above problem lies in the fact that powerful pre-trained models existing in certain modalities, but are lacking in other modalities. For example, the recent advances
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Inspedesst Deshot comparionn
0-2hot moyoles accuracc (66)
0. sccuracy (90
35
18
LIT
CICCL (Duna)
(0urs)
00
120
100
240
300 300
420
Imageleer parra seen (Miiion)
Speech pains soon [Milon]
Figure 1: Comparison of zero-shot transfer performance between baseline CL and proposed CWCL. (Left); zero-shot image classification accuracy measured across training epochs for the image-text modality pair. (Right): zero-shot speech-to-intent classification measured across training epochs for the specch-text modality pair. CWCL consistenly performs better than CL.
in language models have resulted in very powerful models to process text data, while no such models exist for speech and audio data. Unlike text based models that can now generalize to new tasks in zero-shot way, speech and audio models are still trained in a task-specific way (for example, automatic speech recognition (ASR)). Further, collecting labeled datasets in speech domain offers its own set of challenges including quality control, noise, removing silence to name few B 9]. Similarly, even when pre-trained models are available for certain modalities such as images, there might be challenging sub-modalitie (or domains) like medical imaging on which pre-trained models may not be trained on [O]. However, large scale paired datasets maybe available, which connect the above modalities. For example, large datasets of speech and the associated (possibly noisy) transcripts are available easily on the internet. Similary, pairs text and images, pains of medical and raw text [10) maybe more easily available. Based on this observation, methods have been proposed to train image and text encoders by aligning features corresponding to paired image and text data [T. Upon training, these models demonstrate impressive zero-shot performance of downstream tasks such as image classification and image-text retrieval. While in these works both encoders trained from scratch, authors in 2 showed that using a frozen pre-trained image classification model as the image encoder and only training the text encoder significantly boosts downstream zero-shot performance. We observe that this abstract concept of using a pre-trained model in one modality to supervise models in another modality using pairwise data can then be applied to any pair of modalities.
number
Our main focus in this paper is on how best to train such cross-modal models that leverage pre- trained models in one modality. We find that standard contrastive loss used in training such models is inefficient at maximizing the amount of supervision that can be extracted from the pre-trained models. In particular, to learn the embeddings in the "unfrozen" modality, existing methods only use the embedding of the corresponding paired data from the other modality for supervision. However, there maybe many samples from the supervising modality that are similar, and to various degrees of similarity. To address this inefficiency, we propose a new loss function called continuously weighted contrastive loss (CWCL) for contrastive training of multi-modal models. The proposed loss function leads to better supervision and hence better alignment between the two modalities.
We study our proposed loss function using two pairs of modalities, image-text and speech-text. For image-text pair, we find that the proposed loss function leads to an improvement of 6-8 % (absolute) compared to the best bascline zero-shot image classification tasks. For speech-text, it leads to a 20-30% (absolute) improvement on zero-shot speech-to-intent classification and zero-shot keyword spotting tasks. Further, our models achieve performance comparable to models trained with supervision using task-specific datasets. As shown in Figure we find that models trained using the proposed loss function are data and compute-effieient They achieve higher accuracy with fewer pairs of data samples during training. Further, embeddings of downstream test datasets generated using our models show strong alignment among data that belong to the same class, even though the models have never been exposed to these datasets. We show an example in Figure 2 where embeddings extracted from speech signals in the SLURP test dataset show significantly improved sense of similarity for data from the same class, even though no label information was provided to the model.
2
Continuously weighted contrastive loss
2.1 Existing frameworks for contrastive training
Various forms of contrastive learning has been successfully employed in both self-supervised learning 2 13 14] and in supervised learning (15].
12,
Contrastive loss for self-supervised and multi-modal learning: The traditional contrastive loss

---

Similarity matrix with Cl
Simil anyy matrix with CWC)
calendar_quert query
Co calendar.quee quety
socts query
social, query
music_
music query
hot_hue_ lightup
61
12
60
news_query
news_query
-11
calendar set
ca ender set
Calender, sat
newe query
ist hue Hightup
music query
secial quary
calendar, query
calendar sott
neet query
lot huee lightu
music query
social quary
calenster quary
Speech embeddings gg
Speech embedd 2
Figure 2: The similarity matrix between embeddings of the two modalities that are aligned via (Left); baseline CL and (Right): proposed CWCL. The axis labels correspond to the intent of utterances (for example, "news_ currsssuerrrrrr represents utterances with news-related questions). CWCL results in more "block" diagonal pattern than CL, indicating that speech and text samples with the same intent are more aligned while samples with different intents more separated. This can be attributed to the continuous weighting mechanism of CWCL. Note that these embeddings are from a downstream test dataset which was never exposed to the model during training. The visualization confirms that CWCL leads to a higher degree of alignment between similar data samples.
function is used in both single-modality self self-supervised learning as well as multi-modal alignment. We explain the formulation used multi-modal alignment and briefly explain how the same function is used in the single-modality setting. Let B denote a batch of training data consisting of pairs of data samples from two modalities of size N: B {u, r=)).=......NN and e; is from modality V. Let u; and e; be encoded into embeddings denoted as ps, qi respectively. This can be done by separate, modelity-specific encoders or by shared encoder. Then, the traditional contrastive loss function (CL) (to align 21 with V) is defined over B as
where u, is from modality U
LOLU-N
-1 N
Σ log
exp ((p).9))/T 9T) 1=1 ΣjE( exp ((p,, 4)//))
where [N] denotes the set {1,2, JJ. Note that similar loss function LCL.Y-24 maybe defined and the total loss function is given as LCLu-V + LCLV- By minimizing (T). the encoders learn to align pairs of data. Note that in doing so, for each Ug V/ is considered as positive example and all other samples {v, J are considered to be negative examples. This is also illustrated in Figure 4 where the diagonal matrix indicates the set of positive examples chosen (for each row and column). As an example, in 212 for each image, only the corresponding text is used as a positive example and all other text samples are used negative examples (and vice-versa).
Contrastive loss for supervised learning: It is conceivable that in a given training batch, there is more than one "positive" sample. However the information about which samples are related to each other may be missing in self-supervised learning. However, this information is available in a supervised learning setup. Let T denote a batch of training data of size M consisting of samples and labels: T = {xx,, y.). Further, let z; be the embedding generated by the model. Then, it is clear that the set P. {zy. # i y forms set of positive examples. This idea was explored in was proposed to leverage the label information:
15,, where the following loss function D
-1
M
exp ((zi, 21)/T) E exp ((zz, =x)
Csupcon
log
(2)
M
P(1
JEP())
Note that the above loss function can be interpreted as taking the average of pair-wise LCL over the positive set. The authors show that combination of the above loss and the task loss yields better performance than using the task loss alone. However, this method requires labeled datasets.
'The authors also propose another variant of the supervised constrastive loss function. Although we do not discuss it here, it is similar in spirit to 2
3

---

Anchor image
Archor caption
Anchor image
Anchor caption
"A close shot a black sedan'
Self [weight=1)
"A close shot of black sedan'
(weight=1)
Not similar
an sunny day"
Very similar (weight-0.71
dark car on sunny day"
(wnight=O)
Not
(weight-O)
"Here we have big white van"
Quite similar (weight-O.3) 3)
"Here we have big white van"
Not similar
(weight=O)
snow covared park
similar (weight-0)
snow covered park
Standard contrastive learning for image-tert text paired data
Proposed CWCL for image-text paired date
Figure 3: Existing contrastive learning methods treat samples in batch as either strictly positive or negative. However, similarity between data samples has a more continuous and non-binary nature. In this figure, provide example of the nature of similarity in the context of paired image-text data. Note that the 'weight' terms in the figure are contrived for illustration purposes. The proposed CWCL loss function attracts all other data samples degree proportional to their similarity. Similarity itself is measured using intra-modal inner product between samples' embeddings.
In the above two loss functions and other similar variants studied in the literature, find two shortcomings. Firstly, other similar examples that may be present in the training batch are not considered. In the self-supervised setting, all the other similar samples are considered negative examples. In the supervised setting, some classes might be similar to each other (for example, multiple breeds of dogs), but considered be negative examples to each other. Secondly, similarity is considered to be binary. As a result, all "positive examples" are attracted equally, and all "negative examples" are repelled equally: However, we observe that samples in a training batch maybe similar to each other to varying degrees. Some samples might be more similar to each other, a few others less many others may be dissimilar. For more detailed explanation, Figure
2.2
Can we account for non-binary similarity?
To address the above shortcomings, we propose a novel loss function called Continuously Weighted Contrastive Loss (CWCL). We use the same setup as that in multi-modal training used to define 11. The loss function (to align Pt with other gj 's) is defined as
LEWaL u-v
1
Σ JE[N]
wy log exp((pi 9,) ELEIN exp( (pr. 9p)/T)
N
WD EjE(N)
where w 's denote the intra-modal similarity weights between U, and vy in modality V Note that similar loss function to align modality V with modality U maybe defined, with the intra-modal similarity weights computed between u; and uy. We will refer to the intra-modal similarity weights simply as weights for ease of usage and we will drop the superscript, unless the modality needs to be specified. Note that the weights are computed pair-wise, within each training batch.
Before we describe how these weights may be computed, we highlight the properties that they need have. Firstly, we normalize the weights to be between 0 and 1: wiy E [0, 1]. Secondly, "similar" samples from within given domain should have higher weights and dissimlar samples should have lower weights. With these properties, note that LEWCL provides way interpolate between the self-supervised and mully-supervisee variants described earlier. When the weights are given as Wij = 1,1 (j) where 1. denotes the indicator function w.r.t set S, is equivalent to Lec- the other hand, in the supervised setting, if wij is defined as Wij for all pairs i, belonging to the same class, but 0 otherwise, equivalent to Luppoon More importantly, Lewar allows the model to use continuous sense of similarity, by i) computing softmax function for all pairs in the

---

Compute CL. je.g. CLIP, LIT)
Compute CW CL (Ours)
Modality
Encoder
91
42
8b
8M
P-10
Modality
A Apply
Encoder
weighls
VS.
Plas
Mhn
Similanty matrix
Binary weight maltix
Continuous weight malrix
Figure 4: The classical CL-based methods (e.g., CLIP M. LiT 22) etc) can be interpretes as using binary weight matrix for choosing the positive examples. The proposed CWCL utilizes a continuous weight matrix to account for the non-binary nature of similarity for improved alignment.
training batch (inner summation in Equation. 33) and ii) weighting these softmax terms by the similarity weights. Further, that all pair-wise inner products are already computed even in (T). (2) Therefore, computing LeWeL is similar in computational complexity to Le1, and Supponn
2.3 How can we obtain intra-modal similarity weights?
In the traditional self-supervised setting, no information about similarity between training data points maybe available. This might also be the case in multi-modal learning such in [. where the modality encoders are initialized randomly. However, authors in 22 explored the idea of using pre- trained models as initialization in multi-modal models. Further, they find that freezing the pre-trained model (except maybe for a final linear layer) yields the best performance. This setup offers a natural way to obtain the similarity weights. We can measure the similarity between the embeddings from the pre-trained model. We focus on this setup, where modality to train models in another modality. Note that even though model encoding the first modality is frozen, follow-up layers maybe added and trained. Let V be the "frozen" modality with a pre-trained initialization. Then align modality N with using (3. why maybe computed as w = (g: 42)/2 + 0.5 in order for wa E [0, 1]. We do not explore other formulations in this paper. natural question is about how such weights can be computed for the modality U. If the model in modality 4 is also initialized using pre-trained model, the weights may be computed in similar way. However, in this paper, we only focus on the cross-modal transfer. with similarity weights being computed only in the frozen modality initialized with pre-trained model. Assuming the modality V the frozen one, our loss function is given as
use frozen, pre-trained models for one
Leross modal transfer LCWCL, 4-v + LCL, V-+u
Note that the exact configuration and choice of which modality freeze will depend the pairs of modalities being considered, the quality of pre-trained models and paired datasets available.
3 Related work
CLIP like models: CLIP 1 and ALIGN 3 introduced a set of foundational vision-language models where the encoders,
for the image, another for the text modality output embeddings in shared space. In this set of works the encoders for all the modalities are randomly initialized and trained from scratch. Other works have looked at extending the concept other modalities like image-audio [16), others have explored richer embedding [17). adaptive prompt learning (5), architectural advancements like Mixture-of-Ease -xpeerssss
66. Some works also considered the problem setting where embeddings of both image and text are processed together so that specialized text query relevant image embeddings can obtained [18] 19 Another notable work. [4, the authors obtained impressive performance by improving individual encoders by processing image and text embeddings together minimize caption-loss. Additionally, (20) proposed an extension to the cross-modal contrastive loss that can leverage labeled training data by combining it with SupCon [15] Recent
works such as 21 22 23 consider the alignment between images and text that do not belong to the same pair, similar to our proposed method. In [21 23), both encoders are trained from scratch
5

---

by using self- seffdistillation process. Such training process requires careful parameter tuning and generally has lower performance (2T achieves about 42.4% O-shot on ImageNet) compared using pre-trained models, as demonstrated by the metrics. Another difference between our work and the above works is that we consider intra-modal similarity to attract image-text pairs. Owing to the availability of strong pre-trained uni-modal models, intra-modal offers a clear way to identify similar data samples. In [22], the authors consider using a third, object detection model to obtain similarity between images. However, their method is specific to image-text modality pair. It may also lead to performance degradation, as seen in the zero-shot metrics reported.
LiT like models: one domain and aligning the embeddings of another domain the pretrained model's embedding space. Works in this line have looked at extending to multiple domains like imaee-audio text
Alternatively, LiT (2 proposed the idea of leveraging strong pretrained models in
[4], music-audio [25). speech-text for speech translation [2]]; fine-grained query specific image embeddings (27 and benefits of cross-modal alignment for regularizing unimodal classifiers [28]. Along with building model capable of conversation, (29 proposed the use of cross-modal attention layers to improve image-text cross-modal alignment. However, none of these works consider the problem of similarity across samples within the same modality that is explored in our work. Further, all these works are complementary to CWCL and can be improved by it. Handful of works explore similarity amongst samples 30_ 13 30 propose removing certain samples from the negative set used to compute contrastive loss if their average similarity to the other samples in the batch is greater than a certain threshold 30
13 propose using a threshold on similarity to decide which samples are positive pairs and negative pairs and combining it with [[5]. However, the above works still consider similarity as a binary entity, where as CWCL. uses a continuous model.
Incorrect negatives in contrastive learning: Contrastive learning incorrectly assumes that for [31]. In the self-supervised learning one of the remedies proposed is to re-weight the negative part of the contrastive loss' denominator to account for the presence of similar (or positive) samples [32]. However, in the case of cross-modal alignment with pretrained models, the pretrained model is better indicator of the similarity [30 [3]
given sample, every other sample in the dataset is dissimilar
Experiments
In this section, we provide experimental results that demonstrate that CWCL leads to better zero-shot transfer performance. We study two pairs of domains, namely image-text and speech-text. For image-text pair, we demonstrate zero-shot transfer to image classification and image/ text retrieval. On both tasks, CWCL shows improved performance over existing methods for zero-shot transfer. Next, we report results for speech-text modality pair, where we consider the tasks of speech-to- to-intent classification and keyword spotting. Given the difficulties in collecting task-specific speech datasets, we expect CWCL-base main goal is study the effect of using CWCL. We use open source, publicly available and easily accessible datasets for our study and leave the task of training with larger datasets to future work.
zero-shot transfer to have a large impact in this domain. Note that our
4.1
Cross-modal transfer between image and text modalities
Model architecture: Our model architecture follows that in (2 and has a vision encoder and a text encoder. For the vision encoder, we use the ViT-L/16 model architecture [33 pre-trained on ImageNet. We compute the similarity weights using the embeddings from before the final linear layer that is not frozen during training. For the text encoder, we consider two architectures: transformer encoder architecture with 12 luyput dimension 768, and number of heads set to 12 and we also consider the BERT-large architecture.
All our experiments are based on the combination of two publicly available datasets, CC12M and YFCCI5M. The CC12M dataset is a subset of the Conceptual Captions dataset (34 defined in 35
Datasets for contrastive training:
We use a set of 10 million images that are still available in the set of URLs (since the rest of them have been taken down). The YFCC15M dataset is a subset of the Yahoo Flicker Creative Commons dataset (36] defined by I by filtering for high quality English text. It contains set of 15 million image-text pairs. Model training details are provided in A.I
4.1.1 Zero-shot image classification
For zero-shot image classification, we experiment on datasets: ImageNet (37 validation, ImageNet-
v2 (38), ImageNet-R [39. 40), AmageNettA[[4aaa and ObjNet [42), similar to 22. We provide our

---

experimental results in Tables. The results for SimCon [33. and LiT 2 are obtained from our own experimentation. For we use their loss function in our set up. For 2 we use their recommended experimental settings from their paper. Note that the metrics are obtained by using the same model architecture and dataset for the methods being compared. CWCL yields significant boost over the other methods in zero-shot performance. Further, shown in Figure CWCL achieves higher accuracy with fewer image-text training pairs. Owing to the CWCL formulation, the text embeddings generated by our model are designed to be similar to larger set of similar images than the baseline methods, hence leading to better generalization.
Table 1: Zero-shot image classification performance using the ViT-I//6 + 12-layer transformer configuration. CWCL achieves significant improvement zero-shot image classífication across multiple datasets, including out-of-domain datasets such as ObjectNet.
Method
ImageNet (%)
ImageNet-V2($5
ImageNet-R($t)
ImageNet-A(%) ObjNet(S%)
CLIP
31.3
OpenCLIP
34.8
30
SimCon
67.9
58.57
59.32
37.16
44.9
LiT
66.84
58.82
61.28
37.31
45.08
CWCL (Ours)
74.41
66.25
67.37
45.58
50.5
Table 2: Zero-shot image classification using the ViT-L/16 +BERT-large configuration. CWCL-based training achieves state-of-the-art (when trained publicly available datasets) performance all of zero-shot experiments.
Method
ImageNet (%)
ImageNet-V2($5)
ImageNet-R($5)
(mageNet-A(9))
ObjNet(%))
LiT
71.2
62.98
63.8
40.28
48.1
CWCL (Ours)
76.48
67.86
68.7
47.27
52.38
4.1.2 Zero-shot Image-text retrieval
We also examine the zero-shot image-tex: retrieval capabilities of our proposed method. Note that our experiments are only towards comparing standard contrastive loss with CWCL. We leave the task of training with larger datasets 2 and using multi-objective training (which maybe used along with contrastive tuning to obtain better retrieval performance) [34 29 for future exploration. In experiment, we simply compare the performance of models trained with contrastive loss (as done in (2) to that models trained using CWCI. We use the MS-COCO validation dataset (43) to study zero-shot retrieval performance of these models. We report our results in Table Retrieval metrics for the ViT-L/16+1D layer transformer configuration model are provided in Table 6 in the Appendix. Models trained with CWCL outperform those trained using the standard contrastive loss function.
Table 3: Zero-shon shot retrieval results on MS-COCO dataset by using ViT-L/I6+BERT-lagg configura-
tion as the image and text encoders respectively.
Method
T retrieval
T-+ retrieval
R@1 1
R@5
R@ 10
R@1
R@5
@@
LiT
34.58
59.78
70.68
28.49
54.04
65.87
CWCL (Ours)
40.36
66.62
77.76
30.04
54.84
66.06
4.1.3 Robustness templates for zero-shot classification
An added benefit of the proposed CWCL formulation is that our model is robust to the templates/ prompts used in zero-shot tasks. In zero-shot image classification, the labels are converted to text prompts in order adapt the task of classification that alignment. In particular, both [ use a set of 80 "template" sentences convert each label into 80 sentences, extract the text embeddings for all the sentences and use their mean embedding as the representation of the corresponding class. We expect that CWCL leads to robustness w.r.t the choice of such templates or prompts. We study this
7

---

Baseline CL (LiT)
CWCL (Ours)
using templates
using k templates
using all templates
using a temp ates
Top-1 accuracy
accuracy
0696
41
70
70
-4.92
-7.42
80
60
15.56
50
50
Km1
kw10
k=5
Figure 5: Comparison of robustness to templates. (Left): the baseline CL method of LiT (2) (Right): the proposed CWCL approach. The number displayed at each bar reflects the decrease in accuracy due to using only a subset of templates compared to using the full set.
by changing the number of template sentences used to build the classifier embeddings. In particular, we design simple templates such as "this a photo of and "this is an image of and experiment over k 1, 5, 10 templates. We provide further details on the templates in Section A.1.2 We report the results for our model and that of [2] in Figure 5 As can be seen, models trained using CWCL able obtain peak performance with fewer number of templates, whereas models trained using standard contrastive loss require a higher number of templates to build better classifier embeddings. We believe that robustness the choice and the number of template sentences/prompss used is crucial to improve the case of usage of such models.
4.2 Cross-modal transfer between speech and text modalities
The proposed method can be applied speech-text cross-modal learning to transfer semantic knowledge from text embeddings to speech embeddings. Speech models with language understanding are desired for tasks the field of spoken language understanding (SLU) 44 45 SLU differs
from automatic speech recognition (ASR), which simply generates a transcription, but does not have language understanding. SLU models, unlike ASR models can then be used on a wide variety of downstream tasks such as intent classification (in multiple domains) g keyword spotting [46].
In general, speech pre-training schemes usually include information about the phonemes or paralin- guistic information (e.g. speaker, emotion, pathology. etc.), but they do not include semantics in language. While some works have explored the usage of contrastive learning train SLU models, they use the standard contrastive training method [47). However, similar to the image-text case, this may not be efficient. For instance, "Turn on the volume". is closer in meaning "Increase the sound" than "Set an alarm for 7:00 AM tomorrow morning". In this case, the standard cross-modal contrastive loss is unable to learn the cross-modal relationship between the text of the first sentence and the speech of the second sentence since they are considered to be "negative pair". This is precisely what is address by CWCL. As we demonstrate later, CWCL, achieves a significant boost in performance on downstream tasks.
We train speech-text multi-modal model with dataset where speech and its corresponding transcript are available. Note that this is a generic dataset that is not specific to any SLU task. We use a pre- trained, frozen text encoder, owing the availability of strong pre-traue pre-trained language models. A trainable linear layer is added on top of the frozen text encoder to match the dimensionality of the speech and text embedding. We also use pre- pre-trained speech encoder that is robust diverse acoustic condition and further train it using the proposed loss function.
Model architecture: For the speech model, we used the encoder of the pre-trained Whisper ASR [8), which is expected to be robust to different acoustic conditions. For the text models, we found 49 publicly available hugging face models by searching with filters as, task: Zero-shot classification, libraries:
Transformers, and languages: English. We manually added one RoBERTa-based model fine- tuned on MASSIVE 44) data. All 50 models were compared on zero-shot text-to-intent classification using the SLURP dataset ( and the top 2 models were selected. The best model (we call it RoBERTa+S) was the RoBERTa-basse model fine-tuned on MASSIVE data since the data includes SLURP (only the text data) The second model was a BART-based model fine-tuned on Yahoo Answers topic classification (we call it BART+Y)
8

---

Datasets for contrastive training For cross- modal training, we used the Common Voice Corpus 13.0 [50].
This dataset consists of roughly 2400 hours of speech data and the corresponding transcripts obtained using and gender. We use the English subset. Model training details are provided in A.2
and includes speech from diverse set of demographies across age
4.2.1 Zero-shot speech-to-intee to-intent classification
After the cross-modal embedding alignment stage, we evaluated the models on the zero-shot speech- to-intent classification task. The task is classify given speech sequence into one of the intent classes. The main difference between the zero-shot and supervised intent classification is the zero-shot classification can be done without training classifier.
Class embedding generation: Similar to the image-text case, we compute the embeddings of a given speech signal and compute its similarity with the text embeddings for all the intent classes. These class embeddings are obtained as averaged embedding of text sentences' embeddings of the corresponding classes. During inference, the class embedding that has the highest similarity score with the input speech embedding is chosen as the predicted class.
Dataset: We used the SLURP [8] and STOP [5]] datasets for evaluation. In the SLURP dataset, we used all the text sentences in the train subset to generate the class embeddings for 60 intent classes where intent is defined as the concatenation of scenario and action labels, following ESPnet-SLU (52). We did not use the Irain_ sruinsynthetic subset since more than half of the text sentences overlap with the devel and test subsets. On average, 191 text sentences were used per class. We compare the systems by evaluating them on the devel and test subsets. In the STOP dataset, we used the 8 unique domain labels as intent labels. Although not intents in strict sense, the domain labels can be considered simpler version of the intent labels. Since significantly more sentences are available in STOP, we randomly extracted 200 sentences per domain from the training set generate the class embeddings. The evaluation was done on the validation and test sets.
In previous works, speech-to-intent classification has been done with ASR-NLU pipeline system where the speech is first transcribed by ASR (speech-to-text), after which the transcription is classified into an intent using NLU (text-to-intent) [8) We refer to the text-to-intee io-intent performance achieved by the pre-trained text encoders as the "reference" " performance. This provides an estimate of the performance that can be expected from the speech encoder the speech-to-inted task.
Results:
The first results of speech-to-intent classification are shown in the SLURP and STOP (without the superscript #) columns in Table 4 In all cases, multi-modal training with the CWCL loss outperformed the CL loss. On the SLURP dataset, RoBERTa+S has a higher reference performance compared to BART+Y because the fine-tuning data for RoBERTa+S included the SLURP text data. This also leads to a better performance compared to using the BART+Y model as the text encoder.
On the STOP dataet, ROBERTa+S has a lower reference compared to BART+Y, implying that the RoBERTa+S' text model overfits the SLURP data. However, the RoBERTa+S-baune speech intent classification was still better than the BART+Y-based one. This implies that the text model architecture could be another factor that contributes to transferring performance to the speech model. To be specific, the RoBERTa+S was RoBERTa which consists of only encoder layers while the BART+Y was the mecoer-deecoder-bare ROBERTa+S outperforms the text-to- to-intent reference performance on the STOP (87.87 vs. 84.78) dataset. This because, during the cross-modal aligmment stage using CWCL, the speech tower might have learned how to utilize acoustic cues in addition to linguistic information from given speech utterance, to align its embedding to the semantic embedding from the text tower. However, this did not happen in the case of SLURP, because the SLURP dataset includes more intent classes than STOP (60 vs 8 classes), thus being more challenging in transferring knowledge from text to speech during the cross-modal alignment stage.
BART model. Another thing to note is that CWCL with
Experimenting with different templates to generate text embeddings: So far, each class embed- ding used in zero-shot intent classification was generated by averaging all the corresponding text sentences' embeddings from the class the training subset. Although collecting the text data with the intent labels can be less expensive than collecting speech data with the intent labels, the former may not always be possible. To address this, we manually devised fixed set of general templates that were applied to every class. For example, templates are of the form "This audio is about [class]" J", and "The utterance related to [class]", and the text embeddings are averaged to obtain the class embedding. For the exact templates used, readers may refer to Appendix A.2.5 The results are

---

Table 4: Top-1 accuracy for zero-shot speech-to-intent classification (SLURP and STOP) and keyword spotting (GSCV2) after thick vertical line. Superscript # is used indicate use of general templates for class embedding extraction. Supervised results are provided in gray after the double-horizontal
line: [2 is for speech-to-ing to-icettt and [53 54 [55
are for keyword spotting.
Method
Text model
SLURP
SLURP
STOP
STOP*
GSCV2
GSCV2#
CL (baseline)
RoBERTa+S
40.35
23.68
70.13
50.56
64.74
59.65
BART+Y
22.73
8.06
55.67
42.07
56.33
45.54
CWCL
RoBERTa+S
63,80
40.75
87.87
67.77
81,02
82.77
(Ours)
BART+Y
53.12
30.51
80.99
73.08
88.81
89.43
Text-to-intent
RoBERTa+S
88.19
59.86
84.78
69.10
100
98.20
(reference)
BART+Y
77.03
45.93
92.93
79.11
100
100
ESPnet [52]
77.00
Att. RNN
53
93.9
Wav2Vec2
54
96.6
M2D 55
98.5
shown in the SLURP# and STOP# columns in Table 4 We again observe that the proposed CWCL loss outperforms the CL loss.
Comparison to supervised training: We also present results of supervised SLU model on SLURP, based on ESPnet-SLL [52]. Considering our system is zero-shot, the result is noteworthy. For STOP, we could not find previous supervised works that evaluated systems the same way.
Due to lack of space, we present the following results in A.2.2 In Table 7 we found that leveraging pre-trained models was more beneficial than training from scratch for speech-text embedding align- ment. As seen in Table 8 locking the text encoder and fine-tuning the speech encoder gave the best performance, We found that batch size is not critical factor, as shown in Table
4.2.2 Zero-shot keyword spotting (KWS)
We also tested our model for KWS using the Google Speech Command Dataset V2 (GSCV2) where we classified among the 35 keywords in the Google Speech Command. The result is shown in the columns after the thick vertical line in Table + For the results in the first column (without #), we used each keyword as is to extract the class embedding from the text model. For the second column (with #), we used the general template used in the specch-to-intens experiments. The results show that the proposed method outperforms the baseline. With CWCL, the KWS# outperformed KWS. This could be because the text models that generate the class embeddings are usually trained with centence-level samples, not word-level ones whereas the keywords are words, i.e., the KWS class embeddings are extracted from words whereas KWS## are extracted from sentences constructed using templates, thus resulting in better class embedding.
Comparison to supervised training: Interestingly, results achieved with the BART-based text model are comparable to the supervised learning mechanisms of [53,
54 55) Note that the self-supervised
mechanisms use training data to train the final linear classifier (54 55) However, our models without
any training data still achieve close 90% accuracy. This will be useful when defining new keywords as collecting large datasets for keyword classification becomes difficult [9]. Additional results are provided in Table 10 in Table II respectively in Appendix A.2
Conclusion
In this paper, we make the observation that existing contrastive learning based methods for cross- modal alignment using pre-trained models are not efficient in extracting supervision from the pre- trained embeddings. In particular, many similar examples that do not form pairs the training data ignored. We address this by developing a novel loss function that accounts for the continuous nature of similarity and uses information from all similar examples in a training batch. We train models for two pairs of modalities using this loss function, namely image-text and speech-text. In both cases, we observe a significant increase in zero-shot performance on downstream tasks. We believe that the proposed loss function will be impactful in leveraging powerful pre-trained models and transfering the learnt knowledge to other modalities and domains.
10

---

References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al., "Learning transferable visual models from natural language supervision." in International conference on machine learning. PMLR, 2021, pp. 8748-8763.
[2] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer, "Lit: Zero-shot transfer with locked-image text tuning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022,
pp. 18123-18133.
[3] Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, "Scaling up visual and vision- language representation learning with noisy text supervision." International Conference on Machine Learning. PMLR, 2021, pp. 4916
[4] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu, "Coca: Contrastive captioners are image-text foundation models," arXiv preprint arXiv:2205.01917 2022.
[5] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao, "Test-time prompt tuning for zero-shot generalization in vision-language models," " in Advances in Neural Information Processing Systems, 2022.
[6] Basil Mustafa, Carlos Riquelme Ruiz, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby, "Multimodal contrastive learning with limoe: the language-iman mixture of experts." in Advances in Neural Information Processing Systems, 2022.
[7] Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, et al., "Anymal: An efficient and scalable any-modality augmented language model," arXiv preprint arXiv:2309. 16058, 2023.
[8] Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser, "Slurp: A spoken language understanding resource package." arXiv preprint vrXiv:2011.1320. 2020.
[9] Pete Warden, "Speech commands: dataset for limited--ocabularyy speech recognition," arXiv preprint 180vv 002009, 2018.
[10] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher Manning, and Curtis Langlotz, "Contrastive learning of medical visual representations from paired images and text," in Machine Learning Healtheare Conference. PMLR, 2022, pp. 2-25.
[11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Everest Hinton, "A simple framework for contrastive learning of visual representations," 2020.
[12] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He, "Improved baselines with momentum contrastive learning." " arXiv preprint arXiv:2033.04297 2020.
[13] Yash Patel, Yusheng Xie, Yi Zhu, Srikar Appalaraju, and Manmatha, "Simcon loss with multiple views for text supervised semantic segmentation," arXiv preprint arXiv:2302.034322 2023.
[14] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang, "Clap learning audio concepts from natural language supervision," in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1-5.
[15] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan, "Supervised contrastive learning," Advances in neural information processing systems, vol. 33, pp. 18661-18673 2020.
[16] Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello, "Wav2elip: Learning robust audio representations from clip," ICASSP 2022-2022 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 4563-4567.
11

---

[17] Zachary Novack, Saurabh Garg, Julian McAuley, and Zachary C Lipton. "Chils: Zero-shot image classification with hierarchical label sets," arXiv preprint arXIv:2302.02551. 20233
[18] Jonatas Wehrmann, Camila Kolling, and Rodrigo C Barros, "Adaptive cross-modal embeddings for image-lext alignment," in Proceedings of the AAAI conference on artificial intelligence, 2020, vol. 34, pp. 12313-12320.
[19] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi, "Align before fuse: Vision and language representation learning with momentum distillation," Advances in neural information processing systems, vol. 34, pp.
2021.
[20] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao, "Unified contrastive learning in image-text-label space," " in Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, 2022. pp. 19163-19173.
[21] Alex Andonian, Shixing Chen, and Raffay Hamid, "Robust mross-modal representation learning with progressive diff-ditillaation"" in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022, pp. 16430-16441.
[22] Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Wei Liu, Jie Yang, Ke Li, and Xing Sun, "Softclip: Softer cross-modal alignment makes clip stronger," arXiv preprint arXiv:2303. 175611, 2023.
[23] Pedro Morgado, Ishan Misra, and Nuno Vasconcelos, "Robust audio-visual instance discrimina- tion," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 12931-1299999 12945.
[24] Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel, "Audioclip: Extending clip to image, text and audio," " in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 976 976-980.
[25] llriia Manco, Emmanouil Benetos, Elio Quinton, and George Fazekas, "Contrastive audio- language learning for music," in Ismir 2022 Hybrid Conference, 2022.
[26] Rong Ye, Mingxuan Wang, and Lei Li, "Cross-modal contrastive learning for speech translation," in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Seattle, United States, July 2022.
pp. 5099-5113, Association for Computational Linguistics.
[27] Lu Wu, Chenyu Wu, Han Guo, and Zhihao Zhao, "A cross-mon moosl alignment for zero-shot image classification," IEEE Access, vol. 11, pp. 9067-9073, 2023.
[28] Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, and Deva Ramana, "Multimodality helps unimodality: Cross-modes modal few-shot learning with multimodal models," " arXiv preprint arXiv:2301.06267. 2023.
[29] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al., "Flamingo: a visual language model for few-shot learning," Advances in Neural Information Processing Systems, vol. 35, pp. 23716-23736, 2022.
[30] Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, and Thomas Brox, "Crossclr: Cross-modal contrastive learning for multi-modal video representations," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1450-1459.
[31] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi, "A theoretical analysis of contrastive unsupervised representation learning," arXiv preprint arXiv: 1902. 09229, 2019.
[32] Ching- Cho Chuang. Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka, "Debiased contrastive learning." Advances in neural information processing systems, vol. 33, pp. 8765 8775, 2020.
[33] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai Ross Wightman, Jakob Uszkoreit, and Lucas Beyer, "How to train your vit? data, augmentation, and regularization in vision transformers." arXiv preprint 100v:2100.102200, 2021.
12

---

[34] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut, "Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning." " in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), Melbourne, Australia, July 2018, pp. 2556-2565, Association for Computational Linguistics.
[35] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut, "Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts." in Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2021, pp. 3558-3568.
[36] Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li, "Yfec100m: The new data in multimedia research," Communications of the ACM, vol. 59, no. 2, pp. 64-73, 2016.
[37] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, "Imagenet: A large- scale hierarchical image database." in 2009 IEEE conference computer vision and pattern recognition. Icee, 2009, pp. 248-255.
[38] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar, "Do imagenet classifiers generalize to imagenet?," in International conference on machine learning. PMLR, 2019, pp. 5389-5400
[39] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. "Learning robust global repre- sentations by penalizing local predictive power," Advances in Neural Information Processing Systems, vol. 32, 2019.
[40] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al., "The many faces of robustness: critical analysis of out-of-distribution generalization," in Proceedings of the IEEE/CVE International Conference on Computer Vision, 2021, pp. 8340-8349.
[41] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song, "Natural adversarial examples," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 15262-15271,
[42] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz, "Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models," Advances in neural information processing systems, vol. 32, 2019.
[43] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick, "Microsoft coco captions: Data collection and evaluation server," arXiv preprint 1504.003500 2015.
[44] Libo Qin, Tianbao Xie, Wanxiang Che, and Ting Liu, "A survey on spoken language under- standing: Recent advances and new frontiers," arXiv preprint arXiv:2103.0309. 2021.
[45] Jerome R Bellegarda, "Spoken language understanding for natural interaction: The siri experi- ence," Natural Interaction with Robots, Knowbots and Smartphones: Putting Spoken Dialog Systems into Practice, pp. 3-14, 2013.
[46] Axel Berg, Mark O' OoCnor and Miguel Tairum Cruz, "Keyword transformer: A self-attention model for keyword spotting," " in Interspeech, 2021.
[47] Bhuvan Agrawal, Markus Müller, Samridhi Choudhary, Martin Radfar, Athanasios Mouchtaris, Ross McGowan, Nathan Susanj, and Siegfried Kunzmann, "Tie your embeddings down: Cross- modal latent spaces for end-to-end spoken language understanding." in ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
pp. 7157-7161.
[48] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever, "Robust speech recognition via large-scale weak supervision." arXiv preprint arXiv:2212.04356, 2022.
13

---

[49] Jack G. M. FitzGerald, Christopher Leo Hench, Charith S. Peris, Scott Mackie, Kay Rottmann, A. Patricia Domínguez Sánchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh. Swetha Ranganath, I. Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and P. Natara- jan, "Massive: Im-example multilingual natural language understanding dataset with 51 lypologically-divesee languages," ArXiv, vol. abs/2204.08558 2022.
[50] Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber, "Common voice: A massively mulliiiil speech corpus," arXiv preprint arXiv: 2019.
[51] Paden Tomasello, Akshat Shrivastava, Daniel Lazar, Po-Chun Hsu, Duc Le, Adithya Sagar, Ali Elkahky, Jade Copet, Wei-Ning Hsu, Yossi Adi, et al., "Stop: A dataset for spoken task oriented semantic parsing," in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 991-998.
[52] Siddhant Arora, Siddharth Dalmia, Pavel Denisov, Xuankai Chang, Yushi Ueda, Yifan Peng, Yuekai Zhang. Sujay Kumar, Karthik Ganesan, Brian Yan, et al., "Espnet-slu: Advancing spoken language understanding through espnet," in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7167-7171.
[53] Douglas Coimbra De Andrade, Sabato Leo, Martin Loesener Da Silva Viana, and Christoph Bernkopf, "A neural attention model for speech command recognition," arXiv preprint wVXiv:180..08922 2018.
[54] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, "wav2vec 2.0: framework for self-supervised learning of speech representations," Advances neural information processing systems, vol. 33, pp. 12449-12460, 2020.
[55] Daisuke Nüzzmmi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino, "Masked modeling duo: Learning representations by encouraging both networks model the input," in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1-5.
[56] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt, "Openclip." July 2021, If you use this software, please cite below.
[57] Ilya Loshchilov and Frank Hutter, "Decoupled weight decay regularization," in International Conference on Learning Representations, 2019.
14

---

A Appendix A
A.1 Experimental details for aligning image and text modalities
A.1.1 Model training details
We build upon the code repository in [56]. We train models for total of 70 epochs, where each epoch uses a subset of million images. The batch size is set to 16000. Note the number of training steps in this case is equal to 26,250. We train on 4 A100 GPUs. Note that we experimented with different sizes for the subset used in each epoch (ranging from 2 million to the full dataset) and we obtained the best performance when the size was 6 million (for method and the baseline methods that we train). We use a learning rate of 0.001, A optimizer with 31 = 0.9, 32 = 0.999 and a weight decay of 0.0001 [57].
A.1.2 Simple templates to test model robustness
One of the advantages of cross-modal (-shot transfer is the ability of the trained models to be used on downstream tasks without any further training. However, the downstream task still needs to be adapted to the task of modality alignment. We discuss this adaptation in the context of image classification and provide details about our experiments reported in Section 4.1.3)
In T 2] the downstream task of image classification task is solved by first changing the class labels to sentences. The sentences are then converted to embeddings using the text encoder. Given test image, the text embedding that it aligns the most with determines its class. In particular, both works use set of 80 "template sentences" to convert each label to 80 sentences. The text embedding representing a given label is then computed as the average of the embeddings of these 80 sentences.
We observe that the classification accuracy depends on the choice of these template sentences, as also seen in [5) To illustrate this, we formulate = 1, 5, 10 simple template sentences and use them to generate the classifier embeddings. We list these sentence Table 5 Note that for 1, we use the first sentence only and for k = 5, we use the first sentences. Our motivation in choosing simple sentences mimic the process of an end user who may not have the resources to carefully design the template sentences. Our goal is to test our model's robustenss under such a scenario. As shown in Figure(5] model trained using standard contrastive tuning shows poor performance as the number of template sentences is reduced. This shows that to achieve high accuracy, an end user must design template sentences that are complex enough. However, model trained using CWCL maintains its performance across varying number of template sentences, even when only simple templates are used. Our hypothesis is that owing to the continuous nature of the similarity used during training, the model has learnt better cross-modal associations.
Table 5: Simple template sentences that we use to generate classifier embeddings.
a photo of an image of {
picture of {
this a }
a snap [
a shot of { ]
an illustration of {} example of ( ] a { } pictured here
In this picture, we can see
A.1.3 Cross-modal retrieval
We also examine the zero-shot image-text retrieval capabilities of our proposed method. Note that our experiments are only towards comparing standard contrastive loss with CWCL. We leave the task of training with larger datasets 2 3 and using multi-objective training (which maybe used
in conjuntion with contrastive tuning to obtain better retrieval performance) 34 29 19 for future exploration.
15

---

Table 6: CWCL improves upon the CL-based alignment method for image-text retrieval.
Method
I T retrieval
T-xI retrieval
R@1 1
R@s
R@ 10
R@1
R@5
R@ 10
CL
30.42
54.32
65.82
24.17
49.04
61.05
CWCL (Ours)
35.10
61.52
73
25.69
50.04
61.59
In our experiment, we simply compare the performance of models trained with contrastive loss (as done in [2]) to that of models trained using CWCL. We use the MS-COCO validation dataset [43] to study zero-shot retrieval performance of these models. We report our results in Table 6 Models trained with CWCL outperform those trained using the standard contrastive loss function.
A.2
Experimental details for aligning speech and text modalities
In this section, we provide additional details about training the speech-text alignemmt models.
A.2.1 Model training details
We train each model for total of 20 epochs, where one epoch consumes the whole training data equal 1,013,630 samples. We use batch size of 20 with the 12,500 warmup steps and train on 1 A100 GPU. We use a learning rate of 0.00003, AdamW optimizer with 3 0.9, 32 0.999, weight decay 0.0001, and gradient clipping norm 10.
A.2.2 Effects of using pre-trained model weights, locking location, and batch size
Each reported number in this section is Top-1 accuracy (%) on SLURP data for speech intent classification.
Pre-trained speech encoder randomly initialized: In Table training from scratch and from pre-trained weights. The performance is significantly boosted by initializing the speech encoder using weights from the encoder part of the Whisper ASR model [48]. However, regardless of using random weights and pre-trained model weights, training with CWCL results in much better downstream performance.
compared starting multi-
Table 7: Comparison between using randomly initialized weights and pre-trained weights for speech encoders during training: Top-1 accuracy (%) on SLURP data
Method
Random initialization
Pre-trained weights
CL
13.80
22.73
CWCL
26.17
53. 12
Locking location: We have 4 ways to lock our model during multi-modal training since we have pre-trained speech and text models. We compared all the locking options and the result is shown in Table g In both baseline and CWCL losses, locking the text model works best. This can be seen as transferring the knowledge of semantic relationships in models to speech models.
Table 8: Locking location vs. performance: Top-1 accuracy (%) on SLURP data
Locking location
none
speech
text
both
CL
18.77
7.89
24.03
9.82
CWCL
17.50
27.39
53.12
16.70
16

---

Batch size performance: Since the large batch size was shown to improve performance with contrastive loss in computer vision, we also did a similar experiment to see how the batch size affects the performance as it gets larger. As the batch size increases, we also increased the learning rate proportionally, e.g., bs=20 has Ir=1, bs=40 has lr=2. The results are in Table
Table 9: Effective batch size performance: Top-1 I accuracy (%) on SLURP data
batch size
20
40
80
CL
24.03
25.20
24.51
CWCL
53.12
53.94
51.80
A.2.3 Further evidence of modality alignment due to CWCL
In Figure 2 showed the alignment (measured as inner product) between speech features and text features obtained from models trained using just CL and those trained using CWCL. We use the speech and text data from the SLURP test dataset. We illustrated that speech and text embeddings that belong to the same intent class were much more aligned compared to speech and text from mismatched classes. In this section, we provide more examples that support this observation.
In Figures we show the alignment between the speech and text embeddings where the speech and text samples belong to classes other than those used in Figure We again see that the alignment between samples in the same class much higher than that between samples in different classes. In general, observe the same pattern to hold across all the classes in the dataset, thus confirming that our results are not due to sampling bias.
Sirilarrit matrix with CL
Similarity matrix with CWCL
qa factolc
qa_ factoid
Text entredings
Toxt onbaddings
24
news
news query
datetime_quin query
datatime quary
35
play_music
play_music
calendar remove
calandar remove
calendar remova
play music
datetime, stery
news quary
da factaid
lies quey
calendar remevs
play music
detesine quarry
news swery
Rectod
fits query
Speech embeddings
Speech embeddings
Figure 6: similariy between speech and text embeddings obtained by sampling classes randomly from the SLURP test dataset. In this case, the sampled classes are different from those used in Figures and 7
A.2.4 Additional tables for reference
Table TO additionally shows the Top-5 accuracy over speech-text experiments. Since most of the previous works did not report this metric, we only include our own experimental results. Table [T shows the existing supervised model performances where the models are either trained or fine-tuned on the labeled Google Speech Command Dataset V2 for performing the keyword spotting (KWS) task, while our methods did not require any labeled KWS data for performing the task.
A.2.5 General template as a python list
To test the speech-text alignment models, we use a "general" set templates in addition to the one obtained by using the text from the training data itself. This general set of templates aims to mimic scenario where the no example texts maybe available. We list the set of template sentences used in the general set here.
17

---

Similarity matrix with CL
Smillriit matrix with CWC.
hnts_cmmateordd
createradddd
generaljskk
jeneraL_oke
email query
email, query
atamm 1et
aamm set
weather_qusty
woather, query
defirtion
40 deiination
wealfer Rorry atamy,
emall query
grenersl Joie
datt cruaterraao
dofinition
weathhrt quter
alamm fte
amail qiery
genseral jois
createornnd
Speech embrddingg
Speech embedliing
Figure 7: Another example of alignment between speech and text embeddings. The sampled classes are different from those used in Figures and
and
Table 10: Top-5 accuracy for zero-shot speech-to-intent classification (SLURP and STOP) and KWS on Google Speech Command Dataset V2. Superscript # used to indicate use general templates.
Method
Text model
SLURP
SLURP*
STOP
STOP*
KWS
KWS#
CL.
RoBERTa+S
69.57
49.86
98.19
94.03
82,22
82.53
CL.
BART+Y
52.97
24.87
95.27
81.63
84.02
78.14
CWCL (Ours)
RoBERTa+S
84.53
68.58
99.38
96.52
91.20
92.42
CWCL (Ours)
BART+Y
79.48
57.34
99.48
97.71
93.79
94.30
Text-intent
RoBERTa+S
95.66
83.36
98.93
95.20
100
98.20
(upper bound)
BART+Y
99.58
73.82
99.45
98.40
100
100
General template sentences: [ it is about } it about } it will be about } this is about } this was about } this will be about } it is related to { }, it was related related to }, this was related to } this will be related to { }, it is talking about } it was talking about ) it will be talking about I }, this is talking about { this was talking about { } this will be talking about { }, am talking about } I was talking about { } will be talking about } You are talking about } You were talking about )), You will be talking about { } They are talking about { They were talking about { ), They will be talking about ) We are talking about { }, We were talking about }
{ }, it will be related to } this is
will be talking about { } it talks about } it talked about { }, it will talk about } this talks about { } this talked about { }, this will talk about },, I talk about { }, talked about { } I will talk about } You talk about { }, You talked about { } You will talk about { }, They talk about } They talked about },, They will talk about { }, We talk about }, We talked about { }, We will talk about }
Table 11: Keyword spotting Top-1 accuracies on GSCV2 from existing supervised models.
Method
KWS
Attention RNN
553
93.9
KWT-2 46
97.74
Wav2 Vec2
[54
96.6
M2D [55
95.4
M2D - Fine tuned
55
98.5
18


Reviews:
[{"review_id": "ihltX4GaCh", "reviewer": "Reviewer_5jbW", "summary": "They propose a simple but effective method to align the representation space of two self-supervised models using pairs of examples from two modalities. They propose a CWCL loss where they reweight the contrastive loss of example pairs based on the similarity measured in one modality (equation 2). Specifically, let $(v_1, u_1)$ and $(v_2, u_2)$ be two example pairs, the CWCL loss encourage the representation of $v_1$ to be close to $v_2$ if $u_1$ is close to $u_2$. They experiment with a task requiring text-image and a task requiring text-audio alignment in zero-shot settings. The results show that adding CWCL loss to the original contrastive loss outperforms previous methods greatly.\n", "questions": "1. The performance of LiT reported in Table 1 is much lower than the performance reported in the LiT paper. Could you explain what might be the reason for this discrepancy?\n2. For the zero-shot keyword spotting task, because this task seems to be very much at the lexical level instead of the semantic level, I wonder what would be the benefit of using a contextualized pretrained language model? \n3. Instead of having CWCL loss in addition to the original contrastive loss, what about having CWCL loss only but you set the weight of negative pairs to be 0? There seem to be some subtle design choices made for CWCL. I would like to know more.", "limitations": "I didn't find the authors address the limitations of this work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed approach is very simple, but the results show that it brings significant improvements. Especially for the image classification task, their model outperforms the supervised-trained model in the zero-shot setting. \n2. They also show that, compared to a previous method, LIT, their model is more robust to the choice of prompt templates used to generate the text representation for the zero-shot image-classification experiment.\n3. The experiment of audio-text alignment also shows that adding the CWCL loss improves the performance.\n", "weaknesses": "1. The ablation study comparing pure contrastive loss vs contrastive loss plus CWCL is only conducted for the audio-text alignment task.\n2. In section 4.2.1, the author selected the text model very carefully. It thus seems that this method is very sensitive to the choice of the text model.\n3. It may provide more insight if there are some qualitative analyses between the model trained without CWCL and the model trained with CWCL.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1  The ablation study...**\n\nWe provide the comparison between standard contrastive loss and CWCL for the image-text modality in Table. 1. Please note that we have chosen to name the standard contrastive loss-based training as LiT, since we follow their training protocol by first initializing with a pre-trained image model and then using the standard contrastive loss as the training objective.\n    \nThe same comparison for the speech-text modality pair is provided in Table. 2. For standard contrastive loss, we initialize both speech and text encoders with pre-trained models, freeze them, and train the speech encoder only. In this regard, it is similar to LiT. However, we name this method as CL and not LiT, since LiT does not consider this pair of modalities, as its name suggests **L**ocked-**I**mage **T**uning, and we wanted to avoid any misrepresentation of the LiT paper.\n\nHence, the ablation studies presented in Table 1 and 2 are equivalent and consider the image-text and speech-text modalities respectively. We will clarify this further in the revised paper.\n\n**W2 In section 4.2...**\n    \nWe agree that the text model choice could affect the final model performance in the downstream tasks. However, when selecting the two text models used in this paper, we simply selected 2 models that performed the best in the text intent classification on SLURP, among 50 publicly available text models (the details are written in 4.2's model architecture section). Please note the CWCL performance can be expected to be better than CL performance irrespective of the choice of the text model. We chose those text models to demonstrate good performance on downstream tasks. All that said, how to choose the right models for the training in detail is very much worth exploring in a separate work and we appreciate this comment.    \n\n**W3: It may...**\n    \nIn Figures 3, 7, and 8 in the original manuscript, we provide examples of the alignment between the audio and the text embeddings generated using the SLURP speech-to-intent classification dataset. Speech (audio) data in this dataset was never exposed to the model at any stage of training. In this figure, we can observe that using CWCL leads to a strong alignment between speech samples and text samples that have similar semantic meanings. For example, all the speech samples related to queries about the news have a high alignment with the text samples that have similar meanings. (We would like to emphasize that this alignment is observed in a downstream dataset whose speech data was not used during training.) However, we do not observe such an alignment in the models trained with only standard contrastive loss (CL), as seen in the figures. Hence, the speech embedding model trained with CWCL can be considered to have a better language understanding than one trained with CL. \n\nA second set of qualitative results are provided in Section 4.1.2, where we study the robustness to templates in the image-text model. Owing to a stronger alignment between images and text, models trained using CWCL are able to generalize better under variations in the template sentences used. \n\n**Q1: The perf...**\n    \nThis is a good question! It is because of the architecture we used while training the models. In our experiments, we used the ViT-L/16 architecture for the image encoder and a 12-layer transformer for the text encoder. However, LiT achieves its best performance when the BERT-large architecture is used for the text encoder.\n    \n In order to address this, we have now trained models using the ViT-L/16 + BERT-large configuration. We followed the training procedure described in the LiT paper as closely as possible. We provide results with the updated model architecture in Table 13 (the PDF of the global rebuttal). We are able to achieve much higher performance with the new architecture (LiT: 71.2%, CWCL: 76.48%). \n\n**Q2: For the...**\n    \nWe agree that the task of keyword spotting is at a lexical level than a semantic level. However, our goal is to train a versatile model that can be used for multiple tasks, e.g., speech-to-intent classification and keyword spotting.\n \nWhile using our model for keyword spotting, we convert the keywords to sentences and also convert the keyword classification task to the alignment task. This is similar to 0-shot image classification done by CLIP and LiT-like models, which is also a task that does not require contextual language information. Further, the contextualized pre-trained language model could learn richer information in words considering the context in which each word is used. So our model performs well even when the keywords are used as is (without converting them to sentences).\n\nAlthough we followed LiT's training protocol as closely as possible, the LiT model we trained achieves a zero-shot accuracy of 71.2\\% on ImageNet (as opposed to 75.7\\% mentioned in the LiT paper). We hypothesize that this difference is due to some training hyperparameters not fully specified in the LiT paper (such as the details of the Adafactor optimizer) and some possible differences in the training dataset (since some of the URLs in the CC12 and YFCC datasets may have become inactive, hence affecting our download of the dataset). However, under the same training setup, the CWCL model achieves 76.48\\%.  In general, we again find that the CWCL-based models achieve much better performance. \n\n**Q3: Instead...**\n    \nThis is an interesting suggestion. When the weights for the negative pairs are set to 0, the resulting loss function is close to the standard loss function, except that the weights for the positive pairs can still be tuned. This might help with dealing with \"faulty positives\". We thank the reviewer for their suggestion. \n\n In the current paper, we do not explore this and simply set the weights for the positive pairs to be 1, which results in the standard contrastive loss function. \n"}, {"review_id": "G5zfvhhJUR", "reviewer": "Reviewer_yjNF", "summary": "In this work, the authors consider the problem of cross-modal zero-shot transfer and propose a new loss function called continuously weighted contrastive learning (CWCL) that extracts better supervision from pretrained models in a single modality and leads to better alignment between two modalities. They run experiments on two modality pairs, image-text and speech-text, and CWCL yields significant improvements compared to standard contrastive learning (with up to 20-30% absolute improvement on a speech-to-intent classification task).", "questions": "- Did the authors experiment with other weighting functions for w_ij's in Equation 3? Some insights on how w_ij = q_i^T q_j/2 + 0.5 was arrived at would be useful for the reader.\n- As in Figure 5, does CWCL exhibit robustness to templates for the speech-to-intent classification task as well?\n- From Table 1, CWCL outperforms the supervised Resnet50 baseline on ImageNet-V2 as well as ImageNet-R, ImageNet-A and ObjNet by substantially larger margins. Is Resnet50 the appropriate supervised baseline to show for these datasets?\n- From Table 2:\n    - RoBERTa+S performs consistently better than BART+Y on SLURP and STOP, but the trend flips for KWS. Can the authors comment on why that is?\n    - CWCL with RoBERTa+S appears to outperform the text-intent upper bound for STOP (87.87 vs. 84.78). Please explain this result.\n- Just a comment: If the authors are looking to make some space in the draft, Figure 1 can be omitted without loss of clarity.", "limitations": "Limitations have not been explicitly listed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors present a simple and effective contrastive learning variant that makes effective use of pretrained models in individual modalities.\n- They achieve a significant performance boost in zero-shot image classification when compared to existing contrastive learning objectives.\n- They also demonstrate superior results on a zero-shot speech-to-intent classification task.", "weaknesses": "The main contribution of this work (CWCL) can be grounded better in the context of existing work. This work highlights an important shortcoming in existing formulations of contrastive learning, namely that similarity has a binary interpretation. All positive examples and all negative examples are treated equally. This has been identified in prior work, albeit for a single modality, which the authors fail to cite. E.g., \"Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification\", V. Suresh and D. Ong, EMNLP 2021. There are also works on multimodal learning that highlight the importance of using a weighted contrastive loss to downweight faulty positives/negatives such as \"Robust Audio-Visual Instance Discrimination\", Morgado et al., CVPR 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 : The main...**\n    \nBoth references are very interesting and relevant and we will cite them.  We provide a detailed comparison below. \n\n**Comparison with \"Not all Negatives are Equal:..\"**\n     The ideas explored in this paper are similar in spirit to those considered in our paper. However, this paper considers supervised learning. In our work, we consider fully self-supervised learning. This lets us use our models on many downstream tasks in a zero-shot way, whereas in the supervised setting, the resulting models are applicable only to the specific task that they are trained for. \n\n**Comparison with \"Robust Audio-Visual Instance Discrimination\" (soft-XID)}:**\n\nIt is our belief that when models are trained initially using standard contrastive loss and then trained further by using weighted contrastive loss, biases from the initial training stage get further reinforced later on. In the initial stage of training, the encoders are trained to repel all samples except only samples belonging to the same pair. When the same encoders are used to obtain the similarity scores in the second stage, these similarity scores are unreliable and will further reinforce the biases from the first stage.\n\nWe performed an experiment that similar in spirit to the above paper. We trained a pair of image and text encoders using the standard contrastive loss using the CC12M dataset for 20 epochs. Note that this is similar to the setup of the CLIP model. After this, we fine-tuned the model using weighted contrastive loss for another 20 epochs. In the second stage, we obtain the similarity scores using the encoders trained in the first stage, as proposed in \"Robust ...\" paper. We then measure the zero-shot image classification accuracy on ImageNet. We also train a model using the proposed CWCL method for a total of 40 epochs. We provide the results below in Table 11. We further note that in the first experiment (emulating soft-XID method) the 0-shot accuracy decreased upon fine-tuning, further indicating a conflict between the similarity scores generated using the models trained using standard contrastive loss. \n\nOverall, we find that the method of first using standard contrastive loss and then using a weighted contrastive loss yields worse performance as compared to our proposed method.\n\n**Q1 Did...**\n    \nWe first considered a softmax-based weighting function defined as over the intra-modal similarities and secondly our proposed function.      The second formulation performs slightly better than the first one. Further, it results in a nicer interpretation of the proposed CWCL loss function as an interpolation between standard contrastive loss and the supervised contrastive loss function, as mentioned in lines 130-134 in the main paper. Further, note that in the second formulation, embeddings with $\\langle q_i, q_j\\rangle = -1$ result in a weight of 0, thus not resulting in any \"attractive force\" between such samples. \n\n\n**Q2: As in...** Yes, we provide thein Table 10 of the rebuttal PDF. \n\n**Q3: From Table 1...**\n\nOne of the advantages of models such as CLIP, LiT, and CWCL is that the same model can be used on multiple downstream tasks and datasets in a zero-shot way. To perform a more direct comparison to this regime, we choose a supervised and fine-tuned model that achieves a comparable performance on ImageNet and we report the same model's (after fine-tuning) accuracies on the other datasets. We wanted to better demonstrate how using a single model for multiple tasks performs and compare it with the CWCL-based model. Further, we also wish to note that the authors in Zhai et.al.,(LiT) make the same comparison.\n\n**Q4: From Table 2 RoBERTA+S...**\n\nThe pre-trained text model used in RoBERTa+S was trained on the data including SLURP intent classification data (but only the text data. The SLURP speech data was not used in any stage of the training. Further, no data from STOP was used in the pre-training stage). We think that this affected the model to overfit on the task of intent classification in general, outperforming BART+Y (based on performances on SLURP and STOP) while performing worse in keyword spotting (KWS) than BART+Y.\n\n**Q5: From Table 2 CWCL...**\n\nWe would like to clarify that in using the term \"upper bound\", we meant to convey that the text models' performance on intent classification serves as a \"reference\" for what can be expected from the speech encoder after training. We elaborate further below.\n\nIn our method, we \"freeze\" the text encoder and train the speech encoder using a paired (speech-text) dataset. In particular, we use the commonVoice dataset. Note that the text model does not get updated and hence does not get a chance to process this dataset. Further, the speech encoder was initialized using a pre-trained ASR model's encoder. A third difference between the text model and the speech mode is that the speech tower might have learned how to utilize acoustic cues in addition to linguistic information from a given speech utterance, to align its embedding to the semantic embedding from the text tower.  These three differences could contribute to the speech model sometimes performing better than the text model. However, we still expect the text model's performance to be a good reference for what the speech model can achieve after training. \n\nAnother factor that might have led to this is that the number of classes in the STOP dataset is low (8), as opposed to the SLURP dataset (which has 60 classes). Thus, in the case of SLURP, the performance on the speech-intent task is lower than the text-intent task, since it is more challenging."}, {"review_id": "7qFk7oSSbp", "reviewer": "Reviewer_q9nH", "summary": "The authors propose a composite loss in contrastive learning in order to preserve structure between two embedding spaces for different modalities. In the setting in which they study, two unimodal encoders are aligned by freezing one of them and computing a \"continuous\" contrastive weight between samples $(i, j)$ as $0.5 + \\frac{q_i^T q_j}{2}$. With these weights an additional loss for structure preservation is used in addition to the standard self-supervised loss for the unfrozen encoder. The proposed approach outperforms standard approaches such as LiT, SimCLR on 0 shot classification tasks. ", "questions": "- Which continuous pairwise weights did the authors try? Given this is the main contribution it would be helpful to understand performance as this weighting changes in the 0.5 bias used or for nonlinear weighting wrt to the inner products.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The method is simple and efficient requiring a small $\\mathcal(O)(k^2)$ computation for pairwise weights per batch\n- Performance improvements are strong across the experimentation", "weaknesses": "- There is a lot of setup while the entirety of novel contributions seem to be buried in section 2.3.\n- This paper makes more sense for a workshop, the contributions are limited and is a small modification of previous works such as (https://arxiv.org/pdf/1905.12837.pdf, https://arxiv.org/pdf/1811.01459.pdf, https://arxiv.org/pdf/2103.14003.pdf)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: There is a...**\n    \nPlease note the problem setup is only done in Section 2.1. We request the reviewer to elaborate further on what they mean by \"lot of setup\". \n\nFurther, our contributions are multi-fold and are not confined to Section 2.3. The novel CWCL loss function is introduced in Section 2.2, while Section 2.3 discusses how we calculate the similarity weights. \n\nAdditionally, we demonstrate the effectiveness of the proposed methods using many experiments, models, datasets, and modalities. For example, we show the \"template robustness\" property of the proposed methods and we provide qualitative results showing that the proposed loss function leads to better alignment between the modalities. \n\nWe believe that this is the first work to study 0-shot speech-intent classification where no task-specific speech data was used for training. Further, our results show a strong improvement over methods that use standard contrastive loss. We would consider these results to be novel.\n\nIn general, we request the reviewer to elaborate on why they feel that these other aspects are not novel. We believe that in a field like machine learning, an idea followed by extensive experimental results that demonstrate the effectiveness of the idea together contribute to novelty.\n\n\n**W2: This paper...**\n\n*The General Pair-based Weighting Loss for Deep Metric Learning (\\url{https://arxiv.org/pdf/1905.12837.pdf})*:\n\nThe referred paper is very different from ours for the following reasons:\n\nFirstly, it studies the problem of metric learning in a supervised setting, where class information is available in the dataset. This is mentioned explicitly in Section 2.A of the paper where they formulate the problem and in Algorithm 1(authors mine “positive” and “negative” examples using Equations (16, 17) by using class labels). \n\n\nSecondly, they do not consider multi-modality. The algorithms proposed in this paper are geared towards a single modality. Hence, the resulting models cannot be used for other downstream tasks in a zero-shot way. On the contrary, they can only be used for the specific task that they are trained on. \n\nIn our paper, we have already acknowledged a weighted version of contrastive loss has been developed in the supervised learning case by Khosla et. al., (NeurIPS 2020, ref. [14] in our paper). However, extending weighted contrastive loss to self-supervised learning has not been studied before. Further, models such as CLIP and LiT have proven that the scale of data and models have a large impact on downstream performance. But they only use the standard contrastive loss because developing a weighted contrastive loss in the context of the work was not explored before. Thus, our work is not a simple modification of the above paper.\n\n*Deep Metric Learning by Online Soft Mining and Class-Aware Attention\\url{https://arxiv.org/pdf/1811.01459.pdf}*:\n\nThis also addresses the problem of metric learning in a supervised setting. This is highlighted in the main methodology section, where they consider a uni-modal dataset of the form ${x_i, y_i}$ where $y_i$ is the class label. This class label is used to define the positive set $P$ (samples with the same class label) and negative set $N$ (samples with different class labels), used to compute the losses in eqns. (7, 8, 9).\n\n*Rethinking Deep Contrastive Learning with Embedding Memory\\url{https://arxiv.org/pdf/2103.14003.pdf}*:\n\nThis also addresses the supervised learning case; as we quote the introduction section of the above paper: “In this work, we simply adopt MoCo in a supervised manner for DML, which is referred to as s-MoCo. It provides a strong example for us to study the pair weighting in memory-based DML”.\n\n**Q3: Which..**\n    Please refer to the answer we provided to Reviewer 2's questions on the different similarity functions that we considered. \n\n\n**Overall comments on comparison with the above three papers:**\n\nWe understand the reviewer’s concern that these papers study variations of weighted contrastive loss. However, extending weighted contrastive loss from a supervised setting to a self-supervised setting is non-trivial. To summarize the difficulty, all three papers pointed out consider the following question: Using prior information about positive and negative pairs, how to weigh positive pairs that are further apart to bring them closer, and weigh negative pairs that are closer so that they are pushed apart? This is captured in Figure 2. of the third paper pointed out by the reviewer (https://arxiv.org/pdf/2103.14003.pdf). Thus, an optimal solution here is where all positive samples are brought as close as possible, and all negative samples are pushed away as far as possible. On the contrary, in our work, there is no notion of strictly positive or negative. This is a core contribution of the paper, a softer notion of similarity helps us recreate the embedding space in one modality based on a pre-trained model from another modality. \n \nThe difference between bringing together all positive samples and repelling all negative samples (with weighted or unweighted losses) and that of considering similarity as a continuous metric and learning a continuous structure in the embedding space of a modality may be subtle, but the resulting algorithms and models have highly different capabilities and applications. Using supervised labels to construct positive and negative sets makes the models specific to those class labels; features that distinguish two inputs within a class are filtered out and features that distinguish two inputs from different classes are accentuated. However, since our models do not make use of class-specific labels, they retain all features and are capable of zero-shot classification on many downstream tasks.\n\nWe hope we have addressed the reviewer’s concerns regarding the relationship between weighted contrastive loss in a supervised setting and our work and invite them for more discussion.  \n"}, {"review_id": "x4pN2gQwff", "reviewer": "Reviewer_1ozP", "summary": "The paper proposes a novel continuously weighted contrastive loss function for multi-modal models. The proposed method takes consideration of 1) similarity of samples in the same batch 2) the non-binary nature of similarity between input samples,  and therefore utilizes a continuous weight matrix in learning alignment between modalities. In general I think that the ideas on this paper are novel. The explanation of the methods is explicit and concrete. I think it is enough to be reproducible. However the paper may benefit from some revision regard to how the work is presented and focus of experiments.", "questions": "* Is it possible for you to provide results for retrieval compared to the baseline models? This would address a major weakness in the evaluation.", "limitations": "* There is no explicit limitations section in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The proposed idea is novel in resolving two crucial problems in standard contrastive loss.\n* The authors considered modalities of both text-image and speech-image and presented some supportive zero-shot evaluation results.", "weaknesses": "* The experimental results are not complete, it would be good to see a more complete set of results on how this method would perform on the downstream retrieval tasks (T4 compares between CL and CWCL, without indicating performance compared to baseline models such as LiT or CLIP).\n* The experiments regarding the transfer from text to speech are nice however the results in Section 4.2.1 is not clearly presented and is a bit hard to follow.\n* Figure1 is not really informative and maybe can be combined with Figure 4 so that more interesting  and qualitative results can be presented.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Results are incomplete...**\n    \nThank you for raising this point. In Table 4, the row corresponding to CL is indeed the LiT model. We named it CL to indicate that this model was tuned using standard contrastive loss as done in the LiT paper. In fact, the model referred to in this row follows the same training procedure as LiT, with the image encoder initialized using a pre-trained model and frozen during training. We will make this change and rename the row to LiT to clarify this.\n\nFurther, the results reported in Table 4 correspond to using the ViT-L/16 architecture for the image encoder and a 12-layer transformer model for the text encoder; trained using the YFCC15M and the CC12M datasets. However, we noticed that the LiT model achieves the best performance (using a publicly available dataset) when the BERT-large model is used as the text encoder.\n\nThus, we performed further experiments with the BERT-large model as the text encoder. We again observe an improvement in the retrieval performance. We provide the results in Table 12. Further, we also observe performance gains in the zero-shot image classification task as well. We provide the results in Table 13. \n    \nIn the paper, we did not compare CWCL against CLIP on the retrieval task, since we expect CLIP to have lower performance than LiT (and CWCL already performs better than LiT). However, we realize that a direct comparison between CWCL and CLIP might be of interest. To this end, we provide a direct comparison of the two methods using zero-shot retrieval performance on the MS-COCO dataset. Due to the limited time frame, we trained new CWCL and CLIP models on a subset of the training data that consists of samples from/ only CC12M. Further, we trained both models for only 20 epochs. Although this setup results in low absolute performance on both models, we can still obtain a direct relative comparison. We outline the results in Table 14. Interestingly, the models trained on the CC12M dataset (Table 14) perform better at retrieval than models trained on both YFCC15M and CC12M (Table 4). We have noticed such behavior in open-source CLIP models (\\url{https://github.com/mlfoundations/open_clip}) as well. We hypothesize that this is because CC12M is a dataset that has been subject to filtering out bad samples, as opposed to YFCC15M which is not.\n\n\nWe hope that we have addressed the reviewer's concerns about the results of the retrieval task performance of CWCL based approach. We are happy to answer any further questions.\n\n\n**W2: Exp Text to Speech...**\n    \nThank you for this comment. We reviewed and edited the entire section 4.2 carefully considering readers could be less familiar than us with the contents we wrote. We were also short on space and hence combined all results into a single table (Table 2 in the original manuscript). \nIn the revised paper, we will improve this section for readability. \n    \n**W3: Figure 1...**\n\nThank you for this suggestion. We will combine Figures 1 and 4. \n\n**Q1: Is it...**\n\nWe have provided the results of retrieval tasks in our response to one of your earlier comments. Further, we also provide results based on new experiments that studied the ViT-L-16+BERT-large configuration. We appreciate the reviewer's feedback regarding our experiments and we hope we have addressed their concern.\n\n**Summary**\n\nOverall, we appreciate your questions and suggestions. They have helped us improve the paper. We agree that including performance on retrieval tasks adds to the completeness of our results. We will include these in the main paper manuscript. We will also clarify the results in Section 4.2.1. We were a little pressed for space in the original manuscript but we will better manage the space in the revised version by moving Figure 1. This will help us clarify Section 4.2.1 further. \n\nWe hope that we have addressed your concerns about our paper and sincerely hope to discuss further with you during the discussion phase. \n"}]

Author rebuttal:
We would like to thank all the reviewers for their time and effort in reviewing our paper. We appreciate all of the comments and they have helped us improve our paper. Here, we first provide a summary of our paper. Then, we outline the major concerns expressed by the reviewers and explain how we have addressed them. Following this, we also provide a summary of the positive comments made by the reviewers.

**Paper summary:**
 1. We consider the problem of cross-modal transfer, where representations in one modality are learned using a pre-trained model in another modality and a paired dataset that consists of pairs of the two modalities.
2. To address inefficiencies in the training objective used in existing work, we propose a novel loss function called "Continuously Weighted Contrastive Loss" (CWCL) that considers associations between all the pairs in a training batch. 
3. We show that the new loss function leads to a 5-8% (absolute) increase in 0-shot task performance for the image-text modality pair. We also study our method in the context of speech-text modality pair. We provide the first results in 0-shot speech-to-intent classification and keyword spotting, where our models achieve 20-30% (absolute) improvement over existing methods.

Overall, the reviewers had the following positive comments about our paper: The reviewers feel that our work addresses an important shortcoming in existing formulations (yjNF, 1ozP)and that we propose a novel solution that is also simple and efficient (q9nH). All the reviewers expressed that our experimental results are very strong. Reviewers also found the template-robustness property of our models interesting (5jbW).

**Summary of reviewer concerns**
We categorized the reviewers' concerns into broader themes. We describe them below by providing a summary of our responses to each of them.

**Completeness of experiments and interpretation of the results:** Reviewer 5jbW felt that some experiments were missing for the image-text modality pair, yjNF had some questions about experimental results, reviewer q9nH had a question about the design choice for the similarity function, and 1ozP asked for more results on the retrieval task for the image-text modality pair. We summarize our responses below:
 - We provide new experimental results in the rebuttal including a direct comparison between our method, CLIP, and LiT on the retrieval task. We also provide updated results for all the image-text experiments with a new model architecture based on the ViT-L-16+BERT-large configuration. 
- In particular, our new image-text model achieves a 0-shot ImageNet accuracy of 76.5\%, which is better than the best baseline (LiT) when trained on the same dataset.
- We have provided explanations for the choice of our baseline models in our response to reviewer yjNF. We have also provided explanations for some of the results in Table 2 regarding the 0-shot performance in speech-text tasks.
- We have addressed why we chose the similarity function that we use in our paper. 
- We realized that we used two different naming conventions (`CL` and `LiT`) for the baseline method that uses the standard contrastive learning loss. This led to some confusion regarding the equivalence of the experiments between the image-text and speech-text modality pairs. We will correct this to ensure uniformity. 

**Comparison to existing work:** Reviewers yjNF and q9nH pointed us to a few papers that are relevant to our work. We provided details responses comparing our work to them in the rebuttal and we summarize it here.

All but one of the papers pointed out by the reviewers consider the weighted contrastive loss for supervised learning, where class information is available. We have cited the work of Khosla et.al., in our main paper which also considers contrastive loss in the supervised case. In our work, we consider self-supervised learning. In the former case, conditioned on
prior information about which pairs are positive pairs and which are negative pairs, positive pairs that are further apart are weighted highly to bring them closer, and negative pairs that are closer are weighted highly so that they are pushed apart strongly. On the contrary, in our work, there is no notion of strictly positive or negative. This is a core contribution of the paper since a softer notion of similarity helps us recreate the embedding space in one modality based on a pre-trained model from another modality. 

One of the papers considers self-supervised learning and is more similar to our paper. However, since they do not use pre-trained models, they need to use a "self-teaching" process, where a model is trained initially using only standard contrastive loss and then fine-tuned using a weighted contrastive loss.
During the rebuttal phase, we performed experiments with the self-teaching strategy and we found that that performance usually does not improve much or even degrades in our experiments. This may be due to the conflicting nature of the two stages of training. We provide experimental results in our response. Further, The weighted loss they propose is designed to "down-weight" the effect of faulty positives and "up-weight" faulty negatives. In our method, we do not consider samples as positives or negatives.  

**Presentation of the paper:** We also received some comments about the presentation in our paper. We will address these concerns in the revised version of our paper. 

Meta-review: This paper proposes a contrastive objective for cross modal transfer. I think the idea is interesting and the experiments empirically validate its efficacy. Reviewer q9nH has some concerns around the novelty of the proposed method. However, I think the authors have sufficiently addressed this in their rebuttal. All other reviewers provided positive feedback for the work. I recommend accepting the paper to the conference.
Decision: Accept (poster)