Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings
Authors: Yiren Jian
Chongyang Gao
Soroush Vosoughi
Venue: NeurIPS 2022 Accept (2022)

Abstract: Semantic representation learning for sentences is an important and well-studied problem in NLP. The current trend for this task involves training a Transformer-based sentence encoder through a contrastive objective with text, i.e., clustering sentences with semantically similar meanings and scattering others. In this work, we find the performance of Transformer models as sentence encoders can be improved by training with multi-modal multi-task losses, using unpaired examples from another modality (e.g., sentences and unrelated image/audio data). In particular, besides learning by the contrastive loss on text, our model clusters examples from a non-linguistic domain (e.g., visual/audio) with a similar contrastive loss at the same time.  The reliance of our framework on unpaired non-linguistic data makes it language-agnostic, enabling it to be widely applicable beyond English NLP. Experiments on 7 semantic textual similarity benchmarks reveal that models trained with the additional non-linguistic (images/audio) contrastive objective lead to higher quality sentence embeddings. This indicates that Transformer models are able to generalize better by doing a similar task (i.e., clustering) with \textit{unpaired} examples from different modalities in a multi-task fashion. The code is available at https://github.com/yiren-jian/NonLing-CSE.

Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings
Yiren Jian Department of Computer Science Dartmouth College yiren. jian. gr@dartmouth edu
Chongyang Gao
Soroush Vosoughi
Department of Computer Science
Department Computer Science
Northwestern University www.wwwstern.ccommmmmmm edu hooggyaggg220uu...rrrveeeee..
Dartmouth College
soroush. voroush.vosough. edartmouthe edu
Abstract
Semantic representation learning for sentences is an important and well-studied problem in NLP. The current trend for this task involves training Transformer- based sentence encoder through contrastive objective with text, i.c., clustering sentences with semantically similar meanings and scattering others. In this work, we find the performance of Transformer models as sentence encoders can be im- proved by training with multi-modal multi-task losses, using unpaired examples from another modality (e.g., sentences and unrelated image/audio data). In particu- lar, besides learning by the contrastive loss on text, our model clusters examples from a non- non-linguistic domain (e.g., visual/audio) with similar contrastive loss the same time. The reliance of our framework on unpaired non-linguistic data makes it language-agnostic, enabling it to be widely applicable beyond English NLP. Experiments on 7 semantic textual similarity benchmarks reveal that models trained with the additional non-linguistic (images/audio) contrastive objective lead to higher quality sentence embeddings. This indicates that Transformer models are able to generalize better by doing similar task (i.e., clustering) with unpaired examples from different modalities in a multi-task fashion. The code available at https://github.comyyyrennjjjjnn Lin--CSE.
Introduction
Learning semantically meaningful sentence embeddings is important task in Natural Language Processing (NLP), with applications for semantic retrieval and search among other things. The embeddings of sentences are such that semantically similar sentences are located closer to each other in the embedding space. The current state-of-the-art method for learning sentence embeddings is SimCSE 133. which uses contrastive learning. Unsupervised contrastive learning clusters the representations of same examples under different augmentations. SimCSE's main contribution is the use of Dropout augmentation, which they show outperforms other traditional text augmentation methods (c.g.. random deletion, swapping, etc.) for learning sentence embeddings. To improve the results further, supervised SimCSE leverages a labeled Natural Language Inference (NLJ) dataset during the learning. Each entry of the NLII dataset is a triplet of sentences, source sentence and its positive and negative pairs (e.g.. "Two dogs are running" (src), "There are animals outdoors' (pos),
"Contributed as co- first author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Contrast with SimCLR
Contrast with SupCon
[CLS]
B x 768
BERT-base
Text Feature x 32 768
Image Feature x 196 768
Text Embedding Layer
ViT Image Embedding Layer
there exist namber differ enl possible events
this frame contians which are like those destrability
reciplent starts off without the theme in their poseesion words
jo
B
static position
BB
entity
linguistic
Figure 1: Overview of our method VisualCSE. The mini- mini-batch contains two batches, one with sentences from Wikipedia and the other with images from ImageNet. The text batch processed by BERT embedding layer and the image batch is processed by a Vision Transformer (VIT) [ embedding layer separately. The resulted text and image features are fed into the Transformer encoder and we take the outputs at [CLS] position as the representations of examples. Like in SimCSE, the text representations are supervised by the SimCLR loss. We use the SupCon objective to contrast representations from images.
"The pets are sitting on a couch" (neg)). Supervised SimCSE uses hard- nerd-iege and perong-positivee terms in the contrastive learning objective based on the negative and positive examples. Because of the high quality negative and positive pairs in the NLI dataset, the learning signal from these terms has been shown to be very effective in learning sentence embeddings [3].
However, as we show in this paper, the benefits of a labeled dataset (such as NLD) are highly dependent on its size and the annotation quality. The need for such large and high quality datasets limits the applicability of the method to high-resource languages. For instance, the dataset used by SimCSE is a combination of the SNLI [2] and MNLI (57 datasets which contain 570K and 433K manually annotated examples, respectively. Constructing such a dataset is costly and may be impossible in low-resource languages. In fact, it has been shown that the performance of supervised SimCSE highly dependent on the specific labeled dataset used. Specifically, training SimCSE with QQP 43). ANLI (37, Flickr30k (62), and ParaNMT (56) was shown to lead to significant performance drops [3.. Such specific reliance on an NLI dataset is especially problematic as constructing it in new language can be very difficult as it requires annotators to understand each sentence and write seemingly related, but semantically different sentence. Further supporting this claim, Zhang et al. [64] show that if the distribution of the test set differs significantly from the NLI dataset used for training, the quality of the semantic embeddings degraded.
The aforementioned issues show that learning sentence embeddings from unlabeled text data is crucial task that requires further exploration. In this paper, we investigate the addition of non-linguistic supervision (provided by easily obtainable public datasets) contrastive-learingg based sentence

---

embedding learning. Specifically, we propose Transformer-base multi- multi-task framework which jointly learns two independent contrastive tasks (i.c., one textual and one non-textual) using unpaired examples. Our method (shown in Figure D is inspired by Lu et al. 35 which shows the ability of Transformers to transfer knowledge from one modality to another (e.g., they show that a Transformer model pre-trained on text can be fine-tuned on downstream visual tasks). In addition to generating higher-quality sentence embeddings, our investigations shed light on the capability of Transformers to learn from mixed modalities when trained similar tasks. This is complementary to the findings of Lu et al., as those concern the cross-modal transfer ability of Transformers.
Though our method can be used with any modality, we use vision as the modality of choice to describe our method and later show the generalizability of our method audio. As shown in Figure [ our method, VisualCSE (or AudioCSE when learning with audio), has a multi-task loss with two terms: one contrastive learning loss for text, which the same as the one used in SimCSE, and an additional contrastive learning loss to cluster data from images concurrently. Note that the two learning processes are independent of each other, thus allowing us to leverage unpaired text and image/audio datasets, e.g., sentences from Wikipedia, images from ImageNet [9], and audio from LibriSpeech
(39]. Our method is different than other multi-modal representation learning methods (such as CLIP 44 in that it does not require paired examples across modalities. This crucial feature makes our method easy to deploy for low resoureee languages. In addition, our method also uses single model with all parameters shared for tasks in both modalities, highlighting the ability of Transformers to learn from non non-parallel mixed modalities when trained on similar tasks.
The contributions of this work are as follows:
(1) We propose novel framework for introducing non-linguistic supervision for centrastive-earming based sentence embedding learning. Crucially, our framework is language-agnostic does not rely on specific modality, and does not require paired examples across modalities. We show that the supervision provided by other modalities can improve unsupervised sentence embedding learning, enabling sentence embedding learning low-resource languages. We improve SimCSE in several semantic textual similarity benchmarks with 3 different pre- pre-trained language models (LMs).
(2) Complimentary the findings of Lu et al. [35), which demonstrate the capability of Transformers to transfer knowledge from language to vision, we show that Transformer models can generalize better by learning similar task (i.e., clustering) with multi-task losses using non-paralled examples from different modalities.
2 Related Work
Sentence Embedding Learning. Early works on learning sentence embeddings include training model to reconstruct the nearby sentences of an encoded sentence [30]. Sent2 (38] devises an efficient loss for training the sentence encoder, inspired by the word embedding algorithm Word2Vec. Recent works in this area are based on contrastive learning: through taking two different views of examples 14), data augmentation [59], or by passing examples into different encoders 4 28 64).
Post-processing methods also exist for improving the quality of embeddings 23 32 47). The current state-of-the the--rte sentence embeddings method, SimCSE [3] passes the text into an encoder twice and uses Dropout as the augmentation to create niews of examples for contrastive learning. All these methods (and recent works such DiffCSE (8), DebiasedCSE 63), PromptBERT (25) and
many others 3 16 encoders.
22
26
52
solely use text datasets for training their sentence
A concurrent work, MCSE (63 is the first attempt to learn sentence embedding using text with another modality (image). But MCSE requires aligned image-text pairs (i.c., images and captions from Flickr), limiting its applicability to other languages and modalities. We are the first to show that the embedding quality of a sentence encoder can be improved by jointly training with unpaired examples from another modality.
Contrastive Learning. Contrastive learning 19 49 58 60 is a form of self supervupere training using a loss derived from noise contrastive estimation (50 The seminal method SimCLR (5) clusters the same examples under two sets of augmentation (positive pairs), and pulls away different examples in a mini-batch (negative pairs). To alleviate the hardware requirements of SimCLR, MoCo [18] designs moving average memory bank for storing negative samples, which simulates large number
3

---

of negative samples for effective contrastive learning. CMC 49 further extends SimCLR from two-view multi-view contrastive learning. While the contrastive loss was originally designed for unsupervised pre-training, SupCon [27] finds that additionally contrasting examples based on label information beneficial when labels of examples are available. In NL.., contrastive learning mainly have shown it to The contrastive objective perfectly matches the goal of sentence embeddings 13 i.e., semantically similar sentences should have comparable representations. Our method for learning sentence em- beddings contrasts examples from text and examples from another modality (e.g., image or audio) simultaneously. The image and audio datasets we used come with "free" " labels, allowing us to apply a loss based on SupCon, in addition to SimCLR.
applies to sentence embedding learning 38 28 59 (4]. but other works 17 24
be beneficial to few-shot learning [[2] as well.
Learning Representations With Multiple Modalities. Using paired image-text instances, CLIP [44
learns aligned representations of text and images by two separate encoders, achieving superior results in several downstream tasks. Building upon CLIP, SLIP [36] further improves the representations of examples by leveraging self-supervised contrastive objective. UniT (21) learns multi-modal Transformer model with a of aligned tasks from visual and language domains. In video learning, XDC M learns an encoder by cross pseudo-labeling between parallel RGB frames and audio signals.
The data used for training such models includes images and their associated captions (CLIP and SLIP), aligned visual- tasks (UniT), or video clips that come with RGB frames and the corresponding audio (XDC). These methods, well many other works multi-modal learning 34 48, 61, learn better representations by leveraging information shared among different instances (i.e., paired examples). Our method also learns better representations
29.
modes of the of sentences from examples of different modalities. However, the datasets used for training our method do not require parallel examples. Our method performs similar contrastive tasks in two independent domains at the same time, and generalizes better as a sentence encoder by becoming better clustering model.
3 Technical Method
3.1 Overview
Given text dataset Siex (e.g., Wikipedia or NLI), dataset from different modality (e.g., an image dataset Smmage (e.g., ImageNet), or an audio dataset Saudio (e.g.. LibriSpeech)), and a pre-trained LM M (e.g., BERT), our goal is fine-tune M with Seett and / using contrastive losses such that the resulting model can produce high-quality sentence representations.
In the following sections, we use images as the example show how non-linguistic supervision can improve the sentence embedding learning (VisualCSE). A similar approach is used to adapt our learning algorithm audio (AudioCSE), as shown Appendix B Our method trains the sentence encoder with multi-task loss:
Liaat Luxt + Wimage Camage
(1)
where Winage a weighting factor. We provide further details of Cuea and Limage in Sections 3.2 and respectively.
3.3
3.2
Learning from Text
We train the sentence encoder with text using the method laid out by SimCSE. This allows to directly investigate the contribution of additional non-linguistic supervision for training sentence encoders. Here, we briefly summarize how SimCSE works.
SimCSE learns the sentence encoder by contrastive loss, using Dropout as form of data augmenta- tion. For a single unlabeled example of Siex (Wikipedia) in a mini-batch, SimCSE passes x; into the Transformer encoder M twice. The corresponding output hidden states [CLS] position (which are used as sentence embeddings) are denoted as h and h where and zf are two different random masks by Dropout layers of the Transformer model. SimCSE applies the SimCLR loss to contrast examples in mini-batel i.c., the unsupervised loss based on data augmentation is as

---

follows:
N
eaim(h 11 1//
Cunsup
log
(2)
N Ej=1 sim(h, b/
where denotes another example in the mini-batch, N is the size of mini-batch, sim is the operation for computing cosine similarity, and t is temperature scaling factor. The key to SimCSE is using Dropout as a form of augmentation, the rest of its contrastive learning follows SimCLR.
Sexx is a labeled NLI dataset that comes with (ra, of 2. (i.c., source, entailment, contradiction), corresponding to the hidden states (h,, h h2 ), its supervised loss defined as:
N
Sup
log
(3)
(e s b,))/ 1/r + e sim(b,, 1/T)
Following SimCSE, we use 0.05 for both losses.
3.3 Learning from Images
We train our encoder with images using SupCon (main results) and SimCLR (in ablation). For an image yi in Simgge. we apply two sets of data augmentations in the input RGB space. The resulting views of the input (y/, v,") are passed into a ViT embedding layer and further encoded by the same The SupCon contrastive loss is applied on the two output representations (f", f() at the [CLS] position:
Transformer encoder M from Section 3.2
(fim(ff
Σ
SuppCon
log
Ys and ys from same
(4)
Vimage
Σ Y: y5 from different
e simif; ////T
We use the SupCon default 7 = 0.07. Also, we can substitute SupCon loss with the SimCLR:
N
(fmm(f
SimCLR image
log
(5)
Σラ=1
As we show in Section 4.4 SimCLR and SupCon produce similar performances. This is because we use very small batch size (48) due to the limitations of our GPU memory. Thus, the second term in the numerator of SSupCoon nearly non-existent, and Equation reduces to Equation 5 Lexxt
Note that Limage does not make any assumptions on the image dataset. Thus, we can leverage the easily obtainable ImageNet images. Furthermore, CSimCLR can be applied to any unlabeled images from the Internet, at no cost. This allows our framework be easily deployed for learning sentence embeddings in new domain or new language where NLI, other high-quality labeled datasets are available.
3.4 PyTorch Style of Pseudo-Code
We provide the pseudo-code of our algorithm VisualCSE in the style of PyTorch in Algorithm 1 Learning AudioCSE is similar except that we do not apply transform() to the audio data, but instead pass the audio data into the model twice (i.e., augmentation is provided through Dropout).
3.5
Why Does It Work?
As we can see, Liest and Limuge share a similar (if not the same) form, with the only difference being the input data. We speculate that the Transformer model becomes a better contrastive learner by jointly training with similar but independent contrastive objectives in different modalities.

---

Algorithm 1: Visual contrastive sentence embedding learning (VisualCSE)
# Ste_Step:
Number of training steps
# LM:
The language model
# D_text:
Dataloader for text training set
# image:
Dataloader for image training set set of data transformations
# transform:
# SimCLR:
SimCLR loss
# SupCon:
SupCon loss
# opt_text:
Optimizer for learning from text
# ipt_image:
Optimizer for learning from image
for i in Max_Step"
### Retrieve two augmented views of same sent/ing next (ext)
sent_0, sent_ sent, sent
img, = next (D_image) (rinsform(img) transform (img)
img_0, img_1,
##* Learning from text
t_text (LM IM(sent_1))
L. L_text. backward) ppt_text.step()
### Learning from images
__img SupCon (1M(igg0)) IM(ing.1), y)
L_img. backward
opt_image [tt_image.step(( step()
return
LM
Experiments
4.1 Sctup
Evaluation. Following standard practice for sentence embedding, evaluate approach 7 semantic textual similarity (STS) tasks: STS 2012-2016. STS-Benchmark and For fair comparison, use the same evaluation protocol as SimCSE [13, and use Spearman's correlation as the metric.
Models and Training Datasets. We test our method with three different pre-trained Transformer models used in SimCSE: HERT-base-uncased RoBERTa-base and RoBERTa-large. We fine-tune the pre-trained models with our learning objective, shown in Eq.
For learning with Lext, we 10 sentences down-sampled from the Wikipedia English dataset for unsupervised sentence embedding learning (Eq. ( For supervised sentence embedding learning (Eg. B}. we (and SimCSE) use a combined NLÍ dataset with 314K sentences with paired examples labeled as entailment, neutral, and non con-entailment, SimCSE uses entailment examples as strong- positives and non-entailment examples as bard-nega
strong-positives
For learning with Limuge, both unsupervised and supervised sentence embedding settings use down- sampled ImageNet dataset Simage- Images from Simage are chosen randomly from 60 classes with each class having 500 image The images have shape of B x 224 224, thus the image features processed by the ViT embedding layers are tensors of size B x 197 x ht where B is the batch size and h is the hidden dimension of the L.M. Learning with audio is described in Appendix B
Optimization Details. Following SimCSE [13], we train unsupervised models with Adam for one epoch, and supervised models for 3 epochs. We then search batch sizes and learning rates from (64, 128, 256} and {1e-5 5 -5. 3e.--5 for Lieat Moreover, we use fixed batch size of 48 for Cinage (and Lundio) and search learning rates among (5e-6 61- 5e -7 2e -7 le Ne- The models are selected based on the validation set of the STS-Benchmark.
All the unsupervised base LMs are trained on 24GB Nvidia RTX-6000 GPUs, while supervised and large models are trained on 48GB Nvidia RTX-A6000 GPUs. We use pytorch-1.10 with CUDA 11.3,
It takes SimCSE and VisualCSE thousands of steps converge. Thus, we construct a smaller subset of ImageNet so that the model goes the dataset for few epochs.

---

torchvision-0.1 1.3, forchaudio-0.10.. and Huggingface transformers-45500 for our implementation. Training with a different version of software and hardware may lead different results. We use the default random seed 42 (same as SimCSE) of Huggingface for all of our major experiments for reproducibility.
The validated hyper- parem--rraeterss can be found in Appendix A
4.2 Main Results
We show results of our method learning with visual supervision (VisualCSE) in the unsupervised text setting in Table Our method outperforms baseline SimCSE that learns only from unlabeled text with three different pre-trained LMs. Similarly, we show in Table 2 that learning with audio supervision can improve over SimCSE in almost all tasks.
The improvement in performance can be attributed to the fact that SimCSE only clusters text examples, while VisusalCSE or AudioCSE simultaneously cluster text and examples from another modality, making them better contrastive learners. This could indicate that Transformer-basee models are able to generalize better by doing the same task (e.g., clustering) with independent examples from different modalities in multi-task fashion (i.e, with a multi-task loss). Importantly, this learning setup does not require any paired examples across modalities.
Model
STS12
STS13
STS14
STSIS
STS16
STS-B
SICK-R
Avg.
Unsupervised models
SimCSE -EREEBBERT.... hase
68.40
82.41
74.38
80.91
78.56
76.85
72.23
76.25
VisualCSE BERT buse
71.16
83.29
75.13
81.59
80.05
80.03
71.23
77.50
SimCSE-RoBER Täguse
70.16
81.77
73.24
81.36
80.65
80.22
68.56
76.57
VisualCSE-Rolee BoRCCSEE--BBTTaaee
70.41
83.51
74.87
82.79
81.67
81.89
69.95
77.87
SimCSE-RoBERE Taurge
72.86
83,99
75.62
84.77
81.80
81.98
71.26
78.90
VisualCSE-RoBF RoBERT 73.09
84.77
77.09
85.47
82.06
83.26
72.23
79.71
Table 1: Spearman's correlation of different sentence embedding methods using unsupervised text. Comparing to SimCSE that only leverages text, our VisualCSE takes addition supervision from images. results from the SimCSE paper and reproduced by officially released models.
Model
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
Unsupervised models
SimCSE- BASEEBBETT..... hase
68.40
82.41
74.38
80.91
78.56
76.85
72.23
76.25
AudioCSE-BERT hase
71.65
84.27
76.69
83.22
78.69
79.94
70.49
77.85
CHnCSE-RoBERR Tabos
70.16
81.77
73.24
81.36
80.65
80,22
68.56
76.57
68.44
83.96
75.77
82.38
82.07
81.63
70.56
77.83
SimCSE-RoBERR AudioCSF-RooERTIII 72.10
72.86
83.99
75.62
84.77
81.80
81,98
71.26
78.90
84.30
76.74
85.11
82.51
82.94
72.45
79.45
Table Spearman's 's correlation of different sentence embedding methods using unsupervised text. Comparing to SimCSE that only leverages text, our AudioCSE takes addition supervision from audio.
Though not being the main goal sentence embedding, we evaluate our models on transfer tasks in Appendix D. showing that the sentence representations learned using framework can be well- suited to downstream tasks. We also show example sentence retrieval experiments in Appendix C and VisualCSE with lower quality CIFAR images in Appendix b
4.3 Repeated Experiments
SimCSE uses a fixed random seed (42) for training their models, selects models using the STS-B validation set and reports results on all 7 testing sets of the benchmarks. For direct comparison, we follow this setup in Section 4.2 investigate the statistical significance our results, we further run repeated experiments for unsupervised VisualCSE and SimCSE using 5 different random seeds. Because the best validated model is saved one point during the whole learning process, using different random seeds for training VisualCSE/SimCSE can be thought of as simulating scenarios
7

---

where different subsets of the text and image datasets are used for training the models. As shown in Table we see a performance drop for both SimCSE and VisualCSE using BERT-base. However, VSsualCSE, using either model, is shown to be statistically significantly better than SimCSE.
4.4 Ablation on Limage
By default we variant of SupCon (Eq. contrastive objective for images when learning with visual supervision. This allows us leverage the label information available the ImageNet dataset. Besides clustering different views of examples in a mini-bacth, SupCon also tries to pull together examples in mini-bater that share the same label.
Model
Avg.
SimCSE BERT base
75. 75.7440.90
0.0235
VisualCSE BERT base
76.96=0.38
BinCSE-RoBERTaan KTabase
76.36
<0.0001
77.74
However, in practice, due to the limitations of our GPU memory, we train VisualCSE with a batch size of 48. As we have 60 image classes, each with 500 images, in down-size ImageNet dataset used for training, it is not likely that multiple examples from the same class are contained in one mini-batch. Thus, the SupCon loss in practice degenerates to the SimCLR loss. As we can see from Table changing SupCon to SimCLR does not greatly affect the performance (ablation on Laudio shows similar trends as shown in Appendix E).
Table 3: 5 repeated experiments of unsuper- vised experiments.
report means and stan- dard deviations for average Spearman's 's corre- lation the 7 STS benchmarks.
The results with SimCLR show that our framework can be successfully deployed even when only unlabeled non-linguistic datasets are available. Nevertheless, in this paper, since we use labeled image and audio datasets, use SupCon default loss to possibly harvest the label information presented in our datasets (though this unlikely given our batch size).
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
BERT (SimCLR)
71.81
82.57
75.40
82.58
79.34
78.96
70.83
77.36
BERT (SupCon)
71.16
83.29
75.13
81.59
80.05
80.03
71.23
77.50
RoBERTa (SimCLR)
71.02
83.19
74.47
82.59
81.95
81.94
69.87
77.86
RoBERTa (SupCon)
70.41
83.51
74.87
82.79
81.67
81.89
69.95
77.87
Table 4: Comparison of VisualCSE (base) using SupCon and SimCLR as Limage-
4.5 VisualCSE on Different Languages
One of the main advantages of proposed framework its ability to generate higher quality sentence embed- dings than unsupervised SimCSE, without the need for additional labeled datasets (as opposed to supervised SimCSE, which requires large, high-quality, and dif- ficult to annotate NLI dataset, see Section (.6). This is especially important in low-resource languages where NI.I datasets are hard to acquire. Here, we show the generalizability of framework to other languages. Specifically, we experiment on German, French and Russian, using BERT-base models from HuggingFace (see Appendix G We follow the procedure in SimCSE construct million sentences from Wikipedia of Ger- man, French, Russian and Chinese as training datasets. The evaluations are done on validation sets the multi- lingual STS-B from HuggingFace. As shown in Table 5 VisualCSE can improve SimCSE in all three languages by leveraging additional visual supervision, with Russian seeing an improvement of 4.98 in the Spearman's is correlation.
Language
Model
Spearman's
SimCSE
67.34
German
VisualCSE
69.87
SimCSE
67.98
Chinese
VisualCSE
70.05
SimCSE
70.31
French
VisualCSE
72.52
SimCSE
72.50
Russian
VisualCSE
77.48
Table 5: SimCSE and VisualCSE with dif- ferent languages.

---

4.6 Supervised Sentence Embedding
Lastly, we investigate whether supervision from other modalities can improve supervised sentence embedding. In this setting, the text contrastive loss, Luxi, uses entailment pairs as strong positives in the numerator and has an extra term the denominator the hard- -egative learning signal to push away non-entailment pairs in the NLI dataset (see Eq. 3 Since in our image dataset Sinage (or audio dataset Sandio) we do not have such labeled examples to form the hard-negative pairs, we apply the same unmodified Limage (or Lamdio) to the supervised sentence embedding learning. In Table 6 we observe that Visual/AudioCSE can still outperform supervised SimCSE in all the tasks, but with much smaller margins. This shows that the learning signal from the high-quality negative and positive pairs of the NLI dataset are very strong (leading to 5.32 improvement over unsupervised SimCSE) and cannot be supplemented by supervision from other modalities.
Model
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
Supervised models
SimCSE-BRR BARE-BERTiaa base
75.30
84.67
80.19
85.40
80.82
84,25
80.39
81.57
DisuaCCSE-BERY hase
75.39
84.69
80.34
85.76
81.60
84.85
80.39
81.86
AudioCSE-BERTI base
75.73
84.92
80.25
85.90
81.00
84.36
80.75
81.84
Table 6: Spearman's of different sentence embedding learning methods in the supervised setting.
Analysis
5.1 Visual Supervision Versus NLI Supervision
Given the strong performance of supervised SimCSE with the NLI dataset, natural questions to ask are: (1) How much NLI data can be compensated for through visual supervision of unsupervised VisualCSE? (2) How sensitive supervised SimCSE the quality of the labeled dataset used for the hard negatives? Or in other words, at what level of noise in the NLI dataset, does the unsupervised VisualCSE's performance surpass that of supervised SimCSE?
To answer the first question, we down-size the NLI dataset used in supervised SimCSE into different sizes and train SimCSE with these sub-NLI datasets. As shown in Figure our unsupervised VisualCSE works roughly equal to supervised SimCSE with 30K NLI examples.
80, 76
Spearman's comelation
Spesman's correlation
80
70 78.05
79.07
78.25
78.55
78
28-
77:50
77.09
77.88
Uniup ViHAIDEE
75.24
73,65
72,60
74.82
10x
20x
30x
40x
50x
100k
150k
1.2..0.
20.22200
21
10.4.
0.00
10.4:
2.
21
$0.6.
6...0..
2. 21
p.s. 0.00
pa. 2.
labeled examples
noise level
Figure 2: Supervised SimCSE with dif- ferent subsets of the NLI dataset.
Figure 3: Supervised SimCSE with dif- ferent levels of noise. E.g., [0.3,2,2] is applying 30% random deletion, and insertions and swaps.
As Gao et al. show, the performance of supervised SimCSE is sensitive to the dataset used for supervision in Eq. [3]. For instance, they show that replacing the NLI dataset with ParaNMT can reduce the performance by 6 points (i.e., degrading to the performance of unsupervised SimCSE). Furthermore, they show that using only contradiction without entailment of the NLI also drops the performance by 7 points (they also show performance drops with other datasets, such QQP, Flickr30k, and ANLI). This indicates that the performance of contrastive-leanme learniin rearning-based supervised models for sentence embedding relies heavily the high quality of the IL. dataset.
To explore this, we further run experiments on supervised SimCSE with noisy NLI dataset (at differ- ent levels of noise) and observe the point which our unsupervised VisualCSE starts outperforming

---

supervised SimCSE. We add noise through deletion, insertion, and swapping. As shown in Figure 3] the performance of supervised SimCSE clearly goes down as the quality of the data is degraded. going below unsupervised VisualCSE at around 40% random deletions.
5.2 Uniformity and Alignment of the Sentence Embeddings
The quality of learned representations can also be mea- sured by Uniformity and Aligument [53]. Alignment computed based on the expected distance between representations of positive pairs. Lower Alignment in- dicates close representations between positive pairs:
0.300
0.275
811
0.250
Sip SInCSE (81
(78.)
80
0.225
Align )rPpp. fx) f( Uniformity,
on the other hand, measures how well representations
0.200
Nud:CSE (7.5)
are uniformly scattered across the representation space:
0.175
Unsup VisalCSE (7.))
78
Uniform -(a.y))-an. (l1_p)___,___l(()---aaaa -1
0.150
0.125
-77
We show the Uniformity and Alignment plot of our models evaluated on the STS-B in Figure E Our Vi- suaICSE and AudioCSE exhibit better Aligument than the text-only SimCSE models, both in supervised and unsupervised settings. However, note that the unsu- pervised Visual/AudioCSSt suffer in Uniformity, this is especially true of VisualCSE, which though has better Aligmment than AudioCSE, has lower performance due its worse Uniformity. These findings indicate that the non- con-linguistic supervision in our models make them better clusterers (which is basically what Alignment measures).
0.100
-..0
-3.5
-3.0
-2.5
-2.0
-1.5
uniform
Figure 4: Alignment Uniformity of mod- els based on BERT base (ours in red).
6 Discussion and Limitations
We have shown that non- -on-linguistic supervision provided by unpaired data from the visual and audio domain can improve the performance of nnnrastive-leurmin--be based sentence embedding learners in unsupervised settings. Though this extra supervision does not fully compensate for the presence of the large high-quality labeled dataset used in supervised sentence embedding, we show that it can indeed compensate for smaller, noisier versions. Our findings are important from both a technical point of view and their implications for broader impacts. Our findings indicate that Transformer models can generalize better contrastive-learnerss by learning to cluster with independent data from different modalities. Future work could investigate whether this behavior is true for tasks other than clustering (e.g., classification). In terms of broader impact, as we have shown, our framework is language-agnosse enabling to be deployed for low-resource languages where large, high-quality, and hard-annotated labeled datasets may be hard to obtain. This is possible because method does not require paired datasets (i.e., the same image/audio dataset can be used for any language).
A limitation of our method is the reduction in Uniformity of the sentence embeddings. This means that the representations learned may be grouped together in regions of the representation space. Future work could investigate the reason behind this phenomenon and propose methods to alleviate it. Another limitation of our method the increase the training time, takes twice as long train our models since we require an additional forward/backward pass for the non-linguistic data.
7 Conclusion
In this paper, we find that sentence representation learning can be improved by contrasting unpaired examples from non- linguistics domains, in addition to text. Our framework for incorporating such non-linguistic supervision for sentence embedding is language- and modality-agnostic. Specifically, show that both visual and audio supervision from unpaired examples can improve unsupervised (and to a lesser extent, supervised) contrastive-learnin based sentence representation learners across various languages. Our models, Visual/AudioCSE achieve superior performances in standard sentence embedding benchmarks. Due to its reliance on unpaired data from non-linguistic modalities and its language-agnostic nature, our framework has the potential be used for low-resource languages.
10

---

References
[1] Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[2] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning, A large annotated corpus for learning natural language inference. EMNLP, 2015.
[3] Lele Cao, Emil Larsson, Vilhelm von Ehrenheim, Dhiana Deva Cavalcanti Rocha, Anna Martin, and Sonja Horn. PAUSE: Positive and annealed unlabeled sentence embedding. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 10096-10107, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[4] Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellgvist, and Magnus Sahlgren. Semantic re-tuning with contrastive tension. In International Conference on Learning Representations, 2020.
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple frame- work for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020.
[6] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. European conference on computer vision, pages 104-120. Springer, 2020.
[7] Sumit Chopra. Raia Hadsell, and Yann LeCun. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) '05), volume 1, pages 536-546. IEEE, 2005.
[8] Yung- Chuang, Rumen Dangovski. Hongyin Luo. Yang Zhang, Shiyu Chang, Marin Soljacic, Shang- Wen Li, Scott Yih, Yoon Kim, and James Glass. DifICSE: Difference-hase contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4207-4218, Seattle, United States, July 2022. Association for Computa- tional Linguistics.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, Iece, 2009.
[10] William B Dolan and Chris Brockett. Automatically constructing corpus of sentential para- phrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[12] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Association for Computational Linguistics (ACI.), 2021.
[13] Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
[14] John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. DeCLUTR: Deep contrastive learning for unsupervised textual representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Pupers), pages 895,895, Online, August 2021. Association for Computational Linguistics.
11

---

[15] Yuan Gong, Cheng-I Jeff Lai, Yu-An Chung, and James Glass. Ssast: Self-supervised audio spectrogram transformer. arXiv preprint (tt:::211009975550 2021.
[16] Koustava Goswami, Souray Dutta, Haytham Assem, Theodorus Fransen, and John P. McCrae. Cross-lingual sentence embedding using multi-task learning. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 9099-91133 13, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[17] Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning, In International Conference on Learning Representations, 2020.
[18] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[19] Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In Interna- tional Conference on Machine Learning, pages 4182-4199 PMLR, 2020.
[20] Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168-177, 2004.
[21] Ronghang Hu and Amanpreet Singh. Transformer is all you need: Multimodal multitask learning with a unified transformer. CoRR, 10772102.11772, 2021.
[22] James Y. Huang, Kuan-Hao Huang, and Kai-Wei Chang. Disentangling semantics and syntax in sentence embeddings with pre-trained language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1372-1379, Online, June 2021. Association for Computational Linguistics.
[23] Junjie Huang, Duyu Tang. Wanjun Zhong, Shuai Lu, Linjun Shou, Ming Gong, Daxin Jiang, and Nan Duan. WhiteningBERT: An easy unsupervised sentence embedding approach. In Findings of the Association for Computational Linguisties: EMNLP 2021, pages 238 224, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[24] Yiren Jian, Chongyang Gao, and Soroush Vosoughi. Contrastive learning for prompt-based based few-shot language learners. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5577-5587, Seattle, United States, July 2022. Association for Computational Linguistics.
[25] Ting Jiang, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Liangjie Zhang, and Qi Zhang. Promptbert: Improving bert sentence embeddings with prompts. arXiv preprint arXiv:2201.04337, 2022.
[26) Subhradeep Kayal. Unsupervised sentence-embeddingg by manifold approximation and pro- jection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1-11, Online, April 2021. Association for Computational Linguistics.
[27] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18661-18673. Curran Associates, Inc., 2020.
[28] Taeuk Kim, Kang Min Yoo, and Sang-goo goo Lee. Self-guided contrastive learning for BERT sentence representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 2528-2540, Online, August 2021. Association for Computational Linguistics.
12

---

[29] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision and-aanddgaaa language transformer without convolution or region supervision. In International Conference on Machine Learning, pages 5583-5594. PMLR, 2021.
[30] Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. Advances in neural information processing systems, 28, 2015.
[31] A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
[32] Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119-9130. Online, November 2020. Association for Computational Linguistics.
[33] Che Liu, Rui Wang, Jinghua Liu, Jian Sun, Fei Huang, and Luo Si. DialogueCSE: Dialogue- based contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2396-2406, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[34] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi- task vision and language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10437-10446, 2020.
[35] Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. arXiv preprint arXiv:2 2021.
[36] Norman Mu, Alexander Kirillov, David A. Wagner, and Saining Xie. SLIP: self-supervision meets language-image pre-training. CoRR, abs/21 12.1500,, 2021.
[37] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020.
[38] Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. Unsupervised learning of sentence embeddings using compositional n-gram features. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 528-540, New Orleans, Louisiana, June 2015 8. Association for Computational Linguistics.
[39] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In 2015 IEEE international conference acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEEE, 2015.
[40] Bo Pang and Lillian Lee. A sentimental education: sentiment analysis using subjectivity summarization based minimum cuts. In Proceedings of the 42nd Annual Meeting Association for Computational Linguisties, pages 271 171, 2004.
[41] Bo Pang and Lillian Lee, Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting Association for Computational Linguistics, pages 124, 2005.
[42] Nina Poerner, Ulli Waltinger, and Hinrich Schütze. Sentence meta-embeddings for unsupervised semantic textual similarity. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7027-7034, Online, July 2020. Association for Computational Linguistics.
[43] QQP. http://wwww..qooo..c///qqaaaaaa In Quora, 2022.
13

---

[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages PMLR, 2021.
[45] Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512 4525, Online, November 2020. Association for Computational Linguistics.
[46] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning. Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods natural language processing, pages 1631-1642 2013.
[47] Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316 2021.
[48] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-hert: Pre- training of generic visual-linguistic representations. In International Conference on Learning Representations, 2019.
[49] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020, pages 776-794, Cham, 2020. Springer International Publishing.
[50] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. In NeurlPS, 2020.
[51] Hayato Tsukagoshi, Ryohei Sasano, and Koichi Takeda. DefSent: Sentence embeddings using definition sentences. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing VVolume 2: Short Papers), pages 411 111-118 Online, August 2021. Association for Computational Linguistics.
[52] Kexin Wang, Nils Reimers, and Iryna Gurevych. TSDAE: Using transformer-based sequential denoising auto-encoderfor unsupervised sentence embedding learning. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2021, pages 671-688, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[53] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929-9939. PMLR, 2020.
[54] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7, 2019.
[55] Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2):165 210, 2005.
[56] John Wieting and Kevin Gimpel. Paranmt-50m: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451-462. 2018.
[57] Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL HLT 2018, pages 1112-1122. Association for Computational Linguistics (ACL), 2018
14

---

[58] Zhirong Wu, Yuanjun Xiong, X Yu Stella, and Dahua Lin. Unsupervised feature learning via non non-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2018.
[59] Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. Clear: Con- trastive learning for sentence representation. arXiv preprini arXiv:2012. 10n::0002.1146,, 2020.
[60] Yuwen Xiong, Mengye Ren, and Raquel Urtasun. Loco: Local contrastive representation learning. Advances in neural information processing systems, 13:11142-111533 2020.
[61] Xu Yang, Chongyang Gao, Hanwang Zhang, and Jianfei Cai. Auto-parsing network for image captioning and visual question answering. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 2197-2207, 2021.
[62] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67-78, 2014.
[63] Miaoran Zhang, Marius Mosbach, David Adelani, Michael Hedderich, and Dietrich Klakow. MCSE: Multimodal contrastive learning of sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 9959-5969, Seattle, United States, July 2022. Association for Computational Linguistics.
[64] Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. An unsupervised sentence embedding method by mutual information maximization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1601-1610, Online, November 2020. Association for Computational Linguistics.
[65] Kun Zhou, Beichen Zhang, Xin Zhao, and Ji-Rong Wen. Debiased contrastive learning of unsupervised sentence representations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 6120-6130, Dublin, Ireland, May 2022. Association for Computational Linguistics.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes]. [No], or [N/A] . You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes]
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/anssers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
15

---

(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof' of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URI.)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report enror bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "uI75bAU8HmT", "reviewer": "Reviewer_uUn7", "summary": "Summary:\n* This paper proposes a novel contrastive learning method of sentence embeddings, using unpaired examples from another modality. Experiments on semantic textual similarity benchmarks demonstrate that this method is able to produce sentence embeddings higher quality.\n", "questions": "* Did you do hyper-parameters search for both SimCSE baseline and VisualCSE method?\n* Have you tried to compare to SimCSE with “supervised models” setting?\n* What’s the intuition behind sharing everything else besides the embedding layer for image and sentence encoders? \n* Have you tried tasks besides STS?\n\n", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* This paper presents a novel method to conduct multimodal contrastive learning for sentence encoders. \n* This method is simple and generalizable, i.e. not only applied to vision but also to other modalities such as audio. \n\nWeakness:\n* I’m not confident that the improvement is significant. On STS tasks, on average it improves 1% accuracy.  In comparison, the original SimCSE paper improves more than 4% accuracy and is able to push the state-of-the-art in a supervised setting.  I wonder how much variance these datasets have, and how much hyper-parameters sweeps is conducted for both methods. \n* The model setup is a bit unintuitive. In order to conduct multimodal contrastive learning, it assumes that we can replace the embedding layer of a sentence encoder with a visual embedding layer to get an image encoder. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qVx61is4VK", "reviewer": "Reviewer_sC7n", "summary": "This paper studies the non-linguistic supervision to contrastive-learning-based sentence embedding learning. The author proposes a Transformer-based multi-modal framework to learn text-contrastive and vision/audio-contrastive tasks with unpaired samples. Experiments on various semantic textual similarity tasks show the effectiveness of the proposed method.  \n", "questions": "- Could the authors elaborate more on the deficiency of the proposed method in the supervised setting (is this also true for different languages), and could that be alleviated with more unlabeled training data? \n- Could the authors elaborate on the epochs used for training the audio and vision model, and will the quality/amount of image/audio affect the performance of the proposed methods? ", "limitations": "- It would be good for more discussion around the usage (quality and amount) of external multi-modality datasets and will aligned dataset can also be leveraged by the proposed method; \n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Summary Of Strengths\n- The paper is clearly presented and well-written. \n- Extensive empirical experiments with multiple runs not only clearly show the pros/cons for the sts tasks but may suggest the path for future improvement in the low-resource language. \n- A simple yet effective modification is proposed and shows consistent improvements over all systems.  \n\n*Summary Of Weaknesses\n- It seems the paper is missing a comparison with MCSE [1], which is a very relevant work that augments the SimCSE with multimodal contrastive objective; \n- It is unclear, except for the objective and hyperparameters, how other factors like pre-trained image/audio encoder, augmentation methods, dataset, and image/audio quality could affect the final performance of the proposed methods; \n- It is unclear to me how many epochs/images will the proposed methods improve over the SimCSE methods and how will that affect the final quality; \n\n[1] Zhang, Miaoran, Marius Mosbach, David Ifeoluwa Adelani, Michael A. Hedderich, and Dietrich Klakow. \"MCSE: Multimodal Contrastive Learning of Sentence Embeddings.\" arXiv preprint NAACL (2022).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "O2QVaFagt9f", "reviewer": "Reviewer_TGns", "summary": "This work is an extension of existing contrastive sentence representation learning. Instead of only using text data, this work further bring the other modality to improve the model performance. In this work, the author proposed unpaired image or audio data as additional supervision signals. The extra modality data share the same encoding backbone models and similar contrastive learning losses. The author conducted experiments on standard sentence similarity evaluation datasets, and achieved improvements over text-only models. ", "questions": "1. Is it possible to use same CL losses for different modalities?\n2. What if we use both image and audio data together? Can if further improve the perforamnce?\n3. Is there token-level alignment between different modals?\n4. If the proposed mode can further improve the image-text alignment compared with our models, e.g., CLIP?", "limitations": "I do not see important social limitations of this work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The proposed idea is simple and reasonable. By introducing image/audio through the same transformer encoder, the model transferred from text only to modal agnostic model. This is quite different from other works which use separated structures for each modality. One good point of the model is that the image data size (500) is much smaller than the text training data. This means the model extra cost is small. The author also well analyzed the proposed model.\n\nWeakness: The model performance improvement is relatively small. Compared to other CL sentence representation learning work, e.g., DiffCSE, DebiasedCSE, PromptBERT, the proposed method performance with more data is not that significant. The author also did not evaluate the performance on other modalities, then it is hard to give a overall judgment of the model as a modal agnostic mode. The overall training loss on different modality losses are also missing which in my view in also important to understand the contribution of different modalities. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FFUy-dP-M3k", "reviewer": "Reviewer_tnhg", "summary": "This work drives a plausible approach for enhancing unsupervised sentence embedding learning by leveraging unpaired examples from another modality. Motivated by the previous success of Lu et al. [33], Transformer models can transfer knowledge by learning from another modality, e.g. vision or audio. The text part uses the same training paradigm as SimCSE, while an additional image contrastive loss is incorporated by using the same language transformer for image encoding. Results show that the proposed method is language-agnostic and outperforms vanilla SimCSE across various tasks. Analysis versus supervised SimCSE also illustrated that the proposed approach performs better than SimCSE learned with a noisy and smaller NLI dataset.", "questions": "Please refer to weaknesses for score-related questions.\n\n- Can the supervision on non-linguistic dataset (e.g. image labels) further enhance the proposed model?", "limitations": "The authors discussed limitations on the reduction in Uniformity of the sentence embeddings. Other possible limitations could be the marginal improvement with larger non-linguisitic dataset.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea of using non-linguistic supervision for sentence embedding learning is novel.\n- The proposed model does not require paired multi-modal data for training, which can be applied to low-resource language learning.\n- The finding that Transformer models can generalize by learning a similar task across different modalities may shed light on future research on multi-modal representation learning.\n- The experiments demonstrate its success on both VisualCSE and AudioCSE.\n\nWeaknesses:\n- Though the authors attempted to answer the underlying rationale of the proposed model in Sec. 3.4, how the language model can be trained using other modalities remains unclear to me. More analysis experiments are expected to strengthen the theoretical support for the framework.\n- The work has discussed how language dataset supervision can affect the final performance. However, the non-linguistic dataset quality is not discussed. What kind of non-linguistic dataset is suitable for the paradigm? Does ImageNet fit for all language training?\n\nMinors:\n- Line 172: \"thesame\" $\\rightarrow$ \"the same\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2hM7541BN2M", "reviewer": "Reviewer_RnhL", "summary": "This paper presents a contrastive method for learning sentence embeddings that incorporates additional modalities using unaligned data. The method follows the SimCSE approach of clustering similar sentence representations and expands upon it by additionally clustering visual or audio input, named VisualCSE and AudioCSE respectively. Because the approach does not require aligned data between the text and additional modality, the method works for any language and additional modalities can be easily evaluated as well. The authors show that VisualCSE and AudioCSE have clear performance gains in both new approaches across a range of downstream tasks compared to the baseline SimCSE. Additional ablations and experiments also show that the learned sentences embeddings have higher alignment and perform similarly to supervised CSE when training with less data or increasing the level of noise.", "questions": "*clarity:*\n- In equation's 2 and 3, the loss's superscript wasn't immediately clear to me on the first read and doesn't make sense there to me. Can these just be explicitly mentioned in the text (along the lines of \"The unsupervised loss based on data augmentation is as follows\" then \"For a supervised task like NLI, its loss is defined as:\").\n\n*significance:*\n- While the authors showed the benefits of VisualCSE on languages other than English (French, German and Russian), I would have liked to see a low-resource lanugage like TODO. This would likely require using another model framework besides BERT.\n\n*questions:*\n- In the optimization details section, can you expand on how the number of epochs were chosen for training? It appears to be similar to CSE, though 1 epoch seems quite small.\n- Any consideration about intentionally batching images of the same label together to better evaluate the contribution of image labels?\n- Seeing how image labels were used for SupCon, is there a correlate experiment in the audio domain to incorporate labels?\n\n*typos:*\n- L34: dataset -> datasets\n- L43: difference -> different\n- L124: representation -> representations\n- L218: the improve -> the improvement\n- L286: of labeled dataset -> of the labeled dataset\n- L345: extend -> extent\n\n*Citations:*\nMCSE: Multimodal Contrastive Learning of Sentence Embeddings -> This is essentially concurrent work (put on arXiv in April, presented at NAACL), so it's understandable that it's not cited. I would suggest reading and editing a few small aspects of the paper, e.g. saying this work is the first to look at this approach.", "limitations": "yes, limitations are addressed", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n*originality:*\n\n- This is a really interesting concept that I'm a big fan of! It's an interesting take on using multimodal input to benefit language learning by using another modality in a completely unaligned way to improve the language embeddings.\n\n*quality:*\n\n- This paper shows a number of experiments analyzing different aspects of the approach -- raw performance on downstream tasks when compared to SimCSE, performance across 2 modalities compared to text-only training, performance across 3 models, performance across different seeds, performance across different languages, and ablations. Because there are so many experiments, it's easy to analyze and think about the contributions of the approach for different communities.\n- The visual ablation was interesting as it showed whether supervision from the visual domain, in the form of labels, helps performance. Seeing that it did not contribute much, likely due to the smaller batch size, also leaves room for future work.\n- Section 5.1 is another strong ablation, providing a relationship between the number of examples and amount of noise in supervised SimCSE versus unsupervised VisualCSE.\n- Experimental setup is thorough, especially by comparing across seeds and therefore across different subsets of the training data.\n\n*clarity:*\n- The paper is clearly written and I could follow the motivation, experimental setup and results without really needing additional background searches.\n\n*significance:*\n- Not requiring aligned data means virtually any modality can be plugged in here; in addition, the data being used in the existing modalities can easily be expanded or modified. In addition, not being tied to single language is a major contribution! This is especially true when incorporating additional modalities as most work using images is quite limited in the number of languages with annotations.\n- For a quantitative perspective, the results in Table 2 show that VisualCSE and AudioCSE nearly always outperform SimCSE across different models and across downstream tasks. These are impressive gains and really validates the idea of the approach.\n\n**Weaknesses**\n*clarity:*\n- In equation's 2 and 3, the loss's superscript wasn't immediately clear to me on the first read and doesn't make sense there to me. Can these just be explicitly mentioned in the text (along the lines of \"The unsupervised loss based on data augmentation is as follows\" then \"For a supervised task like NLI, its loss is defined as:\").\n\n*significance:*\n- While the authors showed the benefits of VisualCSE on languages other than English (French, German and Russian), I would have liked to see a low-resource language from outside of Europe.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept