Venue: NAACL 2024

KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning

Cong-Duy Nguyen\({}^{1}\), Thong Nguyen\({}^{2}\), Xiaobao Wu\({}^{1}\), Luu Anh Tuan\({}^{1}\)

\({}^{1}\)Nanyang Technological University, Singapore \({}^{2}\)National University of Singapore, Singapore

###### Abstract

Previous work on multimodal sentence embedding has proposed multimodal contrastive learning and achieved promising results. However, by taking the rest of the batch as negative samples without reviewing when forming contrastive pairs, those studies encountered many suspicious and noisy negative examples, significantly affecting the methods' overall performance. In this work, we propose KDMCSE (Knowledge Distillation Multimodal contrastive learning of Sentence Embeddings), a novel approach that enhances the discrimination and generalizability of multimodal representation and inherits the knowledge from the teacher model to learn the difference between positive and negative instances and via that, can detect noisy and wrong negative samples effectively before they are calculated in the contrastive objective. Furthermore, to overcome the limitation of modeling the variation within negative pairs, we introduce a new contrastive objective, AdapACSE (Adaptive Angular Margin Supervised Contrastive Learning for Multimodal sentence embeddings), that enhances the discriminative representation by strengthening the margin within the angular space while capturing varying semantics within the negative. Experimental results on widely used Semantic Textual Similarity (STS) benchmarks demonstrate the effectiveness of our approach. The source code is available at [https://github.com/duyngtr16061999/KDMSE](https://github.com/duyngtr16061999/KDMSE).

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 733–749

June 16-21, 2024 0204 Association for Computational Linguistics

## 1 Introduction

Learning sentence embeddings, which involves translating sentences into consistent-length vectors that capture their semantic connections, remains a pivotal task in the NLP field. While pre-trained language models like BERT Devlin et al. (2019) and RoBERTa Liu et al. (2019) have witnessed immense success, studies indicate that the default sentence embeddings from these PLMs, without specific fine-tuning, may not perform as well as simply averaging Glove vectors Pennington et al. (2014) in capturing semantic similarity Reimers and Gurevych (2019). Consequently, recent works Li et al. (2020); Zhang et al. (2020); Su et al. (2021) are geared towards refining sentence embeddings from PLMs without supervision.

While text-centric models have made significant strides, their depth of understanding of sentence semantics remains a point of contention. True semantic understanding often stems from associations in the real world rather than mere textual statistics Wang et al. (2019); Bender and Koller (2020); Bisk et al. (2020); Wei et al. (2022). Numerous recent studies have explored the enhancement of language representations through visual information Tan and Bansal (2020); Tang et al. (2021); Nguyen et al. (2023b). The recent multimodal contrast in sentence embedding learning Zhang et al. (2022) offers an encouraging direction. This method augments SimCSE with a multimodal contrastive objective, which seeks congruence between sentences and their associated images within a unified space. However, samples drawn randomly from grounded datasets during training can present semantic differences, subtle similarities, or over

Figure 1: Example image-caption pairs in Flickr. Solid lines of the same color talk about the same instance, and a dot line means the additional information that does not occur in the other caption.

 

lapping attributes. For example, in Figure 1, two captions B.1 and B.2 describe the second image B, but also correctly represent the first image A. Due to their inherent noise, such samples can hinder the learning process. Mitigating this issue requires preemptively filtering out these semantically similar samples before loss calculation, ensuring a noise-reduced training environment. In addition, comparing two captions, A.1 and A.2, for the left image reveals that the latter offers a more comprehensive description, detailing aspects such as the individual's gender, attire, sneakers, and even the material of the stairs (concrete). Thus, using only visual images cannot capture the depth of language representations.

Furthermore, MCSE did not consider the distinctions between negative sample pairs while focusing on constructing pairs of positive and negative representations. As shown in Figure 2, although none of the captions perfectly describes the image, their levels of discrimination vary between pairs. On the contrary, the caption \(c\) registers a score of \(0.49\), hinting at its partial relevance to the image. Caption \(d\), with a very weak score of \(0.03\), is completely unrelated. Consequently, within the feature space, both captions \(c\) and \(d\) should be substantially distanced from the accurate depiction. Furthermore, the gap between the true caption and caption d should considerably exceed that between the true caption and the caption \(c\).

To address the aforementioned limitation of noisy negative sampling and complexity of linguistic representation, we present KDMCSE: a Knowledge Distillation Multimodal Contrastive Learning framework for Sentence Embeddings that utilizes CLIP Radford et al. (2021) as the teacher model for both image and text modalities. Our approach harnesses images for multimodal objectives and textual representations derived from the CLIP text encoder. Relying solely on images falls short of capturing the nuances in sentence representation. Emulating the representation of the CLIP's textual encoder - especially when paired with image matching - can augment the richness of linguistic representations, transcending a mere alignment with visual features.

Moreover, in response to the identified challenge of diversity among negative pairs, we introduce AdapACSE: Adaptive Angular Margin Contrastive Learning. This method bolsters discriminative representation by amplifying the margin within the angular space, specifically accounting for the various semantics present in negative samples. Leveraging the teacher model, CLIP, we produce soft labels that signify the similarity between samples. Instead of merely utilizing feature representations as positive or negative samples during the multi-modal objective, we employ CLIP's soft labels to intensely penalize samples exhibiting low similarity and provide leniency to pairs exhibiting certain resemblances.

Our contributions can be summarized as follows.

* We propose a novel multimodal framework for sentence embedding KDMCSE: Knowledge Distillation Multimodal Contrastive learning for Sentence Embedding. The knowledge of the teacher model is transferred to the student model through multimodal contrastive learning, spanning both text and visual modalities.
* We introduce self-supervised contrastive learning: AdapACSE that improves the discriminative representation of samples with varying degrees of similarity.
* We evaluate our approach using standard semantic textual similarity (STS) benchmarks and SentEval transfer tasks. Our method outperforms the performance of earlier state-of-the-art approaches.

## 2 Related Work

Sentence Representation LearningIn previous research, sentence representations were typ

Figure 2: Example image-caption pairs in Flickr. The green caption is a true annotation of the image while 4 red captions is randomly picked from the dataset. The scores on the right are the cosine similarity between its caption representation and image representation extracted from CLIP model.

 

ically learned by extending the principles of word2vec (Mikolov et al., 2013), predicting neighboring sentences (Kiros et al., 2015; Hill et al., 2016; Logeswaran and Lee, 2018) or aggregating n-gram embeddings (Pagliardini et al., 2018), the pre-trained transformer-based model BERT (Li et al., 2020; Reimers and Gurevych, 2019) and its enhancements BERT-Flow (Li et al., 2020) and BERT-Whitening (Su et al., 2021). More recently, various studies have embraced the contrastive learning framework for learning sentence representation. These studies have proposed different methods to form contrastive pairs, either through various data transformation techniques (Yan et al., 2021; Zhang et al., 2020; Giorgi et al., 2021) or through encoders with distinct structures or parameters (Carlsson et al., 2021; Kim et al., 2021). A notable example is SimCSE (Gao et al., 2021), which employs dropout as a data augmentation strategy. Many studies in the field of sentence contrastive learning have made a significant contribution by focusing on the mining of challenging negative samples (Wang et al., 2022b; Zhou et al., 2022; Zhang et al., 2022b; Wang et al., 2022a; Wei et al., 2023; He et al., 2023).

Deep Metric Learning ObjectivesContrastive learning (Chopra et al., 2005) has gained traction across multiple domains (Nguyen and Luu, 2021; Nguyen et al., 2022, 2024, 2023a; Wu et al., 2020, 2022, 2023a, 2024; Wei et al., 2024). Renowned training objectives such as N-Pair Loss (Sohn, 2016), Triplet Margin Loss (Balntas et al., 2016), and ArcCon (Zhang et al., 2022c) are deeply rooted in metric learning principles. In supervised tasks, objectives that leverage softmax have shown efficacy, especially when they integrate class centers and impose penalties on the distances between deep features and their corresponding centers. Among these, Center loss (Wen et al., 2016), SphereFace (Liu et al., 2017), CosFace (Wang et al., 2018), and ArcFace (Deng et al., 2018) stand out and are frequently used in computer vision and natural language processing applications. Nonetheless, these loss functions primarily aim at classification tasks, rendering them unsuitable for regression labels. ArcSEzhang et al. (2022c) presents training objectives using additional margin tailored to enhance the discriminative prowess in pairwise relations and capture the entailment dynamics within triplet sentence structures.

Visually Grounded Representation LearningNumerous studies have underscored the advantages of integrating NLP models with visual insights to improve textual representation learning (Lazaridou et al. (2015); Kiela et al. (2018)). Based on the Skip-Thought model of Kiros et al. (2015b), Bordes et al. (2019) created a unified space that harmoniously accommodates both visual and textual dimensions. Recent pioneering efforts by Tan and Bansal (2020a), Tang et al. (2021) and (Nguyen et al., 2023b) have laid the foundation for vast language models using multimodal guidance, aiming to improve language understanding. Similarly, MCSE by Zhang et al. (2022a) suggested sentence embedding learning using a multimodal contrastive objective to align sentences with their respective images coherently. Distinguishing our work from Zhang et al. (2022a), we present an innovative angular margin contrastive learning framework, building upon the cutting-edge multimodal contrastive learning proposed by Zhang et al. (2022a). By weaving in this new contrastive technique, our goal is to push the envelope in STS by leveraging multimodal semantic data.

## 3 Method

### Background: Unsupervised SimCSE and Multimodal Contrastive Learning MCSE

The idea of unsupervised SimCSE is to use dropout noise as a simple, yet effective, data augmentation strategy. Given a collection of sentences \(\{x_{i}\}^{m}_{i=1}\), we construct a positive pair for each input \(x_{i}\) by encoding it twice using different dropout masks: \(\mathbf{h}_{i}^{z}=g_{\phi}(f_{\theta}(x_{i},z))\) and \(\mathbf{h}_{i}^{z^{\prime}}=g_{\phi}(f_{\theta}(x_{i},z^{\prime}))\), where \(z\) and \(z^{\prime}\) denote different dropout masks1, \(f_{\theta}(\cdot)\) is a pre-trained language encoder such as BERT, and \(g_{\phi}(\cdot)\) is a projection head2 on top of the [CLS] token. The training objective is:

Footnote 1: The standard dropout masks in Transformers are used.

Footnote 2: There is a MLP pooler layer over [CLS] in BERT’s implementation. Gao et al. (2021) use it with reinitialization.

\[\ell_{i}^{\text{S}}=-\log\frac{e^{\operatorname*{sim}(\mathbf{h}_{i}^{k_{i} },\mathbf{h}_{i}^{z^{\prime}_{i}})/\tau}}{\sum_{j=1}^{N}e^{\operatorname*{sim}( \mathbf{h}_{i}^{k_{i}},\mathbf{h}_{j}^{z^{\prime}_{j}})/\tau}}~{}, \tag{1}\]

where \(N\) is the size of the mini-batch, \(\tau\) is a temperature parameter, and \(\operatorname*{sim}(\mathbf{h}_{1},\mathbf{h}_{2})\) is the cosine similarity \(\frac{\mathbf{h}_{1}^{T}\mathbf{h}_{2}}{\|\mathbf{h}_{1}\|\cdot\|\mathbf{h}_{2}\|}\). After training, the [CLS] token outputs of the language encoder are taken as the sentence embeddings.

 

To take advantage of visual and textual information, MCSE (Zhang et al., 2022) adopt SimCSE as the textual baseline and extend it with a multimodal contrastive learning objective:

\[{\mathbf{s}_{i}^{z}=g_{\phi_{1}}(f_{\theta}(x_{i},z)),~{}\mathbf{v}_{i}=g_{\phi_{2}}(f^{ v}(y_{i}))~{},} \tag{2}\]

where \(f^{v}(\cdot)\) is a pre-trained image encoder. \(g_{\phi_{1}}(\cdot)\) and \(g_{\phi_{2}}(\cdot)\) are two projection heads for text and image modalities. They define the multimodal contrastive learning objective as:

\[\ell_{i}^{\rm M}=-\sum_{z\in\left\{z_{i},z_{i}^{\prime}\right\}}\log\frac{e^{ \sin({\mathbf{s}_{i}^{z}},\mathbf{v}_{i})/\tau^{\prime}}}{\sum_{j=1}^{N}e^{\sin({\mathbf{ s}_{i}^{z}},\mathbf{v}_{j})/\tau^{\prime}}}~{}, \tag{3}\]

### Knowledge Distillation Multimodal Contrastive learning for Sentence Embedding

In addition to the multimodal objective, we present a Knowledge Distillation framework designed to harness both visual and textual insights from the vision-language model CLIP. An overview of our KDMCSE model can be found in Figure 3. Given a set of sentence-image pairs represented as \(D=\left\{x_{i},y_{i}\right\}_{i=1}^{m}\), we initially project the sentence \(x_{i}\) and the image \(y_{i}\) into a shared or grounded space.

\[{\mathbf{s}_{i}^{z}=g_{\phi_{g}}(f_{\theta}(x_{i},z)),} \tag{4}\]

\[{\mathbf{t}_{i}=g_{\phi_{l}}(\hat{\mathbf{t}_{i}}),~{}\hat{\mathbf{t}_{i}}=CLIP_{text}(x_{ i})~{},} \tag{5}\]

\[{\mathbf{v}_{i}=g_{\phi_{w}}(\hat{\mathbf{v}_{i}}),~{}\hat{\mathbf{v}_{i}}=CLIP_{visual}(y_ {i})~{},} \tag{6}\]

where \(CLIP_{text},CLIP_{visual}\) are a pre-trained text and image CLIP encoder, \(\hat{\mathbf{v}_{i}}\) and \(\hat{\mathbf{t}_{i}}\) are the visual and text representation extracted from CLIP, \(g_{\phi_{g}}(\cdot)\) is the projection heads for the language student model to project the sentence representation into grounded space, \(g_{\phi_{l}}(\cdot)\) and \(g_{\phi_{w}}(\cdot)\) are distinct projection heads of the teacher model for the text and image modality, respectively. During training, we freeze the pre-trained CLIP model.

**Threshold Filtering** During training, negative samples were randomly taken from data in Eq.3. We not only transfer CLIP's knowledge of the text and visual representation of each sample, but also leverage the similarity within samples in a batch. We calculate the cosine similarity:

\[\alpha_{i,j}^{m,n}=\frac{m_{i}\cdot n_{j}}{\left\|m_{i}\right\|_{2}\left\|n_{ j}\right\|_{2}},\,m,n\in\left\{\hat{v},\hat{t}\right\} \tag{7}\]

where \(m,n\) are the defined modality (text or visual). Thus, as shown in Figure 4, we can generate four similarity mappings: text-text, visual-visual, text-visual, and visual-text. As we strive to align the sentence representation generated by the language encoder (text) with the information gleaned from the teacher model encompassing both sentence (text) and image (visual) data, we exclusively employ 'text-text' and 'text'-visual' pairings for our soft labels. We define a threshold filter function to remove noisy negative samples:

\[\varphi_{i,j}^{m,n}=\begin{cases}0&if~{}~{}~{}~{}\alpha_{i,j}^{m,n}<threshold\\ 1&else\end{cases} \tag{8}\]

where \(threshold\) is a fixed hyperparameter during

Figure 3: The overall architecture of KDMCSE. The upper part is the original SimCSE, the below part is the multimodal contrastive learning approach with knowledge distillation from CLIP model.

 

training. The proposed contrastive objective with Threshold Filtering is defined as follows:

\[\ell_{i}^{\text{m}}=-\sum_{z\in\{z_{i},z_{i}^{\prime}\}}\log\frac{e^{\sin( \boldsymbol{s}_{i}^{z},\boldsymbol{m}_{i})/\tau^{\prime}}}{\sum_{j=1}^{N}\varphi _{i,j}^{t,m}e^{\sin(\boldsymbol{s}_{i}^{z},\boldsymbol{m}_{j})/\tau^{\prime}}}~ {}, \tag{9}\]

where \(m\) is the modality (text or visual), \(m\in\{t,v\}\), \(\boldsymbol{m}_{i}\) is the projected representation of CLIP model, \(\boldsymbol{m}_{i}\in\{\boldsymbol{v}_{i},\boldsymbol{t}_{i}\}\).

### Adaptive Angular margin Contrastive learning

Previous studies Zhang et al. (2022c) introduced the ArcCSE objective. This was designed to increase the discriminative capacity between pairs and encapsulate the entailment relationship among triplet sentences. ArcCSE adds an additive angular margin \(m_{c}\) between positive pairs \(h_{i}^{z}\) and \(h_{i}^{z^{\prime}}\). Compared to Eq.1, it further pushed \(h_{i}\) towards the area where \(\theta_{i,i^{\prime}}\) becomes smaller and \(\theta_{j,i}\) becomes larger, increasing the compactness of sentence representations with the same semantics and enlarging the discrepancy of different semantic representations. ArcCSE is defined as:

\[\ell_{i}^{\mathrm{ArcCSE}}=-\mathrm{log}\frac{e^{\phi\left( \theta_{i,i^{\prime}}+m_{c}\right)/\tau}}{e^{\phi\left(\theta_{i,i^{\prime}}+ m\right)/\tau}+\sum_{j\neq i}^{n}e^{\phi\left(\theta_{j,i}\right)/\tau}} \tag{10}\]

where \(\phi\) is the \(cos\) function, angular \(\theta_{i,j}\) is denoted as follows:

\[\theta_{i,j}=\arccos\left(\frac{h_{i}^{T}h_{j}}{\|h_{i}\|*\|h_{ j}\|}\right) \tag{11}\]

where \(h_{i}\) is the vector representation, in this work, \(h_{i}\in\{\boldsymbol{v}_{i},\boldsymbol{t}_{i},\boldsymbol{s}_{i}^{z}\}\).

There are distinct differences among the negative pairs, especially when we observe that the examples marked in red in Figure 2 serve as negative samples for the green caption. In this grounded space, the distance between the samples becomes crucial. We have refined the contrastive objective to harness the knowledge from the teacher model and to capture the magnitude of the differences between negative pairs. To address this challenge, we introduce a novel training objective, Adaptive Angular margin Contrastive Loss, AdapACSE, which is an adaptation of Eq.10. Instead of fixing margin \(m_{c}\) for all samples in the data, we adapt it to be flexible depending on its negative samples. This new objective improves sentence representation learning by an adaptive margin, \(m_{c}\Delta_{i,j}\), between negative pairs \(h_{i}\) and \(h_{j}\). Thus, if the \(\Delta_{i,j}\) is large and the two samples \(i,j\) are much different, the margin is also large to push the representation away, and vice versa. Our approach is illustrated in the right part of Figure 4, we formalize this objective function as **AdapACSE**:

\[\ell_{i}^{\mathrm{AdapACSE}}=\\ -\mathrm{log}\frac{e^{\phi\left(\theta_{i,i^{*}}\right)/\tau}}{e^{\phi \left(\theta_{i,i^{*}}\right)/\tau}+\sum_{j\neq i}^{n}e^{\phi\left(\theta_{i,j }-m_{c}\Delta_{i,j}\right)/\tau}} \tag{12}\]

Figure 4: The overall framework of knowledge distillation with Adaptive Angular margin contrastive learning. The pipeline first calculates the soft-label similarity scores between text and visual representation, then we apply threshold filtering to remove the noisy negative pairs, and finally, we transfer the soft-label matrices into our proposed AdapACSE to flexibly find the margin. \(s_{\hat{a}}\) is the positive sample for \(s_{a}\), \(v_{b}\) and \(v_{c}\) are its negative counterparts. In particular, the difference in pairs \(s_{a}-v_{c}\) is more pronounced than in \(s_{a}-v_{b}\). As a result, the margin (depicted as a dashed line) for \(c\) (in orange) is greater than for \(b\) (in cyan).

 

where \(\Delta_{i,j}=|1-\alpha_{i,j}|\) is the cosine distance between two samples \(i\) and \(j\). Together with the threshold filtering contrastive learning, we rewrite the AdapACSE objective to be:

\[\ell^{\text{AdapACSE}^{\prime}}_{i}=\ell^{m^{\prime}}_{i}=\] \[-\log\frac{e^{\phi\left(\theta_{i,i^{*}}\right)/\tau}}{e^{\phi \left(\theta_{i,i^{*}}\right)/\tau}+\sum_{j\neq i}^{n}\varphi_{i,j}^{t,m}e^{ \phi\left(\theta_{i,j}-m_{c}\Delta_{i,j}\right)/\tau}} \tag{13}\]

Finally, we define our KDMCSE objective as follows:

\[\ell^{\text{KDMCSE}}_{i}=\frac{\ell^{v^{\prime}}_{i}+\ell^{t^{ \prime}}_{i}}{2} \tag{14}\]

where \(\ell^{v^{\prime}}_{i}\) and \(\ell^{v^{\prime}}_{i}\) are AdapACSE Eq. 13 with \(t,v\) are text and visual modalities, respectively.

## 4 Experiments Setup

### Dataset

We employ Flickr30k Young et al. (2014) and MSCOCO Lin et al. (2014) as our multimodal datasets. Flickr30k contains \(29,783\) training images and MSCOCO has \(82,783\) training images. Every image in these collections comes with multiple captions, usually five captions. As in Gao et al. (2021), we utilize Wiki1M for our text-based dataset, consisting of \(10^{6}\) sentences extracted randomly from English Wikipedia.

### Implementation

Language Encoder - Student modelWe have implemented our work in the Hugging Face Transformers library3 as described by Wolf et al. (2020). We load our language encoder from the checkpoints of bert-base-uncased and roberta-base. Then, we fine-tune these foundation models with our introduced contrastive objective. To evaluate sentence embeddings, we use the 768-dimensional outputs of the [CLS] token preceding the MLP pooler layer from the transformer-based models.

Footnote 3: [https://github.com/huggingface/transformers](https://github.com/huggingface/transformers)

Multimodal encoder - Teacher modelOur teacher model is CLIP Radford et al. (2021), the model's weights are initialized with pre-trained clip-vit-base-patch32, where ViT-B/32 Transformer architecture is built as an image encoder and the patch size \(P\) is 32. Because during training, the teacher model will not be fine-tuned, we will pre-extract image and text features \(\hat{\mathbf{v}},\hat{\mathbf{t}}\) to reduce the computation cost during the training phase.

MLP Projection HeadsWe use 4 different MLP modules \(g_{\{\theta_{1},\theta_{2},\theta_{l},\theta_{v}\}}\) to project different modalities and objectives. For the pure textual objective using the Wiki1M dataset, sentence embeddings are projected into a 768-dimensional space. For CLIP projection, both the sentence embeddings (student and teacher models) and the image feature vectors are projected into a shared 256-dimensional space.

Parameter SettingsIn line with the experimental setup described by Zhang et al. (2022a), we explored two primary training scenarios: _wiki+flickr_ and _wiki+coproco_. Mini-batches were drawn either from the Wiki1M corpus or from the respective caption datasets, with proportions reflecting their relative sizes. Our parameter choices were largely inspired by the guidelines provided by Gao et al. (2021). Specifically, we set the temperature parameters \(\tau\) and \(\tau^{\prime}\), to \(0.05\). For model assessment, we performed evaluations on the STS-B dev set after every \(125\) training iteration, preserving the best performing checkpoint for our final evaluation. Regarding the BERT encoder, we adopted a learning rate of \(3e-5\) and a batch size of \(64\). For RoBERTa, these parameters were adjusted to \(1e-5\) and \(128\), respectively. Our training was executed on an A6000 GPU, with each experimental run spanning roughly 5-6 hours and equivalent to MCSE training time.

### Evaluation

We evaluate our trained models on seven STS (Semantic Textual Similarity) tasks, including STS 2012-2016 Agirre et al. (2012, 2013, 2014, 2015, 2016), the STS Benchmark Cer et al. (2017), and SICK-Relatedness Marelli et al. (2014). Each dataset is made up of pairs of sentences with the aim of assigning a similarity score to each pair. In alignment with Gao et al. (2021), we present the Spearman correlation (multiplied by 100) between the official annotations and our predicted scores in an "all" context. This means that we merge all subsets for each task and then provide a comprehensive Spearman correlation.

 



## 5 Experiments Results

### Main Results

Following the methodology of previous studies, we conducted our model five times using the same hyperparameter settings. Averaged results are delineated in Table 1. A collective overview of our results reveals that our model consistently matches or exceeds the benchmarks set by baseline approaches. MCSE advances beyond SimCSE in the majority of STS evaluations. By integrating auxiliary visual and textual insights derived from the CLIP teacher model, our KDMCSE model showcases pronounced improvements in various downstream tasks. Specifically, when trained on Wiki1M and Flickr30k datasets, KDMCSE increases the performance metrics for BERT (from \(77.3\) to \(78.6\)) and RoBERTa (from \(78.3\) to \(79.1\)). Compared to MCSE-RoBERTa, our model exhibits comparable metrics on STS15 and STS-B, with a slight advancement on STS16. In the context of the _wiki+coco_ dataset, KDMCSE-BERT notably surpasses other methods across most tasks, with scores rising from \(76.6\) to \(77.9\) (STS15 being the exception with a modest increase). Our RoBERTa encoder, when trained with the KDMCSE framework, showed higher average results (an increase from \(77.6\) to \(78.0\)), where tasks such as STS12, STS14, and STS15 registered significant improvements, while others retained or slightly underperformed in comparison.

### Alignment and Uniformity

The concepts of alignment and uniformity are closely linked to contrastive learning, serving as potential metrics to evaluate the quality of the representations derived Wang and Isola (2020). Alignment is a property that encourages encoders to produce analogous representations for instances that are alike. This can be quantified using the expected distance between the embeddings of positively paired instances.

\[{\mathcal{L}}_{align}\triangleq\operatorname*{\mathbb{E}}_{(x,x^{+})} \sim p_{\text{pos}}\left\|f(x)-f(x^{+})\right\|_{2}^{2}~{}. \tag{15}\]

And the _uniformity loss_ prefers a uniform distribution in the hypersphere:

\[{\mathcal{L}}_{uniform}\triangleq\log\operatorname*{\mathbb{E}}_{x,y^{+} \overset{i.e.d.}{\cup}p_{\text{data}}}e^{-2\lVert f(x)-f(y)\rVert_{2}^{2}}~{}. \tag{16}\]

To delve deeper into the mechanics of our methodology, we assess alignment and uniformity metrics using the STS-B development set. In our

\begin{table}
\begin{tabular}{l l c c c c c c c|c} \hline \hline  & **Model** & **STS12** & **STS13** & **STS14** & **STS15** & **STS16** & **STS-B** & **SICK-R** & **Avg.†** \\ \hline \multirow{4}{*}{Test} & SimCSE-BERt\({}^{\diamond}\) & 67.8\(\pm\)\({}_{1.6}\) & 80.0\(\pm\)\({}_{2.1}\) & 72.5\(\pm\)\({}_{1.7}\) & 80.1\(\pm\)\({}_{0.8}\) & 77.6\(\pm\)\({}_{0.8}\) & 76.5\(\pm\)\({}_{0.8}\) & 70.1\(\pm\)\({}_{0.9}\) & 74.9\(\pm\)\({}_{1.1}\) \\  & SimCSE-RoBERTa\({}^{\diamond}\) & 68.7\(\pm\)\({}_{1.0}\) & 82.0\(\pm\)\({}_{0.5}\) & 74.0\(\pm\)\({}_{0.1}\) & 82.1\(\pm\)\({}_{0.4}\) & 81.1\(\pm\)\({}_{0.4}\) & 80.6\(\pm\)\({}_{0.3}\) & 69.2\(\pm\)\({}_{0.2}\) & 76.8\(\pm\)\({}_{0.5}\) \\ \cline{2-10}  & \multirow{4}{*}{KDMCSE-BERT} & SimCSE-BERt\({}^{\diamond}\) & 69.9\(\pm\)\({}_{1.7}\) & 79.8\(\pm\)\({}_{1.5}\) & 72.9\(\pm\)\({}_{0.9}\) & 81.9\(\pm\)\({}_{0.8}\) & 77.8\(\pm\)\({}_{0.9}\) & 76.6\(\pm\)\({}_{1.1}\) & 68.4\(\pm\)\({}_{0.8}\) & 75.3\(\pm\)\({}_{0.9}\) \\  & & MCSE-BERT\({}^{\diamond}\) & 71.4\(\pm\)\({}_{0.9}\) & 81.8\(\pm\)\({}_{1.3}\) & 74.8\(\pm\)\({}_{0.9}\) & 83.6\(\pm\)\({}_{0.9}\) & 77.5\(\pm\)\({}_{0.8}\) & 79.5\(\pm\)\({}_{0.5}\) & 72.6\(\pm\)\({}_{1.4}\) & 77.3\(\pm\)\({}_{0.5}\) \\  & KDMCSSE-BERT & **74.4\({}_{1.4}\)** & **83.1\({}_{0.9}\)** & **76.3\({}_{1.1}^{*}\)** & **83.7\({}_{0.8}\)** & **78.8\({}_{0.9}\)** & **81.3\({}_{0.9}\)** & **73.0\({}_{0.9}\)** & **78.6\({}_{0.8}\)** \\ \cline{2-10}  & \multirow{4}{*}{KDMCSE-RoBERTa\({}^{\diamond}\)} & SimCSE-RoBERTa\({}^{\diamond}\) & 69.5\(\pm\)\({}_{0.9}\) & 81.6\(\pm\)\({}_{0.5}\) & 74.1\(\pm\)\({}_{0.6}\) & 82.4\(\pm\)\({}_{0.3}\) & 80.9\(\pm\)\({}_{0.5}\) & 79.9\(\pm\)\({}_{0.3}\) & 67.3\(>\)\({}_{0.5}\) & 76.5\(<\)\({}_{0.4}\) \\  & & MCSE-RoBERTa\({}^{\diamond}\) & 71.7\(\pm\)\({}_{0.2}\) & 82.7\(\pm\)\({}_{0.4}\) & 75.9\(\pm\)\({}_{0.3}\) & **84.0\(\pm\)\({}_{0.4}\)** & 81.3\(\pm\)\({}_{0.3}\) & **82.3\(\pm\)\({}_{0.5}\)** & 70.3\(\pm\)\({}_{1.3}\) & 78.3\(\pm\)\({}_{0.1}\) \\  & & KDMCSE-RoBERTa & **73.6\({}_{0.7}^{*}\)** & **83.3\({}_{0.6}\)** & **77.4\({}_{0.4}^{*}\)** & **84.0\(\pm\)\({}_{0.3}\)** & **81.5\(\pm\)\({}_{0.7}\)** & **82.3\(\pm\)\({}_{0.6}\)** & **71.2\({}_{0.4}\)** & **79.1\({}_{0.3}\)** \\ \cline{2-10}  & \multirow{4}{*}{KDMCSE-BERT} & SimCSE-BERT\({}^{\diamond}\) & 69.1\(\pm\)\({}_{1.1}\) & 80.4\(\pm\)\({}_{0.9}\) & 72.7\(\pm\)\({}_{0.7}\) & 81.1\(\pm\)\({}_{0.3}\) & 78.2\(\pm\)\({}_{0.9}\) & 73.9\(\pm\)\({}_{0.6}\) & 66.6\(\pm\)\({}_{1.2}\) & 74.6\(\pm\)\({}_{0.2}\) \\  & & KDE-BERT\({}^{\diamond}\) & 71.2\(\pm\)\({}_{1.3}\) & 79.7\({}_{0.9}\) & 73.8\(\pm\)\({}_{0.9}\) & 83.0\(\pm\)\({}_{0.4}\) & 77.8\(\pm\)\({}_{0.4}\) & 72.1\(\pm\)\({}_{1.4}\) & 76.6\(\pm\)\({}_{0.5}\) \\  & & KDMCSE-RoBERTa & **73.2\({}_{1.2}\)** & **80.5\({}_{1.0}\)** & **75.4\({}_{0.9}\)** & **83.2\({}_{0.3}\)** & **79.7\({}_{0.8}\)** & **79.7\({}_{0.7}\)** & **73.7\({}_{1.2}\)** & **77.9\({}_{1.2}\)** \\ \cline{2-10}  & \multirow{4}{*}{KDMCSE-RoBERTa\({}^{\diamond}\)} & SimCSE-RoBERTa\({}^{\diamond}\) & 66.4\(\pm\)\({}_{0.9}\) & 80.7\(\pm\)\({}_{0.7}\) & 72.7\(\pm\)\({}_{1.1}\) & 81.3\(\pm\)\({}_{0.9}\) & 80.2\(\pm\)\({}_{0.8}\) & 76.8\(\pm\)\({}_{0.6}\) & 65.7\(>\)\({}_{0.7}\) & 74.8\(\pm\)\({}_{0.5}\) \\  & & ICSE-RoBERTa\({}^{\diamond}\) &

comparison with MCSE, the results are presented in Table 2. Evidently, when compared to MCSE for both BERT and RoBERTa-based encoders, KDMCSE shows superior performance in both alignment and uniformity metrics. This not only reaffirms the foundational rationale of our method, but also suggests that AdapACSE can effectively enhance the caliber of sentence embeddings.

## 6 Analysis

### Exploring the Impact of Angular Margin

In our AdapACSE loss function, the angular margin, represented as \(m\), is critical to determining the discriminative capability. We conducted an experiment to better understand the impact of \(m\) by varying it in steps of 0.025 radians from 0.025 radians to 0.225 radians. The performance metrics averaged on all tasks are depicted in Figure 7. From the data, we discern an optimal performance when \(m\) is set to 0.125. Any deviation from this optimal point, be it an increase or decrease, results in a performance drop. This aligns with our preliminary hypotheses: A smaller value of \(m\) could have a negligible effect, whereas an overly large \(m\) could negatively skew the representations within the fusion space.

### Threshold Filtering selection

In this section, we will show how to define the threshold hyperparameter during training. We make statistics based on text-text and text-visual similarity scores in the Flickr set, and select based on that statistic. We perform two calculations: (1) the histogram for the similarity threshold and (2) the position of the largest true captions (the annotation captions of that image) that align with the image when sorted with all captions in the dataset with that image. Looking at Figure 5, we see that most of the true captions of an image are in the top 100 (out of about 150,000 captions). However, there are still quite a few captions that, even though they accurately describe that image when sorted, can still be pushed to very far indexes (>200). This may be because there are still many similar images in the dataset, leading to captions that describe another image, but are still correct for this image. The image 6 shows that the scores are in a normal distribution. We focus more clearly on samples with high similarity and see that there are only a few pairs with similarity higher than 0.8. and decreases rapidly as it approaches 1.0. Based on the above statistics, we have chosen a threshold of about 0.85-0.9 to eliminate pairs in the contrastive learning objective.

## 7 Conclusion

In this paper, we introduce KDMCSE, a novel approach to sentence embedding learning that transfers knowledge from a vision and language model with a multimodal contrastive objective to not only align sentences and the corresponding image and text representations of the teacher model but also avoid noisy negative samples. Additionally, our innovative contrastive objective, AdapACSE, ad

Figure 5: Statistic of the maximum index of true captions when sorting the similarity score between image and text of Flickr dataset.

Figure 6: Statistic of the similarity score between image and text of Flickr dataset.

Figure 7: Impact of angular margin \(m\) in AdapACSE. The results are reported on average across 7 tasks, and our experiments are conducted on BERT encoder and in _wiki-flickr_ setting.

 

dresses the challenges associated with capturing nuanced differences within negative pairs, strengthening discriminative representation. Our framework lays the groundwork for continued exploration in this domain, fostering advancements in vision and language applications.

## 8 Limitations

In this study, we shed light on the promising enhancements of KDMCSE in STS benchmarks. However, it is imperative to acknowledge its limitations. In particular, there is a pronounced disparity in both the distribution and volume of word tokens when visual datasets are compared to traditional language corpora. Taking the Book Corpus and Wikipedia as instances, these standard language repositories house billions of words, spanning millions of unique tokens. In contrast, MS COCO, a frequently cited visual-grounded dataset, encompasses merely a million words and about a thousand unique tokens. Another challenge encountered in our methodology is the intricate correlation of the hyperparameter in our contrastive objective, which remains elusive. Its optimal configuration was determined not through systematic understanding, but rather through exhaustive fine-tuning across many experiments. Furthermore, the characterization of "semantic similarity" is highly dependent on the specific task at hand. Although we have seen significant advancements in STS benchmarks, there remains a need to evaluate the disparity in performance between text-only models and their multimodal counterparts across diverse benchmarks. This would provide a more holistic understanding of the efficacy of sentence representations in varied contexts.

## Acknowledgement

We thank all anonymous reviewers for their helpful comments. This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme, AISG Award No: AISG2-TC-2022-005.

## References

* Agire et al. (2015) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Hongo Lopez-Gazpio, Monse Maritxalar, Rada Mihalcea, et al. 2015. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In _Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015)_, pages 252-263.
* Agirre et al. (2014) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. In _Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014)_, pages 81-91.
* Agirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Azirre, Rada Mihalcea, German Rigau Claramant, and Janyce Wiebe. 2016. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016)_, pages 497-511.
* Agirre et al. (2012) Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In
* SEM 2012: The First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_, pages 385-393.
* Agirre et al. (2013) Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013.
* SEM 2013 shared task: Semantic textual similarity. In _Second joint conference on lexical and computational semantics (
* SEM), volume 1: proceedings of the Main conference and the shared task: semantic textual similarity_, pages 32-43.
* Balntas et al. (2016) Vassileios Balntas, Edgar Riba, Daniel Ponsa, and Krystian Mikolajczyk. 2016. Learning local feature descriptors with triplets and shallow convolutional neural networks. In _British Machine Vision Conference_.
* Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 5185-5198.
* Bisk et al. (2020) Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Niisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 8718-8735.
* Bordes et al. (2019) Patrick Bordes, Eloi Zablocki, Laure Soulier, Benjamin Piwowarski, and Patrick Gallinari. 2019. Incorporating visual semantics into sentence representations within a grounded space. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 696-707 

* Carlsson et al. (2021) Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipaa Hellqvist, and Magnus Sahlgren. 2021. Semantic re-tuning with contrastive tension. In _International Conference on Learning Representations_.
* Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Speccia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval 2017)_, pages 1-14.
* Chopra et al. (2005) S. Chopra, R. Hadsell, and Y. LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05)_, volume 1, pages 539-546 vol. 1.
* Deng et al. (2018) Jiankang Deng, Jia Guo, and Stefanos Zafeiriou. 2018. Arcface: Additive angular margin loss for deep face recognition. _CoRR_, abs/1801.07698.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HIT)_, pages 4171-4186.
* Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6894-6910.
* Giorgi et al. (2021) John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021. DeCLUTR: Deep contrastive learning for unsupervised textual representations. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 879-895, Online. Association for Computational Linguistics.
* He et al. (2023) Hongliang He, Junlei Zhang, Zhenzhong Lan, and Yue Zhang. 2023. Instance smoothed contrastive learning for unsupervised sentence embedding.
* Hill et al. (2016) Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data.
* Kiela et al. (2018) Douwe Kiela, Alexis Conneau, Allan Jabri, and Maximilian Nickel. 2018. Learning visually grounded sentence representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HIT)_, pages 408-418.
* Kim et al. (2021) Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021. Self-guided contrastive learning for BERT sentence representations. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP)_, pages 2528-2540.
* Kiros et al. (2015a) Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015a. Skip-thought vectors. _CoRR_, abs/1506.06726.
* Kiros et al. (2015b) Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015b. Skip-thought vectors. In _Advances in neural information processing systems (NeurIPS)_, pages 3294-3302.
* Lazaridou et al. (2015) Angeliki Lazaridou, Marco Baroni, et al. 2015. Combining language and vision with a multimodal skip-gram model. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_, pages 153-163.
* Li et al. (2020) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9119-9130.
* Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In _Proceedings of the European Conference on Computer Vision (ECCV)_, pages 740-755.
* Liu et al. (2017) Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. 2017. SphereFace: Deep hypersphere embedding for face recognition. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 6738-6746.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Logeswaran and Lee (2018) Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations.
* Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivoggli, Raffaella Bernardi, Roberto Zamparelli, et al. 2014. A sick cure for the evaluation of compositional distributional semantic models. In _International Conference on Language Resources and Evaluation (LREC)_, pages 216-223.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. _CoRR_, abs/1310.4546.

 

Cong-Duy Nguyen, Thong Nguyen, Duc Anh Vu, and Luu Anh Tuan. 2023a. Improving multimodal sentiment analysis: Supervised angular margin-based contrastive learning for enhanced fusion representation.
* S. Lee et al. (2021)Long-Duy Nguyen, The-Anh Vu-Le, Thong Nguyen, Tho Quan, and Anh-Tuan Luu. 2023b. Expand bert representation with visual information via grounded language learning with multimodal partial alignment. In Proceedings of the 31st ACM International Conference on Multimedia, MM '23, New York, NY, USA, pp. 5665-5673. External Links: ISBN 978-1-4503-387X Cited by: SS1.
* Y. Nguyen and A. T. Luu (2021)Contrastive learning for neural topic model. Advances in Neural Information Processing Systems34. Cited by: SS1.
* T. Nguyen, X. Wu, A. T. Lau, Z. Hai, and L. Bing (2022)Adaptive contrastive learning on multimodal transformer for review helpfulness prediction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 10085-10096. Cited by: SS1.
* T. Nguyen, M. Wu, C. Dong, C. T. Nguyen, S. Ng, and A. T. Luu (2024)Topic modeling as multi-objective optimization with setwise contrastive learning. In The Twelfth International Conference on Learning Representations, Cited by: SS1.
* M. Pagliardini, P. Gupta, and M. Jaggi (2018)Unsupervised learning of sentence embeddings using compositional n-gram features. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1532-1543. Cited by: SS1.
* J. Pennington, R. Socher, and C. D. Manning (2014)Glove: global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543. Cited by: SS1.
* A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021)Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 8748-8763. Cited by: SS1.
* N. Reimers and I. Grurevych (2019)Sentence-BERT: sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982-3992. Cited by: SS1.
* K. Sohn (2016)Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems, Vol. 29, Curran Associates, Inc.. Cited by: SS1.
* J. Su, J. Cao, W. Liu, and Y. Ou (2021)Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316. Cited by: SS1.
* H. Tan and M. Bansal (2020)Vokenization: improving language understanding via contextualized, visually-grounded supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2066-2080. Cited by: SS1.
* H. Tan and M. Bansal (2020)VGREIZATION: improving language understanding with contextualized, visual-grounded supervision. CoRRabs/2010.06775. Cited by: SS1.
* Z. Tang, J. Cho, H. Tan, and M. Bansal (2021)Vidlankd: improving language understanding via video-distilled knowledge transfer. In Advances in Neural Information Processing Systems (NeurIPS), pp. 24468-24481. Cited by: SS1.
* A. Wang, A. T. Lulu, C. Foo, H. Zhu, Y. Tay, and V. Chandrasekhar (2019)Holistic multi-modal memory network for movie question answering. IEEE Transactions on Image Processing29, pp. 489-499. Cited by: SS1.
* H. Wang, Y. Li, Z. Huang, Y. Dou, L. Kong, and J. Shao (2022)SnCSE: contrastive learning for unsupervised sentence embedding with soft negative samples. Cited by: SS1.
* H. Wang, Y. Wang, Z. Zhou, X. Ji, Z. Li, D. Gong, J. Zhou, and W. Liu (2018)CosFace: large margin cosine loss for deep face recognition. CoRRabs/1801.09414. Cited by: SS1.
* T. Wang and P. Isola (2020)Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning (ICML), pp. 9929-9939. Cited by: SS1.
* W. Wang, L. Ge, J. Zhang, and C. Yang (2022)Improving contrastive learning of sentence embeddings with case-augmented positives and retrieved negatives. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, ACM, Cited by: SS1.
* J. Wei, G. Hu, L. Anh Tuan, X. Yang, and W. Zhu (2023)Multi-scale receptive field graph model for emotion recognition in conversations. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. Cited by: SS1.

 

Jie Wei, Guanyu Hu, Xinyu Yang, Anh Tuan Luu, and Yizhuo Dong. 2022. Audio-visual domain adaptation feature fusion for speech emotion recognition. In _INTERSPEECH_, pages 1988-1992.
* Wei et al. (2024) Jie Wei, Guanyu Hu, Xinyu Yang, Anh Tuan Luu, and Yizhuo Dong. 2024. Learning facial expression and body gesture visual information for video emotion recognition. _Expert Systems with Applications_, 237:121419.
* ECCV 2016_, pages 499-515, Cham. Springer International Publishing.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 38-45.
* Wu et al. (2023a) Xiaobao Wu, Xinshuai Dong, Thong Nguyen, Chaoqun Liu, Liang-Ming Pan, and Anh Tuan Luu. 2023a. Infoctm: A mutual information maximization perspective of cross-lingual topic modeling. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 37, pages 13763-13771.
* Wu et al. (2023b) Xiaobao Wu, Xinshuai Dong, Thong Nguyen, and Anh Tuan Luu. 2023b. Effective neural topic modeling with embedding clustering regularization. In _International Conference on Machine Learning_. PMLR.
* Wu et al. (2020) Xiaoboao Wu, Chunping Li, Yan Zhu, and Yishu Miao. 2020. Short text topic modeling with topic distribution quantization and negative sampling decoder. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1772-1782, Online.
* Wu et al. (2022) Xiaobao Wu, Anh Tuan Luu, and Xinshuai Dong. 2022. Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 2748-2760, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Wu et al. (2024) Xiaobao Wu, Fengjun Pan, Thong Nguyen, Yichao Feng, Chaoqun Liu, Cong-Duy Nguyen, and Anh Tuan Luu. 2024. On the affinity, rationality, and diversity of hierarchical topic modeling. In _Proceedings of the AAAI Conference on Artificial Intelligence_.
* Yan et al. (2021) Yuanming Yan, Runmei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. ConSERT: A contrastive framework for self-supervised sentence representation transfer. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5065-5075.
* Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. _Transactions of the Association for Computational Linguistics (TACL)_, pages 67-78.
* Zhang et al. (2022a) Miaoran Zhang, Marius Mosbach, David Ifeoluwa Adelani, Michael A. Hedderich, and Dietrich Klakow. 2022a. Mcse: Multimodal contrastive learning of sentence embeddings.
* Zhang et al. (2020) Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. 2020. An unsupervised sentence embedding method by mutual information maximization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1601-1610.
* Zhang et al. (2022b) Yanzhao Zhang, Richong Zhang, Samuel Mensah, Xudong Liu, and Yongyi Mao. 2022b. Unsupervised sentence representation via contrastive learning with mixing negatives.
* Zhang et al. (2022c) Yuhao Zhang, Hongji Zhu, Yongliang Wang, Nan Xu, Xiaobo Li, and Binqiang Zhao. 2022c. A contrastive framework for learning sentence representations from pairwise and triple-wise perspective in angular space. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 4892-4903, Dublin, Ireland. Association for Computational Linguistics.
* Zhou et al. (2022) Kun Zhou, Beichen Zhang, Wayne Xin Zhao, and Ji-Rong Wen. 2022. Debiased contrastive learning of unsupervised sentence representations 



## Appendix A Regularization

To validate the significance of our objectives, we select the best-performing models for each dataset and gradually run with three settings: our KDMCSE, KDMCSE without AdapACSE, and KDMCSE without threshold filtering. The results are presented in Table 3, indicating that the incorporation of all the proposed approaches leads to the best performance in all tasks. The second line shows that not applying threshold filtering can marginally harm performance. This result can be attributed to the fact that, during training, noisy negative samples do not often appear in the batch. In the last row of the table, we show that the removal of AdapACSE significantly affects the performance of both BERT and RoBERTa.

## Appendix B Larger experiments on image-text paired data

To mitigate scalability and generalization concerns, we conducted extensive experiments using a significantly larger dataset containing one million image-text pairs sourced from the CC12 dataset, equivalent in sample size to Wiki1M. In Table 4, we present the results of these experiments. Our findings consistently demonstrate that our KDMCSE model surpasses MCSE in terms of average performance (Avg) across STS, across various subtasks within CC12M subsets.

## Appendix C Impact of Adaptive Angular Margin Contrastive objective

To substantiate the significance of our objectives and ensure fairness in our comparisons, we carried out experiments involving MCSE with the inclusion of additional text embeddings from the CLIP text encoder. The results of these experiments are presented in Table 5. Our findings unequivocally indicate that our proposed contrastive objectives have a positive impact on the model's performance, thereby strengthening our core objectives.

\begin{table}
\begin{tabular}{l c c c c c c|c|c} \hline \hline
**Model** & **STS12** & **STS13** & **STS14** & **STS15** & **STS16** & **STS-B** & **SICK-R** & **Avg.\(\uparrow\)** \\ \hline \hline  & \multicolumn{8}{c}{Subset 1} \\ \cline{2-9} MCSE & 71.54 & 77.9 & 73.6 & 80.83 & 78.38 & 79.73 & 74.66 & 76.66 \\ KDMCSE & 73.00 & 78.53 & 74.93 & 82.14 & 79.88 & 81.37 & 74.08 & 77.7 \\ \hline \hline  & \multicolumn{8}{c}{Subset 2} \\ \cline{2-9} MCSE & 71.67 & 78.56 & 74.02 & 81.19 & 78.54 & 80.29 & 73.87 & 76.88 \\ KDMCSE & 73.35 & 79.51 & 75.58 & 83.43 & 80.07 & 81.89 & 73.78 & 78.23 \\ \end{tabular}
\end{table}
Table 4: Ablation results on STS tasks. STS-B: STS Benchmark, SICK-R: SICK-Relatedness, Avg.: average across \(7\) tasks. The models are trained with the BERT encoder and in 1M image-text pairs subsets of the CC12 dataset.

\begin{table}
\begin{tabular}{l c c c c c c c|c} \hline \hline
**Model** & **STS12** & **STS13** & **STS14** & **STS15** & **STS16** & **STS-B** & **SICK-R** & **Avg.\(\uparrow\)** \\ \hline \hline KDMCSE-BERT & 74.4 & 83.1 & 76.3 & 83.7 & 78.8 & 81.3 & 73.0 & 78.6 \\ KDMCSE without AdapACSE & 73.8 & 81.3 & 75.4 & 83.2 & 79.0 & 80.5 & 74.1 & 78.2 \\ KDMCSE without Threshold Filtering & 73.4 & 82.7 & 76.6 & 83.9 & 78.8 & 80.8 & 72.9 & 78.4 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Ablation results on STS tasks. STS-B: STS Benchmark, SICK-R: SICK-Relatedness, Avg.: average across \(7\) tasks. The models are trained with the BERT encoder and in _wiki-flickr_ setting.

\begin{table}
\begin{tabular}{l l l l l l l l|c} \hline \hline
**Model** & **STS12** & **STS13** & **STS14** & **STS15** & **STS16** & **STS-B** & **SICK-R** & **Avg.\(\uparrow\)** \\ \hline MCSE-BERT\({}_{CLIPTcat}\) & 72.60 & 81.86 & 75.28 & 83.28 & 78.74 & 80.77 & 72.78 & 77.90 \\ KDMCSE-BERT & 74.40 & 83.10 & 76.26 & 83.68 & 78.79 & 81.25 & 73.03 & 78.64 \\ \hline MCSE-BERT\({}_{CLIPTcat}\) & 71.39 & 79.25 & 73.84 & 82.65 & 77.96 & 78.85 & 71.74 & 76.53 \\ KDMCSE-BERT & 73.38 & 80.71 & 75.55 & 83.36 & 79.91 & 79.85 & 73.92 & 78.10 \\ \end{tabular}
\end{table}
Table 5: Ablation study on STS tasks. STS-B: STS Benchmark, SICK-R: SICK-Relatedness, Avg.: average across \(7\) tasks.. We train the MCSE models using additional CLIP text embeddings.



ENDFIGURE]

## Appendix D KDMCSE algorithm

To enhance the presentation of our method, we provide a detailed description of our implementation, adhering to the pseudo code outlined in Algorithm 1.

``` Input: Collection of sentences (text-only dataset) \(D=\{x_{i}\}_{i=1}^{n}\) Collection of image-text pairs (multimodal dataset) \(D^{M}=\{x_{i},y_{i}\}_{i=1}^{n_{m}}\) Paired sample step \(p=\left\lceil\frac{|D|}{|D^{M}|}\right\rceil\) for\(t=1,2,3,\ldots,iter\)do if not \(t|p\)then Sample batch \(\{x_{i}\}_{bs}\) from \(D\)  // Forward through language model and projection \(\mathbf{h}_{i}^{s},\mathbf{h}_{i}^{s^{\prime}}=g_{\phi}(f_{\theta}(x_{i}))\)   // Calculate loss \(\frac{1}{bs}\sum_{i}^{bs}\ell_{i}^{S}\) else  Sample batch \(\{x_{i},y_{i}\}_{bs}\) from \(D^{M}\)\(\mathbf{s}_{i}^{s}=g_{\phi_{\phi}}(f_{\theta}(x_{i},z))\)\(\mathbf{t}_{i}=g_{\phi_{o}}(\mathbf{t}_{i}),\mathbf{t}_{i}=CLIP_{text}(x_{i})\)\(\mathbf{v}_{i}=g_{\phi_{o}}(\tilde{\mathbf{v}}_{i}),\hat{\mathbf{v}}_{i}=CLIP_{visual}(y_{ i})\)  // Calculate cosine similarity \(\alpha_{i,j}^{t,t}=\dfrac{\mathbf{s}_{i}\cdot\mathbf{t}_{j}}{\lVert\mathbf{s}_{i}\rVert_{2 }\lVert\mathbf{t}_{j}\rVert_{2}}\)\(\alpha_{i,j}^{t,v}=\dfrac{\mathbf{s}_{i}^{s}\cdot\mathbf{v}_{j}}{\lVert\mathbf{s}_{i}\rVert_{2 }\lVert\mathbf{v}_{j}\rVert_{2}}\)  // Calculate soft-label (text-text 

Figure 8: Sentence 1: A white and black dog and a brown dog in sandy terrain. Sentence 2: Brown , black and white dog standing on a sandy slope. Sentence 3: Two girls stand up against a red wall.

Figure 9: Sentence 1: An elderly man holding a pitchfork and doing some yard work. Sentence 2: An old bearded man is filling the soil with a simple wooden plow , with a wooden fence and an old barn in the background. Sentence 3: A man in a red shirt is performing an aerial trick with a skateboard on a sidewalk.

 

Figure 10: Sentence 1: A person wearing a blue shirt , rides a white horse along a dusty country road. Sentence 2: A man in a blue tee-shirt , wearing a cap , is riding a white horse down a dirt road , in a rural setting of grass and hills. Sentence 3: A man climbing a mountain.

Figure 11: Sentence 1: A dog is jumping into a swimming pool after a duck. Sentence 2: A dog is jumping into a pool to get a duck floating there. Sentence 3: A white curly-haired dog runs with a stick in its mouth.

 


* [13] M. C. B. Kondor, A. J. S. G. Wilson, and R. W. H. Williams, _The Origins of the Universe: An Introduction to the Theory of Relativity_. Cambridge University Press, Cambridge, MA, USA, 2004.
* [14] D. E. Kaplan, _A Course on the History of Quantum Mechanics_. Cambridge University Press, Cambridge, MA, USA, 2005.
* [15] F. L. Wilczek, _Quantum mechanics in quantum physics_. Cambridge University Press, Cambridge, MA, USA, 2006.
* [16] P. W. Shorlin, _Quantum Computation_. Cambridge University Press, Cambridge, MA, USA, 2007.
* [17] T. W. Cover, _Elements of Information Theory_. Cambridge University Press, Cambridge, MA, USA, 2008.
* [18] N. V. Kolmogorov, _Random Fields and Random Processes_. Academic Press, New York, NY, 1964.
* [19] I. A. Kolmogorov, _Probability_. Princeton University Press, Princeton, NJ, 1964.
* [20] I. A. Kolmogorov, _Statistical theory of statistical processes_. Oxford University Press, Oxford, UK, 2004.

[MISSING_PAGE_POST]