DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learning
Authors: Kang He, Yuzhe Ding, Haining Wang, Fei Li, Chong Teng, Donghong Ji
Accepted at: acl25

DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learning

Kang He, Yuzhe Ding, Haining Wang, Fei Li, Chong Teng*, Donghong Ji
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University
{hekang0225, lifei_csnlp, tengchong}@whu.edu.cn

Abstract
Previous multimodal sentence representation learning methods have achieved impressive performance. However, most approaches focus on aligning images and text at a coarse level, facing two critical challenges: cross-modal misalignment bias and intra-modal semantic divergence, which significantly degrade sentence representation quality. To address these challenges, we propose DALR (Dual-level Alignment Learning for Multimodal Sentence Representation). For cross-modal alignment, we propose a consistency learning module that softens negative samples and utilizes semantic similarity from an auxiliary task to achieve fine-grained cross-modal alignment. Additionally, we contend that sentence relationships go beyond binary positive-negative labels, exhibiting a more intricate ranking structure. To better capture these relationships and enhance representation quality, we integrate ranking distillation with global intra-modal alignment learning. Comprehensive experiments on semantic textual similarity (STS) and transfer (TR) tasks validate the effectiveness of our approach, consistently demonstrating its superiority over state-of-the-art baselines.

1 Introduction
Sentence representation learning converts sentences into low dimensional vectors to preserve semantic information and is widely used in NLP tasks, such as semantic similarity (Agirre et al., 2012, 2013), information extraction (Wang et al., 2022a; Zheng et al., 2024), and content analysis (Ling et al., 2022; Wang et al., 2024; Zheng et al., 2025). With the success of pre-trained language models (PLMs) such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), numerous methods (Gao et al., 2021; Wu et al., 2022b; Zhang et al., 2022b; He et al., 2023; Seonwoo et al., 2023; He et al., 2025) have achieved remarkable performance by contrastive learning and different augmentation strategies.

Unfortunately, the methods of constructing positive (Yan et al., 2021; Wu et al., 2022a; Zhuo et al., 2023) and negative (Zhou et al., 2022; Deng et al., 2023; Shi et al., 2023) samples are usually too simple to capture nuanced semantic relationships between sentences deeply. For example, although “A man is skating” and “A man is gliding” are mutually exclusive in common sense, this contradiction is not easily captured through text alone. However, visual information can naturally reveal such contradictions, providing a rich supervision signal for better understanding (Wang et al., 2022b). Incorporating visual signals into language models has been shown to improve performance across various downstream tasks (Bordes et al., 2020; Tang et al., 2021; Nguyen et al., 2023; Huang et al., 2023a). MCSE (Zhang et al., 2022a) leveraged multimodal contrastive learning for cross-modal alignment, and

![Illustration of a batch image-caption pairs from the Flickr dataset. KDMCSE sets a threshold based on sim(I, T) to filter out false negatives. ✓: denotes the sample is correctly classified as false negative or true negative based on image-text similarity. ✗: indicates a sample misclassified as a false negative and erroneously filtered due to its high similarity with the anchor image.](page_768_340_627_312.png)

Figure 1: Illustration of a batch image-caption pairs from the Flickr dataset. KDMCSE sets a threshold based on \( \operatorname{sim}(I, T) \) to filter out false negatives. \( \checkmark \): denotes the sample is correctly classified as false negative or true negative based on image-text similarity. \( \times \): indicates a sample misclassified as a false negative and erroneously filtered due to its high similarity with the anchor image.
KDMCSE (Nguyen et al., 2024) further enhanced alignment by filtering highly similar samples to reduce false negatives and applying adaptive angular contrastive learning to better distinguish negatives. Despite these advances, aligning text and image through semantic similarity still faces two key challenges: cross-modal misalignment bias and intra-modal semantic divergence.

Cross-modal Misalignment Bias (CMB) stems from the inherent asymmetry between modalities when aligning image-text pairs. Text is typically information-dense and selective, focusing on key details, while images capture all components indiscriminately, leading to significant redundancy. As shown in the purple box in Figure 1, the focal object of the image \( I_n \) is “a boat”, which occupies only a small region, with most visual patches containing irrelevant information. Moreover, due to cognitive biases among annotators, a single image may have multiple semantic descriptions (Chun et al., 2021), further amplifying the heterogeneity between modalities. This mismatch causes semantic similarity to misrepresent true alignment, resulting in biased representations.

Intra-modal Semantic Divergence (ISD) refers to the erroneous identification of semantically divergent texts as highly similar due to their shared reference to the same image. Studies (Chun et al., 2022; Parekh et al., 2021) have noted that multiple captions (or images) can describe the same image (or caption) with differing focuses. For instance, in Figure 1, given the anchor image \( I_1 \), caption \( T_1 \) (“Several boys playing football in blue and white uniforms”) emphasizes the players’ appearance and activity, while \( T_p \) (“A group of people are watching a group of boys in football gear”) highlights the spectators. Despite their semantic divergence, both captions exhibit high image similarity, resulting in false negatives. This misalignment undermines intra-modal consistency and degrades sentence representation quality.

To address these challenges, we propose DALR: a Dual-level Alignment Learning Framework for Multimodal Sentence Representation. First, for cross-modal alignment, we introduce an auxiliary cross-modal consistency task that enhances supervision by predicting image-text correspondence through a binary classification framework. This task extracts latent semantic features and constructs a semantic similarity matrix as a soft target to unify representations across modalities. Second, to mitigate intra-modal semantic divergence, we argue that sample relationships are inherently continuous rather than binary. We propose an intra-modal alignment strategy, employing multi-teacher models to generate coarse-grained semantic rankings as pseudo-labels. This strategy incorporates KL divergence to ensure the student model captures global information from the teachers, thereby achieving robust intra-modal alignment.

Experiments on the widely-used STS and TR tasks showcase the considerable effectiveness of DALR. Ablation studies and visualization analysis further validate the existence of CMB and ISD issue and the necessity of joint modality alignment. The main contributions are summarized as follows:

• We introduce DALR to enhance text representations through joint cross-modal and intra-modal alignment.

• We propose a cross-modal alignment method with auxiliary tasks to soften negative samples and improve alignment to mitigate CMB issue.

• We adopt ranking distillation with global alignment learning to capture fine-grained semantic structures for ISD issue.

• Thorough experiments show that DALR improves the performance over all metrics and achieves state-of-the-art on two benchmarks¹.

2 Related Work

2.1 Sentence Representation Learning

Sentence representation learning is a fundamental task in natural language processing. Early methods, such as Skip-Thought (Kiros et al., 2015) and FastSent (Hill et al., 2016), leverage contextual relationships to learn sentence representations. With the progression of PLMs and SimCSE (Gao et al., 2021), the “PLMs + contrastive learning” paradigm has become increasingly prevalent. Data augmentation strategies (Yan et al., 2021; Wu et al., 2022b; Zhuo et al., 2023; He et al., 2023) enhance representation quality by generating diverse positive samples. ConSERT (Yan et al., 2021) uses dropout masking and token shuffling, while PCL (Wu et al., 2022a) adopts multiple augmentation techniques. WhitenedCSE (Zhuo et al., 2023) improves diversity through inter-group whitening. Additionally, advancements in negative sampling (Zhou et al.,

¹https://github.com/Hekang001/DALR.
2022; Deng et al., 2023) and hard negative construction (Shi et al., 2023) further refine sentence representation learning.

2.2 Modality Alignment
Research on modality alignment (Cheng et al., 2023b; Liu et al., 2023b; Zhang et al., 2023; Han et al., 2024) aims to unify feature representations across modalities (e.g., image, text, audio) for enhanced representation learning (Li et al., 2021; Huang et al., 2023b; Zhu et al., 2023), cross-modal understanding (Yu et al., 2023; Li et al., 2023), and generation tasks (Sung-Bin et al., 2023; Tian et al., 2023). Methods like ALBEF (Li et al., 2021) align image-text features through cross-modal attention, while MVPTR (Li et al., 2022) focuses on multi-level semantic alignment. MCSE (Zhang et al., 2022a) integrates visual information into sentence embeddings, and KDMCSE (Nguyen et al., 2024) improves this by leveraging external models for distillation and filtering false negatives. In contrast, our approach balances cross-modal alignment with intra-modal semantic consistency, enhancing visual information utilization and improving sentence representation quality.

3 Methodology

3.1 Preliminary Work
Unsupervised SimCSE Unsupervised SimCSE (Gao et al., 2021) leverages dropout as a minimal data augmentation strategy. Given a sentence set \( T = \{t_i\}_{i=1}^N \), each sentence is encoded twice with different dropout masks, producing two representations \( s_i^{z'} = g_{\varphi_\theta}(f_\theta(t_i, z')) \) and \( s_i^{z''} = g_{\varphi_\theta}(f_\theta(t_i, z'')) \), where \( f_\theta \) is a pre-trained language encoder (e.g., BERT), and \( g_{\varphi_\theta} \) is a projection head. The [CLS] token is used as the final embedding, and the objective is to maximize the similarity between paired representations:

\[
\mathcal{L}_{text} = - \sum_{i=1}^N \log \frac{e^{sim(s_i^{z'}, s_i^{z''})/\tau}}{\sum_{j=1}^N e^{sim(s_i^{z'}, s_j^{z''})/\tau}}
\]

where \( N \) is the batch size and \( \tau \) is a temperature hyper-parameter. \( sim(h_1, h_2) = \frac{h_1^T h_2}{\|h_1\|\|h_2\|} \) is cosine similarity function.

Multimodal Contrastive Learning Given a set of image-text pairs represented as \( C = \{v_i, t_i\}_{i=1}^N \in \mathcal{D} \), MCSE (Zhang et al., 2022a) projects text \( t_i \) and image \( v_i \) into a unified space:

\[
s_i^z = g_{\varphi_\theta}(f_\theta(t_i, z))
\]

\[
h_i^v = g_{\varphi_v}(f_v(v_i)), \quad h_i^t = g_{\varphi_t}(f_t(t_i))
\]

where \( f_v(\cdot) \) denotes a frozen image teacher encoder, and \( f_t(\cdot) \) refers to a frozen text teacher encoder. (More details for image and text teacher encoder are in Section 4.1 and Appendix B.) \( z \) denotes the dropout mask, \( g_{\varphi_\theta}(\cdot) \) is the projection head of the language student model that projects the sentence representation into a shared space, \( g_{\varphi_v}(\cdot) \) and \( g_{\varphi_t}(\cdot) \) are the projection heads of the image and text teacher models, respectively. Therefore, the multimodal contrastive learning objective using InfoNCE (Oord et al., 2018) is expressed as:

\[
\mathcal{L}_{Info} = - \sum_{i=1}^N \log \frac{e^{sim(s_i^v, h_i^v)/\tau}}{\sum_{j=1}^N e^{sim(s_i^v, h_j^v)/\tau}}
\]

3.2 Cross-modal Alignment learning
Figure 2 illustrates the main workflow of DALR. Image and text features exhibit a significant semantic gap, making direct mapping into a shared space for alignment challenging. We propose a cross-modal alignment method with an auxiliary consistency task to capture fine-grained image-text semantics. The generated similarity matrix refines negative samples, providing a guiding signal for enhanced cross-modal contrastive learning.

Cross-modal consistency learning We formulate this module as a binary classification task to predict image-text alignment based on multimodal features. Given the original dataset \( \mathcal{D} \) with aligned image-text pairs, we construct a new dataset \( \mathcal{D}' \) by shuffling images to create mismatched pairs. This enables the model to learn to distinguish between aligned and misaligned pairs. For each image-text pair \( C' = \{v', t'\} \in \mathcal{D}' \), we extract unimodal representations using \( f_v \) and \( f_\theta \), which are then projected into a shared space via modality-specific MLPs, obtaining shared representations \( h_{s'}^{v'} \) and \( s_{s'}^{z'} \) as defined in Eq.2 and Eq.3. We use the cosine embedding loss function with margin m for optimization as follows:

\[
\mathcal{L}_{cons} = \begin{cases}
1 - \cos(h_{s'}^{v'}, s_{s'}^{z'}) & \text{if } y' = 1, \\
\max(0, \cos(h_{s'}^{v'}, s_{s'}^{z'}) - m) & \text{if } y' = 0.
\end{cases}
\]

where \( \cos(\cdot) \) represents the normalized cosine similarity, and \( m \) controls the margin for negative samples, typically set to 0.2 based on empirical findings. The consistency learning task captures deeper semantic relationships by refining the matching between images and texts. It enhances the model’s
Figure 2: The illustration of our proposed framework DALR, consisting of three components: (a) the multimodal contrastive learning module uses the guidance of visual information to obtain sentence representations, (b) the cross-modal alignment module further aligns cross-modal features, and (c) the intra-modal alignment module enhances internal alignment through ranking distillation learning and KL divergence.

multimodal representation, improves the discrimination of negative samples, and reduces noise. Notably, this task can be learned in parallel with contrastive learning, generating cross-modal soft labels to guide alignment.

Cross-modal alignment We use the representation \( s_i^t \) obtained by the language student model and \( h_i^v \) obtained by the visual teacher model to calculate the cosine similarity, and perform normalization to obtain the probability distribution \( P_{ij}^{t2v} \) of pairing \( v_i \) with \( t_j \):

\[
P_{ij}^{t2v} = \frac{e^{\operatorname{sim}(s_i^t, h_j^v)}}{\sum_{k=1}^N e^{\operatorname{sim}(s_i^t, h_k^v)}}, \quad P^{t2v} = (P_{i1}^{t2v}, P_{i2}^{t2v}, ..., P_{iN}^{t2v})
\]

where \( P^{t2v} \) is the probability distribution set composed of \( P_{ij}^{t2v} \) in the same batch. At the same time, we compute the cosine similarity within the teacher text model and normalize it to obtain the probability estimate \( Q_{ij}^{t2t} \) from the teacher model:

\[
Q_{ij}^{t2t} = \frac{e^{\operatorname{sim}(h_i^t, h_j^t)}}{\sum_{j=1}^N e^{\operatorname{sim}(h_i^t, h_j^t)}}, \quad Q^{t2t} = (Q_{i1}^{t2t}, Q_{i2}^{t2t}, ..., Q_{iN}^{t2t})
\]

Similarly, the similarity between \( h_i^v \) and \( h_j^v \) is calculated through the features obtained by the teacher vision model to obtain \( Q_{ij}^{v2v} \). In model training, we promote the alignment between images and texts by minimizing the KL divergence between the target distribution (\( Q_{ij}^{t2t}, Q_{ij}^{v2v} \)) and the predicted distribution \( P_{ij}^{t2v} \):

\[
\mathcal{L}_{CMA} = \frac{1}{2} \sum_{i=1}^N (D_{KL}(Q_{i}^{t2t}||P_{i}^{t2v}) + D_{KL}\left(Q_{i}^{v2v}||P_{i}^{v2v})^T\right))
\]

where \( D_{KL}(\cdot) \) represents KL divergence. Minimizing KL divergence is equivalent to maximizing the mutual information between the teacher and student distributions, which facilitates cross-modal information transfer to some extent. Through the aforementioned two parts, we can capture more cross-modal detailed semantic information and facilitate the learning of sentence representation. The final loss \( \mathcal{L}_{CML} \) of cross-modal contrastive learning is calculated as follows:

\[
\mathcal{L}_{CML} = \mathcal{L}_{cons} + \mathcal{L}_{CMA}
\]

3.3 Intra-modal Alignment Learning

Despite progress in cross-modal alignment, existing methods, such as KDMCSE (Nguyen et al., 2024), overlook the sparsity of image information and the variation in textual focus on different image regions. This results in multiple low-similarity texts aligning with a single image (Chun et al., 2021; Parekh et al., 2021), undermining semantic accuracy, a phenomenon we term “intra-modal semantic divergence”.

To address this, we introduce an intra-modal alignment method featuring two components: ranking distillation for fine-grained semantic capture
and KL-based inter-modal alignment for global distribution learning. Ranking information, which reflects subtle structural differences between sentences, enhances modality alignment. We employ multiple teachers (SimCSE and DiffCSE) to provide comprehensive ranking data, with a weighted combination of [CLS] token embeddings yielding the final representation. The teachers’ similarity score lists act as pseudo-ranking labels, guiding the intra-modal alignment. We apply ListMLE (Xia et al., 2008) to refine ranking learning:

\[
\mathcal{L}_{rank} = - \sum_{i=1}^{N} \log \left( \prod_{j=1}^{M} \frac{\exp \left( S(x_i)_{\pi_i^T(j)}/\tau \right)}{\sum_{k=1}^{M} \exp \left( S(x_i)_{\pi_i^T(k)}/\tau \right)} \right)
\]

where \( S(x_i) \) represents the list of similarity scores generated by the student model for the text input \( x_i \), \( \pi_i^T(j) \) is the index of the \( j \)-th position in the ranking \( \pi_i^T \) generated by the teacher model, and \( S(x_i)_{\pi_i^T(j)} \) represents the score of the student model for the \( j \)-th position in the ranking.

ListMLE directly optimizes ranking order but neglects the probabilistic structure of the score distribution. This simplified approach may fail to capture the global probability information from the teacher model, limiting sentence representation performance. To address this, we introduce KL divergence to minimize the statistical distribution gap between the teacher and student models, aligning pseudo-labels with the student model’s predictions. This reduces confusion between pseudo-labels and model outputs, enhancing learning effectiveness. Specifically, using Eq.6, we can derive the text distribution probability \( P_i^{t2t} \) of the student model:

\[
P_{ij}^{t2t} = \frac{e^{sim(s_i^z, s_j^{z'})}}{\sum_{j=1}^{N} e^{sim(s_i^z, s_j^{z'})}}, \quad P_i^{t2t} = (P_{i1}^{t2t}, P_{i2}^{t2t}, ..., P_{iN}^{t2t})
\]

where \( z, z' \) represent different dropouts, and \( P_i^{t2t} \) is a probability distribution set consisting of a set of probability distributions \( \mathcal{P} = \{P_{ij}^{t2t}\}_{j=1}^{N} \). Finally, we learn a more general distribution by optimizing the KL divergence between the teacher distribution probability \( Q_i^{t2t} \) and the student distribution probability \( P_i^{t2t} \). The objective is as follows:

\[
\mathcal{L}_{IMA} = \sum_{i=1}^{N} (D_{KL}(Q_i^{t2t} \| P_i^{t2t}))
\]

By combining \( \mathcal{L}_{rank} \) and \( \mathcal{L}_{IMA} \), we can ensure that the student model not only matches the overall similarity distribution (KL divergence), but also preserves the critical ranking information (ListMLE). Therefore, the goal of intra-modal alignment learning is:

\[
\mathcal{L}_{IML} = \mathcal{L}_{rank} + \mathcal{L}_{IMA}
\]

3.4 Training Objectives

According to Eq.4, Eq.9 and Eq.13, we can add all losses to a final loss:

\[
\mathcal{L}_{total} = \mathcal{L}_{info} + \lambda \mathcal{L}_{CML} + \mu \mathcal{L}_{IML}
\]

where \( \lambda \) and \( \mu \) are hyper-parameters for weights balance.

4 Experiments

4.1 Experiments Setup

We evaluate our method on two sentence related tasks: semantic textual similarity (STS) and transfer (TR) task. For the STS tasks, we evaluate on seven datasets: STS 2012-2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017) and SICK-Relatedness (Marelli et al., 2014). We use the SentEval toolkit (Conneau and Kiela, 2018) for evaluation and adopt the Spearman’s correlation coefficient (multiplied by 100) as the reporting metric. For the TR tasks, we also use SentEval to evaluate on seven datasets: MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST-2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000) and MRPC (Dolan and Brockett, 2005).

Datasets According to MCSE (Zhang et al., 2022a), we use Flickr (Young et al., 2014) and MSCOCO (Lin et al., 2014) as multimodal sentence embedding datasets. In addition, we follow SimCSE (Gao et al., 2021) and use 1,000,000 sentences randomly selected from Wikipedia as the training dataset.

Baseline Models Following the standard protocol on the two benchmarks (Gao et al., 2021), we compare our model with three baseline models: SimCSE (Gao et al., 2021), MSE (Zhang et al., 2022a), KDMCSE (Nguyen et al., 2024). More details of baseline models are in Appendix A.

Implementation Details During model initialization, we utilize SimCSE and DiffCSE as two text teachers and load the checkpoint of CLIP-ViT-B/32 as the image teacher model. During training, considering that the sizes of the pure-text dataset (with
<table>
  <tr>
    <th>Model</th>
    <th>STS12</th>
    <th>STS13</th>
    <th>STS14</th>
    <th>STS15</th>
    <th>STS16</th>
    <th>STS-B</th>
    <th>SICK-R</th>
    <th>Avg.↑</th>
  </tr>
  <tr>
    <td rowspan="2">wiki</td>
    <td>SimCSE-BERT<sup>○</sup></td>
    <td>67.8±1.6</td>
    <td>80.0±2.1</td>
    <td>72.5±1.7</td>
    <td>80.1±0.8</td>
    <td>77.6±0.8</td>
    <td>76.5±0.8</td>
    <td>70.1±0.9</td>
    <td>74.9±1.1</td>
  </tr>
  <tr>
    <td>SimCSE-RoBERTa<sup>○</sup></td>
    <td>68.7±1.0</td>
    <td>82.0±0.5</td>
    <td>74.0±1.0</td>
    <td>82.1±0.4</td>
    <td>81.1±0.4</td>
    <td>80.6±0.3</td>
    <td>69.2±0.2</td>
    <td>76.8±0.5</td>
  </tr>
  <tr>
    <td rowspan="4">wiki+flcikr</td>
    <td>SimCSE-BERT<sup>†</sup></td>
    <td>69.9±1.7</td>
    <td>79.8±1.5</td>
    <td>72.9±0.9</td>
    <td>81.9±0.8</td>
    <td>77.8±0.9</td>
    <td>76.0±1.1</td>
    <td>68.4±0.8</td>
    <td>75.3±0.9</td>
  </tr>
  <tr>
    <td>MCSE-BERT<sup>†</sup></td>
    <td>71.4±0.9</td>
    <td>81.8±1.3</td>
    <td>74.8±0.9</td>
    <td>83.6±0.9</td>
    <td>77.5±0.8</td>
    <td>79.5±0.5</td>
    <td>72.6±1.4</td>
    <td>77.3±0.5</td>
  </tr>
  <tr>
    <td>KDMCSE-BERT<sup>‡</sup></td>
    <td><b>74.4±1.4</b></td>
    <td>83.1±0.9</td>
    <td>76.3±1.1</td>
    <td>83.7±0.8</td>
    <td>78.8±0.9</td>
    <td>81.3±0.9</td>
    <td>73.0±0.9</td>
    <td>78.6±0.8</td>
  </tr>
  <tr>
    <td>DALR-BERT</td>
    <td>73.9±0.8</td>
    <td><b>84.0±0.7</b></td>
    <td><b>76.5±0.5</b></td>
    <td><b>84.3±0.9</b></td>
    <td><b>80.6±1.1</b></td>
    <td><b>81.8±0.2</b></td>
    <td><b>75.3±0.4</b></td>
    <td><b>79.5±0.7</b></td>
  </tr>
  <tr>
    <td rowspan="4">wiki+coco</td>
    <td>SimCSE-RoBERTa<sup>†</sup></td>
    <td>69.5±0.9</td>
    <td>81.6±0.5</td>
    <td>74.1±0.6</td>
    <td>82.4±0.3</td>
    <td>80.9±0.5</td>
    <td>79.9±0.3</td>
    <td>67.3±0.5</td>
    <td>76.5±0.4</td>
  </tr>
  <tr>
    <td>MCSE-RoBERTa<sup>†</sup></td>
    <td>71.7±0.2</td>
    <td>82.7±0.4</td>
    <td>75.9±0.3</td>
    <td>84.0±0.4</td>
    <td>81.3±0.3</td>
    <td>82.3±0.5</td>
    <td>70.3±1.3</td>
    <td>78.3±0.1</td>
  </tr>
  <tr>
    <td>KDMCSE-RoBERTa<sup>‡</sup></td>
    <td><b>73.6±0.7</b></td>
    <td>83.8±0.6</td>
    <td><b>77.4±0.4</b></td>
    <td>84.0±0.3</td>
    <td>81.5±0.7</td>
    <td>82.3±0.6</td>
    <td>71.2±0.4</td>
    <td>79.1±0.3</td>
  </tr>
  <tr>
    <td>DALR-RoBERTa</td>
    <td><b>73.6±0.4</b></td>
    <td><b>84.4±0.2</b></td>
    <td><b>77.2±0.6</b></td>
    <td><b>84.9±0.7</b></td>
    <td><b>82.0±0.4</b></td>
    <td><b>82.6±0.2</b></td>
    <td><b>74.6±0.7</b></td>
    <td><b>79.9±0.5</b></td>
  </tr>
  <tr>
    <td rowspan="4">wiki+reco</td>
    <td>SimCSE-BERT<sup>†</sup></td>
    <td>69.1±0.8</td>
    <td>80.4±0.9</td>
    <td>72.7±0.7</td>
    <td>81.1±0.3</td>
    <td>78.2±0.9</td>
    <td>73.9±0.6</td>
    <td>66.6±1.2</td>
    <td>74.6±0.2</td>
  </tr>
  <tr>
    <td>MCSE-BERT<sup>†</sup></td>
    <td>71.2±1.3</td>
    <td>79.7±0.9</td>
    <td>73.8±0.9</td>
    <td>83.0±0.4</td>
    <td>77.8±0.9</td>
    <td>78.3±0.4</td>
    <td>72.1±1.4</td>
    <td>76.6±0.5</td>
  </tr>
  <tr>
    <td>KDMCSE-BERT<sup>‡</sup></td>
    <td>73.2±1.2</td>
    <td>80.5±1.0</td>
    <td>75.4±0.9</td>
    <td>83.2±0.3</td>
    <td>79.7±0.8</td>
    <td>79.7±0.7</td>
    <td>73.7±1.4</td>
    <td>77.9±1.2</td>
  </tr>
  <tr>
    <td>DALR-BERT</td>
    <td><b>73.4±1.0</b></td>
    <td><b>82.6±1.2</b></td>
    <td><b>75.6±0.8</b></td>
    <td><b>83.5±0.6</b></td>
    <td><b>80.8±0.7</b></td>
    <td><b>80.5±0.5</b></td>
    <td><b>74.1±0.9</b></td>
    <td><b>78.6±0.9</b></td>
  </tr>
  <tr>
    <td rowspan="4"></td>
    <td>SimCSE-RoBERTa<sup>†</sup></td>
    <td>66.4±0.9</td>
    <td>80.7±0.7</td>
    <td>72.7±1.1</td>
    <td>81.3±0.9</td>
    <td>80.2±0.8</td>
    <td>76.8±0.6</td>
    <td>65.7±0.7</td>
    <td>74.8±0.5</td>
  </tr>
  <tr>
    <td>MCSE-RoBERTa<sup>†</sup></td>
    <td>70.2±1.7</td>
    <td>82.0±0.7</td>
    <td>75.5±1.2</td>
    <td>83.0±0.6</td>
    <td>81.5±0.7</td>
    <td>80.8±1.0</td>
    <td>69.9±0.6</td>
    <td>77.6±0.8</td>
  </tr>
  <tr>
    <td>KDMCSE-RoBERTa<sup>‡</sup></td>
    <td>72.8±1.5</td>
    <td>81.7±0.9</td>
    <td>76.1±1.1</td>
    <td>83.4±1.0</td>
    <td>81.5±0.6</td>
    <td>80.7±0.8</td>
    <td>69.9±0.6</td>
    <td>78.0±0.7</td>
  </tr>
  <tr>
    <td>DALR-RoBERTa</td>
    <td><b>73.1±0.3</b></td>
    <td><b>83.2±0.7</b></td>
    <td><b>76.5±0.9</b></td>
    <td><b>83.9±1.0</b></td>
    <td><b>82.2±0.4</b></td>
    <td><b>81.2±1.1</b></td>
    <td><b>72.0±0.7</b></td>
    <td><b>78.9±0.8</b></td>
  </tr>
</table>

Table 1: Sentence representation performance on STS tasks (Spearman’s correlation, “all” setting). Avg.: average performance across 7 tasks. ○: results from (Gao et al., 2021), †: results from (Zhang et al., 2022a), ‡: results from (Nguyen et al., 2024). We train the models using different seeds and present the average and standard deviations of our findings. We highlight the highest numbers among models with the same pre-trained encoder.

total size \( N_t \) and the multimodal dataset (with total size \( N_m \)) are different, we employed a mixed alternating sampling training strategy. Specifically, each epoch contains the total data from both datasets. By setting the ratio \( N_t / N_m = a \), we load the data as follows: first, we load batches of pure-text data, followed by one batch of multimodal data. In each batch, the model’s loss is updated. We evaluate on the development set of STS-B every 125 steps during training and retain the best checkpoint. All experiments are performed on a NVIDIA Tesla A100 (80GB) GPU. More training details can be found in Appendix B.

4.2 Main Results

Results on STS Tasks Table 1 reports the average STS results over five runs with different random seeds. It is clear that DALR significantly outperforms the previous methods on all PLMs. For example, in the wiki+flcikr setting, compared with KDMCSE, DALR improves BERT\(_{\text{base}}\) from 78.6% to 79.5% (+0.9%) and RoBERTa\(_{\text{base}}\) from 79.1% to 79.9% (+0.8%). Compared to previous state-of-the-art methods, DALR still achieves consistent improvements, demonstrating that DALR provides stronger discriminative representations on the STS tasks. These results also dedicate the effectiveness of our approach in leveraging visual information to boost text representation learning.

Results on TR Tasks We train a logistic regression classifier under the premise of freezing the sentence embedding and evaluate its classification accuracy. As shown in Table 2, the experimental results show that our method achieves the best performance across all tasks on all PLMs, and the overall performance is better than other baselines. Specifically, compared to MCSE, our method achieves absolute improvements of 1.28% and 1.16% on the wiki+flcikr dataset. On the wiki+coco dataset, our approach increases performance from 85.46% to 86.61% with BERT and from 85.85% to 86.73% with RoBERTa. This further verifies the effectiveness of our method in the transfer tasks.

4.3 Ablation Studies

To validate the effectiveness and necessity of the proposed strategies in DALR, we conduct ablation studies using the BERT\(_{\text{base}}\) on the mixed “wiki+flcikr” dataset. As shown in Table 3, when cross-modal alignment learning (CML) is removed, the performance drops significantly across all metrics. This highlights the importance of CML, indicating that incorporating knowledge from other modalities helps in learning more comprehensive representations. A similar degradation is observed when intra-modal alignment learning (IML) is removed, which demonstrates that IML effectively captures fine-grained semantic information and fa-
<table>
  <tr>
    <th>Model</th>
    <th>MR</th>
    <th>CR</th>
    <th>SUBJ</th>
    <th>MPQA</th>
    <th>SST</th>
    <th>TREC</th>
    <th>MRPC</th>
    <th>Avg.↑</th>
  </tr>
  <tr>
    <td rowspan="2">wiki</td>
    <td>SimCSE-BERT<sup>◊</sup></td>
    <td>82.92</td>
    <td>87.23</td>
    <td>95.71</td>
    <td>88.73</td>
    <td>86.81</td>
    <td>87.01</td>
    <td>78.07</td>
    <td>86.64</td>
  </tr>
  <tr>
    <td>SimCSE-RoBERTa<sup>◊</sup></td>
    <td>83.37</td>
    <td>87.76</td>
    <td>95.05</td>
    <td>87.16</td>
    <td>89.02</td>
    <td>90.80</td>
    <td>75.13</td>
    <td>86.90</td>
  </tr>
  <tr>
    <td rowspan="3">wiki+fllickr</td>
    <td>MCSE-BERT<sup>◊</sup></td>
    <td>82.07</td>
    <td>87.28</td>
    <td>94.96</td>
    <td>89.61</td>
    <td>86.58</td>
    <td>84.04</td>
    <td>74.93</td>
    <td>85.64</td>
  </tr>
  <tr>
    <td>KDMCSE-BERT<sup>◊</sup></td>
    <td>82.78</td>
    <td>87.89</td>
    <td>95.37</td>
    <td>90.08</td>
    <td>87.61</td>
    <td>86.08</td>
    <td>75.88</td>
    <td>86.53</td>
  </tr>
  <tr>
    <td>DALR-BERT</td>
    <td><b>82.95</b></td>
    <td><b>88.10</b></td>
    <td><b>95.89</b></td>
    <td><b>90.83</b></td>
    <td><b>88.04</b></td>
    <td><b>86.60</b></td>
    <td><b>76.06</b></td>
    <td><b>86.92</b></td>
  </tr>
  <tr>
    <td rowspan="3">wiki+coco</td>
    <td>MCSE-RoBERTa<sup>◊</sup></td>
    <td>82.82</td>
    <td>88.04</td>
    <td>95.70</td>
    <td>90.13</td>
    <td>87.09</td>
    <td>84.97</td>
    <td>75.51</td>
    <td>86.29</td>
  </tr>
  <tr>
    <td>KDMCSE-RoBERTa<sup>◊</sup></td>
    <td>83.21</td>
    <td>88.16</td>
    <td>95.73</td>
    <td>90.46</td>
    <td>88.05</td>
    <td>86.30</td>
    <td>76.18</td>
    <td>86.87</td>
  </tr>
  <tr>
    <td>DALR-RoBERTa</td>
    <td><b>83.57</b></td>
    <td><b>88.69</b></td>
    <td><b>96.44</b></td>
    <td><b>91.01</b></td>
    <td><b>88.96</b></td>
    <td><b>86.80</b></td>
    <td><b>76.74</b></td>
    <td><b>87.45</b></td>
  </tr>
  <tr>
    <td rowspan="3">wiki+fllickr</td>
    <td>MCSE-BERT<sup>◊</sup></td>
    <td>81.75</td>
    <td>86.89</td>
    <td>94.73</td>
    <td>89.44</td>
    <td>86.81</td>
    <td>83.97</td>
    <td>74.66</td>
    <td>85.46</td>
  </tr>
  <tr>
    <td>KDMCSE-BERT<sup>◊</sup></td>
    <td>82.30</td>
    <td>87.71</td>
    <td>95.04</td>
    <td>89.86</td>
    <td>87.38</td>
    <td>85.68</td>
    <td>75.51</td>
    <td>86.20</td>
  </tr>
  <tr>
    <td>DALR-BERT</td>
    <td><b>82.66</b></td>
    <td><b>87.90</b></td>
    <td><b>95.85</b></td>
    <td><b>90.43</b></td>
    <td><b>87.59</b></td>
    <td><b>86.09</b></td>
    <td><b>75.74</b></td>
    <td><b>86.61</b></td>
  </tr>
  <tr>
    <td rowspan="3">wiki+coco</td>
    <td>MCSE-RoBERTa<sup>◊</sup></td>
    <td>82.24</td>
    <td>87.53</td>
    <td>95.22</td>
    <td>89.76</td>
    <td>87.08</td>
    <td>84.15</td>
    <td>74.96</td>
    <td>85.85</td>
  </tr>
  <tr>
    <td>KDMCSE-RoBERTa<sup>◊</sup></td>
    <td>82.47</td>
    <td>87.88</td>
    <td>95.24</td>
    <td>89.95</td>
    <td>87.51</td>
    <td>85.77</td>
    <td>75.82</td>
    <td>86.37</td>
  </tr>
  <tr>
    <td>DALR-RoBERTa</td>
    <td><b>82.71</b></td>
    <td><b>88.02</b></td>
    <td><b>96.10</b></td>
    <td><b>90.21</b></td>
    <td><b>87.85</b></td>
    <td><b>86.38</b></td>
    <td><b>75.84</b></td>
    <td><b>86.73</b></td>
  </tr>
</table>

Table 2: Transfer task results of different sentence representation models (measured as accuracy). Avg.: average across 7 tasks. ◊: results from (Gao et al., 2021); ◊: reproduce the models (Zhang et al., 2022a; Nguyen et al., 2024) based on publicly available code. We highlight the highest numbers among models with the same PLM.

<table>
  <tr>
    <th></th>
    <th>STS (Avg.) ↑</th>
    <th>TR (Avg.) ↑</th>
  </tr>
  <tr>
    <th>DALR</th>
    <td><b>79.49±0.7</b></td>
    <td><b>86.92±1.0</b></td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{I_{fl}} \)</td>
    <td>78.24±0.9</td>
    <td>85.95±0.4</td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{CML} \)</td>
    <td>78.16±0.8</td>
    <td>85.72±1.1</td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{consistency} \)</td>
    <td>79.13±0.7</td>
    <td>86.70±0.9</td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{CMA} \)</td>
    <td>78.61±0.3</td>
    <td>86.22±0.5</td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{IML} \)</td>
    <td>78.82±0.4</td>
    <td>86.43±0.7</td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{rank} \)</td>
    <td>79.06±0.6</td>
    <td>86.63±1.0</td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{INA} \)</td>
    <td>78.91±0.8</td>
    <td>86.50±1.1</td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{IML} & \mathcal{L}_{CML} \)</td>
    <td>77.17±0.7</td>
    <td>85.54±0.5</td>
  </tr>
</table>

Table 3: Ablation study on our train loss. We quantify the individual contributions of the components: traditional multimodal contrastive loss (\( \mathcal{L}_{I_{fl}} \)), cross-modal alignment loss (\( \mathcal{L}_{CML} \)), and intra-modal alignment loss (\( \mathcal{L}_{IML} \)) (reported avg and std over 5 runs).

cilitates the learning of more accurate and nuanced representations. Pairwise combinations of these components also yield noticeable improvements, highlighting the strength of our approach. Owing to the constraints of space, an in-depth exploration of experiments conducted on the “wiki+coco” dataset is meticulously detailed in Appendix E.1. Additionally, a comprehensive analysis of diverse teacher models is presented in Appendix E.2.

4.4 Analysis and Discussion

Components Analysis To verify the impact of cross-modal alignment (CML) in Eq.8, we integrate the CML into KDMCSE and evaluate its performance on retrieval tasks (details in Appendix E.3). As shown in Table 4, “KDMCSE + CML” outperforms “KDMCSE”, demonstrating that while static threshold filtering reduces false negatives, it fails to fully address cross-modal biases. These biases arise from modality heterogeneity, and simple similarity thresholds are insufficient for aligning global semantic features across modalities.

For deeper analysis, we test intra-modal alignment (IML) on text-based tasks such as re-ranking, retrieval, and classification using the MTEB benchmark (Muenninghoff et al., 2023). Table 5 shows that incorporating IML (“KDMCSE + IML”) significantly improves performance, underscoring the importance of addressing ISD for better sentence representations.

Visualization Analysis To deeply assess the impact of each component effect, we conduct visualize experiments using BERT_base with all components included and with specific components removed. We randomly sample 5,000 image-text pairs from the MSCOCO test set and generate their corresponding text embeddings. These embeddings

<table>
  <tr>
    <th>Model</th>
    <th colspan="2">image → text</th>
    <th colspan="2">text → image</th>
  </tr>
  <tr>
    <th></th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@1</th>
    <th>R@5</th>
  </tr>
  <tr>
    <td>MCSE†</td>
    <td>16.7</td>
    <td>43.5</td>
    <td>22.5</td>
    <td>50.4</td>
  </tr>
  <tr>
    <td>KDMCSE†</td>
    <td>17.9</td>
    <td>45.0</td>
    <td>24.1</td>
    <td>52.8</td>
  </tr>
  <tr>
    <td>w/ CML†</td>
    <td>19.1</td>
    <td>46.4</td>
    <td>25.6</td>
    <td>54.0</td>
  </tr>
  <tr>
    <td><b>DALR†</b></td>
    <td><b>19.5</b></td>
    <td><b>47.6</b></td>
    <td><b>26.7</b></td>
    <td><b>55.9</b></td>
  </tr>
  <tr>
    <td>MCSE‡</td>
    <td>8.8</td>
    <td>26.6</td>
    <td>10.9</td>
    <td>31.2</td>
  </tr>
  <tr>
    <td>KDMCSE‡</td>
    <td>9.4</td>
    <td>27.9</td>
    <td>12.2</td>
    <td>32.7</td>
  </tr>
  <tr>
    <td>w/ CML‡</td>
    <td>9.7</td>
    <td>28.6</td>
    <td>13.3</td>
    <td>33.9</td>
  </tr>
  <tr>
    <td><b>DALR‡</b></td>
    <td><b>10.2</b></td>
    <td><b>29.0</b></td>
    <td><b>13.9</b></td>
    <td><b>34.3</b></td>
  </tr>
</table>

Table 4: Multimodal retrieval results on Flickr30k test set based on BERT_base. † and ‡ denote the settings of wiki+fllickr and wiki+coco, respectively.
Figure 3: The t-SNE of sentence representations learned by DLAR and its three deviants (w/o specific component) using BERT_base. The points are embeddings of sentences sampled from the MSCOCO dataset(Xu et al., 2017). We use K-Means clustering to group similar sentence embeddings and form 50 clusters. (Best viewed in color)

are then projected into a lower-dimensional space using t-SNE (Reif et al., 2019), as shown in Figure 3. The visualization reveals that removing any component disrupts the clustering of similar sentence pairs (indicated by the same color), resulting in poor separation. Conversely, with all components jointly employed, similar samples are effectively clustered, while dissimilar samples remain well-separated. This highlights the ability of our method to improve semantic clustering and reduce representation bias.

Discussion with LLMs Sentence representation methods based on LLMs often rely on supervised signals, such as generating positive and negative samples (Wang et al., 2023; Li et al., 2024) or using instruction tuning (Cheng et al., 2023a), which may lead to unfair comparisons. For example, BGE (Xiao et al., 2024) asymmetrically adds scene descriptions to questions to improve generalization and trains with a large batch size of 19,200, significantly boosting performance. Our study focuses on enhancing sentence representations through images under an unsupervised paradigm similar to SimCSE. Unlike resource-intensive LLM-based approaches, our lightweight model is tailored for retrieval and ranking tasks, prioritizing efficiency and scalability. In many real-world applications, LLMs are impractical due to high computational costs and slower inference, making our method a more efficient and scalable alternative.

More Evaluation Metrics To validate the robustness and generalization ability of our method and scientifically include more diverse experimental evaluation metrics, we further evaluate its performance on additional downstream tasks. As shown in Table 5, our proposed method achieves superior performance compared to baseline models across multiple tasks, including reranking (Re-Rank), retrieval (Retrieval), and classification (CLF). Our

<table>
  <tr>
    <th>Model</th>
    <th>Re-Rank</th>
    <th>CLF</th>
    <th>Retrieval</th>
    <th>STS</th>
  </tr>
  <tr>
    <td>SimCSE<sup>○</sup></td>
    <td>46.47</td>
    <td>62.54</td>
    <td>20.29</td>
    <td>74.33</td>
  </tr>
  <tr>
    <td>MCSE<sup>○</sup></td>
    <td>46.92</td>
    <td>63.20</td>
    <td>21.43</td>
    <td>77.02</td>
  </tr>
  <tr>
    <td>KDMCSE<sup>○</sup></td>
    <td>47.50</td>
    <td>64.83</td>
    <td>22.06</td>
    <td>78.34</td>
  </tr>
  <tr>
    <td>w/ IML</td>
    <td>47.96</td>
    <td>65.32</td>
    <td>22.67</td>
    <td>78.81</td>
  </tr>
  <tr>
    <td>DALR (ours)</td>
    <td><b>48.35</b></td>
    <td><b>67.46</b></td>
    <td><b>23.84</b></td>
    <td><b>79.38</b></td>
  </tr>
  <tr>
    <td>△</td>
    <td>+0.85</td>
    <td>+2.63</td>
    <td>+1.78</td>
    <td>+1.04</td>
  </tr>
</table>

Table 5: Downstream tasks performance among our method and baselines on BERT_base using wiki+lickr. ○: results from (Muennighoff et al., 2023), ◊: reproduce the models based on publicly available code.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Alignment ↓</th>
    <th colspan="2">Uniformity ↓</th>
  </tr>
  <tr>
    <th>flickr</th>
    <th>coco</th>
    <th>flickr</th>
    <th>coco</th>
  </tr>
  <tr>
    <td>MCSE-BERT</td>
    <td>0.293</td>
    <td>0.267</td>
    <td>-2.491</td>
    <td>-2.350</td>
  </tr>
  <tr>
    <td>KDMCSE-BERT</td>
    <td>0.245</td>
    <td>0.261</td>
    <td>-2.387</td>
    <td>-2.383</td>
  </tr>
  <tr>
    <td>DALR-BERT</td>
    <td><b>0.178</b></td>
    <td><b>0.247</b></td>
    <td>-2.215</td>
    <td><b>-2.390</b></td>
  </tr>
  <tr>
    <td>MCSE-RoBERTa</td>
    <td>0.209</td>
    <td>0.195</td>
    <td>-1.721</td>
    <td>-1.418</td>
  </tr>
  <tr>
    <td>KDMCSE-RoBERTa</td>
    <td>0.174</td>
    <td>0.149</td>
    <td>-1.952</td>
    <td>-1.748</td>
  </tr>
  <tr>
    <td>DALR-RoBERTa</td>
    <td><b>0.153</b></td>
    <td><b>0.136</b></td>
    <td><b>-1.977</b></td>
    <td><b>-1.785</b></td>
  </tr>
</table>

Table 6: The alignment uniformity results of the models when using BERT and RoBERTa. All models are trained in the wiki-flickr setting.

comprehensive evaluations not only substantiate the effectiveness of our approach but also guarantee a diverse and exhaustive performance assessment.

Alignment and Uniformity Prior work (Wang and Isola, 2020) has demonstrated that models with better alignment and uniformity can achieve better performance (detailed in Appendix D). We calculate the alignment and uniformity loss on the STS-B development set every 125 training steps. As shown in Table 6, compared to the previous baseline methods, DALR demonstrates superior performance in both alignment and uniformity, particularly in alignment. This indicates that our alignment strategies significantly enhance the alignment of sentence embeddings, thereby improving the overall quality of the embeddings. To further ver-
ify our results, we also conduct experiments on eliminating anisotropy (detailed in Appendix F).

5 Conclusion

In this paper, we propose a dual-level alignment framework (DALR) for multimodal sentence representation learning. DALR extends traditional multimodal contrastive learning by promoting both cross-modal and intra-modal alignment for more robust sentence representations. We introduce an auxiliary task to refine negative sampling and generate similarity matrices for effective cross-modal alignment. Intra-modal alignment is achieved through a combination of ranking distillation and KL divergence-based fine-grained calibration. Extensive experiments on STS and TR benchmarks, supported by detailed analyses, show that DALR consistently outperforms previous state-of-the-art methods.

Limitations

In this paper, the limitations of our work are as follows. Firstly, there are significant differences in the word token distributions and sizes between image-text datasets like MSCOCO and Flickr30k and traditional language corpora (e.g., Wikipedia). While Wikipedia contains billions of words, MSCOCO only contains about 1 million words. Empirically, performance improves with more training data. Secondly, building sentence representation models suited for few-shot learning is a key direction for future research, especially in scenarios where collected data is scarce.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 62176187).

References

Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 252–263.

Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 81–91.

Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 497–511.

Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385–393.

Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In Second Joint Conference on Lexical and Computational Semantics (*SEM). Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 32–43.

Patrick Bordes, Eloi Zablocki, Laure Soulier, Benjamin Piwowarski, and Patrick Gallinari. 2020. Incorporating visual semantics into sentence representations within a grounded space. arXiv preprint arXiv:2002.02734.

Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14.

Qinyuan Cheng, Xiaogui Yang, Tianxiang Sun, Linyang Li, and Xipeng Qiu. 2023a. Improving contrastive learning of sentence embeddings from ai feedback. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11122–11138.

Xize Cheng, Tao Jin, Linjun Li, Wang Lin, Xinyu Duan, and Zhou Zhao. 2023b. OpenSR: Open-modality speech recognition via maintaining multi-modality alignment. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6592–6607.

Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022. DiffCSE: Difference-based contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4207–4218.
Sanghyuk Chun, Wonjae Kim, Song Park, Minsuk Chang, and Seong Joon Oh. 2022. Ecv caption: Correcting false negatives by collecting machine-and-human-verified image-caption associations for mscoco. In European Conference on Computer Vision, pages 1–19. Springer.

Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. 2021. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8415–8424.

Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).

Jinghao Deng, Fanqi Wan, Tao Yang, Xiaojun Quan, and Rui Wang. 2023. Clustering-aware negative sampling for unsupervised sentence representation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8713–8729.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186.

William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).

Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910.

Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. 2024. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26584–26595.

Hongliang He, Junlei Zhang, Zhenzhong Lan, and Yue Zhang. 2023. Instance smoothed contrastive learning for unsupervised sentence embedding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12863–12871.

Kang He, Yuzhe Ding, Bobo Li, Haining Wang, Fei Li, Chong Teng, and Donghong Ji. 2025. Harnessing dimensional contrast and information compensation for sentence embedding enhancement. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5.

Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1367–1377.

Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177.

Jian Huang, Yanli Ji, Yang Yang, and Heng Tao Shen. 2023a. Cross-modality representation interactive learning for multimodal sentiment analysis. In Proceedings of the 31st ACM International Conference on Multimedia, pages 426–434.

Jingjia Huang, Yinan Li, Jiashi Feng, Xinglong Wu, Xiaoshuai Sun, and Rongrong Ji. 2023b. Clover: Towards a unified video-language alignment and fusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14856–14866.

Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. Advances in neural information processing systems, 28.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705.

Linjie Li, Zhe Gan, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Ce Liu, and Lijuan Wang. 2023. Laverder: Unifying video-language understanding as masked language modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23119–23129.

Mingxin Li, Richong Zhang, Zhijie Nie, and Yongyi Mao. 2024. Narrowing the gap between supervised and unsupervised sentence representation learning with large language model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13590–13599.

Zejun Li, Zhihao Fan, Huaixiao Tou, Jingjing Chen, Zhongyu Wei, and Xuanjing Huang. 2022. Mvptr: Multi-level semantic alignment for vision-language pre-training via multi-stage learning. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4395–4405.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755.

Yan Ling, Jianfei Yu, and Rui Xia. 2022. Vision-language pre-training for multimodal aspect-based sentiment analysis. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2149–2159.

Jiduan Liu, Jiahao Liu, Qifan Wang, Jingang Wang, Wei Wu, Yunsen Xian, Dongyan Zhao, Kai Chen, and Rui Yan. 2023a. RankCSE: Unsupervised sentence representations learning via learning to rank. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13785–13802.

Ye Liu, Lingfeng Qiao, Changchong Lu, Di Yin, Chen Lin, Haoyuan Peng, and Bo Ren. 2023b. Osan: A one-stage alignment network to unify multimodal alignment and unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3551–3560.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaela Bernardi, and Roberto Zamparelli. 2014. A sick cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 216–223.

Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037.

Cong-Duy Nguyen, Thong Nguyen, Duc Vu, and Anh Luu. 2023. Improving multimodal sentiment analysis: Supervised angular margin-based contrastive learning for enhanced fusion representation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14714–14724.

Cong-Duy Nguyen, Thong Nguyen, Xiaobao Wu, and Anh Tuan Luu. 2024. KDMCSE: Knowledge distillation multimodal sentence embeddings with adaptive angular margin contrastive learning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 733–749.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.

Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 271–278.

Bo Pang and Lillian Lee. 2005. Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, page 115–124.

Zarana Parekh, Jason Baldridge, Daniel Cer, Austin Waters, and Yinfei Yang. 2021. Crisscrossed captions: Extended intramodal and intermodal semantic similarity judgments for ms-coco. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2855–2870.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. Advances in Neural Information Processing Systems, 32.

Yeon Seonwoo, Guoyin Wang, Changmin Seo, Sajal Choudhary, Jiwei Li, Xiang Li, Puyang Xu, Sunghyun Park, and Alice Oh. 2023. Ranking-enhanced unsupervised sentence representation learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15783–15798.

Zhan Shi, Guoyin Wang, Ke Bai, Jiwei Li, Xiang Li, Qingjun Cui, Belinda Zeng, Trishul Chilimbi, and Xiaodan Zhu. 2023. Osscse: Overcoming surface structure bias in contrastive learning for unsupervised sentence embedding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7242–7254.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642.

Kim Sung-Bin, Arda Senocak, Hyunwoo Ha, Andrew Owens, and Tae-Hyun Oh. 2023. Sound to visual scene generation by audio-to-visual latent alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6430–6440.
Zineng Tang, Jaemin Cho, Hao Tan, and Mohit Bansal. 2021. Vidlankd: Improving language understanding via video-distilled knowledge transfer. Advances in Neural Information Processing Systems, 34:24468–24481.

Zhiliang Tian, Zheng Xie, Fujiang Lin, and Yiping Song. 2023. A multi-view meta-learning approach for multi-modal response generation. In Proceedings of the ACM Web Conference 2023, pages 1938–1947.

Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 200–207.

Haining Wang, Kang He, Bobo Li, Lei Chen, Fei Li, Xu Han, Chong Teng, and Donghong Ji. 2024. Refining and synthesis: A simple yet effective data augmentation framework for cross-domain aspect-based sentiment analysis. In Findings of the Association for Computational Linguistics: ACL 2024, pages 10318–10329.

Qian Wang, Weiqi Zhang, Tianyi Lei, Yu Cao, Dezhong Peng, and Xu Wang. 2023. Clsep: Contrastive learning of sentence embedding with prompt. Knowledge-Based Systems, 266:110381.

Qifan Wang, Yi Fang, Anirudh Ravula, Fuli Feng, Xiaojuan Quan, and Dongfang Liu. 2022a. Webformer: The web-page transformer for structure information extraction. In Proceedings of the ACM Web Conference 2022, pages 3124–3133.

Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of International conference on machine learning, pages 9929–9939.

Weizhi Wang, Li Dong, Hao Cheng, Haoyu Song, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. 2022b. Visually-augmented language modeling. arXiv preprint arXiv:2205.10178.

Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39:165–210.

Qiyu Wu, Chongyang Tao, Tao Shen, Can Xu, Xiubo Geng, and Daxin Jiang. 2022a. PCL: Peer-contrastive learning with diverse augmentations for unsupervised sentence embeddings. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 12052–12066.

Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2022b. ESimCSE: Enhanced sample building method for contrastive learning of unsupervised sentence embedding. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3898–3907.

Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. 2008. Listwise approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning, pages 1192–1199.

Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muenninghoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641–649.

Jiaming Xu, Bo Xu, Peng Wang, Suncong Zheng, Guanhua Tian, and Jun Zhao. 2017. Self-taught convolutional neural networks for short text clustering. Neural Networks, 88:22–31.

Yuanneng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. ConSERT: A contrastive framework for self-supervised sentence representation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5065–5075.

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78.

Tianshu Yu, Haoyu Gao, Ting-En Lin, Min Yang, Yuchuan Wu, Wentao Ma, Chao Wang, Fei Huang, and Yongbin Li. 2023. Speech-text pre-training for spoken dialog understanding with explicit cross-modal alignment. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7900–7913.

Chunhui Zhang, Xin Sun, Yiqian Yang, Li Liu, Qiong Liu, Xi Zhou, and Yanfeng Wang. 2023. All in one: Exploring unified vision-language tracking with multi-modal alignment. In Proceedings of the 31st ACM International Conference on Multimedia, page 5552–5561.

Miaoran Zhang, Marius Mosbach, David Adelani, Michael Hedderich, and Dietrich Klakow. 2022a. MCSE: Multimodal contrastive learning of sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5959–5969.

Yanzhao Zhang, Richong Zhang, Samuel Mensah, Xudong Liu, and Yongyi Mao. 2022b. Unsupervised sentence representation via contrastive learning with mixing negatives. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11730–11738.

Li Zheng, Boyu Chen, Hao Fei, Fei Li, Shengqiong Wu, Lizi Liao, Donghong Ji, and Chong Teng. 2024. Self-adaptive fine-grained multi-modal data augmentation
for semi-supervised multi-modal coreference resolution. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 8576–8585.

Li Zheng, Hao Fei, Ting Dai, Zuquan Peng, Fei Li, Huisheng Ma, Chong Teng, and Donghong Ji. 2025. Multi-granular multimodal clue fusion for meme understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26057–26065.

Kun Zhou, Beichen Zhang, Xin Zhao, and Ji-Rong Wen. 2022. Debiasied contrastive learning of unsupervised sentence representations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6120–6130.

Minghao Zhu, Xiao Lin, Ronghao Dang, Chengju Liu, and Qijun Chen. 2023. Fine-grained spatiotemporal motion alignment for contrastive video representation learning. In Proceedings of the 31st ACM International Conference on Multimedia, page 4725–4736.

Wenjie Zhuo, Yifan Sun, Xiaohan Wang, Linchao Zhu, and Yi Yang. 2023. WhitenedCSE: Whitening-based contrastive learning of sentence embeddings. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12135–12148.

A Baselines Model

We introduce a classic sentence embedding model and two typical multimodal sentence embedding models, which we implement using official code:

• SimCSE (Gao et al., 2021): conducts thorough experiments in both unsupervised and supervised settings using different dropout to obtain positive pairs.

• MCSE (Zhang et al., 2022a): introduces visual information in sentence embedding to enhance SimCSE, and captures the consistency of sentences and their related images in the same space.

• KDMCSE (Nguyen et al., 2024): inherits the knowledge of the teacher model to learn the distinction between positive and negative samples, while also proposing an adaptive angular margin supervised contrastive learning approach to enhance discriminability by reinforcing margins in the angular space.

B Implementation

Teacher Image Model  We employ CLIP as our teacher model, which leverages contrastive learning to derive general visual and language representations from large-scale image-text pairs. The pre-trained weights are loaded from CLIP-ViT-B/32, with the patch size set to 32. After loading the model to obtain the image features, we feed them into a MLP for projection into a shared 256-dimensional space.

Teacher Text Model  We propose using a multi-teacher model weighting strategy to obtain the final teacher representations. In this work, we follow the same setup as RankCSE (Liu et al., 2023a), utilizing SimCSE (Gao et al., 2021) and DiffCSE (Chuang et al., 2022) as teacher models, and the final teacher representation is obtained through weighted aggregation. Additionally, the feature representations are projected into a shared 256-dimensional space. Moreover, other text teacher models such as RankCSE and CLIP (Radford et al., 2021) can also be substituted. A detailed comparison is provided in Appendix E.2.

Student Language Model  The implementation of the language encoder is based on the Transformers library. We start with the checkpoints of bert-base-uncased and roberta-base, fine-tuning
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">KDMCSE</th>
    <th colspan="2">DALR</th>
  </tr>
  <tr>
    <th>wiki+flickr</th>
    <th>wiki+coco</th>
    <th>wiki+flickr</th>
    <th>wiki+coco</th>
  </tr>
  <tr>
    <td>Batch size</td>
    <td>128</td>
    <td>128</td>
    <td>128</td>
    <td>128</td>
  </tr>
  <tr>
    <td>Epoch</td>
    <td>4</td>
    <td>4</td>
    <td>4</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Total time</td>
    <td>290 min</td>
    <td>440 min</td>
    <td>245 min</td>
    <td>370 min</td>
  </tr>
</table>

Table 7: Training Efficiency of KDMCSE and DALR based on BERT_{base}.

the pre-trained models using our proposed training objective in Eq.14. For evaluation, we use the 768-dimensional [CLS] token output prior to MLP pooling layer as the sentence embedding. For the MLP projection head, in the plain text setting (using the Wiki1M dataset), the sentence embeddings are projected into a 768-dimensional space. In the multimodal setting, the feature representations are projected into a shared 256-dimensional space.

More Implementation Details We preform experiments with backbones of BERT_{base} and RoBERTa_{base}. We choose [CLS] embeddings as the final representation. In the plain text setting (using Wiki1M), sentence representations are projected into a 768-dimensional space. In the multimodal setting, the student and teacher models’ feature representations are projected into a shared 256-dimensional space. We use two mixed text and multimodal training scenarios: wiki+flickr and wiki+coco. We evaluate on the development set of STS-B every 125 steps during training and retain the best checkpoint. We implement all experiments with the deep learning framework PyTorch on a NVIDIA Tesla A100 GPU (80GB memory). The temperature parameter \( \tau \) is set to 0.05, and the weight parameters \( \lambda \) and \( \mu \) are set to 0.1 and 0.2, respectively. For BERT_{base} encoder, we use a learning rate of 2e-5 and a batch size of 128 for training; for RoBERTa_{base}, the learning rate is 1e-5 and the batch size is also set to 128. The runtime for each of our experiments is approximately 4 hours, which is shorter than KDMCSE. More details are provided in Appendix C.

C Training Efficiency

We compare the training efficiency of KDMCSE and DALR using BERT_{base}, both tested on a single NVIDIA Tesla A100 GPU (with 80GB of memory). In the experiments, we set the batch size of KDMCSE and DALR to 128, and the training epochs to 4. As shown in Table 7, under the wiki+flickr and wiki+coco experimental settings, DALR completes training in 4 hours and 6.2 hours, respectively.

<table>
  <tr>
    <th></th>
    <th colspan="2">STS (Avg.) ↑</th>
    <th colspan="2">TR (Avg.) ↑</th>
  </tr>
  <tr>
    <th>DALR</th>
    <th>78.64±0.9</th>
    <th>86.61±0.7</th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{info} \)</td>
    <td>77.20±1.0</td>
    <td>85.39±0.8</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{CML} \)</td>
    <td>77.48±0.6</td>
    <td>85.53±0.8</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{consistency} \)</td>
    <td>78.19±0.4</td>
    <td>86.42±0.7</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{CMA} \)</td>
    <td>77.85±0.5</td>
    <td>85.70±0.9</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>wiki+coco</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{IML} \)</td>
    <td>77.89±0.8</td>
    <td>85.74±0.5</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{rank} \)</td>
    <td>78.31±1.1</td>
    <td>86.45±0.7</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{IMA} \)</td>
    <td>78.07±0.9</td>
    <td>86.33±1.0</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{IML} & \mathcal{L}_{CML} \)</td>
    <td>76.75±0.6</td>
    <td>85.07±1.2</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 8: Ablation study on our train loss based on wiki+coco. We quantify the individual contributions of the components: traditional multimodal contrastive loss (\( \mathcal{L}_{info} \)), cross-modal alignment loss (\( \mathcal{L}_{CML} \)), and intra-modal alignment loss (\( \mathcal{L}_{IML} \)) (reported avg and std over 5 runs).

D Alignment and Uniformity

Contrastive representation learning has two key properties: (1) alignment of positive pairs; (2) uniformity on the hypersphere. Wang and Isola (2020) argues that directly optimizing these two metrics can lead to representations with performance comparable to or better than contrastive learning in downstream tasks. Alignment measures the expected distance between normalized representations of positive pairs \( p_{pos} \):

\[
\ell_{align} \triangleq \mathbb{E}_{(x,x^+) \sim p_{pos}} \| f(x) - f(x^+) \|^2,
\]

while uniformity measures the uniform distribution of normalized representations:

\[
\ell_{uniform} \triangleq \log \mathbb{E}_{x,y \overset{i.i.d.}{\sim} p_{data}} e^{-2\|f(x)-f(y)\|^2},
\]

where \( p_{data} \) represents the distribution of sentence pairs. Smaller values for both metrics are better, which aligns closely with the objectives of contrastive learning: positive instances should be as close as possible, indicating smaller alignment, while random instances should be scattered on the hypersphere, indicating smaller uniformity.

E Analysis

E.1 More Ablation studies

Due to space constraints, we present the ablation study results on wiki+coco here. The results in Table 8 demonstrate that all three components are essential, as the absence of any of them leads to a performance drop. Notably, the cross-modal alignment module has the most significant impact on
<table>
  <tr>
    <th rowspan="2">Teacher Model</th>
    <th colspan="2">STS(Avg.)</th>
    <th colspan="2">TR(Avg.)</th>
  </tr>
  <tr>
    <th>Image</th>
    <th>Text</th>
    <th>Image</th>
    <th>Text</th>
  </tr>
  <tr>
    <td>CLIP</td>
    <td>SimCSE</td>
    <td>77.84</td>
    <td></td>
    <td>85.17</td>
  </tr>
  <tr>
    <td>CLIP</td>
    <td>DiffCSE</td>
    <td>78.65</td>
    <td></td>
    <td>86.23</td>
  </tr>
  <tr>
    <td>CLIP</td>
    <td>CLIP</td>
    <td>79.42</td>
    <td></td>
    <td>86.89</td>
  </tr>
  <tr>
    <td>CLIP</td>
    <td>SimCSE+DiffCSE</td>
    <td>79.49</td>
    <td></td>
    <td>86.92</td>
  </tr>
  <tr>
    <td>CLIP</td>
    <td>SimCSE+RankCSE</td>
    <td>79.61</td>
    <td></td>
    <td>87.05</td>
  </tr>
  <tr>
    <td>ResNet</td>
    <td>SimCSE</td>
    <td>77.59</td>
    <td></td>
    <td>85.03</td>
  </tr>
  <tr>
    <td>ResNet</td>
    <td>DiffCSE</td>
    <td>78.28</td>
    <td></td>
    <td>85.97</td>
  </tr>
  <tr>
    <td>ResNet</td>
    <td>CLIP</td>
    <td>79.04</td>
    <td></td>
    <td>86.45</td>
  </tr>
  <tr>
    <td>ResNet</td>
    <td>SimCSE+DiffCSE</td>
    <td>79.32</td>
    <td></td>
    <td>86.76</td>
  </tr>
  <tr>
    <td>ResNet</td>
    <td>SimCSE+RankCSE</td>
    <td>79.36</td>
    <td></td>
    <td>86.80</td>
  </tr>
</table>

Table 9: Comparisons of different image and text teachers based on wiki+lickr setting using BERT_{base}.

performance, as it effectively leverages image information to provide supervisory signals for text representation learning.

E.2 Teacher Model Selection

We conduct extensive experiments to explore the impact of different teacher models (image and text) on DALR’s performance. As illustrated in Figure 9, the results show that combining both cross-modal and intra-modal information can generate more discriminative sentence representations. By comparing various teacher models, we found that stronger teacher models lead to improved performance, which aligns with our expectations. ResNet is trained solely on image data and lacks multi-modal capabilities. As a result, when used as an image teacher model, the sentence representations it helps learn tend to be slightly less effective. A more powerful image teacher model can capture finer details of visual information, while a more advanced text teacher model provides more accurate ranking labels, facilitating more precise ranking knowledge transfer. We also observe an interesting phenomenon: using SimCSE and RankCSE as teacher models yielded even better results than those in our main experiments in Section 4.2. This suggests that further investigation into the selection of teacher models could provide valuable insights for future research.

E.3 Cross-modal Retrieval

To comprehensively evaluate the performance of cross-modal retrieval, we use the R@k metric as the standard for assessing cross-modal retrieval datasets. DALR is designed to learn high-quality sentence embeddings, with a primary focus on semantic similarity tasks. However, its integration of cross-modal contrastive learning and alignment modules also enhances performance in cross-modal retrieval. This outstanding performance further validates the effectiveness and robustness of our model.

F Anisotropy Study

Recent research (Ethayarajh, 2019) has highlighted the anisotropy issue in language representations, wherein learned embeddings are confined to a narrow cone in vector space, severely restricting their expressive capacity. Specifically, the anisotropy in sentence representations results in vectors being densely clustered in specific directions, diminishing their ability to effectively distinguish between different sentences.

To evaluate the impact of our method on mitigating anisotropy, we display the cosine similarity between sentence pair representations calculated on the STS-B test set, and compare them with the gold-standard annotations on STS-B. The Y-axis represents the cosine similarity of the sentence pairs, while the X-axis corresponds to the annotation scores (ranging from 0 to 5), with higher annotation scores indicating greater similarity. In other words, for sentence pairs annotated with a score of 5, the computed cosine similarity should be high. Each light-colored dot represents a sentence pair, and due to the large number of samples, overlapping dots may appear darker.

As shown in Figure 4, the results demonstrate that, for low-scoring sentence pairs, the predicted similarity by our model is significantly lower, outperforming the SimCSE, MCSE, and KDMCSE methods. This outcome also indicates that the anisotropy issue has been alleviated to some extent.

G Qualitative Analysis

We conduct small-scale retrieval experiments using KDMCSE and DLAR based on BERT_{base}. We use 30k captions from the Flickr30k (Young et al., 2014) dataset as the retrieval data and randomly select any sentence from them as a query to retrieve the Top-3 similar sentences (based on cosine similarity). As shown in Table 10, the retrieval results demonstrate that sentences retrieved by DLAR are semantically closer to the query sentences and of higher quality compared to those retrieved by DKMCSE, further demonstrating the effectiveness of DALR.
Figure 4: Scatter plot of the ground truth similarity scores (x-axis) and the cosine similarities (y-axis) between sentence pairs in the STS-B (test set). Each entry in the STS-B includes a text pair and a similarity score from 0 to 5 (gold standard).

<table>
  <tr>
    <th>Rank</th>
    <th>KDMCSE</th>
    <th>DALR (Ours)</th>
  </tr>
  <tr>
    <td colspan="3"><b>Query:</b> A group of men climb ladders outdoors.</td>
  </tr>
  <tr>
    <td>#1</td>
    <td>Two people standing on a roof while another climbs a ladder.</td>
    <td>Two people standing on a roof while another climbs a ladder.</td>
  </tr>
  <tr>
    <td>#2</td>
    <td>A firefighter climbs a ladder towards the fire above him.</td>
    <td>Two men sitting on the roof of a house while another one stands on a ladder.</td>
  </tr>
  <tr>
    <td>#3</td>
    <td>A person is climbing a wooden ladder up a rocky ledge.</td>
    <td>Three people in t-shirt, yellow helmets and harnesses begin to climb ladder.</td>
  </tr>
  <tr>
    <td colspan="3"><b>Query:</b> A man in a white cap and shirt plays the violin with other street performers.</td>
  </tr>
  <tr>
    <td>#1</td>
    <td>A man in a white shirt is playing the flute to someone in a red skirt.</td>
    <td>A man in a white shirt is playing the flute to someone in a red skirt.</td>
  </tr>
  <tr>
    <td>#2</td>
    <td>A man in a white shirt plays an electric violin.</td>
    <td>A man in a white shirt plays an electric violin.</td>
  </tr>
  <tr>
    <td>#3</td>
    <td>A man in a red shirt plays the guitar.</td>
    <td>A man with glasses wearing a tie plays the violin.</td>
  </tr>
  <tr>
    <td colspan="3"><b>Query:</b> A man in a black outfit poses in front of the eiffel tower.</td>
  </tr>
  <tr>
    <td>#1</td>
    <td>A man carrying trinkets with the Eiffel tower in the background.</td>
    <td>A man carrying trinkets with the Eiffel tower in the background.</td>
  </tr>
  <tr>
    <td>#2</td>
    <td>A man wearing black jacket poses with a smile.</td>
    <td>A man in formal wear is posing in front of a building.</td>
  </tr>
  <tr>
    <td>#3</td>
    <td>A man wearing a black long-sleeved shirt is taking a photo of a building.</td>
    <td>A man wearing a black long-sleeved shirt is taking a photo of a building.</td>
  </tr>
  <tr>
    <td colspan="3"><b>Query:</b> Two women wearing ceremonial costumes are walking outside a white building.</td>
  </tr>
  <tr>
    <td>#1</td>
    <td>Two women wearing blue jeans are walking outside.</td>
    <td>Two women wearing dresses are walking by a building.</td>
  </tr>
  <tr>
    <td>#2</td>
    <td>Two women wearing dresses are walking by a building.</td>
    <td>Two people are wearing flower costumes and walking down a street.</td>
  </tr>
  <tr>
    <td>#3</td>
    <td>Men in traditional dress stand outside.</td>
    <td>Two women wearing skirts and heels walking down a sidewalk.</td>
  </tr>
</table>

Table 10: Retrieval examples of retrieved Top-3 sentences from queries by KDMCSE and DLAR from Flickr30k dataset (30k sentences).

Metareview score: 3.5
Metareview: The paper introduces DALR, a dual-level alignment framework designed to enhance multimodal sentence representation learning. It tackles the challenges of cross-modal misalignment bias and intra-modal semantic divergence by introducing a consistency learning module to soften negative samples and employing ranking distillation for fine-grained intra-modal alignment. The work is supported by comprehensive experiments on semantic textual similarity and transfer tasks, where DALR shows consistent improvements over existing baselines.
Summary of reasons to publish: The paper addresses key issues in aligning image and text modalities that negatively affect the quality of multimodal sentence representations. It proposes a novel combination of cross-modal consistency learning with ranking-based intra-modal alignment, which is validated through thorough ablation studies and experimental evaluations. The approach is practical, with results that suggest a meaningful boost in performance on standard benchmarks, pointing to its potential impact in applications requiring robust sentence representations.