miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings
Year: 2023

Abstract:
This paper presents miCSE, a mutual information-based contrastive learning framework that significantly advances the stateof-the-art in few-shot sentence embedding. The proposed approach imposes alignment between the attention pattern of different views during contrastive learning. Learning sentence embeddings with miCSE entails enforcing the structural consistency across augmented views for every sentence, making contrastive self-supervised learning more sample efficient. As a result, the proposed approach shows strong performance in the few-shot learning domain. While it achieves superior results compared to state-of-the-art methods on multiple benchmarks in few-shot learning, it is comparable in the full-shot scenario. This study opens up avenues for efficient self-supervised learning methods that are more robust than current contrastive methods for sentence embedding.1

heading: 1 Introduction
text: Measuring sentence similarity has been challenging due to the ambiguity and variability of linguistic expressions. The community’s strong interest in the topic can be attributed to its applicability in numerous language processing applications, such as sentiment analysis, information retrieval, and semantic search (Pilehvar and Navigli, 2015; Iyyer et al., 2015). Language models perform well on these tasks but typically require fine-tuning on the downstream task and corpora (Reimers and Gurevych, 2019; Devlin et al., 2018; Pfeiffer et al., 2020; Mosbach et al., 2021). In terms of sentence embeddings, contrastive learning schemes have already been adopted successfully (van den Oord et al., 2018; Liu et al., 2021; Gao et al., 2021; Carlsson et al., 2021). The idea of contrastive learning is that positive and negative pairs are generated given
1Source code and pre-trained models are available at: https://github.com/SAP-samples/acl2023-micse/
a batch of samples. Whereas the positive pairs are obtained via augmentation, negative pairs are often created by random collation of sentences. Following the construction of pairs, contrastive learning forces the network to learn feature representations by pushing apart different samples (negative pairs) or pulling together similar ones (positive pairs). While some methods seek to optimize for selecting “hard” negative for negative pair generation (Zhou et al., 2022a), others investigated better augmentation techniques for positive pair creation. In this regard, many methods have been proposed to create augmentations to boost representation learning. Standard approaches for the augmentation aim at input data level (a.k.a discrete augmentation), which comprises word level operations such as swapping, insertion, deletion, and substitution (Xie et al., 2017; Coulombe, 2018; Wei and Zou, 2019). In contrast to that, continuous augmentation operates at the representation level, comprising approaches like interpolation or “mixup” on the embedding space (Chen et al., 2020; Cheng et al., 2020; Guo et al., 2019). Most recently, augmentation was also proposed in a more continuous fashion operating in a parameter level via simple techniques such as drop-out (Gao et al., 2021; Liu et al., 2021; Klein and Nabi, 2022) or random span masking (Liu et al., 2021). The intuition is that “drop-out” acts as minimal data augmentation, providing an expressive semantic variation. However, it will likely affect structural alignment across views. Since positive pairs are constructed from identical sentences, we hypothesize that the structural dependency over the views should be preserved by utilizing drop-out noise. Building on this idea, we maximize the structural dependence by enforcing distributional similarity over the attention values across the augmentation views. To this end, we employ maximization of the mutual information (MI) on the attention tensors of the positive pairs. However, since attention tensors can be very
6159
high-dimensional, computing MI can quickly become a significant burden if not intractable. This paper proposes a simple solution to alleviate the computational burden of MI computation, which can be deployed efficiently. Similar to (Fan et al., 2020), we adopt the Log-Normal distribution to model attention. Empirical evidence confirms this model as a good fit while facilitating the optimization objective to be defined in closed form. In this case, mutual information can be provably reformulated as a function of correlation, allowing native GPU implementation. As discussed above, the proposed approach builds upon the contrastive learning paradigm known to suffer from model collapse. This issue becomes even more problematic when enforcing MI on the attention level, as it tightens the positive pairs via regularizing the attention. Therefore the selection of negative pairs becomes more critical in our setup. To this end, we utilize momentum contrastive learning to generate harder negatives (He et al., 2020). A “tighter” binding on positive pairs and repulsion on ”harder” negative pairs empowers the proposed contrastive objective, yielding more powerful representations.
Combining ideas from momentum contrastive learning and attention regularization, we propose miCSE, a conceptually simple yet empirically powerful method for sentence embedding, with the goal of integrating semantic and structural information of a sentence in an information-theoretic and Transformer-specific manner. We conjecture the relation between attention maps and a form of syntax to be the main driver behind the success of our approach. We speculate that our proposed method injects structural information into the model as an inductive bias, facilitating representation learning with fewer samples. The adopted structural inductive biases provide a “syntactic” prior as an implicit form of supervision during training (Wilcox et al., 2020), which promotes few-shot learning capabilities in neural language models. To validate this, we introduced a low-shot setup for training sentence embeddings. In this benchmark, we finetune the language model only with a small number of training samples. Note that this is a very challenging setup. The inherent difficulty can be attributed to the need to mitigate the domain shift in the low-shot self-supervised learning scheme. We emphasize the importance of this task, as in many real-world applications, only small datasets are often available. Such cases include NLP for
low-resource languages or expert-produced texts (e.g., medical records by doctors), personalized LM for social media analysis (e.g., personalized hate speed recognition on Twitter), etc. Our proposed method significantly improves over the state-of-theart in the low-shot sentence embedding benchmark. This is the first work that explores how to combine semantic and structural information through attention regularization and empirically demonstrates this benefit for low-shot sentence embeddings.
Previous works: Recently, VaSCL (Zhang et al., 2022a), ConSERT (Yan et al., 2021a), PCL (Wu et al., 2022a) and (Chuang et al., 2022) proposed contrastive representation learning with diverse augmentation strategies on positive pair. However, we proposed a principled approach for enforcing alignment in positive pairs at contrastive learning without discretely augmenting the data. Similar to us, ESimCSE (Wu et al., 2021) and MoCoSE (Cao et al., 2022a) proposed to exploit a momentum contrastive learning model with negative sample queue for sentence embedding to boost uniformity of the representations. However, unlike us, they do not enforce any further tightening objective on the positive pairs nor consider few-shot learning. Very recently, authors in InforMin-CL (Chen et al., 2022) and InfoCSE (Wu et al., 2022b)proposed information minimization-based contrastive learning. Specifically, the authors propose to minimize the information entropy between positive embeddings generated by drop-out augmentation. Our model differs from this paper and the method in (Bachman et al., 2019; Yang et al., 2021; Zhang et al., 2020; Sordoni et al., 2021; Wu et al., 2020), which focuses on using mutual information for selfsupervised learning. A key difference compared to these methods is that they estimate MI directly on the representation space. In contrast, our method computes the MI on attention. Other related work include (Zhang et al., 2022b; Zhou et al., 2022b; Zhang et al., 2022c; Liu et al., 2022).
The contributions of the proposed work are: First, we propose to inject structural information into language models by adding an attention-level objective. Second, we introduce Attention Mutual Information (AMI), a sample-efficient selfsupervised contrastive learning. Third, we introduce low-shot learning for sentence embedding. We show that our method performs comparably to the state-of-the-art in the full-shot scenario and significantly better in few-shot learning.

heading: 2 Method
text: The proposed approach aims to exploit the structure of the sentences in a contrastive learning scheme. Compared to conventional contrastive learning that solely operates at the level of semantic similarity in the embedding space, the proposed approach injects structural information into the model. This is achieved by regularizing the attention space of the model during training. We let D denote a dataset consisting of string sequences (sentences) from corpus X with D = {x1,x2, ...,x|X |}, where we assume xi to be a tokenized sequence of length n with xi ∈ Nn. For mapping the input data to the embedding space, we use a bi-encoder fθ parametrized by θ. Bi-encoders entail the computation of embeddings for similarity comparison, whereby each sentence in a pair is encoded separately. Hence, the instantiation of a bi-encoder on augmented input data induces multiple views. For the following, we let v ∈ {1, 2} denote the index of the view, where each view corresponds to a different augmentation. Consequently, encoding a data batch Db yields embedding matrices Ev ∈ R|Db|×U , where U denotes the dimensionality of the embeddings. Employing a Transformer, encoding the input data yields the embedding matrices and the associated attention tensors Wv. Then learning representation of the proposed approach entails the optimization of a joint loss:
min θ LC(E1,E2) + LD(W1,W2) (1)
with (E1,W1), (E2,W2) = fθ(Db). Here, LC is responsible for the semantic alignment, corresponding to the standard InfoNCE (van den Oord et al., 2018) loss that seeks to pull positive pairs close together while pushing away negative pairs in the embedding space. In contrast, LD is responsible for the syntactic alignment, operating on the attention space. However, in comparison to LD is employed only on positive pairs’ attention tensors.

heading: 2.1 Embedding-level Momentum-Contrastive Learning (InfoNCE)
text: The InfoNCE-loss seeks to pull positive pairs together in the embedding space while pushing negative pairs apart. Specifically, InfoNCE on embeddings pushes for the similarity of each sample and its corresponding augmented embedding. Negatives pairs are constructed in two ways, reflected by the two terms in the denominator of Eq. 2. First, inbatch negative pairs are constructed by pairing each
sentence with another random sentence (sharing no semantic similarity), pushing for dissimilarity. Second, using embeddings obtained from a momentum encoder known as MoCo (He et al., 2020; Cao et al., 2022a). The momentum encoder is a replication of the encoder fθ, whose parameters are updated more slowly. Specifically, while the parameters of fθ encoder are updated via back-propagation, the parameters of the momentum encoder are updated using an exponential moving average from the former. The negative embeddings are produced from samples from previous batches, which are stored in queue Q and are forward-passed through the momentum encoder. Then the InfoNCE (van den Oord et al., 2018) loss (LC) is defined as:
− |Db|∑
i
log d(ei, +ei) |Db|∑ j:i ̸=j d(ei, ej) + |Q|∑ j d(ei, qj) , (2)
where ei ∈ E1 and +ei ∈ E2 denote the embeddings of different augmentations of xi. Furthermore, d(x,y) = exp(sim(x,y)/τ) with sim(.) the cosine similarity metric, qj denoting representations obtained from momentum encoder, and τ ∈ R is a temperature scalar.

heading: 2.2 Attention-level Mutual Information (AMI)
text: Preliminaries and notations: We first briefly review the attention mechanism and explain the notation used in the rest of this section. A Transformer stack consists of a stack of L layers, with input data cascading up the layer stack. Each layer comprises a self-attention module and a feed-forward network in its simplest form. Passing sentences through the encoder stack entails simultaneous computation of attention weights. These attention weights indicate the relative importance of every token. To this end, key-value pairs are computed for each token of the input sequence within each self-attention module. This entails the computation of three different matrices: key matrix K, value matrix V , and query matrix Q. The values of the attention weights W are obtained according to W = softmax(f(Q,K)) ∈ Rn×n, where f(.) is a scaled dot-product. Output features are then generated as obtained according to WV . To attend to different sub-spaces (Vaswani et al., 2017) simultaneously, the attention mechanism is replicated H times, referred to as multi-head attention. During training the encoder, the selfattention tensors W values are subject to a random
Mul$-Dropout
deterministic process, with randomness arising due to drop-out. Hence, the proposed approach seeks to optimize structural alignment by maximizing mutual information between the attention tensors Wv = [w1, ...,w|Db|] of the augmentation views. We propose a four-step pipeline to regularize the joint attention space. For a schematic illustration of the AMI pipeline, see Fig. 1.
1) Attention Tensor Slicing: Given that augmentation has different effects on the attention distribution depending on the depth (layer) and the position (head) in the Transformer stack, we propose to slice the attention tensor. Chunking the attention has multiple advantages. On the one hand, this allows for preserving the locality of distribution change. This is important as it can be empirically observed that distribution divergence between views decreases with increasing depth in the encoding stack. On the other hand, restricting the space permits using a simple distributional model such as bivariate distribution compared to a mixture distribution for the whole stack.
For the sake of economy in notation and avoid notational clutter, we will restrict the attention tensor of a single encoded sample in the following. To this end, a slicing function π : RL×H×n×n → RR×n×n cuts the attention tensor for each input sample into R (indexed) elements: π(wi) = [w1i , ...,w R i ] ∈ Rn×n with wri = (wj,k)1≤j,k≤n and r ∈ R. For a schematic illustration of how the attention tensor is sliced into tiles, see Fig. 2.
2) Attention Sampling: Different sentences in the batch are typically in token sequences of different lengths. To accommodate the different lengths and facilitate efficient training, sequences are typically padded with [PAD]-token for length equality. Although this allows for efficient batch encoding on GPU, attentions arising from [PAD]-tokens have to be discarded when looking at statistical relationships. To accommodate for the different lengths of tokenized sequences, perform a sampling step for attention values within each grid cell wri . To this end, we leverage multinomial distribution Pmult(p1, .., pn2), where s correspond to the
number of non-padding tokens with 1 ≤ s ≤ n. Specifically, we sample from the s2 attention values pool, each with a probability of 1
s2 , with the
remaining elements associated with probability 0. As a result, we obtain a set Jr = {j1, ..., jm} consisting of m indices of the attention tensors for each slice r ∈ R:
Jr ∼ Pmult( 1/s2, ..., 1/s2︸ ︷︷ ︸ 1, .., s2 ,
(n− s)2, ...,n2︷ ︸︸ ︷ 0, ..., 0 ) (3)
It should be noted that for the same slice r across the views, the same index set is used for sampling: w̃r =
⋃ j∈Jr w r[j] and +w̃r = ⋃ j∈Jr +w r [j].

heading: 3) Attention Mutual Information Estimation:
text: We propose using mutual information to measure the similarity of attention patterns for different views. Specifically, we follow (Fan et al., 2020) and adopt the Log-Normal distribution for modeling the attention distribution, which is prudent for several reasons. First, Empirical observation confirms attention asymmetry. Second, by utilizing a non-symmetric distribution, it becomes possible to break down the attention tensor W into K and Q, thereby allowing for non-symmetrical attention. Third, adopting the log-normal models facilitates the optimization objective to be defined in closed
form and hence easy to optimize, particularly on GPUs. Mutual information for two normally distributed tuple vectors (z1, z2) can be written as a function of correlation (I.M. and A.M., 1957):
I(z1, z2) = − 1
2 log(1− ρ2) (4)
where ρ corresponds to the correlation coefficient computed from from z1 and z2. Hence, we compute the mutual information for each slice r and sample xi as MIri = I(log(w̃ r i ), log( +w̃ r i )). The log(.) function accommodates the Log-Normal to Normal random variable transformation. For details on the implementation, see Alg. 1.
4) Mutual Information Aggregation: To compute the loss component for attention regularization, we need to aggregate the distributional similarities for the entire tensor. Aggregation is obtained by averaging the individual similarities obtained for each slice r ∈ R and each sample xi in the batch. With λ ∈ R some weighting scalar, the attention alignment loss term is:
LD(W1,W2) = − λ
|R| · |Db|
Db∑
i
R∑
r
MIri (5)

heading: 3 Experiments
text: In this section, we describe the experimental setting used for the evaluation, present our main results, and discuss different aspects of our method by providing several empirical analyses.

heading: 3.1 Experimental Setup
text: Model and Hyperparameters: Training is started from a pre-trained transformer LM. Specifically, we employ the Hugging Face (Wolf et al., 2020) implementation of BERTbase. For each approach evaluated, we follow the same hyperparameters proposed by the authors. In the InfoNCE loss, we set τ = 0.05. In order to determine the hyperparameter λ a coarse grid search {1.0, 0.1, ..., 1.0e−5} was conducted to assess the magnitude. Upon determination, a fine grid search was conducted once with 10 steps. We set λ = 2.5e − 3 for training 100% of the data in a single episode with a batch size of 50 at a learning rate of 3.0e−5 and 250 warm-up steps. The number of optimization steps is kept constant for training the different dataset sizes. For the training set of size 106(= 100%), we train for 1 epoch; for the size of 105(= 10%),
Semantic Textual Similarity (STS) Benchmark
Model STS12 STS13 STS14 STS15 STS16 STS-B SICK-R Avg.
BERT 21.54 32.11 21.28 37.89 44.24 20.29 42.42 31.40 BERT♢(first-last avg) 39.70 59.38 49.67 66.03 66.19 53.87 62.06 56.70 GloVe♣(avg.) 55.14 70.66 59.73 68.25 63.66 58.02 53.76 61.32 BERT-flow♢ 58.40 67.10 60.85 75.16 71.22 68.66 64.47 66.55 BERT-whitening♢ 57.83 66.90 60.90 75.08 71.31 68.24 63.73 66.28 IS (Zhang et al., 2020) 56.77 69.24 61.21 75.23 70.16 69.21 64.25 66.58 SG-OPT (Kim et al., 2021) 66.84 80.13 71.23 81.56 77.17 77.23 68.16 74.62 CT (Carlsson et al., 2021) 67.43 79.18 69.05 76.92 74.62 73.24 68.38 72.69 SCD† (Klein and Nabi, 2022) 66.94 78.03 69.89 78.73 76.23 76.30 73.18 74.19 Mirror-BERT† (Liu et al., 2021) 69.10 81.10 73.00 81.90 75.70 78.00 69.10 75.40 SimCSE (Gao et al., 2021) 68.69 82.05 72.91 81.15 79.39 77.93 70.93 76.15 MoCoSE†(Cao et al., 2022b) 71.58 81.40 74.47 83.45 78.99 78.68 72.44 77.27 InforMin-CL† (Chen et al., 2022) 70.22 83.48 75.51 81.72 79.88 79.27 71.03 77.30 MixCSE† (Zhang et al., 2022b) 71.71 83.14 75.49 83.64 79.00 78.48 72.19 77.66 ConSERT†,∗large (Yan et al., 2021b) 70.69 82.96 74.13 82.78 76.66 77.53 70.37 76.45 VaSCL†,∗ (Wang et al., 2022) 69.08 81.95 74.64 82.64 80.57 80.23 71.23 77.19 DCLR†,∗ (Zhou et al., 2022a) 70.81 83.73 75.11 82.56 78.44 78.31 71.59 77.22 ArcCSE†,∗ (Zhang et al., 2022c) 72.08 84.27 76.25 82.32 79.54 79.92 72.39 78.11 PCL†,∗ (Wu et al., 2022a) 72.74 83.36 76.05 83.07 79.26 79.72 72.75 78.14 ESimCSE†,∗ (Wu et al., 2021) 73.40 83.27 77.25 82.66 78.81 80.17 72.30 78.27 DiffCSE†,∗ (Chuang et al., 2022) 72.28 84.43 76.47 83.90 80.54 80.59 71.29 78.49
we train for 10 epochs, etc. The training was conducted using an NVIDIA V100 with a training time of around 1.5h. The overall GPU budget from experimentation and hyperparameter optimization is estimated to be around 500 GPU/hours. The momentum encoder is associated with a sample queue of size |Q| = 384. The momentum encoder parameters are updated with a factor of 0.995, except for the MLP pooling layer, which is kept identical to the online network. Additionally, we increase the drop-out for the momentum encoder network from the default rate (0.1) to 0.3. Data and Evaluation: Following (Gao et al., 2021), we train the model unsupervised on sentences from Wikipedia. We create random sample sets of different sizes {106, 105, 104, 5.0·103, 103} to train the model in a few-shot learning scenario. We repeated the training set creation for each size 5 times with different random seeds. Mutual Information Estimation: Following the observations in (Voita et al., 2019), we restrict the computation of the mutual information to the upper part of the layer stack. Specifically, we select the layers between 8 and 12 (= last layer in BERTbase). To accommodate input sequences of varying lengths and make computation more efficient, we pool together pairs of adjacent heads (without overlap) while preserving the layer separation. From each of the (4× H2 ) chunks of pooled attentions, we random sample 150 joint-attention pairs for each embedding of the bi-encoder.

heading: 3.2 Experimental Results
text: Unsupervised Sentence Embedding: We compare miCSE to previous state-of-the-art sentence embedding methods on STS tasks. For comparisons, we favored comparable architectures (bi-encoder) that facilitate seamless integration of the proposed approach and methods of comparable backbone. We also added methods that employ explicit discrete augmentation to provide a full picture of existing techniques for sentence embedding. For semantic text similarity, we evaluated on 7 STS tasks: (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017) and SICK-Relatedness (Marelli et al., 2014). These datasets come in sentence pairs with correlation labels in the range of 0 and 5, indicating the se-
Algorithm 1 Mutual Information estimation Input: Batch Db, encoder fθ, multinomial sampler pmult Output: Average mutual information 1|R|·|Db| ∑Db,R i,r MI r i
(E1,W1), (E2,W2)← fθ(Db) ▷ Transformer encoding creating views for i← 1...|Db| do
wi, +wi ← EXTRACT(W1,W2, i) ▷ Extract attention tensor for each sample {(+)w1i , ..., (+)wRi } ← π((+)wi) ▷ Slicing the attention tensors s← number of text tokens in xi for r ← 1...|R| do
Jr ← pmult(1/s2, ...1/s2, 0) ▷ Sampling indices of valid attentions MIri ← AMI( ⋃ j∈Jr w r i [j], ⋃ j∈Jr
+wri [j]) end for
end for procedure AMI(w1,w2)
z1, z2 ← log(w1), log(w2) ▷ Log-Normal to Normal transform ρ← cos(z1 − z̄1, z2 − z̄2) ▷ Compute correlation coefficient on centered attentions Return −12(1− ρ2) ▷ Mutual information for tensor slice end procedure
mantic relatedness of the pairs. Specifically, we employ the SentEval toolkit (Conneau and Kiela, 2018) for evaluation. All our STS experiments are conducted in a fully unsupervised setup, not involving any STS training data. The benchmark measures the relatedness of two sentences based on the cosine similarity of their embeddings. The evaluation criterion is Spearman’s rank correlation (ρ). For comparability, we follow the evaluation protocol of (Gao et al., 2021), employing Spearman’s rank correlation and aggregation on all the topic subsets. Results for the sentence similarity experiment are presented in Tab. 1. As can be seen, the proposed approach is slightly lower in terms of average performance than state-of-the-art algorithms such as DiffCSE. However, it should be noted that these aforementioned methods use extensive discrete augmentation techniques, such as word repetition, deletion, and others, while the proposed method in this work does not employ any form of discrete data augmentation. This renders the proposed method more general and less ad-hoc in nature. While it is technically feasible for our method to incorporate discrete augmentation, it was deliberately excluded in this study for the sake of generalization with the intention of further exploration in future research. A more in-depth analysis shows the best performance on the SICKR benchmark, where it outperforms the second-best approach SCD by (+0.44) and third-best PCL by (+0.87). We highlight the comparison to the closest method SimCSE, where the proposed approach has an average gain of (+3.94). This improvement is due to the two additional components (i.e., AMI and MoCo) we add to this baseline method. Low-shot Sentence Embedding: In this experiment, the performance of several SOTA sentence embedding approaches is benchmarked elaboratively. Similar to Sec. 3.2, we evaluate 7 STS tasks, STS Benchmark, and SICK-Relatedness with Spearman’s ρ rank correlation as the evaluation metric. However, in contrast to the previous section, models are trained on different subsets of the data, namely {100%, 10%, 1%, 0.1%} of the Wikipedia dataset used in (Gao et al., 2021). Results for the low-shot sentence similarity experiment can be presented in Fig. 3. As can be seen, the proposed approach gains by increasing the training set size and consistently outperforms all the baselines in all training subsets. Interestingly, our proposed method reaches the performance of SimCSE trained on the entire dataset with only 0.5% of the data. We believe it shows the impact of exploiting structural information for data augmentation during training. It should be noted that the performance gain is most significant when conducted on a single token rather than token averaging. We attribute this to token averaging, which to a certain degree,
0.01% 0.1% 1.0% 10.0% 100.0%
Trainingset Size
50
55
60
65
70
75
S p
ea rm
an ra
nk co
rr el
at io
n (ρ
)
miCSE SimCSE
(A) Few-shot Performance
0.1% 0.5% 1.0% 10.0% 100.0%
Trainset Size
0
1
2
3
4
5
6 7 ∆ S p ea rm an ’s ra nk co rr el at io n
SimCSE
+AMI +MoCo
(B) Component Analysis
FIGURE 4. Few-shot performance analysis of models trained with different ratios of dataset size. Performance is shown in Spearman’s correlation average of the STS benchmark. Left: Few-shot performance of SimCSE (Gao et al., 2021) ( ) and the proposed approach miCSE ( ). Right: Few-shot ablation study with y-axis showing change (∆) in Spearman’s rank correlation ρ, showing the effect of adding components w.r.t. the SimCSE baseline.
is equivalent to attention regularization. On the extremely low data regime, the proposed approach shows very strong performance up (+11) compared to SimCSE - see Fig. 4a. It suggests resilience of our method to very small batch training.

heading: 3.3 Experimental Analysis of components
text: Given that AMI is a regularizer on Transformer attention, we evaluate the applicability in conjunction with other contrastive learning methods. We evaluate the following approaches CT (Carlsson et al., 2021), Mirror-BERT (Liu et al., 2021), and SimCSE (Gao et al., 2021). Evaluation is conducted on 7 STS tasks, STS Benchmark, and SICKRelatedness with Spearman’s ρ rank correlation as a metric. Results for the low-shot sentence similarity experiment are presented in Tab. 2. As can be seen, our proposed AMI can boost the performances of all approaches in all settings. Additionally, it shows the most significant boost in performance in combination with SimCSE. In addition, we observe that the impact of AMI grows with declining training set size. Combined with SimCSE, AMI leads to a performance gain of up to (+5.91) at 0.1% of the data. We also observe that adding AMI to all the approaches significantly reduces the variance for all methods. This can probably be attributed to the regularization effect of the proposed AMI component. In addition, we conducted an ablation study to assess the effect of AMI and MoCo w.r.t. the baseline SimCSE - see Tab. 3. As shown in Fig. 4b, AMI and MoCo improve the baseline at different data ratios. Again, AMI provides a particularly strong performance boost in the low-data regime. In contrast, the impact of MoCo diminishes with decreasing training set size.
We emphasize that our approach gets the best of both worlds by integrating these two components. This can be directly exploited for different few-shot setups by adjusting the hyper-parameter λ.
Discussion on the Structure and Attention: The proposed approach aligns the attention patterns for drop-out augmented input pairs. We posit that conducting such a regularization enforces constraints w.r.t. the structure (e.g., syntax) of the sentence embeddings. This is motivated by recent literature findings, which suggest that the Transformer's attention captures structural information such as syntactic grammatical relationships of the sentences (Ravishankar et al., 2021; Clark et al., 2019; Raganato et al., 2018; Voita et al., 2019). Additionally, recent research explicitly targets the extraction of topologies from attention maps for diverse tasks on syntactic and grammatical structure (Kushnareva et al., 2021; Cherniavskii et al., 2022; Perez and Reinauer, 2022). Although no “one-to-one” mapping connects syntactic structures and attention patterns, the attention tensor, at the bare minimum, encodes a “holistic notion” of the syntactic structure of sentences. While this study refrains from making any definitive claim on the matter, a preliminary analysis wrt. role of syntax in our proposed method is conducted (see Appendix).
Discussion on the discrete argumentation: Discrete augmentation serves as a suitable strategy for expanding datasets to enhance learning robustness and partially address the issue of data scarcity. Although augmentation contributes to improved robustness, additional measures are required to tackle the information gap challenge in few-shot learning scenarios. Therefore, our current study deliberately excluded discrete augmentation to minimize any interference it may have with our low-shot learning algorithm. The primary rationale behind this decision is that while discrete augmentation is known to alleviate data scarcity by replicating
missing information, it often leads to a superficial correlation between test and training data, rather than enhancing the model’s few-shot learning capability. Consequently, we excluded augmentation to maintain control over miCSE’s behavior and validate its effectiveness without any negative consequences. The significant superiority of miCSE over augmentation-based approaches (such as DiffCSE) in the low-shot setup is evident from Fig. 3. Nevertheless, the proposed approach inherently facilitates the integration of discrete augmentation, offering the potential to enhance results in both few and full-shot learning scenarios. However, it is crucial to acknowledge that their structural similarities must be respected when applying augmentation strategies to positive pairs. One promising option is to utilize the augmentation strategies proposed by ESimCSE (Wu et al., 2021), which involve word duplication and deletion to address length biases. This can be followed by enforcing AMI on the shared attention subspaces of the augmented instances. Although we do not explore this approach in our current paper, it presents an intriguing avenue for future research.

heading: 4 Conclusion
text: We proposed a method to inject structural similarity into language models for self-supervised representation learning for sentence embeddings. The proposed approach integrates the inductive bias at the level of Transformer attention by enforcing mutual information on positive pairs obtained by drop-out augmentation. Leveraging attention regularization makes the proposed approach much more sample efficient. Consequently, it outperforms methods with a significant margin in low-shot learning scenarios while having state-of-the-art performance in full-shot to comparable approaches.

heading: 5 Limitations
text: The proposed AMI component is effective in the low-data regime but cannot be generalized to all cases. Future work will investigate the role of syntax in the structural regularization of attention and the extension of the proposed approach to discrete augmentation.

heading: A Appendix
text: In the following sections, we add additional details omitted in the main paper due to space restrictions. First, we show an analysis of the relationship between syntactic structure and semantics. Next, we illustrate the cosine similarity distribution according to human judgment (ground truth) in Sec. C. Next, in Sec. D, we visualize the 2D histogram of joint distributions between views. In Sec. E, we present detailed results of the few-shot performance of miCSE in contrastive and non-contrastive setup. Finally, the exact relation between mutual information and correlation is presented in Sec. F.

heading: B Analysis on Structure vs. Semantic
text: In light of the lack of a rigorous benchmark for analyzing structure(syntax) in sentence embedding, we performed two qualitative analyses visualized in Fig. 5 and Fig 6.
Let us consider the following three sentences and their linearized syntax tree to understand better the notions of negatives and (dis-)similar syntax.
Anchor / Positive: Life is good Negative (similar Syntax): Good is expensive Negative (dissimilar Syntax): Live a good life
For each sentence, we computed the dependency tree. Subsequently, we linearize the tree structure for comparison, as can be done with tools such as spaCy2. Positive samples have an identical tree and negative samples have non-identical trees with their part-of-speech tags:
Anchor / Positive: nsubj(1,0) – ROOT(1,1) – acomp(1,2) – punct(1,3). Negative (similar Syntax):
2https://spacy.io/
nsubj(1,0) – ROOT(1,1) – acomp(1,2) – punct(1,3). Negative (dissimilar Syntax): ROOT(0,0) – det(3,1) – amod(3,2) – npadvmod(0,3) – punct(0,4).
Here nsubj corresponds to "nominal subject," acomp to "adjectival complement," det to "determiner," npadv to "noun phrase as adverbial modifier" and punct to "punctuation."
Our empirical observations are:
Observation (i) There is a higher semantic and syntactic similarity between positive pairs compared to the negative pairs: Our contrastive learning approach assumes that positive pairs exhibit more syntactic similarity than negative pairs (i.e., syntactic inductive bias). To validate this hypothesis, we plot the semantic similarity against syntactic similarity for both positive and negative pairs. Specifically, we analyzed the embeddings and attention values of the trained model with SimCSE and the proposed approach. Input to the models was randomly sampled sentences from Wikipedia. Interestingly enough, although training the proposed model involves maximization of MI over the attention w.r.t. positive pairs, we also observe the reflection of syntactic information in the negative pairs. As shown in Fig. 5, the negative pairs end up in the low left corner, whereas the positive pairs are in the upper right corner.
Observation (ii): Negative pairs with similar syntax show higher attention similarity, compared to pairs with dissimilar syntax: For a more in-depth analysis of this, we further sub-divided the negative pairs into two groups: a) negative pairs with similar dependency trees, b) negative pairs with dissimilar dependency trees. For simplicity, we adopted a binary similarity scheme - “similar” implies an identical dependency tree, whereas “dissimilar” corresponds to a non-identical dependency tree. To highlight the inter-group syntax similarity, samples of each group were normalized w.r.t. the centroid of the opposite group. As shown in Fig 6 (by the increased distance between the cluster centers), the proposed approach encodes a notion of syntactic similarity. Note that this margin appeared solely due to enforcing the AMI on attention for the positive pairs, leading to a notion of “syntax” on negative pairs.

heading: C Cosine-similarity Distribution
text: To directly show the strengths of our approaches on STS tasks, we illustrate the cosine similarity on embeddings distributions of STS-B pairs in combination with human ratings in Fig. 7. The STS dataset comes in sentence pairs with correlation labels in the range of 0 and 5, indicating the semantic relatedness of the pairs. Here, the x-axis is the sample similarity of sentences according to human judgment (ground truth), and the y-axis represents the cosine similarity between pairs using embeddings. Color coding corresponds to ground-truth similarity. Compared to the baseline model (SimCSE), miCSE better distinguishes sentence pairs with different levels of similarities, as can be seen from the stronger correlation between embedding distance and human rating. This property leads to better performance on STS tasks. In addition, we observe that miCSE generally shows a more scattered distribution while preserving a lower variance on semantically similar sentence pairs. This observation further validates that miCSE can potentially achieve a better alignment-uniformity balance.
D Visualization of Joint Distribution
To analyze the impact of the proposed approach compared to the baseline SimCSE at the attention level, we visualized the joint distribution of the attention values created by the two views created by the bi-encoder. The joint distribution and mutual information are closely related. More specifically, given two random variables X and Y , the associated mutual information can be expressed in terms of the joint distribution as:
I(X,Y ) = ∑
x,y
p(x, y) log p(x, y)
p(x)p(y) , (6)
where p(x, y) denotes the joint-distribution and p(x), p(y) the marginals. Assuming random variables are normally distributed, the joint distribution of random variables is distinctly shaped depending on the correlation coefficient ρ. See Sec. F details on the relationship between entropy and the correlation coefficient. In the extreme case of totally unrelated marginals ρ = 0, the joint distribution assumes a circular shape having the lowest possible mutual information. On the other end of the
spectrum, in the case of perfect correlation, the joint distribution assumes collinearity (45◦ diagonal), with mutual information assuming maximal value. We sliced the attention tensor into 12 slices to avoid visual clutter, pooling together every 3 adjacent heads and every 4 adjacent layers. Slicing the tensor at a higher resolution leads to visually very similar results. The axes of the joint distribution (2d histogram) correspond to the marginals’ distribution. As miCSE maximizes the mutual information, one can observe a reduction in the scatter of the joint distribution compared to SimCSE.

heading: E Detailed Comparison with SimCSE
text: Our proposed method is built on top of contrastive learning. Thus it intrinsically relies on the existence of the negative pairs. To complement the performance comparison of contrastive learning in Fig. 4a, we designed an experiment to analyze the extent to which attention regularization alone (AMI) can compensate for the lack of negative pairs. To that end, we conducted training with positive pairs only. See Tab. 4 and Fig. 9 for results. The integration of mutual attention information boosts the performance by up to (+15) across all training set sizes. It suggests the potential application of our proposed attention regularization for non-contrastive learning.

heading: F Bivariate Normal Mutual Information
text: General Log-Normal Properties: Similar to the normal distribution, the log-normal distribution logN (w|µw,σ2w) has two parameters µw and σw capturing mean and variance. It follows that applying the log transformation on a random variable w, we yield random variable z = log(w), which is
normally distributed: z ∼ N (µz,σ2z). Mutual Information: Given a vectors of tuples (X1,X2) containing i.i.d. points sampled the joint bivariate normal distribution of p(A,B) = N (µ, Σ) with µ ∈ R2, Σ ∈ R2×2. It can be shown that there exists an exact relationship between mutual information and the correlation coefficient ρ (I.M. and A.M., 1957) derived from X1 and X2. To that end, we expand the notation:
µ = ( µ1 µ2 ) , Σ =
( σ21 ρσ1σ2
ρσ1σ2 σ 2 2
) (7)
The marginal and the joint entropy terms for Gaussian distributed variables can be written as:
H(Xi) = 1
2 log(2πeσ2i ) =
1 2 + 1 2 log(2π) + log(σi), i ∈ {1, 2}
(8)
H(X1,X2) = 1
2 log
[ (2πe)2|Σ| ] =
1 + log(2π) + log(σ1σ2) + 1
2 (1− ρ2).
(9)
Given that Mutual Information can be written in terms of entropy as:
I(X1,X2) = H(X1) +H(X2)−H(X1,X2) (10)
Then it follows by inserting Eq. 8,9 in Eq. 10:
I(X1,X2) = − 1
2 (1− ρ2) (11)
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 5
A2. Did you discuss any potential risks of your work? Not applicable. No potential risk
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
3 A4. Have you used AI writing assistants when working on this paper? Grammarly for grammar correction and spelling correction
B 3 Did you use or create scientific artifacts? Section 3.1, 3.2
3 B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. All open-source
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3.2
C 3 Did you run computational experiments? Section 3.1 + Section 3.2
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 3.1 + Section 3.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3.1 + Section 3.2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3.1 + Section 3.2
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Future work will investigate the role of syntax in the structural regularization of attention and the extension of the proposed approach to discrete augmentation