Towards Better Understanding of Contrastive Sentence Representation Learning: A Unified Paradigm for Gradient (2024)

Abstract: Sentence Representation Learning (SRL) is a crucial task in Natural Language Processing (NLP), where contrastive Self-Supervised Learning (SSL) is currently a mainstream approach. However, the reasons behind its remarkable effectiveness remain unclear. Specifically, many studies have investigated the similarities between contrastive and non-contrastive SSL from a theoretical perspective. Such similarities can be verified in classification tasks, where the two approaches achieve comparable performance. But in ranking tasks (i.e., Semantic Textual Similarity (STS) in SRL), contrastive SSL significantly outperforms non-contrastive SSL. Therefore, two questions arise: First, what commonalities enable various contrastive losses to achieve superior performance in STS? Second, how can we make non-contrastive SSL also effective in STS? To address these questions, we start from the perspective of gradients and discover that four effective contrastive losses can be integrated into a unified paradigm, which depends on three components: the Gradient Dissipation, the Weight, and the Ratio. Then, we conduct an in-depth analysis of the roles these components play in optimization and experimentally demonstrate their significance for model performance. Finally, by adjusting these components, we enable non-contrastive SSL to achieve outstanding performance in STS. 1

heading: 1 Introduction
text: Sentence Representation Learning (SRL) explores how to transform sentences into vectors (or “embeddings”), which contain rich semantic information and are crucial to many downstream tasks in Natural Language Processing (NLP). In the era of Large Language Models (LLMs), SRL also
∗Corresponding author 1Our code is available at https://github.com/
BDBC-KG-NLP/UnderstandingCSE.
plays an important role in providing the embedding models for Retrieval-Augmented Generation (RAG, Gao et al., 2023). The quality of sentence embeddings is usually measured through Transfer tasks (TR) and Semantic Textual Similarity tasks (STS) (Conneau and Kiela, 2018).
Contrastive Self-Supervised Learning (SSL) is now a prevalent approach in SRL, which is introduced by Gao et al. (2021) and Yan et al. (2021). It optimizes the representation space by reducing the distance between a sentence (or “anchor”) and semantically similar sentences (or “positive samples”), as well as increasing the distance between the sentence and semantically dissimilar sentences (or “negative samples”).
While the mechanisms underlying contrastive SSL can be intuitively understood, its effectiveness in SRL has not been thoroughly explored. Specifically, there are still some conflicts between the existing conclusions: (1) In theoretical perspective, machine learning community has found that contrastive SSL shares many similarities with non-contrastive SSL (e.g. alignment & uniformity (Wang and Isola, 2020), Barlow Twins (Zbontar et al., 2021), and VICReg (Bardes et al., 2022)) (Balestriero and LeCun, 2022; Tao et al., 2022) (2) However, in practical applications, con-
14506
trastive and non-contrastive SSL show comparable performance only in classification tasks (e.g., in Visual Representation Learning (VRL) and TR in SRL (Farina and Pappadopulo, 2023)). In contrast, for ranking tasks (i.e., STS in SRL), contrastive SSL seems be the only effective method, significantly outperforming non-contrastive SSL (Nie et al., 2023; Klein and Nabi, 2022; Xu et al., 2023). These inconsistent conclusions suggest that to make the obtained representations suitable for STS, certain unique factors must be present in the optimization objectives, which have rarely been explored in the existing literature.
In this work, we attempt to identify the key factors that enable contrastive SSL to be effective in STS. Specifically, we would like to answer two questions: (1) What commonalities enable various contrastive losses to achieve superior performance in STS? (2) How can we make non-contrastive SSL, which is similar to contrastive SSL but ineffective in STS, effective? We first analyze the commonalities among four effective losses (Oord et al., 2018; Zhang et al., 2022b; Nie et al., 2023) in SRL from the perspective of gradients. From this analysis, we find that all gradients can be unified into the same paradigm, which is determined by three components: the Gradient Dissipation, the Weight, and the Ratio. By statistically analyzing the values of these three components under different representation space distributions, we propose three conjectures, each corresponding to the role of a component in optimizing the representation space. Subsequently, we construct a baseline model to empirically validate our conjectures and demonstrate the significance of these components to model performance by varying them in the baseline.
After understanding the key factors that enable contrastive losses to be effective, we are able to analyze the reasons behind the poor performance of non-contrastive SSL in STS from the perspective of three components in the paradigm. We find that these ineffective losses do not perform as well as effective ones across these components. Therefore, by adjusting these components, we manage to make them function and achieve improved model performance in STS (refer to Figure 1).
Briefly, our main contributions are as follows:
• We propose a unified gradient paradigm for effective losses in SRL, which is controlled by three components: the Gradient Dissipation, the Weight, and the Ratio (Section 2);
• We analyze the roles in optimization for each component theoretically. Further, we propose and validate the conjectures on their effective roles in performing STS tasks (Section 3);
• With the guidance of our analysis results, we modify four optimization objectives in noncontrastive SSL to be effective in STS by adjusting the three components (Section 4).

heading: 2 A Unified Paradigm for Gradient

heading: 2.1 Preliminary
text: Given a batch of sentences {si}Ni=1, we adopt a encoder f(·) to obtain two l2-normalized embeddings (i.e., hi and h′i) for two different augmented views of sentence si. Regarding hi as the anchor, the embedding from the same sentence (i.e., h′i) is called the positive sample, while the embeddings from the other sentences (i.e., hj and h′j , j ̸= i) are called negative samples. Then, what contrastive learning does is to increase the similarity (typically cosine similarity) between the anchor and its positive sample and decrease the similarity between the anchor and its negative samples.

heading: 2.2 Gradient Analysis
text: To understand the optimization mechanism rigorously, we choose four contrastive losses used in recent works and derive the gradient for them. Note that all the loss functions have been proven to be effective in SRL and can obtain competitive performance based on the model architecture of SimCSE (Gao et al., 2021). InfoNCE (Oord et al., 2018) is a widely used contrastive loss introduced to SRL by ConSERT (Yan et al., 2021) and SimCSE (Gao et al., 2021). It can be formed as
Linfoi = − log eh
⊤ i h ′ i/τ
∑N j=1 e h⊤i h ′ j/τ ,
where τ is a temperature hyperparameter. The gradient of Linfoi w.r.t hi is
∂Linfoi ∂hi =
∑N j ̸=i e h⊤i h ′ j/τ (h′j − h′i)
τ ∑N
k=1 e h⊤i h ′ k/τ
. (1)
ArcCon (Zhang et al., 2022b) improves InfoNCE by enhancing the pairwise discriminative power and it can be formed as
Larci = − log ecos(θii′+u)/τ
ecos(θii′+u)/τ + ∑N
j ̸=i e h⊤i h ′ j/τ
,
where θij = arccos(h⊤i hj) and u is a hyperparameter. The gradient of Larci w.r.t hi is
∂Larci ∂hi =
∑N j ̸=i e h⊤i h ′ j/τ (h′j − sin(θii′+u) sin(θii′ ) h′i) τ(ecos(θii′+u)/τ + ∑N
k ̸=i e h⊤i h ′ k/τ )
. (2)
MPT and MET (Nie et al., 2023) are two loss functions in the form of triplet loss in SRL, which have the same form:
Ltrii = max(0, d(hi, h′i)−max j ̸=i d(hi, h ′ j) +m),
where m is a margin hyperparameter and d : Rd × Rd → R is metric function. Specifically, the only difference among the three loss functions is d(a, b), which is −aT b for MPT and ||a − b||2 for MET. Similarly, we can obtain their gradient w.r.t hi:
∂Lmpti ∂hi = I× (h′j − h′i), ∂Lmeti ∂hi = I× ( h′j ∥hi − h′j∥2 − h ′ i ∥hi − h′i∥2 ) , (3)
where j = argmaxh⊤i h ′ k(k ̸= i) and I is an indicator function equal to 1 only if d(hi, h′j) − d(hi.h ′ i) < m satisfies.
Comparing the above gradient forms, we find that all gradients guide the anchor hi move towards its positive sample h′i and away from its negative samples h′j . It implies that these various forms of loss functions have some commonalities in the perspective of the gradient. After reorganizing the gradient forms, we find that they can all be mapped into a unified paradigm:
∂Li ∂hi = GD(·) N∑ j ̸=i W(·)(h′j − R(·)h′i). (4)
Three components control this paradigm:
• GD(i, {hk, h′k}Nk=1) is the Gradient Dissipation term, which overall controls the magnitude of the gradient;
• W(i, j, {hk, h′k}Nk=1) is the Weight term, which controls the magnitude of the contribution of negative samples to the gradient;
• R(i, j, {hk, h′k}Nk=1) is the Ratio term, which controls the magnitude of the contribution of positive samples to the gradient;
Table 1 shows the specific form of three components in each loss, which all have been converted into the angular form for subsequent analysis.

heading: 3 Role of Each Component
text: 3.1 Theoretical Analysis
Although the above losses are unified into the same paradigm, the specific forms of the three components in different loss functions are various. Therefore, we first demonstrate that their roles are consistent despite their varied forms. To validate this, we record the trends of each component with the anchor-positive angle θii′ and anchor-negative angle θij′ and confirm whether there are consistent trends under different forms.
Specifically, we assume that θii′ and θij′ follow normal distribution N (µpos, σ2pos) and N (µneg, σ2neg), respectively. Based on experimental evidence, we draw µpos from [ π20 , π 2 ], and µneg from [ π20 , π], with σpos fixed at 0.05 and σneg fixed at 0.10. After validating the consistency among different forms, we propose intuitive conjectures regarding the roles played by each component.

heading: 3.1.1 Gradient Dissipation
text: To study the role of the gradient dissipation term, we experiment with three steps: (1) For both µpos and µneg, we divide their intervals equally into 100 parts to obtain 10,000 µpos-µneg pairs, covering all situations of the entire optimization process; (2) For each µpos-µneg pair, we sample 1,000 batches, each comprising 1 sampled from θii′ and 127 sampled from θij′ ; (3) The average value of the gradient dissipation term is calculated across these batches. The results are plotted in Figure 2 with heatmaps.
There are two forms of gradient dissipation: one is implemented through fraction functions (including InfoNCE and ArcCon), and the other is implemented through indicator functions (including MPT and MET). The results in Figure 2 show that both the two forms of the gradient dissipation term exhibit a similar pattern: when µpos and µneg are close, the value is 1; when µneg is larger than µpos to some extent, the value rapidly decreases to 0. Intuitively, this term avoids a great distance gap between the anchor-positive and the anchor-negative pairs. Recall that the semantic similarity is scored on a scale from 0 to 5 in traditional STS tasks, rather than being binary classified as similar or dissimilar. Such a great gap may deteriorate the performance of sentence embeddings in STS tasks. Therefore, we propose
Conjecture 1. The effective gradient dissipation term ensures that the distance gap between µneg and µpos remains smaller than the situation trained without gradient dissipation.

heading: 3.1.2 Weight
text: To study the role of the weight term, we also experiment with three steps: (1) For µneg, we divide its intervals equally into 100 parts, while for µpos, we fixed its value to π6 . Then we can obtain 100 µpos-µneg pairs to analyze the relative contribution among the negative samples; (2) For each µposµneg pair, we sample 1000 batches, each comprising 1 sampled from θii′ and 127 sampled from θij′ ; (3) The average proportion of the weight for the
hardest negative sample (i.e., the one with the highest cosine similarity to the anchor) is calculated. The results calculated under different temperatures are plotted in Figure 3.
There are two forms of weight: one is implemented through the exponential function (including InfoNCE and ArcCon), and the other is implemented through the piecewise function (including MPT and MET). For the piecewise-form weight, the value is non-zero if and only if the negative sample is the hardest. Therefore, to prove the consistency between these two forms, it is only need to examine the proportion of the hardest negative samples in the exponential-form weight. The results show that the exponential-form weight also becomes focused solely on the hardest negative examples during the training process, similar to the piecewise-form weight. Therefore, we propose
Conjecture 2. The effective weight term ensures that the hardest negative sample occupies a dominant position in the gradient compared to other negative samples.
The ratio term can be categorized into two types: One is the static ratio (including InfoNCE and MPT), keeping the value of 1; The other is the dynamic ratio (including ArcCon and MET), with
values dependent on θii′ and θij′ . To investigate the value range of the dynamic ratio, we follow the first two steps in Section 3.1.1 and calculate the values of different dynamic ratio terms. The results are plotted in Figure 4. As shown in the figure, the dynamic ratio varies significantly. However, when considering the impact of gradient dissipation and the fact that µpos is typically smaller than µneg, it can be found that the values of the dynamic ratio mostly fall between 1 and 2.
There exists an interesting phenomenon where sentence embeddings trained with ArcCon outperform those trained with InfoNCE, and MET does the same for MPT (Refer to Table 3). However, as observed in previous analyses, the gradient dissipation term and the weight term of these losses are essentially consistent. Therefore, we believe that it is precisely because ArcCon and MET can achieve larger ratios during the training process that these losses exhibit better performance, which can be explained through the following lemma. Lemma 1. For an anchor hi and its positive sample h′i and negative sample h ′ j , assume the angle between the plane Ohih′i and the plane Ohih ′ j is α. When hi moves along the optimization direction λ(rh′i − h′j), r must satisfy
r > 1
λ +
sin θij′ cosα sin θii′ − √ 1 λ2 − sin 2 θij′ sin 2 α sin2 θii′ .
to ensure the distance from hi to h′i becomes closer after the optimization step.
The larger ratios in ArcCon and MET enable them to meet the condition in more situations, thereby exhibiting better performance. We propose Conjecture 3. The effective ratio term can meet the condition in Lemma 1 more frequently, and ensure that the distance from the anchor to the positive sample is closer after optimization than that before optimization.

heading: 3.2 Empirical Study
text: To validate the conjectures in Section 3.1 and further investigate the impact of each component on the model performance, this section conducts experiments based on
Li = GD(·) N∑
j ̸=i W(·)
( h⊤i h ′ j − R(·)h⊤i h′i ) , (5)
where GD(·), W(·), and R(·) do not require gradient. It can be easily verified that the gradient
of Li w.r.t hi is the paradigm in Equation 4. We select GD(·) = I{h⊤i h′i−maxNk ̸=i h⊤i h′k<m}, W(·) =
e h⊤i h ′ j/τ
∑N k ̸=i e h⊤ i h′ k /τ , R(·) = r, and m = 0.3, τ = 0.05, r = 1 as baseline. We adopt BERTbase (Devlin et al., 2019) as backbone and utilize commonly used unsupervised datasets (Gao et al., 2021) as training data. In validating conjectures, we divide the training data into two parts, where 90% is used for training and 10% is held out for statistical analysis. In investigating the impact of each component on performance, we vary each component in the baseline and train with all data, with Spearman’s correlation on the STS-B (Cer et al., 2017) validation set as the performance metric.

heading: 3.2.1 Validation of Conjecture 1
text: To verify Conjecture 1, we compare the distribution of θii′ and θij′ in sentence embeddings obtained from baseline training and from training without gradient dissipation (i.e., setting GD(·) = 1). The results, presented in Figure 5, indicate that the distribution of θii′ is closer to that of θij′ when gradient dissipation is applied, compared to the scenario without it, therefore validating the conjecture.
To investigate the impact of the gradient dissipation term on performance, we vary m in GD(·) from 0.3 to 1 and plot the corresponding performance changes in the first graph of Figure 6. An increase in m implies a weakening effect, and the model performance decreasing as m increases proves the importance of effective gradient dissipation term for model performance.

heading: 3.2.2 Validation of Conjecture 2
text: To verify Conjecture 2, we calculated the average proportion of the hardest negative samples in the exponential-form weight under different temperature τ , with results presented in Figure 7. In the original BERT, since embeddings are confined within a smaller space (Li et al., 2020), a very small
τ is required to allow the hardest negative samples to occupy a higher proportion. However, in models trained with the baseline, the spatial range of embeddings increases, enabling a commonly used setting (τ = 0.05) to also allow the hardest negative samples to dominate the optimization direction. These findings validate the conjecture.
To investigate the impact of the weight term on performance, we vary τ in W(·) from 0.01 to 3.0 and plot the corresponding performance changes in the second graph of Figure 6. An increase in τ means the overall proportion of the hardest negative samples in the gradient decreases. When τ is increased to 0.3, the proportion begins to sharply decline (refer to Figures 3 and 7), at which point there is also a sharp drop in model performance. This demonstrates the necessity of an effective weight term for model performance.

heading: 3.2.3 Validation of Conjecture 3
text: To verify Conjecture 3, we first record the distribution of the minimum values required to meet the conditions in Lemma 1 (presented in Figure 8a). The results show that, across different models and values of λ, a ratio greater than 1 consistently meets more conditions. Then, we record the values of three dynamic ratios during the training process (presented in Figure 8b). The results show that ArcCon and MET maintain a ratio greater than 1 throughout the training process, thus better fulfilling the lemma’s conditions. Finally, we record the
0.25 0.50 0.75 1.00 λ
0.0
0.5
1.0
1.5
2.0
R at io
0.25 0.50 0.75 1.00 λ
(a) Distribution of the minimum ratio that satisfies the condition under different λ for the original BERT model (left) and the BERT model fine-tuned with the baseline loss (right).
0 1000 3000 5000 7000 Step
1.3
1.4
1.5
1.6 1.7 R at io
ArcCon MET
(b) Variations in the dynamic ratios in ArcCon and MET across training steps.
0 1000 3000 5000 7000 Step
0.70 0.75 0.80 0.85 0.90 0.95 C os in e
2.00 1.00 0.50 0.25
(c) Variations in average cosine similarity across training steps for anchor-positive pairs.
Figure 8: Results for validating Conjecture 3.
average cosine similarity between the anchor and positive sample during the training process under different ratios (presented in Figure 8c). The results show that a larger ratio can indeed result in a closer anchor-positive distance. Together, the above results validate the conjecture.
To investigate the impact of the ratio term on performance, we vary r in R(·) from 0.25 to 2.0 and plot the corresponding performance changes in the third graph of Figure 6. An increase in r means the conditions in Lemma 1 can be met more frequently, and the model performance improving as r increases proves the significance of an effective ratio term for model performance.

heading: 4 Modification to Ineffective Losses
text: After understanding the properties that the gradients of the effective losses in SRL should possess, we can make some ineffective optimization objectives in non-contrastive SSL effective. Alignment & Uniformity (Wang and Isola, 2020) are two metrics used to assess the performance of
the representation space, yet directly using them as optimization objectives results in poor performance. Alignment can be represented as
La = 1 N
N∑
i=1
∥hi − h′i∥22,
and uniformity can be represented in many forms (Liu et al., 2021), one of which is the Minimum Hyperspherical Energy (MHE)
Lu,MHE = log 2 N(N − 1)
∑
1≤k<l≤N e−∥hk−hl∥ 2 2 ,
and another is the Maximum Hyperspherical Separation (MHS)
Lu,MHSi = −min j ̸=i ∥hi − hj∥2.
Another two widely used optimization objectives in non-contrastive SSL are Barlow Twins (2021)
LB = D∑
k=1
(CBkk − 1)2 + νB D∑
k ̸=l (CBkl) 2,
where CB = 1N ∑N i=1 hih ′⊤ i , and VICReg (2022):
LV = La + νV,1 ( v({hi}Ni=1) + v({h′i}Ni=1) )
+ νV,2 ( c({hi}Ni=1) + c({h′i}Ni=1) ) ,
v({hi}Ni=1) = 1
D
D∑ k ̸=l (CV({hi}Ni=1)kl)2,
c({hi}Ni=1) = 1
D
D∑
k=1
max ( 0, γ − σ({hi}Ni=1)k ) ,
where CV({hi}Ni=1) = 1N−1 N∑ i=1 (hi − h̄i)(hi − h̄i) ⊤, and σ(·) is the sample standard deviation. They are both popular in Visual Representation Learning (VRL), yet their performances are relatively poor when applied to SRL.
The gradient w.r.t hi for alignment and uniformity (La+νuLu,MHE and La+νuLu,MHSi ), Barlow Twins (LB), and VICReg (LV) can also be mapped into the paradigm. We present the results in the upper part of Table 2, where the gradient for LB and LV is derived by Tao et al. (2022).
Components in the paradigm of ineffective losses do not perform in the same manner as those of effective losses. For gradient dissipation terms, since their values are 1 in all ineffective losses, it implies that they have no effect. For weight terms, except for MHS, other losses cannot ensure that the hardest negative samples dominate in the gradient. For ratio terms, since they are dynamic and cannot guarantee an effective value throughout the optimization process, it is better to adopt a static ratio. Therefore, we adjust these terms, with the results presented in the lower part of Table 2.
We take Barlow Twins as an example to introduce the modification method. For the specific modification process of other optimization objectives, please refer to Appendix E. Based on gradients, the optimization objective of Barlow Twins is equivalent to
LeBi = −h⊤i wBp,ih′i + νB N∑
j ̸=i h⊤i w B n,ijhj ,
where wBp,i = 2(I−(1−λ)Wdiag) N , w B n,ij =
2h′⊤i h ′ jI
N2 ,
and both of them do not require gradient. To adjust the gradient dissipation term, we stop the gradient of anchor hi that does not meet the condition (h⊤i h ′ i −maxNk ̸=i h⊤i h′k < m), which is equivalent to multiplying the loss by the indicator function
di = I{h⊤i h′i−maxNk ̸=i h⊤i h′k<m}.
To adjust the weight term, we first set νB = 1. Then, we modify wBn,ij to an exponential form:
wmBn,ij = eh
′⊤ i h ′ j/τ
∑ k ̸=l e h′⊤k h ′ l/τ .
To adjust the ratio term, we set the loss to have a static ratio r, by modifying wBp,i to
wmBp,i = N∑
j ̸=i wmBn,ij =
r ∑N
j ̸=i e h′⊤i h ′ j/τ
∑ k ̸=l e h′⊤k h ′ l/τ ,
which do not require gradients. Finally, the modified Barlow Twins can be represented as
LmBi = di(−h⊤i wmBp,i h′i + N∑
j ̸=i h⊤i w mB n,ijhj). (6)
We evaluate all the modified optimization objectives using SentEval benchmark (Conneau and
Kiela, 2018), with the results for the Semantic Textual Similarity (STS) tasks presented in Table 3 and the Transfer (TR) tasks in Table 4. In the STS tasks, the modified optimization objectives show a significant improvement compared to the original ones and achieved performance comparable to mainstream contrastive SSL methods. These results indicate that our modifications successfully make these non-contrastive self-supervised learning optimization objectives effective for STS tasks. Additionally, in the TR tasks, the modified optimization objectives also show overall improvement compared to the original ones, suggesting that our modifications do not introduce any additional negative impacts on the optimization objectives.
Finally, we examine the performance improvements brought by modifying each component of the optimization objectives. Specifically, we gradually modify each component of the optimization objectives in the order of the Gradient Dissipation term, the Weight term, and the Ratio term. Then, we measure the changes in the average performance on seven STS tasks, with the results shown in Table 5. The results indicate that as each component is modified, the performance of the optimization objectives gradually improves, validating the importance of each effective component for the optimization objectives.

heading: 5 Related Work

heading: Contrastive Sentence Representation Learning
text: is initially investigated by Gao et al. (2021) and Yan et al. (2021), followed by numerous efforts (2022a; 2022; 2023; 2024) that enhance the method’s performance on STS tasks. Beyond these performancefocused studies, Nie et al. (2023) explores the reasons for the success of contrastive SRL, identifying the significance of gradient dissipation in optimization. The distinction of our work lies in further identifying other critical factors and making previously ineffective losses effective. Similarities between Contrastive and Noncontrastive Self-Supervised Learning are explored in numerous works (2022b; 2023) in Com-
puter Vision (CV). Zhang et al. (2022b) study the similarities between the two methods from the perspective of spectral embedding. Garrido et al. (2023) point out that under certain assumptions, the two methods are algebraically equivalent. Our work makes non-contrastive SSL effective in SRL, revealing that similar parallels also exist in NLP. Gradients of Self-Supervised Learning are investigated in CV by Tao et al. (2022), which is similar to our methods. The distinction of our work lies in that we work in the field of NLP, and that we explore the impact of different gradient components on optimizing the representation space.

heading: 6 Conclusion
text: In this paper, we propose a unified gradient paradigm for four different optimization objectives in SRL, which is determined by three components: the Gradient Dissipation term, the Weight term, and the Ratio term. We uncover the roles these components play in optimization and demonstrate their significance to the model performance on the STS tasks. Based on these insights, we succeed in making ineffective non-contrastive SSL optimization objectives effective in STS. Our work advances the understanding of why contrastive SSL can be effective in SRL and guides the future design of new optimization objectives.

heading: Limitations
text: First, our work currently focuses only on the impact of optimization objectives on model performance. This means that the results of this study cannot be applied to analyze the impact of model architecture on model performance. Secondly, we conduct experiments only on STS tasks in the domain of SRL, without extending to other ranking tasks or modalities. These areas are left for future work to explore. Lastly, modifying the gradientequivalent form of optimization objectives results in significant differences in the form of optimization objectives before and after modification. To ensure consistency in form, one should make modifications based on observations and experience (see examples in Appendix E).

heading: Acknowledgements
text: This work was supported by the National Science and Technology Major Project under Grant 2022ZD0120202, in part by the National Natural Science Foundation of China (No. U23B2056), in part by the Fundamental Research Funds for the Central Universities, and in part by the State Key Laboratory of Complex & Critical Software Environment.

heading: A Experiment Details

heading: A.1 Distribution of Representation Space
text: In analyzing the role that each component plays in optimizing the representation space (Section 3.1), we assume that θii′ and θij′ follow normal distributions N (µpos, σ2pos) and N (µneg, σ2neg), respectively. These assumptions are consistent with the phenomena observed in subsequent experiments (Section 3.2). Specifically, Figure 5 demonstrates that in the representation space, the distributions of θii′ and θij′ closely approximate normal distributions, and the θii′ distribution in Figure 8c is within the range of our assumptions. Furthermore, we estimate µpos, µneg, σpos, and σneg during trained with the baseline. The results are presented in Figure 9. From these results, it is evident that our assumed ranges for µpos ([ π20 , π 2 ]) and µneg ([ π 20 , π]) adequately cover the actual scenarios that may occur. Furthermore, the values of σpos and σneg (0.05 and 0.10, respectively) are also close to the practical observations.

heading: A.2 Details of the Empirical Study
text: In validating the conjectures, we randomly hold out 10% of the data for analyzing the representation space. To obtain results in Figure 5 and Figure 8c, we directly use the held-out data, and Figure 5 is plotted through Kernel Density Estimation (KDE). To obtain the results in Figure 7, Figure 8a, and Figure 8b, we split the held-out data into batches
with size N = 128 and calculate the corresponding values within each batch.

heading: A.3 Training Details
text: Our implementation of training the representation space is based on SimCSE (Gao et al., 2021), which is currently widely used in research. Our experiments are conducted using Python3.9.13 and Pytorch1.12.1 on a single 32G NVIDIA V100 GPU.
Following Gao et al. (2021), we obtain sentence embeddings from the “[CLS]” token and apply an MLP layer solely during training. The embeddings are trained with 1,000,000 sentences from Wikipedia, which are also collected by Gao et al. (2021), for one epoch. Note that, in validating the conjectures, we randomly hold out 10% of the data for analysis and only use 90% for training.

heading: A.4 Parameter Setting
text: For experiments in Section 3.2, all experiments are conducted with BERTbase as the backbone and batch size = 128, learning rate = 1e-5.
For the modified losses in Section 4, we first perform a grid search on batch size ∈ {64, 128, 256, 512}, learning rate ∈ {5e-6, 1e-5, 3e-5, 5e-5} with m = 0.30, τ = 5e-2, and r = 1. Then, based on the best combination of batch size and learning rate from the first grid search, we perform the second grid search on m ∈ {0.27, 0.30, 0.33, 0.37}, τ ∈ {1e-2, 3e-2, 5e-2}, and r ∈ {1.00, 1.25, 1.50, 1.75, 2.00}. The results are presented in Table 6 and Table 7.

heading: A.5 Evaluation Protocol
text: In Section 3.2 and Section 4, we adopt a widely used evaluation protocol, SentEval toolkit (Con-
neau and Kiela, 2018), to evaluate the performance of SRL. SentEval includes two types of tasks: the Semantic Textual Similarity (STS) tasks and the Transfer tasks (TR). The STS task quantifies the semantic similarity between two sentences with a score ranging from 0 to 5 and takes Spearman’s correlation as the metric for performance. There are seven STS datasets included for evaluation: STS 2012-2016 (2012; 2013; 2014; 2015; 2016), STS Benchmark (2017), and SICK Relatedness (2014). The TR task measures the performance of embeddings in the downstream classification task and takes Accuracy as the metric. There are also seven datasets included for the evaluation of TR task: MR (2005), CR (2004), SUBJ (2004), MPQA (2005), SST2 (2013), TREC (2000), MRPC (2005). Note that in Section 3.2, we only use STS Benchmark validation set for evaluation, and it is conventional to use only this dataset when comparing the performance under different hyperparameters (Gao et al., 2021).
B Illustrations of the Role of Each Component
We first present the results of experimenting the gradient dissipation term for InfoNCE and MET (refer to Section 3.1) in Figure 12a.
Then, we provide three illustrations in Figure 12 to facilitate an intuitive understanding of the roles played by different components in optimizing the
representation space. In the diagram, black dots represent the anchor, gray dots represent the positions of the anchor after optimization, green dots represent positive samples, and red dots represent negative samples. Among the negative samples, the lighter the shade of red, the harder the negative sample is.

heading: C Proof of Lemma 1
text: For the anchor point hi, let us denote the angle between it and the positive sample h′i as θii′ , and the angle between it and the negative sample h′j as θij′ . Let α represent the dihedral angle between the planes Ohih′i and Ohih ′ j . Under the influence of the optimization direction λ(rh′i−h′j), the new position of the anchor point after movement is denoted as hni . Although all embeddings lie on a hypersphere, to simplify the proof process, we orthogonally project all embeddings onto the tangent plane at the anchor hi (as shown in Figure 11). In this case, the length of line hih′i is equal to sin(θii′), the length of line hih′j is equal to sin(θij′), and the angle ∠h′ihih′j is α. This simplification is reasonable because the optimization of the anchor hi on the hypersphere can in fact be considered as optimization on the tangent plane, which is then projected back onto the hypersphere. For the optimization direction λ(rh′i−h′j), we also project it onto the tangent plane, where −λh′j corresponds to the line hic, and λrh′i corresponds to line chni .
Then, to ensure that the anchor-positive distance does not increase after optimization (i.e., hni h ′ i ≤ hih ′ i), we assume h n i h ′ i = hih ′ i = sin(θii′),
thereby obtaining the necessary boundary condition that satisfies the requirement. Specifically, by deriving the relationship between r and the other elements at this time, we can obtain the lower bound of r that satisfies the requirements. To this end, we add an auxiliary point b on the line chni , such that the quadrilateral bhni h ′ ihi forms a rhombus (two possible situations when adding the auxiliary point are given in Figure 11), and we denote the length of the line bc as x. By applying the cosine theorem, we can obtain the following: under Situation 1, we have
x1 =− λ sin(θij′) cos(α)
+ √ sin2(θii′)− λ2 sin2(θij′) sin2(α);
under Situation 2, we have
x2 =λ sin(θij′) cos(α)
− √ sin2(θii′)− λ2 sin2(θij′) sin2(α).
At this point, we can express r as follows:
r = sin(θii′)− x1 λ sin(θii′) = sin(θii′) + x2 λ sin(θii′)
= 1
λ +
sin θij′ cosα sin θii′ − √ 1 λ2 − sin 2 θij′ sin 2 α sin2 θii′ ,
which is the lower bound of r that satisfies the requirements, thus proving Lemma 1.

heading: D Gradients of Optimization Objectives

heading: D.1 Gradients of effective Objectives
text: In this section, we provide the derivation process of gradients in Section 2. The gradient of Linfoi w.r.t hi is
∂Linfoi ∂hi = ∂ ∂hi
 − log e h⊤i h ′ i/τ
∑N j=1 e h⊤i h ′ j/τ
 
= ∂(log
∑N j=1 e h⊤i h ′ j/τ )
∂hi − ∂h
⊤ i h ′ i/τ ∂hi
= ∂ ∑N j=1 e h⊤i h ′ j/τ/∂hi
∑N j=1 e h⊤i h ′ j/τ
− h ′ i
τ
=
∑N j=1 e h⊤i h ′ j/τh′j τ ∑N
j=1 e h⊤i h ′ j/τ
− h ′ i
τ
=
∑N j ̸=i e h⊤i h ′ j/τ (h′j − h′i)
τ ∑N
j=1 e h⊤i h ′ j/τ
.
The gradient of Larci w.r.t hi is
∂Larci ∂hi = ∂ ∂hi  − log
e cos(θii′+u) τ
e cos(θii′+u) τ + N∑ j ̸=i e h⊤ i h′ j τ
 
= ∂ log(ecos(θii′+u)/τ +
∑N j ̸=i e h⊤i h ′ j/τ )
∂hi
− ∂ cos(θii′ + u)/τ ∂hi
= ∂(ecos(θii′+u)/τ +
∑N j ̸=i e h⊤i h ′ j/τ )/∂hi
ecos(θii′+u)/τ + ∑N
j ̸=i e h⊤i h ′ j/τ
+ sin(θii′ + u) τ · ∂(θii′ + u)
∂hi
= ecos(θii′+u)/τ · sin(θii′+u)sin(θii′ ) h ′ i
τ(ecos(θii′+u)/τ + ∑N
j ̸=i e h⊤i h ′ j/τ )
+
∑N j ̸=i e h⊤i h ′ j/τh′j
τ(ecos(θii′+u)/τ + ∑N
j ̸=i e h⊤i h ′ j/τ )
− sin(θii′ + u) τ sin(θii′) h′i
=
∑N j ̸=i e h⊤i h ′ j/τ (h′j − sin(θii′+u) sin(θii′ ) h′i) τ(ecos(θii′+u)/τ + ∑N
j ̸=i e h⊤i h ′ j/τ )
.
The gradient of Ltrii w.r.t hi is
∂Ltrii ∂hi =I{−d(hi,h′i)+d(hi,h′j)<m}
× ( ∂d(hi, h ′ i)
∂hi −
∂d(hi, h ′ j)
∂hi
) ,
where j = argmink ̸=i d(hi, h′k). Therefore, The gradient of Lmpti w.r.t hi is
∂Lmpti ∂hi = I{h⊤i h′i−h⊤i h′j<m}(h ′ j − h′i),
and the gradient of Lmeti w.r.t hi is ∂Lmeti ∂hi =I{∥hi−h′i∥2−∥hi−h′j∥<m}
× (
h′j ∥hi − h′j∥2 − h ′ i ∥hi − h′i∥
) .

heading: D.2 Gradients of Ineffective Objectives
text: In this section, we provide the gradients of objectives in Section 4.The gradient of La w.r.t hi is
∂La ∂hi = − 2 N h′i, (7)
(a) The gap between the anchor-positive and anchor-negative distances is larger in the representation space trained without gradient dissipation (left) compared to that with gradient dissipation (right).
(b) When not amplified by the weight term, the contribution of the hardest negative samples to the optimization direction is minimal (left). Once amplified by the weight term, the hardest negative samples dominate the optimization direction (right).
(c) When the ratio term does not meet the conditions in Lemma 1, anchor-positive distance increases after optimization. When the conditions are met, anchor-positive distance decreases after optimization.
Figure 12: Illustrations for the role in optimization of each component.
the gradient of Lu,MHE w.r.t hi is
∂Lu,MHE ∂hi =
∑N j ̸=i 2e
2h⊤i hjhj ∑
1≤k<l≤N e 2h⊤k hl
, (8)
and the gradient of Lu,MHS w.r.t hi is ∂Lu,MHS
∂hi =
1
∥hi − hj∥2 hj , (9)
where j = argminj ̸=i ∥hi − hj∥. The gradients of LB and LV w.r.t hi have been derived by Tao et al. (2022):
∂LB ∂hi = 2 N (−Ah′i + νB
N∑
j ̸=i
h′⊤i h ′ j
N hj), (10)
where A = I − (1 − νB)CBdiag, and CBdiag is the diagonal matrix of CB, and
∂LV ∂hi = 2 N (−h′i + νV
N∑
j ̸=i
h⊤i hj N hj), (11)
where νV = 2ν V,1N2
D(N−1)2 . Note that the original Barlow Twins adopts a batch normalization, and VICReg adopts a de-center operation, which is different from the commonly used l2 normalization in SRL. But Tao et al. (2022) verify that these operations have a similar effect in training, therefore we use l2 normalization for all losses for consistency.
One more thing to note is that the gradients of ineffective losses are mapped into the following form:
∂Li ∂hi = GD(·) N∑ j ̸=i W(·)(hj − R(·)h′i), (12) which uses hj instead of h′j in Equation 4. Since hj and h′j are mathematically equivalent with respect to hi, we consider Equation 4 and 12 to be consistent and only present Equation 4 in the main body of the paper.

heading: E Modifications to Ineffective Losses
text: We offer two methods of modification: (1) modifying the gradient-equivalent form of the optimization objective, which is the method to modify Barlow Twins in Section 4, and (2) directly modifying the optimization objective itself.
For VICReg, we use the first method for modification as we do for Barlow Twins. Based on gradients, the optimization objective of VICReg is equivalent to
LeVi = −h⊤i wVp,ih′i + νV N∑
j ̸=i h⊤i w V n,ijhj ,
where wVp,i = 2I N , w V n,ij = 2h⊤i hjI N2
, and both of them do not require gradient. To adjust the gradient dissipation term, we stop the gradient of anchor hi that does not meet the condition (h⊤i h ′ i −maxNk ̸=i h⊤i h′k < m), which is equivalent to multiplying the loss by the indicator function
di = I{h⊤i h′i−maxNk ̸=i h⊤i h′k<m}.
To adjust the weight term, we first set νV = 1. Then, we modify wVn,ij to an exponential form:
wmVn,ij = eh
⊤ i hj/τ
∑ k ̸=l e h⊤k hl/τ .
To adjust the ratio term, we set the loss to have a static ratio r, by modifying wVp,i to
wmVp,i = N∑
j ̸=i wmVn,ij =
r ∑N
j ̸=i e h⊤i hj/τ
∑ k ̸=l e h⊤k hl/τ ,
which do not require gradients. Finally, the modified VICReg can be represented as
LmVi = di(−h⊤i wmVp,i h′i + N∑
j ̸=i h⊤i w mV n,ijhj). (13)
For alignment and uniformity, we use the second method for modification. To adjust the gradient dissipation term, we stop the gradient of anchor hi that does not meet the condition (h⊤i h ′ i − maxNk ̸=i h ⊤ i h ′ k < m) as before. As for the weight term, since the weight term of (La + νuLu,MHS) does not require adjustment, we only adjust the weight term for (La + νuLu,MHE). Specifically, We set νu = 1 and introduce a temperature parameter τ to Lu,MHE:
Lu,mMHE = log 2 ∑ 1≤k<l≤N e −∥hk−hl∥22/(2τ)
N(N − 1) .
To adjust the ratio term, we modify both of them to have a static ratio r by prepending a coefficient to the alignment:
La,mMHEi = r ∑N j ̸=i e h⊤i hj/τ
2τ ∑
1≤k<l≤N e h⊤k hl/τ
︸ ︷︷ ︸ no gradient
∥hi − h′i∥22,
La,mMHSi = r
2maxj ̸=i ∥hi − hj∥2︸ ︷︷ ︸ no gradient
∥hi − h′i∥22.
Finally, the modified losses can be represented as
LmMHEi = di(La,mMHEi + Lu,mMHE), (14) LmMHSi = di(La,mMHSi + L u,MHS i ). (15)
In order to verify the correctness of the modification, we present the gradients of all the modified losses here. The gradient of LmMHEi w.r.t hi is
LmMHEi ∂hi = di
∑N j ̸=i e
h⊤i hj/τ (hj − rh′i) τ ∑
1≤k<l≤N e h⊤k hl/τ
. (16)
LmMHEi ∂hi = di (hj − rh′i) ∥hi − hj∥2 ,
j = argmin k ̸=i
∥hi − hk∥2 (17)
LmBi ∂hi = di
∑N j ̸=i e
h′⊤i h ′ j/τ (hj − rh′i)∑N
k ̸=l e h′⊤k h ′ l/τ
. (18)
LmVi ∂hi = di
∑N j ̸=i e
h⊤i hj/τ (hj − rh′i)∑N k ̸=l e h⊤k hl/τ . (19)

heading: Future Work
text: These areas are left for future work to explore. Lastly, modifying the gradientequivalent form of optimization objectives results in significant differences in the form of optimization objectives before and after modification. To ensure consistency in form, one should make modifications based on observations and experience (see examples in Appendix E)