On The Inadequacy of Optimizing Alignment and Uniformity in Contrastive Learning of Sentence Representations
Authors: Zhijie Nie
Richong Zhang
Yongyi Mao
Venue: ICLR 2023 poster (2023)
Keywords: Sentence representation learning
Contrastive learning
Alignment
Uniformity

Abstract: Contrastive learning is widely used in areas such as visual representation learning (VRL) and sentence representation learning (SRL). Considering the differences between VRL and SRL in terms of negative sample size and evaluation focus, we believe that the solid findings obtained in VRL may not be entirely carried over to SRL. In this work, we consider the suitability of the decoupled form of contrastive loss, i.e., alignment and uniformity, in SRL. We find a performance gap between sentence representations obtained by jointly optimizing alignment and uniformity on the STS task and those obtained using contrastive loss. Further, we find that the joint optimization of alignment and uniformity during training is prone to overfitting, which does not occur on the contrastive loss. Analyzing them based on the variation of the gradient norms, we find that there is a property of ``gradient dissipation'' in contrastive loss and believe that it is the key to preventing overfitting. We simulate similar "gradient dissipation" of contrastive loss on four optimization objectives of two forms, and achieve the same or even better performance than contrastive loss on the STS tasks, confirming our hypothesis.

Published as a conference paper at ICLR 2023
ON THE INADEQUACY OF OPTIMIZING ALIGNMENT AND UNIFORMITY IN CONTRASTIVE LEARNING OF SENTENCE REPRESENTATIONS
Zhijie Nic Richong Zhang--2, Yongyi Mao
SKLSDD, Beihang University, Beijing, China
"hhogguanncun Laboratory, Beijing, China
i School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada 'Shen Yuan Honors College, Beihang University, Beijing, China {niez), zhangre]@act wwa.. edu. cn, ymao@uortawa.ca
ABSTRACT
Contrastive learning is widely used in areas such visual representation learning (VRL) and sentence representation learning (SRL). Considering the differences between VRL and SRL in terms of negative sample size and evaluation focus, we believe that the solid findings obtained in VRL may not be entirely carried over to SRL. In this work, we consider the suitability of the decoupled form of contrastive loss, i.e., aligument and uniformity, in SRL. We find a performance gap between sentence representations obtained by jointly optimizing aligument and uniformity on the STS task and those obtained using contrastive loss. Further, we find that the joint optimization of alignment and uniformity during training is prone overfit- ting, which does not occur on the contrastive loss. Analyzing them based on the variation of the gradient norms, we find that there is a property of "gradient dissi- pation" in contrastive loss and believe that it is the key to preventing overfitting. We simulate similar "gradient dissipation" of contrastive loss on four optimiza- tion objectives of two forms, and achieve the same even better performance than contrastive loss on the STS tasks, confirming our hypothesis.!.
INTRODUCTION
Unsupervised contrastive learning (Wu et al., 2018) is originated from visual representation learn- ing (VRL) (Chen et al., 2020; He et al., 2020; Grill et al., 2020), and has achieved impressive performances therein. Briefly, contrastive learning forces the representation of an input instance (or "anchor") to be similar to that of an augmented view of the same instance (or "postive example") and to differ from that of some different instances (or "negative examples"). One plausible justification of this approach is that minimizing the loss of contrastive learning (or "contrastive loss") is shown to be equivalent to simultaneously minimizing an "alignment loss" and a "uniformity loss", where the former dictates the representation similarity between an instance and its positive examples, and the latter forces the representations of all instances to spread uniformly on the unit sphere in the repre- sentation space (Wang& & Isola, 2020). Notably this decomposition of the contrastive loss relies on the condition that the number of negative examples participating in the contrastive loss approaches infinity. For VRL, arguable that such a condition holds reasonably well since usually a large number (e.g., 65536 (He et al., 2020)) of negative examples are used in training.
In recent years, contrastive learning has also been adapted to sentence representation learning (SRL), by fine-tuning the representations obtained from a pretrained language model (e.g. BERT (Devlin et al., 2018)) (Yan et al., 2021; Giorgi et al., 2021; Gao et al., 2021; Zhang et al., 2022b). Such approaches have demonstrated great performances not only for downstream classification tasks, but also for semantic textual similarity (STS) tasks. It is noteworthy that significantly contrast-
Cooresponningg author
'The codes and checkpoints are released for study https: //g thub, com/ BDBC//BCCKKK BDBC-KG-MM KG-PPP/ TTLR2023-Gradieee saipation.ii.

---

Published as a conference paper at ICLR 2023
ing VRL, which are primarily evaluated using downstream classification tasks (Russakovsky al., 2015: Krizhevsky et al., 2009), or via "extrinsic" protocols (Chiu et al., 2016; Faruqui et al., 2016), SRL particularly emphasizes the STS tasks, or "intrinsic" protocols, when evaluating the quality of learned sentence representations (Reimers & Gurevych, 2019; Li et al., 2020; Yan et al., 2021; Zhang et al., 2022c). This is because the representations obtained from the pretrained language models have already shown strong transfer capability to downstream tasks (Devlin et al., 2018; Liu al., 2019) while their similarities are rather poorly correlated with the human-rated similarities (Reimers & Gurevych, 2019; Li et al., 2020).
Following the justification of contrastive learning in VRL(Wang & Isola, 2020), some works (Gao et al., 2021: Zhang et al., 2022b) attribute the success of contrastive learning on the STS tasks to a good balance between alignment and uniformity. Consequently, alignment and uniformity losses are adopted widely as the key metrics for evaluating the goodness of sentence representations learned from contrastive learning (Gao et al., 2021; Zhang et al., 2022b;e;a; Klein & Nabi, 2022).
Noting that contrastive learning for SRL in fact only uses a rather small number of negative examples (e.g, 63 (Gao et al., 2021; Zhang et al., 2022b) or smaller (Zhang et al., 2022c)), in this paper, we question whether the "decomposition principle". or jointly optimizing alignment and uniformity, adequately explains the performance gain in the STS tasks brought by contrastive learning.
After extensive experiments, we find that optimization using alignment and uniformity losses pro- duces lower performance than that with contrastive loss in the STS tasks. Moreover, this perfor- mance degradation is not reflected by the alignment and uniformity metrics. Interestingly, observe the same phenomenon in contrastive learning with another loss function, Decoupled Con- trastive Loss (DCL) (Yeh et al., 2021), in which the optimization objectives are very similar to aligument and uniformity. Our further studies also show that training with such decoupled forms of contrastive loss cause severe overfitting, which does not occur training with the standard con- trastive loss. These observations suggest that alignment and uniformity losses might not serve suit- able substitutes for contrastive loss in SRL and that the success of the standard contrastive learning for SRL can not be adequately explained in terms of aligument and uniformity properties or some delicate balance between the two.
also
This paper focuses on uncovering other important factors, beyond aligument and uniformity, that contribute to the success story of contrastive learning in SRL and explain the performance gap between the training scheme using the contrastive loss and those using a decoupled contrastive loss (in terms of alignment and uniformity or their equivalent). Specifically, we hypothesize that the training dynamics of the standard contrastive learning for SRL plays essential role in its effectiveness. To that end, we decompose the gradient of the contrastive loss into an alignment component and uniformity component and compare the norms of these two components with their counter-parts in training with the decoupled contrastive losses. Interestingly, we observe a distinct "gradient dissipation" phenomenon in training with the standard contrastive loss: the gradient signal quickly drops and vanishes as soon as the negative example is adequately further away from the anchor than the positive example, where the adequacy appears be reflected by rather moderate threshold. Notably such phenomenon does not appear in the training schemes using decoupled contrastive loss, when the negative sample size small. This observation led us to believe that "gradient dissipation" plays an essential role in standard contrastive learning.
To validate this hypothesis, we construct two new loss functions, both capable of inducing "gradient dissipation" in their training dynamics. We test them experimentally and observe that indeed train- ing with both losses gives comparable or even better performance in the STS tasks than the standard contrastive loss. Interestingly. similar to the contrastive loss, training with these new loss functions also eliminates the alleviates the overfitting problem observed in training using decouple contrastive losses. This confirms that gradient dissipation serves key role in the success of contrastive learn- ing for SRI and suggests that properly conditioning the training dynamics is more important than optimizing alignment and uniformity when not many negative examples used.
Along our development, we provide insights as to why gradient dissipation is a desirable property. We also provide additional theoretical justifications on the effectiveness of the new loss functions by showing that they are in fact upper bounds of the standard contrastive loss. Due to page limit, some results, derivations and discussions are presented in Appendix.
2

---

Published as a conference paper at ICLR 2023
2 PRELIMINARY
2.1 CONTRASTIVE LEARNING IN SENTENCE REPRESENTATION LEARNING
Let C be set of sentences. For each Fς E C, we use x to denote an augmented view of z,. With respect to anchor J4, xy is referred to a positive example, and any 2y or z'J (i # i) is referred as a negative example. There an encoder mapping each z; and x to their representations h; and lig, which are vectors in Rd Considering the metric invariance, we usually normalize them to obtain h4 and Rt, which are constrained on the unit hypersphere 5 centered at the origin. InfoNCE Loss (Oord et al., 2018) or "contrastive loss" for anchor 2; is defined by
exp (KTbL/T)
Let log
(I)
(TTT + M exp
( /T
exp
where 7 is the temperature hyperparameter and N represents the number of negative samples. The quality of sentence representations is shown to insensitive to N (Gao et al., 2021). and a small N, such 63 (Gao al., 2021: Zhang et al., 2022b) or smaller (Zhang et al., 2022c) has shown to be the same sufficient as the bigger one.
2.2 THE DECOUPLE VERSION OF CONTRASTIVE Loss
Alignment and uniformity (Wang & Isola, 2020) decoupled from contrastive loss shown to be two significant properties. The losses based on the two properties are defined as
Lake (1 Catm ACmiform
(2)
Calign (Jaa) (aa)
E h.pppppppp
[
M11
a> 0
(3)
Cunitorm (f;t) 2 log
E fid (hakk))-Pu
[e-1
E>0
(4)
where a, and À three hyperparameters. the meantime, we follow another work (Yeh al., 2021) on decoupled contrastive learning, which removes the positive sample part from the denomi- nator of contrastive loss:
(TTh/TT exp
Lasl = - llgg
-KT log
Σ exp (hT (KE /T JJ
(5)
En exp
(1F (1/ /T
aligament
uniformnly
Note that the alignment parts in the two works recognized equivalent (Ych et al., 2021) and we proved the uniformity part of them have the same lower bound (Appendix D), which corresponds to the optimization objective of Minimum Energy Problem the hypersphere (Kuijlaars & Saff, 1998; Liu et al., 2018). Therefore, the above two decoupled forms of contrastive loss are treated equally in this paper.
2.3 PERFORMANCE COMPARISON
We compare the performance of contrastive loss and its two decoupled forms based on SimCSE (Gao et al., 2021), and evaluate with seven datasets on semantic textual similarity (STS) tasks and seven downstream classification datasets on transfer (TR) tasks from the SentEval toolkit (Conneau & Kicla, 2018). We report the average Spearman correlation for the STS tasks and the average accuracy for the TR tasks in Table Please refer to Appendix A for experimental details.
As can be seen from the results, we can find that (1) the performance on both STS and TR tasks is better than original pretrained models after optimizing with any one of the three loss functions; (2) the main improvement over the original pretrained models is on the STS tasks, while the im- provement on the TR tasks is relatively weak; (3) the decoupled forms obtain the same better performance on TR tasks as contrastive loss, but there still have a performance gap between them on STS tasks. These observations reflect to replace contrastive loss on TR tasks, but it also implies that there are some other factors playing a key role in the improvement of STS tasks, which have been neglected in the previous studies.
extent the validity of alignment and uniformity to
3

---

Published as a conference paper at ICLR 2023
Method
BERT
base TR.Avg
BERT
large
RoBERTas
Tabase TR.Avg
RoBERTa
STS.Avg
STS.Avg
TR.Avg
STS.Avg
STS.Avg
TR.Avg
Pretrained
56.70
85.34
54.11
85,52
56.57
83.20
53.90
83.76
Lel 76.04
86.38
76.83
86.14
77.31
85.23
78.52
85.71
Ludon
72.62
87.52
74.58
87.77
72.64
85.09
73.02
85.68
Ldet
71.13
85.18
72.73
87.06
73.18
85.48
72,43
86.05
Daat
75.25
86.95
77.40
87.85
76.33
84.52
75.63
85.57
Empt 77.25
87.56
77.35
87.71
76.42
85,10
78.84
86,51
Liet
78.38
87.94
78.38
87.94
77.38
85.74
78.71
86,42
fmat
77.76
88.65
77.76
88.65
76.95
85.64
78.82
87.06
Table Performance of the optimization objectives studied this paper. All results reported are the average value obtained from three runs. STS.Avg represents the average of Spearman correlation on seven semantic textual similarity datasets. TR.Avg represents the average accuracy on seven downstream classification datasets.
THE INADEQUACY OF OPTIMIZING ALIGNMENT AND UNIFORMITY
3.1 PROBLEM ANALYSIS
To gain deeper understanding for the above performance gap, we record several metrics during the training and evaluation process, the images of which are shown in Figure 1.
Figure Ia shows the Luniform- Calign scatterplot of sentence representations on STS-B (Cer et al., 2017) development set, where the colors represent the average Spearman correlation of the STS tasks. important to note that the scatterplots show that the proposed method is able to achieve better balance between alignment and uniformity than other methods (Gao et al., 2021: Zhang et al., 2022a;b;c; Klein & Nabi, 2022). Indeed, the scatterplot shows that the sentence representations whose (Luniform: Latign) located in the middle of the image perform better than those whose location in the top left or bottom right. However, the performance gap between different loss functions are reflected by the locations in this figure. Specifically, the sentence representations pretrained by La and its decoupled forms can obtain the almost same values of aligument and uniformity, but an obvious performance gap on the STS tasks.
this style are widely used in recent works to
Figure lb shows the numerical variation of Lel. Calign and Luuiform on the training set and devel- opment set during the optimization process with Lel. In terms of trends, all metrics on both the training set and development set first decrease and then remain flat, which can be seen as a reference for normal training process in the study. Figure and 1d show the same metrics during the opti- mization process with Laku and Ldet. Comparing with Figure 1b, the different variation trends the training set and the development set imply some degree overfitting. Specifically, when Laku is applied for optimization, all three metries except Lalign show first decrease and then increase in the development set, while the similar situation occurs in all three metrics when Cast is applied.
3.2 OPTIMIZATION DYNAMICS STUDY
Since Lasu and Las can be treated as equivalent La only when negative samples tend infinity, we need to pay attention to what exactly is different between them when the number of negative samples is small. To gain more insights on their differences, we investigate the gradient property between contrastive loss and its decoupled forms in optimization dynamics.
The gradient of Lei for the anchor hs can be split into terms:
8C.l MM exp (AT (GTTTT) (bt (61--.) I- Ma.
DA exp CosP(ATii (KThG/T) (he -t) I- Mas
Bh: 7 exp S (KTH))) )+ EN exp (RTIGHT) [[hxl
exp (ETRA) + EN exp (1T Ihill
(6)
where 7 is the identity matrix and Mr is the projection matrix hy-
4

---

Published as a conference paper at ICLR 2023
76
Contrasiiee Los8
Angrmnnn Los3
Un formityyy Losa
pretrained
fect Gel
74
7300
0.5
722
Tniern Step
Trareing Stop
Trah Step
(b) Metrics recorded during training via Lel.
70
Contrasivve Losa
gigmmand
Uniformity Loss
Egign
68
trait
66
0.3
Tnarr Stea
Traiin Strp
ring Stee
-64
(c) Metrics recorded during training via Laseu.
0.2
62
Contract
Urilomiiyy
Orar
34V
60
-3.0
-2.5
-2.0
-1.5
Cuniform
(a) Cuniform Lalige scatterplot. The colors of the points represent the aver- age Spearman correlation on STS tasks.
66 12c
O6
CE
Traning
Traine
(d) Metrics recorded during training via Laet-
The difference between contrastive loss and its two decoupled forms in the training and evaluation phase. All above images are plotted based on the optimization process for BERT BERT-as base-
Figure 1:
When training is driven by such a gradient signal, the first term PP0s points from hy h/ pulling the anchor and the positive sample close to optimize alignment, while each Vucs) in the second term points from ht to hú, pushing the anchor away from the negative samples to optimize uniformity. Then the gradient norm on alignment is computed:
E exp (coe (1/177) [6; (.r.)
EVies exp (cos0 //) sin Osrr
Poz
exp (cos /TT) + DN exp (y0
[lk, eexp [cow Gest (r) EN 3.3 exp (con (rje/>)
||
m
where 0r represents the angle between the anchor h and the positive sample R/ and fiy represents the angle between the anchor h and the negative sample h2. Likewise, we can calculate the gradient norm of Calign and Ldel on alignment, which are expressed by |/ Caligull |l and || separately:
0 Calign Ohs
sin (0w (2))2
1i4 cos Ocr ai (0w (2)) sino
| Latignll
||
(8)
[4:
pos
|i cos cos0g
sin Ow |hhlll
(9)
[lbal]
Comparing equation 7, 8 and 9, we find that the gradient directions of these three parts are the same, but a key difference among them is that 0.y' is contained V but not || Laligal| neither in || Il. which leads to a difference in whether the gradient signal is related to Bigr.
Figure 2a visualizes this difference by plotting the heatmaps of || V with OurOiy. The leftmost four plots show the | under the different number of negative samples, while the rightmost two plots show [.Vigg and || under the condition of 64 negative sample size. Observing the leftmost four plots, part (or the area with weak gradient signals) gradually decreases as the number negative sample increases. Therefore, it is conceivable that when the number of negative samples tends to infinity, these images will gradually become identical to the two rightmost images.
||. [/V and |
find that the area of the shaded
Comparing the leftmost plot with the rightmost two plots, we can clearly the difference between them, where the former has a "gradient dissipation" situation that the latter does not have. So let us think what will happen during training. 0zy will increase and 0gr will decrease gradually. For
5

---

Published as a conference paper at ICLR 2023
0
0
By
60'
By
Oy
0
|| (N=644)
||val|| (N=
||va
(N=643)
(N=64*)
[|VCatgll (N=64)
x (N=64)
(a) Comparison among contrastive loss and its decoupled forms in the alignment part of the gradient
102
1.00
13
155
15
275
0.55
t.0
6
030
By
By
By
Byy
By
By
[/Ver-11 (N=64)
|/ (m=0.10)
I/vi (m=0.23)
|| (m=0.80)
|| 0m=00445)
[/V (m=0.15m)
(b) Comparison among the proposed loss functions for validation in the alignment part of the gradient norm.
Figure 2: Gradient norms of contrastive loss and its decoupled forms with respect the align- ment part. The hyll II in all equations are ignored when the images are plotted. All Ogy in one equation treated the same value. All syper-parameters in equations are consistent with the columns corresponding to BERT base in Table 2.
I1 the gradient signals of some anchors will suddenly dissipate at a certain training phase and only the anchors whose Ozy not too larger than 0Fr' will still receive the gradient signals. For Lalinnl || and || I|. since they are not constrained by 0,y', their gradient signals will not vary with the horizontal axis. All anchors except those with absolutely small 0,r will receive the continuous gradient signals during training.
We point out that the above qualitative conclusion does not fail to hold if the angle is replaced with generic distance function p(..-) on the hypersphere. Then the difference on "gradient dissipation" can intuitively described as: "When the number negative samples is small, contrastive loss tries to keep p(hi, (G)) small relative to p(hs, R2) by some value, while its decoupled forms try to keep p(h,, h!) absolutely small". Here, we unthinkingly hypothesize that the "gradient dissipation" key property in the performance gap between contrastive loss and its decoupled forms, and more insights on this hypothesis will be provided Section 5.
Due to the space limitation of the main text, only the gradients related to the alignment part analyzed here. Please refer to Appendix B for the analysis the uniformity part and Appendix C for the derivation of all equations this section and the subsequent ones.
4 EXPERIMENTAL VERIFICATION
4.1 VALIDATION VIA THE INTRODUCTION OF Laet
To validate hypothesis, we try simulate this property with the smallest changes. Then two facts noticed: (1) Edd will gradually become negative during training; (2) the "gradient dissipation" only occurs when 0,y is larger enough with respect to 0w. These two facts inspire us to introduce a ReLU function (Glorot et al., 2011) on top of Lael to force the truncation of the gradient signal provided to the anchor when Laet < 0:
[-MFR// log (EN exp (RTR) /=)
Las = mar
(10)
Likewise, obtain its gradient norm associated with alignment:
sin 0,y'
- log E exp (6T /T >
Poe dart
||,
(11)
0,
A M / log ΣN exp ( /7 5
The leftmost plot of Figure 2b shows | Comparing with the leftmost plot of Figure 2a, find that the images of | and |
dart under the condition of 64 negative sample size.

---

Published as a conference paper at ICLR 2023
almost identical, which proves that the added ReLU function can indeed simulate the property of "gradient dissipation". More importantly, the experimental results in Table demonstrate the effectiveness of det+. which obtains 3%-7% improvement over Lael STS tasks in the different pretrained models.
4.2 VALIDATION VIA THE UPPER BOUNDS FOR Lel
In fact, Ldet+ is introduced not only for the above mentioned observations, but also because Caet+ + log 2 can be proved be the upper bound of Let. Further, we find that another upper bound with the property "gradient dissipation", which can be derived after a second relaxation based on L
L log max (- RTT4/T + log Σ ENSA exp ( (g/T)/ .0) logg facs
(12)
S 2 - 5
1 max
(-14 M4
max 3.3
(6 (bTh))) + log(NV 1).0)
where 7 log((N - 1) can be replace as the margin hyperparameter m. Then we can get new loss function:
Lmpt max (-AT max h. may m.0) JJ
(13)
where the hardest negative sample is selected for optimization. Since this new loss function is very close in form to Triplet Loss (Weinberger & Saul, 2009). we express this loss function as fompt (Minimum dot Product Triplet Loss). To further increase the diversity of gradient norm variations, we replace the dot product with the Euclidean distance to obtain Lieet (Minimum Fuclidean distance Triplet Loss) and the angle to obtain mas (Minimum Angle Triplet Loss):
Lmot max [|:: Killa
min [h: Mýlla + m2 0}
(14)
33
Lima max
min0.. m,
(15)
3.3
Their performance on STS tasks and TR tasks are reported in Table and find that the performance of these loss functions ahead of the decoupled forms of contrastive loss, and is comparable to or better than those of contrastive loss. Likewise, the gradient norm variations associated with alignment, | [|. Weelll and || Vanalll are plotted in Figure 4c. As among the gradient norms variations due to the choice of different distance functions, but it does not bring significant difference their performance, which proves that it is the "gradient dissipation" property and not a specific distance function that is at work.
see, there are some difference
4.3 MECHANISTIC ANALYSIS OF GRADIENT DISSIPATION
To gain a better understanding of the property of "gradient dissipation", we develop interest in Cmpt because the degree of "gradient dissipation" can be adjusted by the only parameter (Figure 2b) and independent of the number of negative samples. With this more flexible form, how the margin value effects the performance can be quantitatively observed. Recalling that the "gradient dissipation" occurs when dij. i.e. |p(h, ht) min p(h;, K(] goes outside certain range and we can obtain different final di by selecting different m during training. Figure 3a shows the performance on STS tasks and TR tasks with different dij. It can be clearly observed that the performance of STS tasks are much more sensitive to di than that of TR task. Specifically, too large or too small dı will cause the performance of the STS task to drop sharply. In contrast, the performance on TR tasks does not vary significantly with dij. which seems to explain to some extent why the decoupled forms can work well in VRL. Surprisingly, no trade-offs are needed to guarantee the good performance on STS and TR tasks and suitable margin value helps to achieve a double- best performance for both STS and TR tasks, which indicates that sentence representation quality can achieve both intrinsic and extrinsic excellence.
For a more in-depth look, we set m to 0.10, 0.23 (corresponding the best performance) and 0.80 respectively, and record the numerical variation of Le), Latign and Luniform the training and development sets. Comparing Figure 3c and 3d first, we find that La can drop to lower on both training and validation sets when is 0.23, and this difference is magnified by the Luuiforms while Lalign remains almost constant. This observation is consistent with our intuition: when m is too small, the gradient signals may be weak at the beginning of the training, leading no improvement
7

---

Published as a conference paper at ICLR 2023
90
Contrait ve Losa
Al gramenn Loss
Uniiomm Lons
Spearman Correlation
60
80
Accurscy
730
trai"
70-
60
5vg.Avg
50
TR.
50
tranein 50ea
Train ng Stop
Traning
(c) Metrics recorded during training via Empe (m = 0.10).
0.8
0.9
1.0
1.1
1.2
1.3
Contrasiive Loss
grimend Leas
Un farmalll Loss
(a) Effects of di performance.
tan
train
1.1
Distance Gap
0.9
"nasiin Step
Traini Stop
Traning Step
0.8
(d) Metrics recorded during training via Lmpt (m = 0.23).
Get
Cori
0.7
Case
Ciree
0.6
Edd
Contract
Urilomiiy
4t
Training Step
12c
162
24V
(b) The average di record during training with different loss functions adopted for training.
Traning
Ec
12c
Traini Stop
CE
(c) Metrics recorded during training via Lamps (m 0.80).
The plots of the loss functions proposed in this work. dj in (a) and (b) presents Euclidean distance. We first calculate the average distance of all negative samples the anchor, then subtract the distance of positive samples to anchor from it to get the distance gap, and finally plot the average distance gap of all anchor in the mini-batch in the figure. All above images are plotted based on the optimization process for BERT base-
Figure 3:
in the uniformity of the representation vectors. Then turn the attention to Figure 3e, which plots the same metrics when m is 0.80, the curves as a whole is highly similar to Figure 1d, with the same overfitting phenomenon. We think it can be interpreted an excessively large margin leads the optimization process exceptionally difficult, preventing the "gradient dissipation" phenomenon from occurring during training.
To summarize, the above phenomenons further illustrate that the timing of gradient dissipation is very important; too early "gradient dissipation" will result in under-tuned models, while too late "gradient dissipation" will result in degraded model performance and overfitting during training. We further study the connection between Lmpt and the Mixup- based approaches (Kalantidis al., 2020; Zhang et al., 2022b) in Appendix E.
5 DISCUSSION
In this section, share some thoughts and observations to explain why we believe that the "gradient dissipation" plays a key role in SRL. We start by considering it in the relation to the STS tasks. For consistency with the retrieval scenario in practice, the STS tasks care about the orders among the semantic similarity of the anchors rather than the specific values among them (Reimers & Gurevych, 2019). In the ideal case, we should let the samples closer to the anchor if they are more semantically similar to the anchor, and no need to care about their absolute distance to the anchor. Intuitively, the property of "gradient dissipation" only works to puts negative samples further away from the anchor compared to positive samples, which is consistent with the need for STS tasks. On the other hand, alignment and uniformity only describe the relation between the anchor and its samples, while ignoring the relations between the positive samples and the negative samples. In other words, this optimization form overemphasizes the characteristics of the individual sentence itself and ignores the actual connections between the semanties. In early exploratory trials, we find that day is continuously enlarged when the decouple forms are adopted for training (Figure 3b), which is far beyond the level when trained by contrastive loss. Considering the limitations of data augmentation to generate
8

---

Published as a conference paper at ICLR 2023
positive samples and the noise in sampling negative samples, we suspect that the large distance gap brought by the decouple forms compromises the original semantic information of the pretrained models to some extent.
Combining on the above thoughts and observations, we hypothesize that the "gradient dissipation" property is responsible for the better performance in the STS evaluation.
6 RELATED WORK
Sentence representation learning (SRL.) (Kiros et al., 2015; Conneau et al., 2017; Reimers & Gurevych, 2019; Li et al., 2020) of the fundamental tasks in NLP,, aiming at learning semanti- cally rich high-dimensional representation at the sentence level. The good sentence representations need satisfy both (1) high correlation with human-rated similarities (intrinsic evaluation) and (2) good transferability (extrinsic evaluation) (Chiu et al., 2016; Faruqui et al., 2016).
The pretrained language models (Devlin et al., 2018; Liu et al., 2019) were once regraded as the source for obtaining universal sentence representations, and the representations obtained by pre- trained models have shown extraordinary performance on tranfer tasks. However, these representa- tions obtained from the pretrained models even get lower performance than the average Glove em- beddings (Pennington et al., 2014) on semantic textual similarity (STS) tasks (Reimers & Gurevych, 2019). Then more studies have found the word representation space of pretrained models, such as BERT (Devlin et al., 2018) and ELMo (Peters et al., 2018), are anisotropic (Ethayarajh, 2019), where word embeddings are concentrated
high-dimensional conical space (Gao et al., 2018).
Early works (Li et al., 2020; Su et al., 2021) try to diminish the anisotropy of the pretrained repre- sentation space using the whitening transformation (Su et al., 2021) or the flow function (Li et al., 2020). These approaches are easy to implement, but limited improvement for STS tasks. Recently, contrastive learning methods based on instance discrimination (Zhang al., 2022a;b;c) are intro- duced to SRL. However, great deal of works (Yan et al., 2021: Gao et al., 2021; Zhou et al., 2022; Zhang et al., 2022b) focus on how obtain or generate better positive and negative samples. At the same time, we find that even though there are significant differences in current contrastive learn- ing methods in VRL and SRL, such as differences in pretraining and fine tuning, and differences in negative sample sizes (see Section for details), few works have dabbled in the optimization mechanisms of contrastive learning in SRL, Instead, large number of findings from VRL., such as aligument and uniformity (Wang & Isola, 2020; Gao et al., 2021), momentum updates (He et al., 2020; Wu et al., 2021) and bootstrap (Grill et al., 2020; Cao et al., 2022), have been directly applied.
Although there are a large number of works (Gao et al., 2021: Zhang al., 2022b;c;;; Klein & Nabi, 2022) in SRL that use alignment and uniformity as evaluation metrics, to our knowledge, this work is the first work to study their inadequacy as the optimization objectives in SRL.
7 CONCLUSION
In this paper, we focus on the performance gap between contrastive loss and its decoupled forms, i.e. alignment and uniformity in SRL. Our series of new findings contribute deeper understand- ing of how contrastive loss can improve the quality of sentence representation on STS tasks: (1) Alignment and Uniformity Loss or their equivalent are not suitable as alternative loss functions for contrastive loss in SRL due to their lower performance and overfitting problem; (2) The "gradient dissipation" property of contrastive loss under a small number of negative samples plays key role in the performance on STS tasks and preventing overfitting; (3) The "gradient dissipation" prop- and megoriveeeetive samples,
erty works to control a suitable distance gap between anchor-posite while alignment and uniformity are the properties that control the absolute large distance of them; (4) Other two loss functions with "gradient dissipation" property also can solve the overfitting prob- lem, and obtain the same or even better performance than contrastive loss, even if their "gradient dissipation" properties are not sensitive to the number of negative samples. We hope these findings to build a better understanding the key properties of contrastive loss in SRL. Further, some new loss functions that are not bound to the form of contrastive loss can be designed via these propertics, improving the quality of the sentence representation synthetically.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work was supported in part by the National Key R&D Program of China under Grant 2021ZD01 10700, in part by the Fundamental Research Funds for the Central Universities, and in part by the State Key Laboratory of Software Development Environment.
REFERENCES
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. Semeval-2012 task 6: A pilot on semantic textual similarity. In * SEM 2012: The First Joint Conference Lexical and Computa- tional Semantics-Volle Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop Semantic Evaluation (SemEval 2012), pp- 385-393, 2012.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor LonzzalzzAAgiree and Weiwei Guo. sem 2013 shared task: Semantic textual similarity. In Second joint conference lexical and computational se- mantics (* SEM), volume I: proceedings of the Main conference and the shared task: semantic textual similarity, pp. 32 43, 2013.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor González-Agire, Wei- wei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. Semeval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop Semantic Eval- uation (SemEval 2014), pp. 81-91, 2014.
Encko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez- Agirre, Wei- wei Guo, Inigo Jopez-Gazpio Montse Maritxalar, Rada Mihalcea, et al. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pp. 252-263, 2015.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor González-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. Semeval-2016 task Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop Semantic Evaluation (Semfival-2016), pp. 497-511, 2016.
Rui Cao, Yihao Wang, Yuxin Liang. Ling Gao, Jie Zheng, Jie Ren, and Zheng Wang. Exploring the impact of negative samples of contrastive learning: A case study of sentence embedding. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 3138-3152, 2022.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-201) 2017), pp. 1-14, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Billy Chiu, Anna Korhonen, and Sampo Pyysalo. Intrinsic evaluation of word vectors fails predict extrinsic performance. In Proceddings of the 1st workshop on evaluating vector-space represen- tations for NLP, pp. 1 1-6, 2016.
Alexis Conneau and Douwe Kiela. Senteval: evaluation toolkit for universal sentence represen- tations. arXiv preprint arXiv:1803.05445 2018.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 670-680, 2017.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018.
William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop Paraphrasing (IWP2005). 2005.
10

---

Published as a conference paper at ICLR 2023
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geom- etry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP--JCNNIP), pp. 55-65, 2019.
Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, and Chris Dyer. Problems with evaluation of word embeddings using word similarity tasks. ACL 2016, pp. 30, 2016.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations, 2018.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. EMNLP. 2021.
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. Declutr: Deep contrastive learning for un- supervised textual representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 Ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 879-895, 2021.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 315-323, JMLR Workshop and Conference Proceedings, 2011.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pp. 9726-9735 IEEE Computer Society, 2020.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 168-177, 2004.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances Neural Information Processing Systems, 33:21798 21809, 2020.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. Advances in neural information processing systems, 28, 2015.
Tassilo Klein and Moin Nabi. SCD: Self-contrastive decorrelation of sentence embeddings. Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 394-400, Dublin, Ireland, May 2022. Association for Computational Lin- guistics. doi: 10. 1cl acl-short. 44.
URL ht http///caaaatttoggggoooo org/2022.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Arno Kuijlaars and E Saff. Asymptotics for minimal discrete energy on the sphere. Transactions of the American Mathematical Sociery, 350(2):523-538, 1998.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence em- beddings from pre-trained language models. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9119-9130) 2020.
Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. Advances in neural information processing systems, 31, 2018.
11

---

Published as a conference paper at ICLR 2023
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1r7.v:190..11692, 2019.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A sick cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference Language Resources and Evaluation (LREC' pp. 216-223,20 2014.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv:1807.0333 03748, 2018.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pp. 271-278, 2004.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL, (05), pp. 115-124, 2005.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543 2014.
Matthew E. Peters, Mark Neumann, Mohit Iyyer. Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Papers), pp. 2227-22-222 2227, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 1.118653/1/N18-1200 URL https:/ httt::://acllanhhlloggooo wtt.llogo..oooo11o1//1
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-
IJCNLP). pp. 3982-3992, 2019.
Nils Reimers, Philip Beyer, and Iryna Gurevych. Task-oriented intrinsic evaluation of semantic textual similarity. In Proceedings of COLING 2016, the 26th International Conference on Com- putational Linguistics: Technical Papers, pp. 87-96, 2016.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211225) 2015.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103. 15316,, 2021.
Joseph John Thomson. Xxiv. on the structure of the atom: an investigation of the stability and periods of oscillation of a number of corpuscles arranged at equal intervals around the cireumference of a circle; with application of the results to the theory of atomic structure. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 7(39):2372265, 1904.
Ellen M Voorhees and Dawn M Tice. Building question answering test collection. In Proceed- ings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 200-207, 2000.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9939-9939. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2023
Kilian Q Weinberger and Lawrence K Saul. Distance metric learning for large margin nearest neigh- bor classification. Journal of Machine Learning Research, 10:207-2444 2244, 2009.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2):165-21) -10, 2005.
Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu, Esim- cse: Enhanced sample building method for contrastive learning of unsupervised sentence embed- ding. arXiv preprint oxXiv2209.043000 2021.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. Consert: A contrastive framework for self-supervised sentence representation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5065-5075, 2021.
Chun-Hsiao Yeh, Cheng-Yao Hong. Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning, arXiv preprint (.Kiv:2110006844 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Miaoran Zhang, Marius Mosbach, David Adelani, Michael Hedderich, and Dietrich Klakow. MCSE:
Multimodal contrastive learning of sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pp. 5959-5969, Seattle, United States, July 2022a. Association for Computa- 436. URL https: http:::s///aalatthllloooooooooo
tional Linguistics. doi: oxq/2022. naacl mmani 4.6.
Yanzhao Zhang, Richong Zhang, Samuel Mensah, Xudong Liu, and Yongyi Mao. Unsupervised sentence representation via contrastive learning with mixing negatives. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 8220-8230, 2022b.
Yuhao Zhang, Hongji Zhu, Yongliang Wang, Nan Xu, Xiaobo Li, and Binqiang Zhao. A contrastive framework for learning sentence representations from pairwise and triple-wise perspective in an- gular space. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 4892-4903, 2022c
Kun Zhou, Beichen Zhang, Wayne Xin Zhao, and Ji-Rong Wen. Debiased contrastive learning of unsupervised sentence representations. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6120-6130, 2022.
A EXPERIMENT DETAILS
A. EXPERIMENT SETUP
Following recent works in SRL (Gao et al., 2021: Yan et al., 2021; Zhang et al., 2022b), we use the SentEval toolkit (Conneau & Kiela, 2018) to evaluate the quality of sentence representations.
For intrinsic evaluation, i.e., correlation to human judgments, we use seven Semantic Textual Sim- ilarity (STS) datasets (STS12-16 (Agirre et al., 2012; 2013; 2014; 2015; 2016), STS-B (Cer et al., 2017) and SICK-R (Marelli et al., 2014) in SentEval. For extrinsic evaluation, i.e., transfer ability to downstream tasks, we also select seven downstream task datasets (MR (Pang & Lee, 2005), CR (Hu & Liu, 2004), SUBJ (Pang & Lee, 2004), MPQA (Wiebe et al., 2005), SST-2 (Socher et al., 2013), TREC (Voorhees & Tice, 2000) and MRPC (Dolan & Brockett, 2005)) to verify the migration ability of the sentence representations. We followed the default configuration recommended by SentEval for training and evaluation.
13

---

Published as a conference paper at ICLR 2023
A.2 TRAINING DETAILS
We validate the effectiveness of all loss functions based on the SimCSE (Gao et al., 2021) as it is effective but simple enough, so the performance is not easily influenced by other factors.
Following Gao et al. (2021), we observe the following points during training: (1) No data any STS training is used for training: (2) 1,000.000 sentences sampled from the English Wikipedia* are used for training instead: (3) Spearman correlation on STS-B development set is recorded each 125 steps; (4) The checkpoints corresponding the highest Spearman correlation will be saved for evaluation; (5) The training period is one epoch for all pretrained models.
We implement the codes using Python3.7 and Pytorch1. 12.00 and experiment with the single 32G NVIDIA V100 GPU.
Method
Parameter
BERT base
BERT large
RoBERTapase
RoBERTan Jarge
T
0.05
0.05
0.05
0.05
Le
learning rate batch size
le-5
le-5
le-5
le-5
64
64
128
128
a
2
2
t
Laken
X
0.1
0.1
0.2
0.4
learning rate batch size
le-5
1e-5
1e-5
1e-5
64
64
64
64
0.03
0.01
0.02
0.02
Lde
learning rate
le-5
5e-5
1e-5
le-5
batch size
64
64
64
64
0.17
0.18
0.15
0.17
Laet+
learning rate
3e-5
5e-5
le-5
le-5
batch size
64
64
m7
0.23
0.24
0.29
0.31
Lmpt
learning rate batch size
le-5
1e-5
7e-6
7e-6
64
64
64
64
m
0.45
0.50
0.43
0.45
Cmet
learning rate
le-5
1e-5
7e-6
7e-6
batch size
128
128
128
128
0.15
0.17x
0.14m
0.13m
Cmat
learning rate batch size
le-5
3e-5
7
7e-6
128
128
128
128
Table 2: The parameters corresponding to the best results of the STS tasks, which are also corre- sponding the reported results and the plotted figures in this paper.
A.3 PARAMETER SETTING
For all loss functions, we perform grid search on learning rate =(7e-6, 1e-5, 3e-5, 5e-5) and batch size =644, 128, 256, 512]. For other hyperparameters in every optimization objectives, we first narrow the interval with extensive search, then the grid search is conducted in the following ranges:
Lel and Lael
(0.03, 0.05,
Laku
A (0.1.0.3.0...0....000 0.7,0.9)
weteeraaagg..oooooooooooooooooee for simese.txt
14

---

Published as a conference paper at ICLR 2023
(1.0,2.0,3.0) t e [2] step size is
L
-rE [0.10, 0.20), step size is 0.01
Cmpt
- [0220, 0.35], step size is 0.01
-
Lmmet
m E [0.40, 0.50], step size is 0.01
Lmma
m E {0.107, 0.20T). step size is 0.0l
The optimal parameters are shown in Table 2, which are adopted to report all results and plot to all plots in this paper.
A.4 EVALUATION PROTOCOL
We report the performance of the sentence representation the STS task using the Spearman's 's rank correlation, which has been widely used in recent works. Compared with the Pearson's 's correlation, Spearman's rank correlation focuses relative ranking instead of absolute scores, which is more in line with practical retrieval applications of text similarity matching (Reimers et al., 2016).
B THE ANALYSIS ON UNIFORMITY PART
0.04
0.
8002
0.003
51
B
B/
8y
6p
6g
||
(N=64+)
|| ||
(N=642)
(N=64?)
||
(N=64*)
I|VC/
(N=64)
(N=64)
(a) Comparison among contrastive loss and its decoupled forms in the uniformity part of the gradient norm.
1.3
0.22
0.5
3.5
By
0g
0g
[1963 (N=64)
||
||l
||
(b) The unifor- mity part of the gradient Last.
(c) Comparsion among three loss functions like Tripet Loss in the uniformity part of the gradient norm. The horizontal axis represents the minimum angle between the anchor and the negative samples.
Figure 4: Gradient norms of all the loss functions studied in the main text with respect to the uni- formity part. All plotted images ignore h/||/s|| and reflect only the gradient contribution of a single negative sample to the anchor.
Similar to the main text, we study the uniformity part of the gradient norm of all the loss functions mentioned in the main text, and their formulas are shown below:
neg
exp (cos f,,, /T) sin Buy'
1
(16)
exp (cos (w//T) + Σn exp (cos O [hs]
21 exp (2t cos (0,,) sin sin 1 Σn Eiize exp (21 cos (0:3)) [hal
V cs
uniform
(17)
15

---

Published as a conference paper at ICLR 2023
1 exp (cos Ozyr /r) sin Bug
(18)
Liszs exp (cos 02y/T)) |1b:]]
exp (cos Dey /T) sin O,y
RT/T/T log Σ
exp (i?
/T <
E exp (cos Oeye /T) ||eell
V det
5.3
(19)
RT/T r log Σ exp (12 hg) /T 2
3.344
sin (Dey)
if M + max h ht m
V pos
|/4
(20)
0, BT M4 + max
h?? htg 2 m
3331
cos (0egr/2)
11: Mlla
min |l higle l| < m
V
3.333
(21)
met
0,
[1: M4ll
min
II - killa 2 m
3.33
O,r- min O,y' < m
||all
poe
(22)
maxt
0,
0,2
min0.y' 2 m
3.53
The derivation of all the above equations can be found in Appendix C, and their trends with Oue--Oa3 are plotted in Figure 4. The findings of the uniformity part highly similar to those of the alignment part:
V neg; el
approaches | Lniiforrmll and /
|| as the negative sample size increases.
When the number of negative samples is small, "gradient dissipation" exists for || V ||.
while [ ||
and V do not exist.
Nege exists the similar "gradient dissipation" property of / 1.
dei+
VPos ||. |/ and || also exist the propety of "gradient dissipation", but differs || 7C28 in the trend of the gradient norms.
from
C FORMULA DERIVATION
C.1 THE DERIVATION OF EQUATION
exp (BTRC/T) R Σ En exp
(KTR) hi ais
OLd
id
+
dh,
exp (1Tf6/7)) + EN exp
Th,
1i exp (REFs/T)
(B1 (Ri-R; R;) -- -Mn
(ATRt/T + EN exp (RTRG/T) ||bsl|l
exp
exp hg/TT (Ri h - -h:) 1 Ma
D exp (hTht/T) + Ei exp (bTh)/T) [hall
(23)
ΣN exp (RT (g/hT) (ki- I- M
T
exp (NTiG/T)
ΣN exp (KTTetT) ||bill
1 EN exp [ (g/R//)) (ha- Ri))
1- 1-Ma,
7 exp (itht/t) + Eiyors exp (RThg/T)
1hall
EN muxy
16

---

Published as a conference paper at ICLR 2023
C.2
THE DERIVATION OF EQUATION 7 AND EQUATION 16
ENAA exp (RTB//T) (R: (ha)
I Ma
exp (iffc.T)) + -ENe exp (Rg
|1bell
Exst: exp (R/
(1:) - fes) (I Mani
1
(hf h(/T) ΣEses exp (g/ exp
l|kell
(24)
> 2. exp (cos (cos /T)
|R4 - hos cos Oeg'
T exp (cos 0:2/ /T) + Enses exp (cos (T)
[| ||
Erspe exp (cos Pez/ /iissinnin sin 0se T exp (cos Oes /T) + E 3iss exp (coss 02y//T) l1hal
exp (R/R3/7 (hs- (hs-Ri) R3)) 7 exp (bfht/T) Eist exD (KFRs/T)
neg,
Mun
11hal1
exp (//
(1 -h²) (1 (
(iift/t)= + Σ N مe exp (nT
l|ball
exp
(25)
Ih, cos Oey' - hig
exp (cos / NN exp (cos 0. /T) + 2 5. सi exp (cos 0igs/T)
||A=||
exp (cos Beg' /T) sin Oey'
exp (cos /T) + Egi exp (cos Oigi 02//T) ||te|/|
C.3
THE DERIVATION OF EQUATION 8 AND 17
Lalign
(10: [6]] (
= (2 2h: ht) 8
-2-2h;
(26)
aLalign
(2 -2- -2h2 Ri)
2 (-2ht) 1 MM
Oh:
2
Thall
(27)
a (2 siT
R1
MRL
||Aell
|i4 - hhe cos O.e
&Lalign Ōh,
= ax 2 cos 0.v)
||
= =<(4sin (0wr 2 sin ONv |hill
(28)
= a<(2sin (0w//2) /2) 02 sin Os! ||aa||
L ||
N Σ
Σ exp (- ---lic-- - ng 111))
Cuniform
log N(N - 1)
(29)
2t ΣN sizz exp [ (-ellh-- 1iii -Ril13)
(hi (R:) Mns
8 Luniform ah;
(-ellie- lix RR. nn(l2)
1|hell
ΣN Σnjzi exp
(30)
uniform
17

---

Published as a conference paper at ICLR 2023
- (hi- - h.))
28 exp
VC
uniform
1 Mh;
Σ; 2ises exp -t] [ - -hs 11:)
||/=11
[1... -15--00 Ris) (I Man)
exp (-4t sin (0 /2))
L3.s exp (- 41 sin (0.y /2))
hhsll
(31)
[ī. - hi cos 0:g
2xexp (-41 sin ? (0.F 22)
Σ; Σ exp (- sin (0.gi (2)
||/s||
2t exp (-4t sin (0,, /2) sin Bag'
Σ, Enyges exp (-4t sin 2 (0.g) 22) |/all
C.4 THE DERIVATION OF EQUATION 9, 11, AND 19
0Cad
R1 h1 MR.
Oh:
del
ΣN Sists exp (i(hg//)
(hu - R6)) 1 Mr.
(32)
Σ exp ((TrtT)
23. Hocks
EN
[2g h.)) (1 - Mnd
poa del
||/all
(it) his) - )
(33)
/1//ll
1 sin Ozy
1 exp h kt/TT (RG- hu)) T Mn:
del
- Σxsm exp (h2
[hall
exp (AT (gTRg/T))
(Ri) ha)) (T - Mra)
7
D exp (R/))//)))
[|hlll
(34)
||A - -h cos 0gg
exp (cos /T)
N
Eis exp (cos 0:ge/77
||h.ll
exp (cos Oup' /T) sin Ogy
Diase exp (cos Ozg I
18

---

Published as a conference paper at ICLR 2023
C.5
THE DERIVATION OF EQUATION 12
Let log
exp (RTi//T)) + Exiee exp (h2
1 Σgsee exp (6TR)TT)
= log
exp (KTR2/T)
Σ exp 1.374
(M/R/TT
= loog
+ exp
N
loog
1 exp
N/// + log
Σ
exp
(h
(1/
(35)
< log max
-MT T log
Σ exp
(hTis/T) .0
J.JØti
< log max
-KT + log
(N (N- 1) max exp (n. :0)
3.323
log 2 max
(h max
3:3 (kTRt) /T log((V 1),
(- max (bT + Iog(NN 1),0
log 2 + max
3.33
where the first inequality sign holds due to the following inequality:
f(z)
log(1 exp(x))
log
Σ exp(t)
(36)
(0.x)
< log(2 exp(max(())
log 2 max(x, 0)
C.6
THE DERIVATION OF
poss mpt
pas
V
mps
AND
Consider a generic form of the optimization objective first:
Lmnht max p Rhs. minp (he, Ri) + m
(37)
where p(...) represents function used to calculate the similarity. Then the gradient of Lmde with respect to h; calculated and decomposed:
Op [Ti, h6)
Ominp (he, Ri)
- Mh.
Mns
(he,, Ril - mino (hr < m
Ommet Th,
Oh.
||/2/|
Oh.
3:33
Poeee
Dong
(hi, Ri) - minp (hy big 2 m
0,
(38)
where Vneg is only contributed by the negative sample with the closest distance to the anchor. By specifying p(...) as dot product, 12-morm and angle, we can get the gradient norm of Cmpts Lmet and Lina with respect to the part of alignment respectively:
19

---

Published as a conference paper at ICLR 2023
sin (0.y)
plhe, R() = -if
and
p hus Rt)
minp
(hi, (isshs) < m
|fuil
30 ming jo mino
cos (0:)/2)
p(hi, R4) = - |1ūs -Rill and
p fuc. Rt)
(hi, hig) < m
[hall
V
mdle
(39)
p(hs, K.) = Oey
and
p (he R6)
hi. Rg < m
J.
0,
p hhe k4)
min
hi, h2 2
JU
Similarly, we can derive the gradient norm of each loss function corresponding the part of unifor- mity respectively:
sin (min0.y")
plhi, hi) RT. itg and P (his (is
mino 3.572
(hes K1) m
||a||
cos [mmnn Bgr 12)
3.14/4
p(hi, h1) - -11a,- Rigl1 and p (ha, (hi,hhs)
mino
(hi hig) 2 m
V maat
|hall
P(bs.,t,) Ouy and
[his, (hs,hh)
minp
fu.. Rig) < m
|4:11
3:3
0,
(hi, his
minp
(hg k(g m
(40)
D THE CONNECTION BETWEEN THE TWO DECOUPLED FORMS
In this paper, two important study objectives are two decoupled forms of contrastive loss. The first one is the alignment and uniformity proposed by Wang & Isola (2020):
Latign (J:a) A
[lhe - M11
a >>
(41)
Lunitorm (fit) log
E sad (ha Para
[exp(-elihs - h1 h1]
t>
(42)
And the second one is decoupled contrastive loss (DCL, Laai) proposed by Yeh et al. (2021):
exp
N
Last
log
87 KTR/T log Σ
exp (KTT) /T
(43)
Eni exp (kTT))) /T
where the first term of equation 43 is acknowledged by DCL's authors to be equivalent to equation 41, while the difference between the second term and Cuniform is only the order of the logarithmic function and the first summation operation when the losses are calculated for all samples in the same mini-batch. Their lower bounds can be obtained through Jensen's 's Inequality:
Lunifom
log M(N 1)
exp( =1 K3133]]
M
N
log
Eepp(
-ellis
R3112)
(44)
M
N
Σ
Σ
l|,,
-6,1
M(N
J.
For a mini-bate batch of data, the total loss is the mean value of the loss calculated for each anchor in the batch:
del
log
exp
( K3) /T
(45)
20

---

Published as a conference paper at ICLR 2023
M
N
log(NN 1)
log
exp(h Kg)lT
M 1
-1
2 TM(N - 1)
1
Σ
(46)
M
MT Σ
Σ 3,39
lh. 1hh-hh -2
M 1)
where M is batch size. Comparing equation 44 and 46, their common optimization objective is to maximize the sum of the squares of the Euclidean distance of all pairwise samples. Further, this optimization objective corresponds to a specific form of the minimum energy problem on the hy- persphere (Kuijlaars & Saff, 1998; Liu et al., 2018), which is generalized from traditional Thomson Problem (Thomson, 1904) in physics.
E RELATION WITH MIXUP-BASED METHODS
We focus on class of Mixup-based methods (Kalantidis et al., 2020; Zhang al., 2022b) in con- trastive learning which have similar optimization objectives to mmpt- These methods use Mixup (Zhang et al., 2018) generate hard negative samples for robustness and performance improve- ment, indicating the effeteness in VRL (Kalantidis et al., 2020) and SRL (Zhang et al., 2022b). It should to be noted that such methods are the improvements at the sample level, while do not change the property of "gradient dissipation" in contrastive loss.
Existing works generate mixup negative samples with two methods. The first one is generating by linearly weighting the representations of two hard negative samples (Kalantidis et al., 2020). If we regard h2 the anchor, the mixup negative sample for the anchor can be defined as:
1/g Xi'm + (1 -1-10) (1-1))a, m + i x F
(47)
where Nnn Rr are two hard negative samples and 1/g. represents the mixup hard negative sample and is the weight parameter. The second method is generating by linearly weighting the representations of positive sample and a random negative sample (Kalantidis et al., 2020; Zhang et al., 2022b):
i/ = >hς + (1 - 2iile
k #
(48)
where igg is a random negative sample and X should be less than 0.5 to avoid generating pseudo- negative samples with high probability.
Then we discuss the relation between both two sample generation methods with Lmpt. When M mixup negative samples are added the original contrastive loss, the following inequality holds:
exp [ (R/TR//)
Lmix -log
(hFit/T) + ΣE exp (hT + EN exp (hTit/T))
exp
Σx exp (RTRG/T + EN exp (ATRE/T)
log
1
exp (TT
(49)
log
[ exp
(-ATRC/T) [> exp (KTRt/T) + EN exp (RTTa/T))
log
[1 + exp
(-ATRA/T + log
E> exp (h (hTit/T) + DD exp (hT
E log 2 + max
(-AT R log
(Σ) exp ((TTt + + exp (1/
Then we need to perform a secondary relaxation, and the process needs to be discussed on the following sub-conditions:
21

---

Published as a conference paper at ICLR 2023
Condition 1 If iig is generated using the way in equation 47, the maximum value of the N + M terms the log function must be taken in the first N terms, and the following inequality holds:
(-i, log [ exp (n/ Σx exp ( (iTRk/)) .0)
Lxix I log 2 max
log max <
(-R max [hf (h/ /T log(NN N + M),
(50)
log 2 max
(-h h1 + max [ h -
- log(N + M),0
Condition 2.1 If Mgg is generated using the way in equation 48 and the maximum value of the + M terms in the log function is taken in the first N terms, the inequality 50 still holds.
Condition 2.2 If M4g is generated using the way in equation 48 and the maximum value of the N+M terms the log function is taken in the latter M terms, the another inequality holds:
(- M/T log (E) exp (R/R)/T) EN exp (K (NTRt/T))
Luxx < log 2 max
< log 2 max
(-h Kt/T - max (b. hv) /T log(N + +)),0)
log 2 max
--TTM//T max (AT (AM (1 (1-)) /T) + log(N + M).
(51)
log 2 max
1- Fit h4
1 1-1 X max
(R. + log(NV + M).0)
k.
log 2
max
-h -hThh + max (hT + - log(NV + +),0)
These observations show the connection between Lmpt and the mixup-based contrastive methods: (1) if mixup negative samples generated with the first method, the optimization objective of fmpt will be equated to some extent with that of Lmix by appropriately adjusting m; (2) if mixup negative samples are generated with the second method, Lmix will have two upper bounds with different margins, 7 log([N + M) and - log + M). during training. On the contrary, the margin vn in Lmpt will no longer change during training. Therefore, the optimization objective of Empt can be similar equal to the mixup-based methods by using appropriate m (slightly larger than T log( N MM)) which provides a new perspective to explain the well-performance of Cmpt -lkee loss functions.
22


Reviews:
[{"review_id": "IWs6Nmetew", "reviewer": "Reviewer_qhtV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to investigate the secrets of why contrastive learning for sentence representation learning (SRL) works from the perspective of alignment and uniformity (Wang & Isola, 2020).\n\nWhile alignment & uniformity are usually utilized in the literature as a tool for demonstrating the effectiveness of some SRL approaches, this work attempts to directly optimize sentence embedding models based on the metrics, showing that this way of training results in inferior performance compared to that of typical contrastive learning.\n\nIn addition, the authors claim that one of the reasons why the performance gap exists is that contrastive learning exhibits the \"gradient dissipation\" property (especially when the number of negative examples is relatively small), which is not shown when one trains SRL models directly using alignment & uniformity.\n\nTo back up their claim, the authors propose several revised learning objectives that can simulate the \"gradient dissipation\" property even in the case we do not use the original form of contrastive learning and shows that such loss functions can improve the performance of SRL models.\n\n\n", "summary_of_the_review": "To summarize, although the paper presents an attractive story that reveals a part of the secrets of why contrastive learning is effective in SRL, it is still a little bit unclear whether the claims made by the authors are absolutely technically correct, especially considering some concerns I explained in the above Weaknesses part.", "strength_and_weaknesses": "Strengths\n- This work attempts to directly train SRL models based on alignment & uniformity which are usually used as diagnostic metrics to evaluate the effectiveness of SRL models, and from the trial, the authors reveal some intuitive findings on why contrastive learning works in SRL.\n- The paper presents a series of experiments with the newly proposed learning objectives that persuasively back up the claim that the \"gradient dissipation\" property is a key to the success of contrastive learning for SRL.\n\nWeaknesses\n- Conditioned on Figure 2 (a), it seems that the \"gradient dissipation\" property appears especially in the case where the number of negative examples for contrastive learning is relatively small and that the property vanishes as the number of negative examples increases. Therefore, if the property is truly a key to success for contrastive learning, we would arrive at a weird conclusion that it is better to reduce the number of negative examples, which contradicts the wisdom in the literature that it is almost always better to adopt more negative examples.\n- Although the storyline presented in the work was quite persuasive, it is still unclear whether we can make an assertive conclusion only relying on the information obtained from the gradient norm of models.\n- From Table 1, while the proposed losses ($L_{mpt}, L_{met}$, and $L_{mat}$) are better than the typical contrastive learning ($L_{cl}$) when evaluated on BERT, they do not show any superiority when tested on RoBERTa, which weakens the authors' powers of persuasion.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "EbzWi4k8rR", "reviewer": "Reviewer_DG2P", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper identifies an issue with decomposed contrastive losses: a lack of gradient dissipation, which the paper claims leads to overfitting. Gradient dissipation means once negative examples are sufficiently far, those examples effectively no longer receive gradient. To fix this, a series of margin-based contrastive losses are proposed, with different energy functions: dot product, Euclidean distance, and angular difference. Models are then evaluated on semantic similarity tasks and other classification tasks (detailed in Appendix A.1).", "summary_of_the_review": "I am not convinced that the proposed margin-based contrastive losses are novel, and found multiple papers that I found to be very similar [1,2,3].", "strength_and_weaknesses": "*Weaknesses*\nThe contribution of a margin-based contrastive loss has been previously explored. See the Novelty section below.\nAdditionally, I find it concerning that BERT consistently outperforms RoBERTa in the downstream transfer tasks. Why is this the case?\n\n*Additional questions and comments*\n1. The bolding in Table 1 is not consistent across columns.\n2. Is the reason gradient dissipation helps because it allows the model to focus on more difficult examples, rather than continuing to optimize easy ones which already satisfy the margin constraint?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EHfqxhPvxAY", "reviewer": "Reviewer_gozg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyzes previously proposed contrastive learning methods for image representation learning in the context of learning sentence representations. They consider the standard InfoNCE loss and two \"decoupled\" contrastive losses \"A&U\" (Wang and Isola 2020) and \"DCL\" (Yeh et al 2021) whose losses contain separate terms for alignment (decreasing the distance between an anchor and positive example) and uniformity (increasing the distance between the anchor and negative examples). They first show that the decoupled versions underperform InfoNCE and then present justification using gradients of the loss function with respect to the anchor representation. Their primary argument suggests that decoupled losses keep decreasing the alignment losses even when the negative examples are all far away whereas InfoNCE stops after a point. To counteract these issues, they propose two new losses akin to hinge losses and triplet losses which improve the performance to match the performance or outperform InfoNCE. \n\n-------------------\nScore updated after rebuttal.", "summary_of_the_review": "While the paper shows the inadequacy of previously proposed alternatives to InfoNCE and proposes simple changes to address them however the optimization dynamics study is not entirely convincing of the justification of the introduced new losses. Further the new losses don't seem to improve over the baseline InfoNCE for larger more common underlying models such as RoBERTa. Hence I would not recommend acceptance in current form.", "strength_and_weaknesses": "Strengths\n1. The paper shows interesting analysis and experiments showing the alternatives to InfoNCE proposed in the literature may not be as robust as previously presented, at least not outside of image representation learning. \n2. The proposed new losses dcl+ and mpt are simple changes made to previous losses and still show improvements over baselines.\n3. The experimental setup for different losses is thorough with extensive grid search done on different hyperparameters.\n\nWeaknesses:\n1. The gradient norm discussion in section 3.2 is not very convincing. In eq (6), the gradient wrt h_i is shown as a sum of a \"positive\" term \\nabla^{pos} and a \"negative\" term \\nabla^{neg} both of which have terms related to the anchor representation h_i. All the plots and justifications are given with respect the norms of these individual terms. It's not clear why it even makes sense to analyse the two terms separately given that the entire gradient term contains directions both of which update anchor-positive and anchor-negative relationships not just the infoNCE loss which has it in the \"positive\" term (in addition to the negative term\"). I would prefer to see plots similar to figure (2) for the entire loss, not just individual components. \n\n2. It is not clear entirely clear to me what the contribution of this work is. Is it a new loss function? is it showing that prior decoupled losses are not good for sentence representation learning? or something else? Additionally, the new losses presented in the paper still do not seem to outperform the infoNCE baseline for a larger (and much more widely used) model RoBERTa, so why should they be used? \n\n3. Not too much of a concern but the writing is confusing in certain places (especially 3.2) and for example the paragraph right after eq (5) does not seem to serve a purpose. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BOH1e1obD0", "reviewer": "Reviewer_Rqm3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates contrastive learning in the context of building sentence representations. It finds that the decoupled form that decomposes the contrastive loss into aligment and uniformity components can have a detrimental effect in this setting, due to different training dynamics compared to the InfoNCE loss. These effects can be overcome by truncating the gradients once the negative examples are sufficiently further away compared to the positive example. Crucially, the amount of thresholding is controllable independently of the batch size.", "summary_of_the_review": "Overall my assessment is that despite the very thorough experiments in the paper regarding different loss formulations, ultimately the scope of the paper is narrow compared to typical papers at ICLR.", "strength_and_weaknesses": "Strengths:\n- The paper investigates different variants of contrastive learning in the context of concrete tasks with natural language\n- It is able to identify a problem with the decoupled contrastive learning approach in this setting, and propose a fix that both addresses this problem and restores performance on downstream tasks\n- The paper proposed loss formulations close to the Triplet loss that typically achieve performance comparable or better than the InfoNCE loss\n\nWeaknesses:\n- After reading the paper, I didn't come away with a clear understanding regarding why the investigation into different contrastive losses is important. One way of looking at the situation is that for any given task there will always be some hyperparameter choices that perform better or worse than others, and in this situation the InfoNCE loss works well and the decoupled versions aren't as good. In that view, this key takeaway from the paper would simply be the advice to always use the InfoNCE objective for sentence representation learning. Clearly there is more going on, but I worry that my impression here points to potentially limited impact of the paper, or at least that the paper doesn't sufficiently convey the importance to the reader. For example, some of the past work was dealing with very real computational limitations related to large batch sizes, but that doesn't seem to be at issue here.\n- ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster