DistillCSE: Distilled Contrastive Learning for Sentence Embeddings
Authors: Jiahao Xu
Wei Shao
Lihui Chen
Lemao Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: sentence embedding
self-training
contrastive learning
TL;DR: This paper improves self-training for sentence embeddings by mitigating the issue induced by high variance teacher logits, and achieves the state-of-the-art performance.

Abstract: This paper proposes the DistillCSE framework, which performs contrastive learning under the self-training paradigm with knowledge distillation. The potential advantage of DistillCSE is its self-enhancing feature: using a base model to provide additional supervision signals, a stronger model may be learned through knowledge distillation. However, the vanilla DistillCSE through the standard implementation of knowledge distillation only achieves marginal improvements. The quantitative analyses demonstrate its reason that the standard knowledge distillation exhibits a relatively large variance of the teacher model's logits due to the essence of contrastive learning. To mitigate the issue induced by high variance, this paper accordingly proposed two simple yet effective solutions for knowledge distillation: a Group-P shuffling strategy as an implicit regularization and the averaging logits from multiple teacher components. Experiments on standard benchmarks demonstrate that the proposed DistillCSE outperforms many strong baseline methods and yields a new state-of-the-art performance.

DistilICSE: Distilled Contrastive Learning for Sentence Embeddings
Jiahao Xu'
Wei Shao?
Lihui Chen'
Lemao Liu"
Nanyang Technological University, "City Univeristy of Hong Kong, Tencent Al Lab jiahao004@e ntu. edu. sg cemo--c@ee ctt.a/duuccccitcceeuuuuue 3 iredmondd.i@teenent.cco
elhchen@ntu.cn sd
we
Abstract
et al., 2019) for training by pulling positive samples closer and pushing in-batch negatives apart, lead- ing to state-of-the-ar performance. Subsequently, a plethora of sentence embedding methods has been developed based on the SimCSE framework (Zhang et al., 2020; Yan et al., 2021; Giorgi et al., 2021; Kim et al., 2021; Carlsson et al., 2021; Zhou et al., 2022; Chuang et al., 2022; Clark et al., 2020; Dangovski al., 2021; Zhang et al., 2022; Deng et al., 2019: Xu et al., 2023). Unfortunately, one problem with contrastive learning for sentence em- beddings is that the construction of positive and negative sample pairs is often too simple, making it easy for the model to distinguish between posi- tive and negative pairs. As a result, the model may not learn very informative knowledge, leading to sub-optimal performance (Tian et al., 20206; Wang and Qi. 2022).
This paper proposes the DistilICSE framework, which performs contrastive learning under the self-training paradigm with knowledge distilla- tion. The potential advantage of DistilICSE is its self- oef-ennhnncing feature: using a base model provide additional supervision signals, stronger model may be learned through knowl- edge distillation. However, the vanilla Dis- tilICSE through the standard implementation of knowledge distillation only achieves marginal improvements due to severe overfitting. The further quantitative analyses demonstrate the reason that the standard knowledge distilla- tion exhibits a relatively large variance the teacher model's logits due the essence of con- trastive learning. To mitigate the issue induced by high variance, this paper accordingly pro- posed two simple yet effective solutions for knowledge distillation: a Group-P shuffling strategy as an implicit regularization and the averaging logits from multiple teacher compo- nents. Experiments on standard benchmarks demonstrate that the proposed DistilICSE out- performs many strong baseline methods and yields a new state-of-the-art performance.
To this end, inspired by self-training (Yarowsky, 1995; scu, 1965), this paper proposes a framework -DistilICSEE which performs contrastive learning under the self-training paradigm with knowledge distillation (Hinton et al., 2015). The advantage of DistilICSE is its self-enhaneing feature: using base model to provide additional supervision signals, stronger model can be learned through knowledge distillation. Specifically, our framework can be divided into three steps ($2): First, learns a base model as a teacher using standard contrastive Icarning: Second, it learn a stronger student model through knowledge distillation (Gao et al., 2023); Thrid, it iteratively repeats the process of knowl- edge distillation by treating the student model as a teacher.
1 Introduction
Sentence embedding aims to encode the sentence's semantic information from a discrete language space to continuous dense vectors which preserves the semantic meaning of the original sentences. It plays a central role in many downstream NLP tasks, for instance, document summarization (Gao al, 2019), corpus mining (Bennani-Smires et al., 2018), and machine translation (Wang et al.. 2017). Recently, with the emergence of contrastive learning (CL), SimCSE (Gao et al., 2021) pioneers the current mainstream approaches of sentence em- beddings. It combines CL with pre-trained lan- guage models (PLMs) (Devlin et al., 2019; Liu
However, it is far from easy to put DistilICSE into practice: our preliminary experiment shows that the vanilla implementation of the proposed framework only achieves marginal improvements. We identify that the vanilla distillation method suf- fers from the severe overfitting on the training cor- pus (See Table 9 later). This motivates us to investi-
"The source code is available at tttps://githut com/ Jishao084/DistillCE.

---

student model has a equal or smaller capacity than gate the in-depth reason why overfitting occurs. One possible intuition is that in the contrastive learning scenario, the logits of the teacher model are defined on a pair of examples; whereas in the usual scenario these logits are defined on a single example. This essential difference may lead to a rel- atively large variance of the teacher model's logits in the contrastive learning scenario. To demonstrate our insight, two metries are designed to quantify the variance of logits: one variance measures the change of logits between training and testing ex- amples which controls over-fitting, and the other measures the change of logits with respect to dif- ferent teacher models. Through our quantitative analysis, it is observed that logits defined on an example pair indeed have much larger variance than those defined on single example ($3.1). To mitigate these two high-variance problems, we re- spectively proposed two simple yet effective solu- tions for the knowledge distillation step: group-p shuffling strategy as an implicit regularization to prevent overfitting issue ($3.2) and the averaging logits from multiple teacher components to reduce the second variance w.r.t different teacher mod- els ($3.3). Experiments on standard benchmarks demonstrate that the proposed DistilICSE outper- forms many strong baseline methods and yields new otate-of-the-art performance ($4).
the teacher model. Self-training via knowledge dis- tillation involves two main steps in the context of sentence embeddings as follows.
Step 1: training teacher model In the initial phase, teacher model is trained for sentence em- beddings. The mainstream method for sentence embeddings is SimCSE (Gao et al., 2021), which incorporates contrastive learning (CL) principles into the learning process. SimCSE maximizes the agreement of samples with its positive instances while pushing away all the in-batch negative exam- ples using the CL loss defined as follows:
b O/T
Eat log
(I)
Σ ==
sim(ha.3)/+
where h; = [(x) represents the embedding of sentence x; generated by the embedding model f()), and h denotes the embedding of the positive instance of 24. The function sim(-.- calculates the cosine similarity between two embeddings.
Step 2: distilling the student model For the sec- ond phase, the student model learns from scratch mimic the output of the teachers. In this paper, we specifically focus on using a homogeneous model structure for both the teacher and student models (i.e. same capacity). To achieve this, we utilize the vanilla knowledge distillation loss (Hinton et al., 2015), which aims to minimize the cross-entropy of the output logits between the teacher and student. Formally, given sentence z; and its correspond- ing set of bntchtch sentence pairs { x distillation minimizes the cross entropy loss be- tween the teacher distribution q and the student distribution according to the following objective:
In summary, our contribution is three-fold:
1. We first pinpoint an important issue about knowledge distillation for contrastive learn- ing: teacher logits exhibit high variance due to the essence of contrastive learning.
2. We propose two methods: group-p shuffling regulation and logit mean pooling, to mitigate the variance on datapoints and variance across distillation teachers respectively.
ldistill
((ts)) logp(sis)
3. Experimental results demonstrate our pro- posed method surpasses the distillation base- line and achieves a new SOTA, which illus- trates the effectiveness of our proposed meth- ods.
1=1 JデİJ=1
p(sis) = Σk 0
(2)
elus/ee
a(tin) = EN E claa/
2 DistilICSE for Sentence Embeddings
Here, Siyj and füj represent the cosine similarity logits of sentence pairs calculated by the student and teacher models, respectively. тa and 7; denote the corresponding temperatures for the student and teacher models, and N denotes the number of sen- tences within a mini-batch.
Self-training utilizes a trained model generate synthetic labels for examples and trains a student model from scratch (Yarowsky, 1995; scu, 1965). This approach is can be implemented by knowl- edge distillation (Hinton et al., 2015) when the

---

In general, the student model is commonly trained jointly with the teacher model's training objective. Hence, the objective for self-training on sentence embeddings combines both objectives using a trade-off factor A:
logits (i.e. each element's magnitude of embedding vector h2) are the Ist-order logits with respect the random data sample.
However, in sentence embedding settings, we uti- lize the cosine similarity 4.j for distillation, which is production of sentence embeddings from sen- tence pair æt): 44.j h hy (assuming both h; and hy are l2 normalized). Therefore, the magni- tude of logits 4.5 depends on both æ; and æj, and thereby tuy represents the 2nd-order logits with re- spect to the random data sample. However, since (Ig, æyy) is randomly paired during the training pro- cess, this randomness induces two potential issues.
f let + Aedistill
(3)
Iterative self-training Since single-round self- training enhances the student model performance beyond the baseline teacher, natural assumption is employ the student as the teacher for the next round of distillation. Consequently, the next-round student could be further improved, which is widely supported by almost all existing findings (Allen- Zhu and Li, 2023; Mobahi et al., 2020). More specifically, iterative self-training employs the r-th round student the + 1-th round teacher:
Variance of logits w.r.f data points The testing set of STS tasks includes a wide range of sentence pairs with varying degrees of semantic relationship. ranging from strong to weak. However, training samples are randomly gathered from the corpus to form batch, and thereby the sentences within each batch are completely unrelated. In simpler terms, the level of similarity between the unsuper- vised training corpus and the testing set may differ significantly from each other.
(1ag) x p² (81.)
(4)
Vanilla distillation tends to overfitting How- ever, our preliminary experiments on vanilla dis- tillation does not obtain significant improvements, which is in line with the finding in Gao et al. (2023). In particular, our further analysis shows that there is a large gap between the loss of the student model on the training and testing sets (See Table 9), which indicates the student model overfits the training cor- pus. Therefore, we first explore the cause of the overfitting issue and try to tackle such problem in the following sections.
To demonstrate such distinction, we compare the sample similarity logits within a batch and the sim- ilarity logits within STS-B development set. We assess the magnitude of the logit values by sort- ing them in descending order, as shown in Fig. 1. The distribution of Ist-order embeddings appears similar between the training and testing sets while the 2nd-order similarity logit distribution differs significantly: logits in training sentences exhibit greater concentration, whereas the logits for testing sentences are more moderate and uniform.
3 High Variance Issues and Solutions
this section, we will first point out the high vari- ance from the logits that causes the overfitting for knowledge distillation in contrastive learning (i.c., Step 2 in DistilICSE framework) and then propose two simple yet effective solutions to mitigate such issues.
We also quantify such difference in Table 1 by comparing the KL divergence between the training and testing sets. The results reveal that there is not much difference in the 1st-order embedding log- its, whereas the 2nd-order similarity logits display significant variation. Consequently, there is a con- siderable distribution discrepancy in the similarity logits L.j. which may lead to overfitting.
3.1 Issues about High Variance of Logits
Conventional self-training involves utilizing teacher information through the equation: p(u1 - e h, /Ex en0g bi, where Wh represents an entry of learnable parameter matrix W, and yı is the task related label. For instance, in language generation tasks, W is the vocabulary embedding matrix and y1 is the token, and in classification tasks, W represents the weighting matrix and y is the label in the classifier. Since the embedding hi is solely related to a single data example F;, the
Variance of logits w.r.t teacher models Variance may also comes from the teacher model. Since 4.j represents the second-order logits for data samples, its variance theoretically corresponds to the sec- ond order of the variance in embeddings. In other words, the variance from teacher embeddings is magnified when transformed into similarity logits. Table 2 illustrates that the variance in similarity

---

110"4
training of the student model. Therefore, we try to improve the learning process by proposing two strategies to mitigate the issue.
3.2 Regularization by Shuffling Logits
230
100
930
500
1900
1300
1400
0.
For the first issue, inspired by the dropout (Srivas- tava et al., 2014) technique where dropout noise is introduced on the training set to prevent overfitting, we could introduce an outsourced noise into distil- lation logits to alleviate the overfitting issue caused by high variance. Therefore, a possible solution is to shuffle the teacher logits at certain interval, which introduces shuffling noise. This approach helps mitigate overfitting within the interval while preserving valuable information outside such an interval.
0.4
320
003
600
1100
Figure 1: Sharpness of logits between in- batch sen-
tences and STS-B development set.
t SEinh.
4.J
Datapoint
mean
std.
mean
std.
Training
-7.43E-04
5.25B-05
2.67E-01
6,40B-02
Testing
-7.29E-04
4.61E-05
6.96E-01
1.81E-01
We conducted series of experiments to ver- ify our hypothesis. Specifically, we categorized the logits into five groups by their magnitudes (i.e. model confidence), and for each group, we ran- domly shuffled the teacher logits during the distil- lation process. The experimental results are pre- sented in Table 3. It demonstrates that shuffling a subset of the teacher logits effectively addresses the overfitting problem and consequently enhances the performance during testing, especially when the top 12 logits are shuffled. Such results provide evidence that shuffling logits is an effective strategy for mitigating the overfitting issue.
div.
0.0037
3.8794
Table 1: KL divergence on datapoints between training and testing corpus separately measured embeddings and logits. d represents embedding dimension and D is the total number of embedding dimensions. (i.e. for base model D=768)
logits is significantly greater than the variance in sentence embeddings. Additionally, the differential entropy for similarity logits is even higher com- pared to embeddings. Moreover, we analyze the impact of this magnified variance: we consider the top 12 logits and calculate their Spearman score across teachers. We find that the Spearman score only 48.56% for 16 SimCSE teachers across 100 mini-batches. This clearly shows that the variance in embeddings severely disrupts the similarity log- its across teachers.
Group-P shuffting To further refine the shuffling regulation, we propose Group-P shuffling strategy, which adaptively divides logits into groups and conducts shuffling within each group. Formally, given a sequence of similarity logits for in-batch sentence pairs T; = {tt, the logits probability, and its corresponding sorted cumulative probability distribution G((z))::
{ ドー= we first compute
In conclusion, the variance comes from per- spectives: variance data points and variance across teachers. In other words, the predicted scores for different input samples and teachers vary significantly, which can have implications for the
chick
G(ta))= Σ 12226,
(5)
EiaLy cha
where, the cumulative distribution G(tgg) the summation of all logits probability that are larger than tz.j. Then, we divide by probability inter- val i.e. {p, 2p, ... 1}. and finally we randomly shuffle the similarity logits, whose G(tr)) is within the same interval,
Variable
std.
Diff. Entropy
1st order 2nd order
1. EbD
h.a
2.177c-5
-9.3160
lis
0.0406
-1.7853
Table 2: Average standard deviation and differential en- tropy for embeddings and similarity logits fr.j across 16 single teachers. 1: Differential entropy can be negative since the logarithm of the PDF can be negative, result- ing in negative value for the integral.
(6) where p the hyperparameter to trade-off the group size. Large increase group size for shuf- fling and prevents the overfitting to teacher logits,
f. Shuffle (G(Ia))),p)

---

Methods
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
SimCSE + Vanilla Distill
70.85
83.49
74.84
81,52
78.19
78.60
71.69
77.03
Shuffle top 1-12
71.70
83.37
75.62
82.28
79.23
79.65
73.09
77.85
Shuffle top 13-24
71.50
83.75
75.34
81.83
78.69
78.77
71.74
77.37
Shuffle top 25-36
71.30
83.67
74.94
81.92
78.23
78.50
72.21
77.25
Shuftle top 37-48
71.51
83.78
75.34
81.82
78.67
78.75
71.66
77.36
Shuffle top 49-63
71.23
83.66
74.96
81.97
78.21
78.50
72.17
77.24
Table 3:
Performance of distillation logits using single SimCSE checkpoint under 64 batch size.
while small p reduce the group size and encourage the reliance on the teacher.
Eq. to jointly conduct CL and distillation learning from the SimCSE teacher.
3.3 Averaging Logits from Multiple Teacher Components
Dataset We use the default one million randomly sampled sentences from English Wikipedia for un- supervised training, previous studies (Gao et al., 2021; Chuang et al., 2022; Zhang et al., 2022; Wu et al., 2022) are all conducted on this corpus ! We do not conduct any data selection or sampling strategy during the training.
For the second issue, the logits variance across teacher models could be reduced by multiple teacher components: According to the Central Limit Theorem (Fischer), the M samples' average converges its expectation with 1 1/M variance of its original variance. Therefore, we could simply use the following mean sampling to generate the teacher logits t.y:
Evaluation We evaluate our model on 7 sentence semantic textual similarity (STS) tasks, which in- cludes STS tasks 2012-2016 (Agirre al., 2012), STS Benchmark (Cer al., 2017), and SICK- Relatedness (Marelli et al., 2014). We follow Sim- CSE (Gao et al., 2021) settings of MLP layers and employ MLP on top of [CLS] token representa- tion for training while removing MLP for evalua- tion. We evaluate the model for every 125 updating steps based on the STS-B development set, without any gradient accumulation. And evaluate the best checkpoint at the final evaluation on test sets.
4.
+
(7)
M
where M is the total number of teachers, and [ the output similarity logit from the m-th teacher. In summary, we propose our DistilICSE distilla- tion method: We still employ the Eq. 2 as the distil- lation objective. However, we reduce the variance across teachers by averaging logits from multiple teacher components and regulate the student model by group-p shuffling for teacher logits.
Implement details We conduct the experiments using pre-trained checkpoints from BERT (De- vlin et al., 2019) and RoBERTa (Liu et al., 2019) with Huggingface Transformer (Wolf et al., 2020) framework. We employ the current mainstream CL framework SimCSE train teachers.
4
Experiment
4.1 Setups
Baselines We compare several sentence repre- sentation methods on STS tasks, which include GloVe embeddings (Pennington et al., 2014), Skip- thought (Kiros et al., 2015), BERT embeddings with pooling aggregation (Devlin et al., 2019), BERT-Flow (Li et al., 2020), and BERT- (Su et al., 2021). We also compare with several re- cently proposed CL-based sentence representation methods: ISBERT (Zhang et al., 2020), CT-BERT (Carlsson al., 2021), ConSERT (Yan et al., 2021), together with the current mainstream SimCSE (Gao al., 2021) and current SOTA DiffCSE (Chuang et al., 2022). We also conduct the vanilla distil- lation baseline by ourselves, which leverages the
During the training, the CL temperature т, learn- ing batch size, and maximum sequence length are set to 0.05, 64, and 32 respectively, which are the same as the default SimCSE setting. We train the model for I epoch. The learning rate for the BERT base model is 3e while for the large model is le-5, and - for RoBERTa model. The model optimized by Adam (Kingma and Ba, 2017) op- timizer with default settings without any gradient accumulation or momentum CL strategies.
'https://huggingface https:///uuggingffa...ooaaatttte ctttaaasttectttateesssssssssssss princeton-nlp htwps//oooosiiiiiceees/e/////aii wiki m_r for__iicceeee...meee txt

---

Method
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
GloVe emheddings (avg.)
55.14
70.66
59.73
68.25
63.66
58.02
53.76
61.32
BBRTbase (fasststtaal avg.)
39.70
59.38
49.67
66.03
66.19
53.87
62.06
56.70
DERTTbas--flee
58.40
67.10
60.85
75.16
71.22
68.66
64.47
66.55
BERTbas--hiitennn
57.83
66.90
60.90
75.08
71.31
68.24
63.73
66.28
IS IS-BERTbase
56.77
69.24
61.21
75.23
70.16
69.21
64.25
66.58
CT-BERTaase
61.63
76.80
68.47
77.50
76.48
74.31
69.19
72.05
ConSERT-BERY base
64.64
78.49
69.07
79.72
75.95
73.97
67.31
72.74
DitiCSE-BERR
72.28
84.43
76.47
83.90
80.54
80.59
71.23
78.49
SimCSE-BERTDar
68.40
82.41
74.38
80.91
78.56
76.85
72.23
76.25
DCLR- BAAR-BERTiass
70.81
83.73
75.11
82.56
78.44
78.31
71.59
77.22
ArCCSE-BERT base
72.08
84.27
76.25
82.32
79.54
79.92
72.39
78.11
Monilla-Disiil--BRRRR
70.85
83.49
74.84
81.52
78.19
78.60
71.69
77.03
*DistilICEE-BERE
73.56
84.09
77.39
84.06
80.68
80.86
73.02
79.09
*+Teacher Components
73.14
84.36
77.05
83.64
79.94
80.21
72.15
78.64
*+Group-P Shuffling (p=0.1)
72.39
83.51
75.71
82.97
78.87
79.48
73.24
78.02
DDisillCSE-BERT base (2nd Round)
74.54
84.51
77.67
84.87
80.70
81.48
72.16
79.42
SimCSF-BERT
70.88
84.16
76.43
84.50
79.76
79.26
73.88
78.41
BOL.RBERTiarge
71.87
84.83
77.37
84.70
79.81
79.55
74.19
78.90
ArcCSS-BEKTTior 1arge
73.17
86.19
77.90
84.97
79.43
80.45
73.50
79.37
Vamilla-Distil-BBRRTaaa
72.27
85.56
77.65
84.82
80.36
80.53
75.05
79.46
*DisillCCS--BRIT
75.18
86.32
78.92
85.89
81.18
81.97
75.33
80.68
"DittllCSE-BERTTE large (2nd Round)
75.08
86.64
79.53
86.45
81.29
82.72
76.17
81.13
BinCSE-RoBERTTA.
70.16
81.77
73.24
81.36
80.65
80.22
68.56
76.57
DCLR-RoBERTa...
70.01
83.08
75.09
83.66
81.06
81.86
70.33
77.87
Vanilla- Distill-RoEERTia
71.14
82.49
73.67
81.18
81.58
81.24
68.74
77.15
BaaallcsE-RoBERT..
71.45
83.33
75.53
83.19
82.47
82.38
69.44
78.26
SinCSE-RoBERTanan RoBERTaaarg
72.86
83.99
75.62
84.77
81.80
81.98
71.26
78.90
COLL-RoBERTaaaaa
73.09
84.57
76.13
85.15
81.99
82.35
71.80
79,30
72.96
84.50
76.68
85,41
82.29
82.83
71.89
79.51
DDssilICSE-RoBERTTI DDsiilCSEERRoBERTaaese (2nd Round)
74.86
85.72
78.15
86.42
83.35
84.96
73.20
80.95
73.41
85.89
78.81
86.59
83.96
84.98
74.43
81.15
Table 4: Experimental results on standard Semantic Textual Similarity tasks. Our proposed method is marked
with *** and Vanilla-Distiin are the performance of direct distillation baselines. DistilICSE outperforms Vanilla- Distillation across all types of pre-trained language models with p < 0.005.
4.2 Main Results
orthogonal and their gains could be further com- bined. Finally, we achieve a new SOTA perfor- mance on the standard STS tasks across BERT and RoBERTa backbone.
We conduct the experiments with our proposed DistilICSE method for distillation, and the results shown in Table 4. First, it shows that pro- posed DistilCSSE-BETTees achieves 79.09% av- erage Spearman score on STS tasks, which out- performs the distillation baseline Vanilla-Distill- BERT base by 2.08%, and further surpasses its teacher SimCSE-BERThas by 2.87%. Second, we also separately conduct the experiments for shuffling and teacher components. It shows that both proposed strategies yield better performance compared with the distillation baseline, which fur- ther demonstrates the effectiveness of our proposed method. Third, combining both strategies finally achieves the best performance across all the base- lines, which illustrates that both two strategies are
Discussion on efficiency Since our proposed method involves multiple teachers for distillation. the main computation overhead arises from infer- ring the teachers for in-batch negative similarities. To address this, conduct parallel computation across GPUs. As result, the overall training over- head negligible and the training time is compara- ble to the baseline.
4.3 Ablation Study
Group-P shuffling We search the p value in set (0.05, 0.08, 0.1, 0.12, 0.15) respectively. Table 6 shows the best performance is given by p = 0.1.

---

Method
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
SimCSB Distill
70.85
83.49
74.84
81.52
78.19
78.60
71.69
77.03
+ Teacher Comp.
73.14
84,36
77.05
83.64
79.94
80.21
72.15
78.64
Ensemble Teacher Comp.
68.85
82.46
74.07
81.21
78.95
78.92
70.66
76.45
Table 5: Reducing the variance of teacher logits improves the distillation performance while a simple ensemble of teacher components only achieves comparable performance with baseline SimCSE.
Weightage trade-off' factor X Under the distilla- tion baseline settings, conduct the experiments search the in Eq. 3, and the optimal value is 1.
Avg. Spearman to
Methods
STS-B
S. T.
O.
O.S.
non-shuffle
83.69
98.81
95.49
96.04
shuffle
83.80
98.56
95.92
96.68
p
0.1
0.12
STS-B
83.902
84.09
85.22
83.97
83.94
Table 8: Distillation model Spearman correlation with other models.
A
0.1
0.2
0.5
2
STS-B
83.46
83.46
83.47
83.48
83.43
Table 6: Searching p and Å on STS-B development set.
demonstrates shuffling alleviates the overfitting is- sue. As a consequence, the performance of the shuffling method continues increasing after the non- shuffling method achieves its best performance. Further, we investigate the distilled student checkpoints in Table 8. We compute the Spear- correlation score STS-B development set between the student model and: self teacher (S.T.), which is the teacher model used to distill the student; 2) other teachers (O.T.), which are other SimCSEs not used to distillate the current student; 3) other students (O.S.), which are students distilled from other teachers.
Distillation temperatures Table 7 shows that the distillation performance is robust the distillation temperatures. Hence, we the т, and 7, 0.02 and 0.01 respectively, and fix these temperatures across all the experiments.
T,1 T.
0.05
0.02
0.01
0.05
77.01
77.03
77.07
0.02
76.81
76.81
77.13
0.01
76.37
76.83
77.06
Table Vanilla- Vanilla-Distill formance on STS tasks with different distillation tem- peratures.
baseline average per-
First, for the S.T. column, the non-shufiling stu- dent has high correlation with its teacher while the shuffling method reduces the correlation. This indicates shuffling prevents the student from over- fitting its own teacher. Second, for the O.T. col- umn, the non-shuffingg has a low correlation while the shuffling obtains high correlation with other
4.4 Empirical Justification on Two Strategies
Teacher components We analyze the perfor- mance from multiple teacher components in Ta- ble 5. It shows that employing teacher compo- nents will result performance increasing to 78.64 Spearman score on average while the ensemble of them only achieves SimCSEs' performance.
Shuffling logits We empirically show that group- p shuffling is a regulation that prevents students overfit to teacher model. Fig. 2 shows the loss curve and development set performance during the distil- lation. It shows that the non-shuffling distillation loss decreases immediately within the first several steps, which implies the model overfits the train- ing corpus. Different from direct distillation, loss for shuffling strategy continues decreasing, which
102
620
1100
Figure 2: Distillation loss and performance curves be- tween shuffling and non-shuffling strategies.

---

Data Split
Vanilla
Group-P
+Logits Avg.
based on the strong baseline of SimCSE (Gao et al., 2021). Under the SimCSE framework, sev- eral studies focus on constructing hard contrastive pairs (Zhang et al., 2020; Yan et al., 2021; Giorgi et al., 2021; Kim et al., 2021). Some studies aim to counter the PLMs bias towards sentence represen- tations (Carlsson et al.. 2021; Zhou et al., 2022), while others introduce more effective CL frame- work (Chuang al., 2022; Clark et al., 2020; Dan- govski et al., 2021; Zhang al., 2022; Xu et al., 2023; Liu al., 2023).
Training 1.3058
2.8715
3.1377
Testing 6,5503
6.4410
6,2712
Table 9: Model's loss value on training and testing set. The large gap between training and testing loss for the vanilla distillation indicates severe overfitting issue. Our proposed two regulations, i.c. Group-P shufffing and Logits Avg., alleviate the overfitting issue the vanilla distillation framework.
teachers, which indicates shuffling is helpful to find the common optimum shared across teachers, and it is robust to the specific choice of teacher. Third, the O.S. column demonstrates distillation baseline has low correlation to others while shuf- fling increases such a correlation, which indicates shuffling is helpful to achieve the global optimum.
The development of sentence embeddings has clear clue: introducing stronger pre-trane pre-training tasks for PLMs along with CL. The reason is model that achieves better language modeling performance (i.e. token-context alignment) usually has a better ability to capture semantic information and thereby leads to better STS task performance. In contrast to the prior work, this paper aims to study self- training with distillation strategy in sentence em- beddings and mainly focuses on the investigation of the factors that affect the model performance. Instead of introducing pre-training tasks for PLMs, we identify the variance from teachers that signifi- cantly affect the learning performance and thereby propose methods to tackle those issues.
Two methods alleviate overfitting To verify the statement that our proposed two methods alleviate students from overfitting, we measure the student loss value on training and testing datasets in Ta- ble 9, respectively. It shows that although vanilla distillation achieves lower training loss, it has higher loss value on the testing set. While our pro- posed Group-p shuffling is able to bring the two loss values closer. This phenomenon shows that introducing noise through group-p p shuffle has great potential to effectively alleviate the overfitting.
5.2 Contrastive Learning
The importance of contrastive learning (CL) has long been recognized. (Chen et al., 2020; Gidaris et al., 2018; Oord et al., 2018; Wu et al., 2018; Tian et al., 2020a). In NLP research fields, CL is introduced into sentence representations (Giorgi et al., 2021; Wu al., 2020), text classification (Fang et al., 2020), information extraction (Qin et al., 2021), machine translations (Pan al., 2021), question answering (Karpukhin et al., 2020) etc.
5 Related Work
5.1 Sentence Embeddings
Early studies for sentence representations inherited the word2vec (Mikolov et al.) ideas: a sentence's 's contexts shares similar semantic information and such information can be captured by predicting a sentence from its surrounding sentences (Kiros et al., 2015; Hill al., 2016; Logeswaran and Lee, 2018). Pagliardini al. (2018) aggregates the gram embeddings using a pooling strategy, which achieves a strong result. With the development of lerge-scale pre-trained language models (Devlin et al., 2019; Liu et al., 2020), sentence represen- tation methods begin to utilize PLMs' strong lan- guage representation ability. For example, Reimers and Gurevych (2019) employs siamese network with PLMs for supervised sentence representation, while Li et al. (2020) and Su al. (2021) apply post-prooet prosessoc on top of PLM's representations. Recent studies on sentence embeddings are
For example, CL has proven its effectiveness on lask-agnostic sentence representations and is further used improve faithfulness and factual- ity to generation and summarization. Shu et al. (2021) design rule-based augmentation method on logic-to-text generation, and Cao and Wang (2021) on faithful and factual consistency. In NLP in- terpretability, Gardner et al. (2020) evaluate local decision boundaries contrast sets, and Jacovi et al. (2021) develop contrastive explanations for classification models. In contrast the prior stud- ies that aim to improve performance through CL,, we mainly focus on the default CL in a self- trainini manner, and it is employed as an additive objective

---

in self-training.
Conclusion
In particular, Gao et al. (2023) study knowl- edge distillation for contrastive learning on sen- tence sembeddings similar to our work. However, our work differs theirs in three major aspects. First, our teacher and student are of the same model size whereas they aims to distill small model from very large model. Second, we analyze the in-depth reason why vanilla distillation does not work well for contrastive sentence embeddings and propose novel methods to make it successful accordingly. Third, our distillation does not required supervised corpus during the training, making ours more gen-
In this paper, propose self-training with the knowledge distillation framework for contrastive sentence embeddings. We identify that vanilla dis- tillation suffers from severe overfitting issue. The reason for this problem lies in the significant vari- ance of the output logits of the base model in self- training, both among data points and across teach- ers. Furthermore, reducing the variance will lead to better student performance. Consequently, we propose group-p shuffling to regulate the variance and mean sampling from multiple teacher compo- nents to reduce the logit variance. Experimental results on standard benchmarks demonstrate the ef- feetiveness of our proposed method, which yields a new state-of-the-art performance.
eral.
5.3 Knowledge Distillation
Knowledge distillation (Hinton et al., 2015) in- volves training compact model, often referred as student model, to mimic the behavior and knowledge of a larger, more complex model known as the teacher model. It has been successfully ap- plied to various tasks, such as language modeling (Zhuang et al., 2021), text classification (Heinzer- ling and Strube, 2018; Chia et al., 2019), named entity recognition (Zhou et al., 2021), machine translation (Tan et al., 2019), language generation (Melas-Kyrial et al., 2019).
Limitations
This paper identifies variance that significantly af- fects the distillation learning performance. For variance on data points, a more effective strategy is needed to regulate such variance in logits; For variance across teachers, a more lightweight strat- egy is needed for teacher components. Besides, the performance of our proposed method could be further improved if more advanced knowledge distillation framework is introduced.
Teacher model knowledge guide students in mul- tiple ways during the distillation. Its predictions or soft targets, attention weights, or hidden repre- sentations, could all be used to guide the training of the student model. Consequently, the student is provided with stronger training signals from the teacher and achieves even superior performance. For example, Zhuang et al. (2021) directly mim- ies the output logits vocabulary while Jiao et al. (2020) utilizes both hidden representations and at- tention matrix.
Ethics Statement
This study focuses on the self-training methods for contrastive learning sentence embeddings. The pro- posed objective and methods aim achieve better performance on general domain tasks. The training corpus randomly sampled from Wikipedia and benehmark datasets are open source. None of them contain any personally sensitive information; For language models, We employ widely applied pre- trained language models, i.e. BERT and RoBERTa, with commonly used contrastive learning and dis- tillation strategies, thereby having no impact the political, social, or natural environment.
Different from those studies, we employ knowl- edge distillation as an element of our self-training framework. Therefore, we focus on the most funda- mental and general form of distillation which only minimize the cross entropy of prediction logits dis- tribution between teacher and students. We use the homogencous structure model for both the student and teacher model for distillation. Our research mainly focuses on the output logit distribution from the teacher instead of a special distillation frame- work. Therefore, our method is generic for more advanced distillation technologies.
References
1965. Probability of error of some adaptive pattern- recognition machines. IEEE Transactions on Infor- mation Theory, 11(3):363-377
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In #SEM 2012:

---

The First Joint Conference on Lexical and Compu- tational Semantics- Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385- 393, Montréal, Canada. Association for Computa- tional Linguisties.
Kevin Clark, Minh-Thang Luong, Quoe Le, and Christopher D Manning. 2020. Electra: Pre- tre-training text encoders as discriminators rather than genera- tors.
Rumen Dangovski, Li Jing, Charlotte Loh, Seung- wook Han, Akash Srivastava, Brian Cheung. Pulkit Agrawal, and Marin Soljačić. 2021. Equivariant con- trastive learning. arXiv preprint wtXiv:2111.00899
Zeyuan Allen-Zhne and Yuanzhi Li. 2023. Towards un- derstanding ensemble, knowledge distillation and self-distillation in deep learning. In The Eleventh International Conference on Learning Representa- tions.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690-4699.
Kamil Bennani-Smires, Claudiu Musat, Andreea Hoss-
mann, Michael Baeriswyl, and Martin Jaggi. 2018. Simple unsupervised keyphrase extraction using sen- tence embeddings. In Proceedings of the 22nd Con- ference Computational Natural Language Learn- ing, pages 221-229
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Shuyang Cao and Lu Wang. 2021. CLIFF: Contrastive
learning for improving faithfulness and factuality in abstractive summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6633-6649, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Hongchao Fang, Sicheng Wang. Meng Zhou, Jiayuan Ding, and Pengtao Xie. 2020. Cert: Contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.1 12766
Fredrik Carlsson, Amaru Cuba Gyllensten, Evan- gelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2021. Semantic re-tuning with contrastive tension. In International Conference on Learning Representations.
Hans Fischer. A history of the central limit theo- rem: from classical to modern probability theory. Springer.
Chaochen Gao, Xing Wu, Peng Wang, Jue Wang. Liangjun Zang, Zhongyuan Wang. and Songlin Hu. 2023. Distilese: Effective knowledge distillation for contrastive sentence embeddings.
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-
Gazpio, and Lucia Specia. 2017. SemEval-2017 task Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1 Itth International Workshop on Semantic Evaluation (SemEval-2011) pages 1-14, Vancouver, Canada, Association for Computational Linguistics.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, 10, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In In- ternational conference on machine learning, pages 1597-1607. PMLR
Yang Gao. Yue Xu, Heyan Huang, Qian Liu, Linjing Wei, and Luyang Liu. 2019. Jointly learning topics in sentence embedding for document summarization. IEEE Transactions on Knowledge and Data Engi- neering, 52(4):688-699
Yew Ken Chia, Sam Witteveen, and Martin Andrews, 2019. Transformer to cnn: Label-scarce distillation for efficient text classification.
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala. Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco. Daniel Khashabi, Kevin Lin, Jiangming Liu, Nel- son F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models' local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo,
Yang Zhang. Shiyu Chang, Marin Soljacic. Shang- Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022.
DifiCSE: Difference-based contrastive learning for sentence embeddings. In Proceedings of the 2022
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4207-4218, Seattle, United States. Association for Computational Lin- guistics.

---

1307-1323, Online. Association for Computational Linguistics.
representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 2528-2540, Online. Association for Computational Linguistics.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. 2018. Unsupervised representation learning by pre- dicting image rotations. In International Conference on Learning Representations.
Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization.
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021. DeCLUTR: Deep contrastive learning for un- supervised textual representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 879-895, Online. Association for Computational Linguisties.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralaa, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre- pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119-9130, Online. Association for Computa- tional Linguistics.
Benjamin Heinzerling and Michael Strube. 2018. BPEmb: Tokenization-free pre-trained subword em- beddings in 275 languages. In Proceedings of the Eleventh International Conference on Language Re- sources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Jiduan Liu, Jiahao Liu, Qifan Wang, Jingang Wang. Wei Wu, Yunsen Xian, Dongyan Zhao, Kai Chen, and Rui Yan. 2023. RankCSE: Unsupervised sen- tence representations learning via learning to rank. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 13785-13802, Toronto, Canada. Association for Computational Linguisties.
Felix Hill, Kyunghyun Cho, and Anna Korhonen.
2016. Learning distributed representations of sen- tences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1367-1377, San Diego California. Association for Computational Linguisties.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean, 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2020. Rerber: robustly optimized (bert) pretraining approach.
Alon Jacovi, Swabha Swayamdipta. Shauli Ravfogel, Yanai Elazar, Yejin Choi, and Yoav Goldberg. 2021. Contrastive explanations for model interpretability. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 1,97-1611, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
I Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence represen- tations. International Conference on Learning Representations.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.
TinyBERT: Distilling BERT for natural language un- derstanding. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 4 4163- 4174, Online. Association for Computational Lin- guistics.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zam- parelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference Language Resources and Evaluation (LREC'14), pages 216-223, Reykjavik, Iceland. European Lan- guage Resources Association (ELRA).
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781. Online. Association for Computational Linguistics.
Luke Melas-Kyriazi, George Han, and Celine Liang, 2019. Generation distillationnn for efficient natural language understanding low-data settings. In Pro- ceedings of the 2nd Workshop on Deep Learning Approaches for Row-Resource NLP (DeepLo 2019), pages 124-131, Hong Kong, China, Association for Computational Linguistics.
Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021. Self-guided contrastive learning for BERT sentence

---

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1955
Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. 2020. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Pro- cessing Systems, 33:3351-3361.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou 2021. Whiteming sentence representations for bet- semantics and faster retrieval. arXiv preprint arXiv:2103.15314
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.03744.
Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tic- Yan Liu. 2019. Multilingual neural machine trans- lation with knowledge distillation. arXiv preprint a902 10461.
Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2018. Unsupervised learning of sentence embed- dings using compositional n- n-gram features. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long Papers), pages 528 5400 New Orleans, Louisiana. Association for Computational Linguis- tics.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020a. Contrastive multiview coding. In European confer- ence on computer vision, pages 776-794. Springer.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. 2020b. What makes for good views for contrastive learning? Ad- vances in neural information processing systems, 33:6827-6839.
Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021. Contrastive learning for many-to-many multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume I: Long Papers), pages 258. Online. Asso- ciation for Computational Linguistics.
Rui Wang, Andrew Finch, Masao Utiyama, and Eiichiro Sumita. 2017. Sentence embedding for neural ma- chine translation domain adaptation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 560-566
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing ( (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Xiao Wang and Guo-Jun Qi. 2022. Contrastive learning with stronger augmentations. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of of-the-atr art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng L.,, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. 2021. ERICA: Improving entity and rela- tion understanding for pre-trained language models via contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3350-3363, Online. Association for Computational Linguistics.
Qiyu Wu, Chongyang Tao, Tao Shen, Can Xu, Xi- ubo Geng, and Daxin Jiang. 2022. Pcl: Peer- contrastive learning with diverse augmentations for unsupervised sentence embeddings. arXiv preprint arXiv:2201. 12093..
Nils Reimers and Iryna Gurevych. 2019, Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018 Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pat- tern recognition, pages 3733-3742.
Chang Shu, Yusen Zhang, Xiangyu Dong, Peng Shi, Tao Yu, and Rui Zhang. 2021. Logic-consistency text generation from semantic parses. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 4414 4426, Online. Association for Computational Linguistics.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. arXiv preprint 2:Xiv:0012.15466

---

Jiahao Xu, Wei Shao, Lihui Chen, and Lemao Liu. 2023. +imCSE++: Improving contrastive learning for sen- tence embeddings from two perspectives. In Proceed- ings of the 2023 Conference Empirical Methods in Natural Language Processing. Association for Com- putational Linguistics.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. ConSERT: con- trastive framework for self-supervised sentence repre- sentation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume I: Long Papers), pages 5065-5075, Online. Association for Computational Linguistics.
David Yarowsky. 1995. Unsupervised word sense dis- ambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pages 189-196.
Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. 2020. An unsupervised sentence embedding method by mutual information maximiza- tion. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1601-1610, Online. Association for Computational Linguistics.
Yuhao Zhang, Hongji Zhu, Yongliang Wang, Nan Xu, Xiaobo Li, and Binqiang Zhao. 2022. A contrastive framework learning sentence representations from pairwise and triple-wise perspective in angular space. In Proceedings of the 60th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pages 4892-4903.
Kun Zhou, Beichen Zhang, Xin Zhao, and Ji-Rong Wen. 2022. Debiased contrastive learning of unsupervised sentence representations. In Proceedings of 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 6120- 6130, Dublin, Ireland. Association Computational Linguistics.
Xuan Zhou, Xiao Zhang, Chenyang Tao, Junya Chen, Bing Xu, Wei Wang, and Jing Xiao. 2021. Multi- grained knowledge distillation for named entity recognition. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computarional Linguisticx: Human Language Technologies, pages 9716-5716.
Honglei Zhuang, Zhen Qin, Shuguang Han, Xuanhui Wang, Mike Bendersky, and Mare Najork. 2021. En- semble distillation for bert-based ranking models. In Proceedings of the 2021 / ACM SIGIR International Conference the Theory of Information Retrieval (ICTIR 21).


Reviews:
[{"review_id": "QQ0q6nIH9x", "reviewer": "Reviewer_jiJV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### R1: problem with vanilla distillcse is not clear. Why not compare RoBERTa or XLNet?\n**[Vanilla distillation tends to overfit]** The main contributions are that: 1) it points out the overfitting issue for vanilla distillation and explains why the overfitting occurs via measuring the variance for simCSE; 2) it proposes simple yet effective methods to address the overfitting and makes the DistillCSE successful. Therefore, the problem with vanilla distillCSE is the overfitting which is further explained through measuring the variance for simCSE. \n\nTo quantify that the vanilla distillCSE suffers from overfitting and our proposed method can alleviate the overfitting, we add new experiments to compare their training loss and testing loss as the following table. \n\n| **Data Splits** | **Vanilla** | **Group-P Shuffling** | **+Logits Avg.** |\n|---|:---:|:---:|:---:|\n| Training Set | 1.3058 | 2.8715 | 3.1377 |\n| Testing Set | 6.5503 | 6.4410 | 6.2712 |\n\n\nAs shown in this table, although the vanilla method has low training loss, its testing loss is higher than the group-p shuffling method, which indicates the overfitting to the training set. Moreover, integrating both group-p shuffling and teacher logits averaging yields the lowest testing loss, The large discrepancies between training and test data samples can be the root cause of overfitting on the training set.  The proposed method is able to bring the two loss values closer, and hence reduce the discrepancies.\n\nConsequently, inspired by the dropout technique where dropout noise is introduced on the training set to prevent overfitting, we introduce a regulation on variance (or variance adjustment) into the training set logits via group shuffling, to address overfitting.\n\nWe did report the performance of RoBERTa in our main experiments in Table 4. Here, we re-summarize our existing results based on both BERT and RoBERTa from Table 4 as follows:\n\n| **Method** | **Avg. STS (confidence interval)** |\n|---|:---:|\n| **SimCSE-BERT-base** | 76.25 (0.1271)* |\n| Vanilla-Distill | 77.03 (0.1165) |\n| *Ours | 79.09 (0.1035)* |\n| *Ours (2nd Round) | 79.42 (0.1035)* |\n| **SimCSE-BERT-large** | 78.41 (0.1164)* |\n| Vanilla-Distill | 79.46 (0.0961) |\n| *Ours | 80.68 (0.1043)* |\n| *Ours (2nd Round) | 81.13 (0.1075)* |\n| **SimCSE-RoBERTa-base** | 76.57 (0.1042)* |\n| Vanilla-Distill | 77.15 (0.1018) |\n| *Ours | 78.26 (0.0969)* |\n| **SimCSE-RoBERT-large** | 78.90 (0.1075)* |\n| Vanilla-Distill | 79.51 (0.0969) |\n| *Ours | 80.95 (0.0888)* |\n| *Ours (2nd Round) | 81.15 (0.0953)* |\n\nWe employ vanilla distillation as the target system for the t-test, and models marked with “*” are significantly different from the vanilla distillation model with p <0.005.\n\nFor XLNet, however, we find that SimCSE-XLNet does not converge under the unsupervised SimCSE framework in our experiments. Specifically, we observe that the initial XLNet performance on the STS-B development set is around 50%, and the performance continues to decrease during the training. Finally, it stabilized at around 20% Spearman score. In contrast to unsupervised settings, the supervised SimCSE on top of XLNet achieves on-par performance with unsupervised SimCSE-BERT model.  In addition, to the best of our knowledge, we did not find any existing papers that successfully implement SimCSE on top of XLNet.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea appears to be innovative, although the actual contribution might be relatively minor. Nevertheless, the authors have tested their model on various tasks, and based on this aspect, I believe their work deserves serious consideration.", "reasons_to_reject": "The problem with vanilla distillcse is not clear. \nThey compared their model with BERT but why not Roberta or XLNet.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "DistillCSE is a sentence embedding learning method that leverages pseudo-labeled data and knowledge distillation. The model is built on three main concepts:\n\nContrastive Learning: This self-supervised learning technique aims to identify similarities and dissimilarities between sentences. Similar sentences are represented close to each other in the embedding space, while dissimilar ones are kept further apart.\n\nSelf-Training Paradigm: The model generates pseudo labels and relearns its weights using both the pseudo labels and correct labels. This allows the utilization of both unlabeled and labeled data.\n\nKnowledge Distillation: In this technique, smaller models are trained to mimic larger models, transferring knowledge from teachers to students in a more compact manner.\n\nAlthough the original form of DistillCSE does not yield the expected results, the authors propose two solutions. The first is Group-p shuffle, which involves grouping similar sentences, shuffling them, and presenting them to the teacher for pseudo labels. The second solution is to take the teacher's logits for similar sentences and average them.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "\n“Second, it learn a stronger student model through knowledge distillation;” It should be “learns”.\n“Thrid” should be Third\n"}, "extra_text": {}}, {"review_id": "MPcCQUcdFy", "reviewer": "Reviewer_JpHQ", "summary": "", "questions": "Questions about Reject 1. Reason.  \n1) Why is it distinguished from many other noising methods for improving over fitting? Especially, are there any specific reasons to categorize logits into confidence-level based groups? To believe that it is safe and understand where to apply it,  we need a deeper explanation  about why the specific inductive bias is positively working beyond just showing its effectiveness in experiments,  ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### R1: How the shuffling solves the variance problem\n**[Problems with vanilla distillation is overfitting]**\nThe goal of this work is to address the overfitting problem in vanilla distillation. We noticed that the existing distillation method suffers from overfitting i.e. low training loss but high testing loss.  Large discrepancies between training and test data samples can be the root cause of overfitting on the training set.  Therefore, inspired by dropout technique where dropout noise is introduced on the training set to prevent overfitting, we introduce a regulation on variance (or variance adjustment) into the training set logits via group shuffling, to address overfitting.\n\nTo support the above statement about overfitting, we measure the loss on the training and testing sets in the table below. It shows that although vanilla distillation achieves lower training loss, it has a higher loss value on the testing set. While our proposed Group-p shuffling is able to bring the two loss values closer.  This phenomenon shows that introducing noise through group-p shuffle has great potential to effectively alleviate the overfitting problem.\n\n| **Data Split** | **Vanilla** | **Group-p Shuffling** | **+Logits Avg.** |\n|---|:---:|:---:|:---:|\n| Training set | 1.3058 | 2.8715 | 3.1377 |\n| Testing set | 6.5503 | 6.4410 | 6.2712 |\n\n\nThe effectiveness of the group-shuffling strategy is also demonstrated and reported in Sec.4.4.  It shows that group shuffling prevents students from overfitting to a particular teacher and encourages students to reach the optimal in global view.\n\n### R2: Contribution\n**[Contribution]**\nThe main contributions are that 1) it points out the overfitting issue for vanilla distillation and explains why the overfitting occurs via measuring the variance for SimCSE; 2) it proposes simple yet effective methods to address the overfitting and finally makes the DistillCSE successful.\n\nPointing out the vanilla DistillCSE which does not work well is easy, but analyzing its failure in-depth and making the improved DistillCSE successful are solid contributions. \n\n### R3: novelty of raised problem, approach and methods are weak\n**[Novelty]** DistillCSE, for the first time, studies the distillation sentence embeddings and provides an in-depth analysis of the high variance issue, which can make a significant impact on distillation performance and does not depend on a specific distillation framework. Therefore, our study is generic and significantly contributes to various types of distillation sentence embedding frameworks in future work.\n\n### Q1: Why shuffling alleviate over fitting?\n**[Group-p Shuffling learns group-level similarity distribution]** The reason is that group-level similarities are more reliable while sentence-level similarities are noisy. This hypothesis is verified in Table 2 where the sentence-level similarities noise std is around 0.04. We provide an example to further illustrate the point: For instance, the similarity distribution of two sentence pairs with similar similarities (e.g. 0.31 and 0.32) are of great noise since std is 0.04, therefore their ground-truth similarities could be 0.31+0.04 and 0.32-0.04, which maybe totally different to the relative scale from teacher’s estimation. Hence **similar similarities** are unreliable due to the noise, and the student tends to overfit to such a noise.  On the contrary, the distribution of sentence pairs with **dissimilar similarities** (e.g. 0.5 and 0.3) is more reliable. Therefore, group-shuffling shuffles the intra-group (similar similarities) distribution while preserving the inter-group (dissimilar similarities) distribution. Consequently, the model learns reliable group-level information while the shuffling noise for similar similarities regulates the overfitting issue.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. providing empirical evidence of  the variance issues of second order similarity distribution of logits. \n2. method is sufficiently effective in the benchmarks\n", "reasons_to_reject": "1. It is not sufficiently demonstrated how the shuffling solves the variance problem. \n2. Proposing a DistillCSE as a framework is difficult to  see as a contribution. (Applying CL to representation learning is well-known regularization. Applying CL to Teacher in Self-training framework via KD is simply implemented.) \n3. the novelty of raised problem, approach and methods are  weak. ", "questions_to_authors": "Questions about Reject 1. Reason.  \n1) Why is it distinguished from many other noising methods for improving over fitting? Especially, are there any specific reasons to categorize logits into confidence-level based groups? To believe that it is safe and understand where to apply it,  we need a deeper explanation  about why the specific inductive bias is positively working beyond just showing its effectiveness in experiments,  ", "paper_topic_and_main_contributions": "problem: In self-training through KD  + Contrastive Learning, high variance of teacher’s logits. \nsolution: Group-P shuffling + teachers’ logit averaging \nresult: Effective in Semantic Textual Similarity benchmarks + alpha on BERT and RoBERTa base models. \n", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XqypiJv6ye", "reviewer": "Reviewer_f9XB", "summary": "", "questions": "On Table 5, which model did you use? Also, the best results don't seem to correspond to any results reported on Table 4, why is this the case? If this is just due to randomness, could you provide std over 5 runs?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### R1: only Bert-base being above the baselines with a large margin; it is not clear if other results are significant; hyper-parameters tuning was done on Bert-base only\n\n**[STD]** We employ vanilla distillation as the target system for the t-test, and models marked with “*” are significantly different from the vanilla distillation model with p <0.005.\n| **Method** | **Avg. STS (confidence interval)** |\n|---|:---:|\n| **SimCSE-BERT-base** | 76.25 (0.1271)* |\n| Vanilla-Distill | 77.03 (0.1165) |\n| *Ours | 79.09 (0.1035)* |\n| *Ours (2nd Round) | 79.42 (0.1035)* |\n| **SimCSE-BERT-large** | 78.41 (0.1164)* |\n| Vanilla-Distill | 79.46 (0.0961) |\n| *Ours | 80.68 (0.1043)* |\n| *Ours (2nd Round) | 81.13 (0.1075)* |\n| **SimCSE-RoBERTa-base** | 76.57 (0.1042)* |\n| Vanilla-Distill | 77.15 (0.1018) |\n| *Ours | 78.26 (0.0969)* |\n| **SimCSE-RoBERT-large** | 78.90 (0.1075)* |\n| Vanilla-Distill | 79.51 (0.0969) |\n| *Ours | 80.95 (0.0888)* |\n| *Ours (2nd Round) | 81.15 (0.0953)* |\n\n**[all types of models outperform the vanilla distillation by p<0.005]**  We provided the performance with t-test confidence interval, it shows that our method on all types of models outperforms both vanilla distillation and SimCSE by a significant margin with respect to the confidence interval value. Specifically, on BERT-large, RoBERTa-base and RoBERTa-large models we increase the performance over vanilla-distill by 1.22, 1.11 and 1.44 respectively, which are substantial margins compared with existing papers.\n\n**[p searching]** Searching specific hyperparameters for each model will surely result in even stronger results for our proposed model. However, we did not do this because the current performance already achieves SOTA performance across models which demonstrates our methods are robust to hyperparameters. Besides, it is a common practice to search the hyperparameters on the BERT-base setting and apply them across all settings in sentence embeddings [1]. Therefore, from a fair comparison perspective, we did not search specific hyperparameters for each model. Tuning the hyperparameters is the last thing to do with the proposed method.\n\n\n\n### Q1: Model used on Table 5.\n\nThe results in Table 5 are surely different from Table 4 because Table 5 are development set results: Table 5 is hyperparameter searching conducted on the STS-B development set, rather than the testing set performance reported in Table 4.Specifically, in Table 5, the p value is searched based on our proposed methods: DistillCSE-BERT-base settings, while the lambda value is searched based on vanilla distillation and we fixed the lambda value across all experiments for a fair comparison between vanilla distillation and our proposed DistillCSE framework. \n\nWe provide the 5 runs’ performance of our DistillCSE framework for the first round distillation mean performance (1st row) with the corresponding std (second row) shown below. Our proposed DistillCSE framework outperforms the vanilla distillation method and shares a small performance variance across all types of models.\n\nHowever, since the second round of distillation requires different first-round distillation checkpoints, which would be a total of 64 checkpoints. We could not finish the experiments in time, but we will add them in the later version.\n\n\n\n| **Methods** | **STS12** | **STS13** | **STS14** | **STS15** | **STS16** | **STS-B** | **SICK-R** | **Avg.** |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| **SimCSE-BERT-base** | 68.53 | 80.69 | 73.76 | 81.37 | 78.19 | 77.12 | 70.93 | 75.80 |\n|  | 0.84 | 1.76 | 1.62 | 1.00 | 0.67 | 0.88 | 1.14 | 0.82 |\n| Vanilla | 70.96 | 83.93 | 75.12 | 81.79 | 78.59 | 78.51 | 71.74 | 77.24 |\n|  | 0.11 | 0.65 | 0.52 | 0.46 | 0.35 | 0.14 | 0.09 | 0.25 |\n| Ours | 72.81 | 84.14 | 76.88 | 84.00 | 80.52 | 80.72 | 72.42 | 78.79 |\n|  | 0.35 | 0.37 | 0.03 | 0.24 | 0.11 | 0.11 | 0.07 | 0.11 |\n| **SimCSE-BERT-large** | 71.17 | 84.89 | 76.50 | 84.25 | 79.22 | 79.23 | 74.17 | 78.49 |\n|  | 0.76 | 0.62 | 0.39 | 0.33 | 0.45 | 0.27 | 1.10 | 0.23 |\n| Vanilla | 73.46 | 85.73 | 78.13 | 85.01 | 80.66 | 81.02 | 74.13 | 79.73 |\n|  | 0.76 | 0.42 | 0.32 | 0.24 | 0.14 | 0.50 | 0.55 | 0.10 |\n| Ours | 74.28 | 86.36 | 78.76 | 85.73 | 81.02 | 81.61 | 75.16 | 80.42 |\n|  | 0.58 | 0.22 | 0.21 | 0.26 | 0.17 | 0.21 | 0.21 | 0.17 |\n| **SimCSE-RoBERTa-base** | 69.84 | 81.93 | 74.07 | 82.37 | 80.86 | 81.07 | 68.88 | 77.00 |\n|  | 1.24 | 0.25 | 0.50 | 0.50 | 0.22 | 0.37 | 1.06 | 0.25 |\n| Vanilla | 71.60 | 82.20 | 73.39 | 81.31 | 81.68 | 81.14 | 68.97 | 77.18 |\n|  | 0.64 | 0.42 | 0.40 | 0.18 | 0.13 | 0.15 | 0.32 | 0.04 |\n| Ours | 71.77 | 83.38 | 75.36 | 83.32 | 82.18 | 82.13 | 69.25 | 78.20 |\n|  | 0.41 | 0.23 | 0.20 | 0.24 | 0.17 | 0.17 | 0.40 | 0.10 |\n| **SimCSE-RoBERTa-large** | 71.13 | 83.64 | 75.71 | 84.46 | 81.20 | 82.01 | 70.58 | 78.39 |\n|  | 1.48 | 0.24 | 0.47 | 0.48 | 0.28 | 0.34 | 0.70 | 0.23 |\n| Vanilla | 73.16 | 84.55 | 76.74 | 85.31 | 82.07 | 82.66 | 71.62 | 79.44 |\n|  | 0.28 | 0.06 | 0.08 | 0.15 | 0.32 | 0.25 | 0.39 | 0.10 |\n| Ours | 74.39 | 85.67 | 78.08 | 86.32 | 83.28 | 84.55 | 73.01 | 80.76 |\n|  | 0.34 | 0.31 | 0.17 | 0.19 | 0.18 | 0.41 | 0.22 | 0.14 |\n\n\n### Reference\n[1] Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The tackled issue is an interesting one, especially when the authors dive into the reasons why high variance occurs when training a model with contrastive learning. The proposed solutions seem relevant to the task, and the overall paper is clear and well organised.", "reasons_to_reject": "Despite effectively reducing the variance, results seem somewhat limited, only Bert-base being above the baselines with a large margin. Standard deviations are not provided so it is not clear if other results are significant. Moreover, it seems, according to Table 5, that the hyper-parameters tuning for p and $\\lambda$ was done on Bert-base, potentially leading to this larger increase in the results for this model only.\n\nEdit: the std have been provided and are small enough for statistical significance, I have updated my scores accordingly.", "questions_to_authors": "On Table 5, which model did you use? Also, the best results don't seem to correspond to any results reported on Table 4, why is this the case? If this is just due to randomness, could you provide std over 5 runs?", "paper_topic_and_main_contributions": "This paper investigates a potential issue with knowledge distillation: the high variance that can arise from the teacher's model trained with Contrastive Learning. To tackle this issue, the authors propose two simple solutions, namely Group-P shuffling and averaging logits from multiple teachers. They show that these two methods do reduce the variance, as well as outperforming existing methods.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "On Table 4, DCLR-RoBERTa-base outperforms DistillCSE-RoBERTa-base on both STS15 and SICK-R. Yet, the bold values are not set accordingly.\n\nOn Figure 2, the legend is overlapping with the lines, making it hard to read."}}]

Author rebuttal:

Meta-review: The paper deals with improving sentence embedding learning. The paper demonstrates that the key issue with knowledge distillation is high variance in the teacher model due to the nature of contrastive loss.  The paper  solves this problem by introducing additional regularization into contrastive training and knowledge distillation. It shows experimentally that the proposed techniques improve BERT performance in a standard sentence similarity benchmark. In rebuttal response the authors show that this method also works with RoBERTa, while experiments with XLNet show that this model does not perform well on this task and cannot be improved within the proposed paradigm.
The reviewers are generally agree in their view on the paper. All raised questions of novelty and meaningful comparison, especially comparison to other regularization techniques.
I would add to concerns raised by the reviewers that the paper does not fit very well into the "Semantics" category, because it does not discuss anything specific to sentence semantics. I would put it under "Machine learning" category, since it discusses a general topic of knowledge distillation with contrastive loss. Thus, I believe the paper might be stronger if the proposed regularization techniques were tested on a broader range of knowledge distillation tasks beyond sentence similarity (and maybe even beyond the NLP realm).
Decision: Accept-Findings