Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations
Authors: Fangyu Liu
Yunlong Jiao
Jordan Massiah
Emine Yilmaz
Serhii Havrylov
Venue: ICLR 2022 Poster (2022)
Keywords: self-supervised learning
sentence embeddings
sentence representations
knowledge distillation

Abstract: In NLP, a large volume of tasks involve pairwise comparison between two sequences (e.g. sentence similarity and paraphrase identification). Predominantly, two formulations are used for sentence-pair tasks: bi-encoders and cross-encoders. Bi-encoders produce fixed-dimensional sentence representations and are computationally efficient, however, they usually underperform cross-encoders. Cross-encoders can leverage their attention heads to exploit inter-sentence interactions for better performance but they require task fine-tuning and are computationally more expensive. In this paper, we present a completely unsupervised sentence representation model termed as Trans-Encoder that combines the two learning paradigms into an iterative joint framework to simultaneously learn enhanced bi- and cross-encoders. Specifically, on top of a pre-trained Language Model (PLM), we start with converting it to an unsupervised bi-encoder, and then alternate between the bi- and cross-encoder task formulations. In each alternation, one task formulation will produce pseudo-labels which are used as learning signals for the other task formulation. We then propose an extension to conduct such self-distillation approach on multiple PLMs in parallel and use the average of their pseudo-labels for mutual distillation. Trans-Encoder creates, to the best of our knowledge, the first completely unsupervised cross-encoder and also a state-of-the-art unsupervised bi-encoder for sentence similarity. Both the bi-encoder and cross-encoder formulations of Trans-Encoder outperform recently proposed state-of-the-art unsupervised sentence encoders such as Mirror-BERT and SimCSE by up to 5% on the sentence similarity benchmarks.

Published as a conference paper at ICLR 2022
UNSUPERVISED SENTENCE-PAIR MODELLING THROUGH SELF- AND MUTUAL-DISTILLATIONS
Fangyu Liu Yunlong Jiao² Jordan M
Emine Yilmaz² Serhii Havrylov²
University of Cambridge 2Amazon f1 f13990 wa..ecam.c..kk.......... {jyunl boyun jormas, emi ney, hwwrysavvasaaaocss hevryyayaezyyyzonccoom
ABSTRACT
In NLP, large volume of tasks involve pairwise comparison between two se- quences (e.g., sentence similarity and paraphrase identification). Predominantly, two formulations are used for sentence-pair tasks: bi-encoders and cross-encoders. Bi-encoders produce fixedddimensional sentence representations and are compu- tationally efficient, however, they usually underperform cross-encoders. Cross- encoders can leverage their attention heads to exploit inter-sentence interactions for better performance but they require task finetuning and computationally more expensive. In this paper, we present a completely unsupervised sentence-pan pair that combines the two learning paradigms
model termed into an iterative joint framework to simultaneously learn enhanced bi- and cross- encoders. Specifically, on top of a pre-trained language model (PLM), we start with converting it to an unsupervised bi-encoder, and then alternate between the bi- and cross-encoder task formulations. duce pseudo-labels which are used as learning signals for the other task formulation. We then propose an extension to conduct such self-distillation approach on multiple PLMs in parallel and use the average of their pseudo-labels for nutual-distillation. TRANS-ENCODER creates, to the best of our knowledge, the first completely un- supervised cross-encoder and also state-of-ther art unsupervised bi-encoder for sentence similarity. Both the bi-encoder and cross-encodee formulations of TRANS- ENCODER outperform recently proposed state-of-the-ar unsupervised sentence encoders such as Mirror-BERT (Liu et al. 2021) and SimCSE (Gao et al.) 2021) by up to 5% on the sentence similarity benchmarks. Code and models are released at httttps:///
each alternation, one task formulation will pro-
com/ amzn/ trans encoder
INTRODUCTION
Comparing pairwise sentences is fundamental to a wide spectrum of tasks in NLP such as information retrieval (IR), natural language inference (NLI), semantic textual similarity (STS) and clustering. Two general architectures usually used for sentence- -piir modelling are bi-encoders and cross-encoders
In cross-encoder, two sequences are concatenated and sent into the model (usually deep Transformers like BERT/RoBERTa; Devlin et al. [219 Liu et al. (019)) in one pass. The attention heads of Transformers could directly model the inter-sentence interactions and output classification/rellaaace score. However, a cross cross-encoder needs to recompute the encoding for different combinations of sentences in each unique sequence pair, resulting heavy computational overhead. is thus impractical in tasks like IR and clustering where massive pairwise sentence comparisons are involved. Also, task finetuning is always required for converting PLMs to cross-encoders. By contrast, in bi-encoder, each sequence is encoded separately and mapped to common embedding space for similarity comparison. The encoded sentences can be cached and reused. A bi-encoder is thus much more efficient. Also, the output of a bi-encoder for other downstream tasks. That said, it is well-known that in supervised learning, bi-encoders underperform cross-encoders (Humeau et al. 2020 Thakur al. 2021) since the former could not
be used off-the-she as sentence embeddings
Work done during internship Amazon.

---

Published as a conference paper at ICLR 2022
Unlabelled sentence-paire
Self-labelles data (from bi-encoder)
cross- encoder
PLM
bi-encoder
SimCSE/ Minor-BERT
Self-labelled data (from cross-encoder)
Unlabelled sentence-pan Santence--pair
Figure I:
A graphical illustration of the self-distillation learning scheme in TRANS-ENCODER, Notice that the blue boxes represent the same model architecture trained sequentially.
explicitly model the interactions between sentences could only compare them in the embedding space in post hoc manner.
In this work, we ask the question: can we leverage the advantages of both bi- and cross-encoders and bootstrap knowledge from them in unsupervised manner? Our proposed TRANS-ENCODEES addresses this question with the following intuition: As a starting point, we can use bi-encoder representations tune cross-encoder. With more powerful inter-sentence modelling, the cross- encoder should resurface more knowledge from the PLMs than the bi-encoder given the same data. In turn, the more powerful cross-encoder distil its knowledge back the bi-encoder. We can repeat this cycle to iteratively bootstrap from both the bi- and cross-encoders.
2 TRANS-ENCODER
The general idea of TRANS-ENCODEE ENCODER is simple yet extremely effective. In $2.1 we first transform an off-the the-shelf PLM to strong bi-encoder, serving initialisation point. Then, the bi-encoder produces pseudo-labels and the PI.M subsequently learns from these pseudo-labels in cross-encoder manner 2.2).. Consecutively, the cross-encoder further produces more accurate pseudo-labels for bi-encoder learning ($2.3). This self-distillation process is visualised in Fig. I] Then in, $2.4 we propose further extension called nutual-distillation that stabilises the training process and boosts the encoder performance even more.
2.1 TRANSFORM PLMS INTO EFFECTIVE BI-ENCODERS
Off-the-shelf PLMs are unsatisfactory bi-encoders have reasonably good starting point, we leverage simple contrastive tuning procedure to transform existing PLMs bi-encoders. This approach is concurrently proposed in both (Liu et al. 202T and (Gao et al. 202T).
Let f(.) denote the encoder model (a Transformer-baser model); X' be a batch of randomly sampled raw sentences. For any sentence T; E X, we send to the encoder twice to create two views of the same data point: f(x;) and fzz)). Note that the two encodings slightly differ due to the dropout layers the encoder We then use the standard InfoNCE loss (Oord et al. 2018) to cluster the positive pairs together and push away the negative pairs in the mini-batch:
|2
exp(cos (f (z.)
CinfONCE Σ
log
(s))
23 2,EM
т denotes a temperature parameter: NG denotes all positive & negatives of r; within the current data
batch. Intuitively, the numerator the similarity of the self-duplicated pair (the positive example)
In this work, bi-encoder is the under bi-encnder architecture, same encoder is used (i.c., the two-branches bi-enender share weights). "Miroo-BFRR BERT has the option to also randomly mask span in and using drophead (Zhou al. 2020) to replace dropout hidden states augmentation.
universal encoder. When embedding a pair of sentences
2

---

Published as a conference paper at ICLR 2022
and the denominator is the sum of the similarities between v; and all other negative strings besides Ty The loss encourages the positive pairs to be relatively close compared to the negative ones. In experiments, use checkpoints released by SimCSE and Mirror-BERT But in principle, any techniques could be used here as long as the method serves the purpose of transforming BERT to an effective bi-encoder and does not require additional labelled data (see $6
2.2 SELF-DISTILLATION B1- TO CROSS-ENCODER
After obtaining a sufficiently good bi-encoder, we leverage it to label sentence pairs sampled from the task of interest. Specifically, for given sentence-pair (sentl, sent2), input them to the bi-encoder separately and get two embeddings (we use the embedding of [CLS] ] from the last layer). The cosine similarity between them is regarded their relevance score. In this way we have constructed a self-labelled sentence-pia partence--pirre scoring dataset in the format of (sentl, sent2, score).
We then employ the same model architecture learn from these score, but with cross-encoder formulation. The cross-encoder weights are initialised from the original PLM For the sentence-pair (sent1, 2ent2), we concatenate them to produce " "[CLS] sent [SEP] sent [SEP]" and input it to the cross-encoder. A linear layer (newly initialised) then map the sequence's encoding (embedding the [CLS] token) to scalar. The learning objective of the cross-encoder minimising the KL divergence between its predictions and the self-labelled scores from the bi-encoder. This is equivalent to optimising the (soft) binary cross-entropy (BCE) loss:
LECE
E (un (oglog((((z)) + (1 (n-yn) yn) N
1
log(1 o (Zn))
where N is the data-batch size; o()) is the sigmoid activation; 2,, is the prediction of the cross-encoder; Yn is the self-labelles ground-truth score from the bi-encoder.
Note that while the cross-encodes is essentially learning from the data produced by itself (in a bi-encoder form), usually, it outperforms the original bi-encoder on held-out data. The cross-encoder directly discovers the similarity between two sentences through its attention heads, finding more accurate cues to justify the relevance score. The ability of discovering such cues could then generalise to unseen data, resulting in stronger sentence-pan scoring capability than the original bi-encoder. From a knowledge distillation perspective, we can view the bi- and cross-encoder as the teacher and student respectively. In this case the student outperforms the teacher, not because of stronger model capacity, but smarter task formulation. By leveraging this simple yet powerful observation, we are able to design learning scheme that iteratively boosts the performance of both bi- and cross-encoder.
2.3 SELF-DISTILLATION CROSS- TO B1- BI-ENCODER
With the more powerful cross-encoder at hand, natural next step is distilling the extra gained knowledge back to a bi-encoder form, which is more useful for downstream tasks. Besides, and more importantly, a better bi-encoder could produce even more self-labelled data for cross-encoder learning. In this way could repeat 2.2 and 22.3 continually bootstrapping the encoder performance.
We create the self-labelled sentence-scoring dataset in the same way as $2.2 except that the cross- encoder is used for producing the relevance score. The bi-encoder is initialised with the weights after SimCSE training from 1 For every sentence pair, two sentence embeddings are produced separately by the bi-encoder. The cosine similarity between the two embeddings regarded their predictions of the relevance score. The aim is to regress the predictions to the self-labelled scores by the cross-encoder. We use mean square enror (MSE) loss:
N Σ 1=1
LMSE = N
In
yr
(3)
We have design choice to make here, we could either (1) directly use the weights of the hi-enender (2)
use the weights of the original PL.. will show in experiments ($5.2) that (2) is slightly better than (1).
"Again, we have the choice of using the cross-encnder weights from 2.2 (the extra linear layer disregarded).
We will discuss more in experiments.
3

---

Published as a conference paper at ICLR 2022
PLM-1
bi-encoder-1
cross-
encoder-1
Average predictions both models
PLM-2
bi-encoder-2
cross-
encoder-2
Figure 2: A graphical illustration of the mutual- distillation learning scheme in Note that, for simplicity, only the to cross-encoder mutual-distllationn shown. We also conduct cross- to bi-encoder nutual-distillation in the same manner.
where N is the batch size; In is the cosine similarity between sentence pair; yn. is the self-labelled ground-truth, In experiments, we will show that this resulting bi-encoder is more powerful than the initial bi-encoder. Sometimes, the bi-encoder will even outperform its teacher (i.e., the vross-encoder).
discrepancy the key. Intuitively, MSE allows for more accurate distillations since it punishes any numerical discrepancies between predicted scores and labels on the instance level. However, in practice, we found that using MSE for bi-to- cross distillation aggravates the overfitting issue for cross-encoder: the cross-encoder, with its strong sentence-pais modelling capability, completely overfits to the mapping between concatenated sentence pairs and the pscudo scores. This elimination discrepancy between model predictions and pseudo labels harms generalisation and the iterative learning cycle cannot continue (as the predicted scores by the student model will be the same as the teacher model). We thus use BCE loss for bi-to-cross since BCE is more forgiving loss function compared with MSE. According to our gradient derivation (App. $A.2), BCE essentially is a towards numerical discrepancies. This prevents cross-encoders from overfitting to the pseudo labels completely. Similar issue does not exist for the cross-to-bi distillation as for bi-encoders, two input sequences are separately encoded and the model does not easily overfit to the labels. We have more thorough discussion in App. $A.2 and Tab. 8 to highlight the rationales of the current configuration.
Choice of loss functions: maintaining
version of MSE, which is more tolerant
2.4 MUTUAL-DISTILLATION
The aforementioned self-learning approach has a drawback: since the model regresses to its previous predictions, it tends to reinforce its errors. To mitigate this problem, design simple mutual- distillation approach to smooth out the errors/biases originated from PLMs. Specifically, we conduct self-distillation on multiple PLMs in parallel (for brevity, RoBERTa, however the framework itself is compatible with any number of PLMs). Each PLM does communicate/synchronisss with each other except when producing the self-labelled scores. In mutual mutual-distillation, we use the average predictions of all models as the ground-truth for the next round of learning. graphical illustration is shown in Fig. 2
in this paper: BERT and
In the following, we call all self-distillation models mutual-distillaan models TRRANS-ENCODER-EE TENC (-mutual) training produces one best bi-encoder, called TENC (-mutual) (bi), and one best cross-encoder, called TENC (-mutual) (cross), based on dev set. We report numbers of both.
(or TENC for short); all (or TENC-mutual for short). Note that one
3
EVALUATION TASKS
Evaluation task: semantic textual similarity (STS), Following prior works (Reimers & Gurevychi
2019 Liu et al. 2021 Gao al. 2021), we consider seven STS datasets: SemEval STS 2012-2016
We also experimented with a sequential ensemble strategy: initialising the cross- and bi-encoder with BERT/ROBERTA alternatively (1 cycle of BERT then cycle of RoBERTa) but found the performance worse than -istilliistillation..
4

---

Published as a conference paper at ICLR 2022
(STS12-16 16, Agirre et al. 2012 2013; 2014 2015 2016), STS Benchmark (STS-B. Cer et al. 2017) and JICK-Relatedness S.CK-R. Marelli et al. 2014). .. In all these datasets, sentence pairs are given a human-judged relevance score from I to 5. We normalise them to 0 and and models are asked to predict the scores. We report Spearman's 's p rank correlation between the two.
Evaluation task: binary classification. We also test TRANS-ENCODER on sentence-pair binary classification tasks where the model has to decide a sentence- pair has certain relations. We choose (I) the Quora Question Pair (QQP) dataset, requiring a model to judge whether two questions are duplicated: (2) a question-answerte
entailment dataset QNLI (Rajpurkar et al., 2016 Wang et al. 2019) in which given a question and sentence the model needs to judge if the sentence answers the question; (3) the Microsoft Research Paraphrase Corpus (MRPC) which asks a model to decide if two sentences are paraphrases of each other. The ground truth labels of all datasets are either or 1.
Following (Lii et al., 2020; Liu et al., (02T), we compute Area Under Curve (AUC) scores using the binary labels and the relevance scores predicted by models.
4 EXPERIMENTAN SETUP
Training and evaluation details. For each task, use all available sentence pairs (from train, development and test sets of all datasets combined) without their labels as training data. The original QQP and QNLI datasets
extremely large. We thus downsample QQP to have 10k, 1 and 10k pairs for train, dev and test; QNLI to have 10k train set. QNLI does not have public ground truth labels for testing. So, we use its first Ik examples in the official dev set as our dev data and the rest in the official dev set as test data. The dev set for MRPC is its official dev sets. The dev set for STS12-16, STS-B and SICK-R is the dev set of STS-B. We save one checkpoint for every 200 training steps and at the end of each epoch. We use the dev sets to select the best model for testing @ Dev sets are also used to tune the hyperprameters in each task.
For clear comparison with SimCSE and Mirror-BERT, we use their released checkpoints as initialisa- tion points (i.e., we do not train models in $2. ourselves). We consider four SimCSE variants. Two base variants: SimCSE-BERTI BERT--ase,
and two large variants: SimCSE-BERT large, RoEEST---a We consider two Mirror-BERT variants: Mirror-RoBERTa-baar and Miiror-RRBBRTT TaaBBRRT--aas-drophhe baoeedropheadd brevity, analysis in the main text focuses on SimCSE models. We list Mirror-BER BERT results in Appendix. We train AANS-ENCODER models for 3 cycles on the STS task and 5 cycles on the binary classification tasks. Within each cycle, all bi- and cross- cross-encoders are trained for 10 and I epochs respectively for the STS task; 15 and 3 epochs for binary classification (8) All models use AdamW (Loshchilov & Hutter 2019) as the optimiser: In all tasks, unless noted otherwise, we create final representations using [CLS]. We train our base models on a server with # V100 (16GB) GPUs and large models on a server with 8 A100 (40GB) GPUs. All main experiments have the same fixed random seed. All other hparams are listed in Appendix.
Nutual-distillation setup. For the two models used for mutual-distillation they are either (1) the base variant of BAnCSE-BERS and RoBBRTa or (2) the large variant of the two. Theoretically. we could mutually distil even more models but we keep the setup simple for fast and clear comparison. Also, since mutual catual-distillation models use information from both PLMs, we also list (in tables) ensemble results that use the average of the predictions from two TRANS-ENCORDE
5 RESULTS AND DISCUSSION
5.1 MAIN RESULTS
STS results Tab. 1). The main results for STS are listed in Tab. Compared with the baseline SimCSE, CRANS-ENCODER has brought significant improvements across the board. With various variants of the SimCSE as the base model, TRANS-ENCODES consistently enhances the average score
NNte that checkpoints of bi- and cross-encoders are independently saved and evaluated on dev set. In the end, one best bi-eneoder and one best cross-cncoder are obtained.
We do not consider BERT-bassd checkpoints by Liu et al. (2021) since they adopt mean pooling over all tokens. For brevity, we only experiment with encoders trained with [CLS].
We use fewer epachs for cross-encoders since they usually converge much faster than bi-encoders.
5

---

Published as a conference paper at ICLR 2022
#
dataset->
STS12 STS13 STS14 STSI5
STSI6 STS-B B SICK-R R avg.
single-model results
SimCSE-BERT-ba
68.64
82.39
74.30
80.69
78.71
76.54
72.22
76.21
1.1
+ TENC (bi)
72.17
84.40
76.69
83.28
80.91
81.26
71.84
78.65
1.2
+ TENC (cross)
71.94
84.14
76.39
82.87
80.65
81.06
71.16
78.32
1.3
+ TENC-mutual (bi)
75.09
85.10
77.90
85,08
83.05
83,90
72.76
80.41
1.4
+ TENC-mutual (cross)
75.44
85.59
78.03
84.44
82.65
83.61
69.52
79.90
SimCSSE-RoBERRTabbbe
68.34
80.96
73.13
80.86
80.61
80.20
68.62
76.
2.1
+ TENC (bi)
73.36
82.47
76.39
83.96
82.67
82.05
67.63
78.36
2.2
+ TENC (cross)
72.59
83.24
76.83
84.20
82.82
82.85
69.51
78.86
2.3
TENC-mutual (bi)
75.01
85.22
78.26
85.16
83.22
83.88
72.56
80.47
2.4
+ TENC-mutual (cross)
76.37
85.87
79.03
85.77
83.77
84.65
72.62
81.15
3
SinCSE-BERT-lagg
71.30
84.32
76.32
84.28
79.78
79.04
73.88
78.42
3.1
+ TENC (bi)
75.55
84.08
77.01
85,43
81.37
82.88
71.46
79.68
3,2
+ TENC (cross)
75.81
84.51
76.50
85.65
82.14
83.47
70.90
79.85
3.3
+ TENC-mutual (bi)
78.19
88.51
81.37
88.16
84.81
86.16
71.33
82.65
3.4
+ TENC-mutual (cross)
77.97
88.31
81.02
88.11
84.40
85.95
71.92
82.52
4
SimCSEE-RoBERTa-aare
71.40
84.60
75.94
84.36
82.22
82.67
71.23
78.92
4.1
+ TENNC (bi)
77.92
86.69
79.29
87.23
84.22
86.10
68,36
81.40
4.2
+ TENC (cross)
78.32
86.20
79.61
86.88
82.93
84.48
67.90
80.90
4.3
+ TENC-mutual (bi)
78.15
88.39
81.76
88.38
84.95
86.55
72.31
82.93
4.4
TENC-mutual (cross)
78.28
88.31
81.94
88.63
85.03
86.70
71.63
82.93
ensemble results (average predictions two models)
SimCSE- ensemble
70.71
83.49
76.45
83.13
81.79
81.51
71.94
78.43
1.1+2.1
TENC-base (bi) ensemble
73.58
85.01
78.35
85.02
83.21
84.07
70.93
80.03
1.2+2.2
TENC-base (cross) ensemble
74.25
84.92
78.57
85.16
83.25
83.52
70.73
80.06
1.3+2.3
T'NC--mttul--rre (bi) ensemble
75.29
85.34
78.49
85.28
83.43
84.09
72.75
80.67
1.4+2.4
TENC-mutual-Be base (cross) ensemble
76.60
86.18
79.09
85,49
83.64
84.67
71.96
81.09
344
SimCSE-large ensemble
73.22
86.03
78.15
85.95
82.83
83.05
73.86
80.66
3.1+4.1
TENC-large (bi) ensemble
78.49
87.02
80.31
87.85
84.31
86.54
72.39
82.41
3.2+4.2
TENC-large (cross) ensemble
77.93
86.91
79.83
87.82
83.74
85.58
72.02
81.98
3.3+4.3
TENC-mutual- (bi) ensemble
78.42
88.60
81.91
88.38
85.01
86.52
72.23
83.01
3.4
TENC-mutual-large (cross) ensemble
78.52
88.70
81.90
88.67
84.96
86.70
72.03
83.07
Table 1: English STS. Spearman's 's p rank correlations are reported. TENC models use only self- distillation while TENC-mutual models use muttal-distillation well. Blue and red denotes mutual-distillation models that are trained in the base and large group respectively. Models without colour are not co co-trained with any other models. "Note that for base encoders, our results can slightly differ from numbers reported in (Gao al., 2020 l since different evaluation packages are used.
by approximately 4-5% 5 Self-distillation usually brings an improvement of 2-3% (e.g., compare line 1.1, 1.2 with in Tab. 1). Further, nuttualdistilllatoo brings another boost of 1-3% (e.g., compare line 3.3, 3.4 with 3.1, 3.2 in Tab. T}. Besides single-model results, the ensemble of mutual-distilled models are clearly better than the ensemble models of either (1) naively averaging the initial SimCSE models' scores or (2) averaging predictions two self-distilled models a post hoc manner. This demonstrates the benefit of allowing models communicate/snncroniie with each other in all stages of the self-distillation training.
Binary classification results TTb.. 2). On QQP, QNLI, and MRPC, observe similar trends as the STS tasks, and the improvement is sometimes even more significant (see full table in Appendix Tab. IT). We conjecture it is due the fact that TRANS-ENCODER training also adapts models the task domain (since tasks like QQP and QNLI are very different from Wikipedia data, which is where SimCSE models are trained on). We will discuss the domain adaptation effect in more details in analysis. Another point worth discussing is the benefit of mutual-distillation is more inconsistent on binary classification tasks, compared with STS tasks (again, see full table Tab. in Appendix). We
Similar trends observed Mirror-BERT, Appendix App. $A.1

---

Published as a conference paper at ICLR 2022
dataset->
QQP
QNLI
MRPC
avg.
SimCSE-BERT-base
80.38
71.38
75.02
75.59
TENC (bi)
82.10
75.30
75.71
77.70
+ TENC (cross)
82.10
75.61
76.21
77.97
TENC-mutual (bi)
84.00
76.93
76.62
79.18
TENC-mutual (cross)
84.29
77.11
77.77
79.72
Table 2: Binary classificattin task results. AUC scores are reported. We only demonstrate results for
model for brevity. Full table can be found in Appendix Tab. 11
dataset +
QQP
QNLI
MRPC
avg.
base
81.82
73.54
75.06
76.81
TENC (bi)
82.56
71.67
74.24
76.16
TENC (cross)
83.66
79.38
79.53
80.86
TENC-mutual (bi)
81.71
72.78
75.51
76.67
TENC-mutual (cross)
83.92
79.79
79.96
81.22
Table 3: A domain transfer setup: testing TRANS- models trained with STS data directly on binary classification tasks. We only demonstrate results for one model for brevity. Full table can be found in Appendix Tab. 12
suspect it is due to the performance imbalance between the BERT and RoBERTa variants on these tasks (a significant performance gap exists in the first place).
Domain transfer setup (Tab. 3). One of the key questions we are keen to find out is how much of TRANS-ENCODER'S success on in-domain tasks generalises/ransfers to other tasks/domains. Specifically, does TRAAS--ENODDER create universally better bi- and cross-encoders, or is it only lusk/domain-speciffie To answer the question, we directly test models trained with the STS task data binary classification tasks. For bi-encoders, i..., TENC (bi), the results are inconsistent across setups. These results hint that training on in-domain data is important for optimal performance gains for unsupervised bi-encoders. This is in line with the finding of (Liu et al. 2021). However, for cross-encoders, surprisingly, we see extremely good transfer performance. W/ or w/o mutual- distillation, TENC (cross) models outperform the SimCSE baselines by large margins, despite the fact that tasks like QQP and QNLI are of a very different domain compared with STS. In fact, they are sometimes even better than models tuned on the in-domain task data (c.f. Tab. III. This finding has deep implications. It hints that the cross-encoder architecture (cross-attention is by design very suitable for sentence-pal modelling (i.e., strong inductive bias is already in the architecture design), and once its sentence-pat modelling capability is 'activated', it is an extremely powerful universal representation that can be transferred to other tasks. The reason that STS models outperform in-domain models in some tasks hint that more sentence-pairs as raw training data is welcome (since the STS train set is larger than the binary classification ones). As future work, we plan to explore mining sentence pairs for TRANS- -ANN-EE learning from general corpus (such as Wikipedia). Both lexical- (such as BM25) and semantic-level (neural models such as bi-encoders) IR systems could be leveraged here.
5.2 DISCUSSIONS AND ANALYSIS
In this section we discuss some interesting phenomena we observed, justify design choices we made an empirical way, and also show more fine-grained understanding of what exactly happens when using -ONND-E.CDDER...
Initialise with the original weights or train sequentially? Fig. 3a As mentioned in $2.2 and 2.3 we initialise bi-encoders with SimCSE/Minor-BERR weights and cross-encoders with PLMs weights (we call this strategy refreshing). We have also tried maintaining the weights of bi- and cross-encoders sequentially. Le., initialise the cross-encoder's 's weights with the bi-encoder which just created the pseudo labels and vice versa (we call this strategy sequential). The benefit of doing so is keeping all the legacy knowledge learned in the process of self-distillation in the weights. As
7

---

Published as a conference paper at ICLR 2022
SinCSE BERT
SmCSE RoBERTo base
SimCSE: BERT
Sponnau' tho en STS (e0g)
79.5
Linnc sog
SinCSE-RoB EeTobbae ammirdded.
79.0
*TEne (crous)
"TEne (cross) ref.
+TEne (b)
28.5
+TEne (cnoss)
78.0-
on
27.5
27.0
72-
26.5
26.0
SimC SE-BEIKT-bbe
SimC B-nCSB-Roo1220
SimCSE-BERTT-oo
SimCSE-RoBERR
(a) Sequential vs. refreshing (on STS).
(b) TENC vs.standard self-distillation (on QNLI).
Figure 3: Ablation studies of TRANS- regarding design choices.
0.88
BERRT-base
Spearmen's rbo
0.88
RoBERTa-hase
BERT-large
ALC
0.80
ReBERTa-lagge
0.86
0.84
bi-encoder
0.75
cross-cncoder
0.84
0.82
cycle
eycle
(a) STS
(b) QQP
(c) QNLI
Figure 4: TENC's performance against distillation cycles under different base models and tasks.
suggested in Fig. 3a (also in Appendix Tab. 13), refreshing is slightly better than sequential. We suspect it is because initialising with original weights alleviate catastrophic forgetting problem in self-supervised learning (SSL), similar to the moving average and stop-gra gradient strategy used in contrastive SSL methods such as MoCo (He et al. 2020).
Self-distillation without alternating between different task formulations? Fig. 3b If disregard the bi- and cross-encoder alternating training paradigm but using the same bi-encoder architecture all along, the model is then similar to the standard self-distillation model (c.f. Figure by MMbahi et al. ()200). Theoretically, standard self-distillation could also have helped, especially considering that it adapts the model in-domain data. However, as seen in Fig. 3b (and also Appendix Tab. 14), standard self-distillation lags behind TRANS-ENCODER ENCODER by significant amount, and sometimes underperforms the base model. Standard self-distillation essentially ensembles different checkpoints (i.e.. the same model at different training phases provides different 'views" of the data). TRANS- ENCODER can be seen as a type of self-distillation model, but instead of using previous models to provide views of data, different views of the data. The fact that standard self-distillation underperforms TRANS-ENCODER by large margins suggests the effectiveness of proposed bi- and cross cross-encoder training scheme.
use different task formulations (i.e., bi- and cross-encoder) to provide
How many cycles is optimal? Fig. 4) Our approach iteratively bootstraps the performance of both bi- and cross-encoders. But how many iterations (cycles) are enough? We find that heavily depends on the model and dataset, and could be unpredictable (since the 'optimality' depends on a relatively small dev set). In Fig. 4. we plot the TENC models' (self-distillation only) performances on dev sets in three tasks: STS, QQP, and QNLI. In general, the patterns are different across datasets and models.
Does more training data help? (Fig. 5) We control the number of training samples drawn and test the
model (using SimCSE-BuR CEESSE-ERT-bbaee on STS test sets. The average performance on all STS tasks are reported in Fig. 3] with three runs of different random seeds. There are total 37,081 data points from STS. We only draw from these in-domain data until there is none left. It can seen that for in-domain training, more data points are usually always beneficial. Next, we test with more samples drawn from another task, SNLI (Bowman al.) 2015), containing abundant sentence-pairs. The performance of the model further increases till 30k extra data points and starts to gradually decrease after that point. However, it is worth mentioning that there will be discrepancy of such trends across tasks and models. And when needed, one can mine as many sentence-paie as desired from general corpus (such Wikipedia) for -ANS-EN learning.
8

---

Published as a conference paper at ICLR 2022
79.5
79.0-
Figure 5: STS performance against num- ber of training samples. 'full' means all STS in-domain data are used. After 'full', the samples are drawn from the SNLI dataset. Note that each point in the graph is an individual run.
73.5
75.0-
legend
57.5
SinC86
77.0-
N-anedde
76.5
creessmookk
waining
76.0
180k
Asample
RELATED WORK
Our work is closely related to (1) unsupervised sentence representation learning; (2) sentence-pair tasks using bi- and cross-encoders; and (3) self-distillation. Each has large body of work which we can only provide highly selected summary below.
Unsupervised sentence representations. Prior the emergence of large PLMs such BERT, popular unsupervised sentence embedding methods included Thi-Thoughts (Kiros et al. 2015). FastSent (Hill et al., 2016), and Quick-Thoughts (Logeswaran Lee 2018). They all exploit the co-occurrence statistics of sentences in large corpora (i.e., sentences under similar contexts have similar meanings). recent paper DeCLUTR (Giorgi et al. 202T) follows such idea and formulate the training contrastive learning task. Very recently, there has been growing interest in using individual raw sentences for self-supervieed contrastive learning on top of PLMs. Contrastive Tension (Carlsson et al., 2021). Mirror- BERT (Liu et al., 2021), SimCSE (Gao et al. 2021), Self-Guuee Guided Contrastive Learning (Kim et al. 2021). ConSERT Yan et al., 2021). BSL (Zhang et al. 202I), inter alia, all follow such an idea. Specifically, data augmentations
either input or feature space used to create two views of the same sentence for contrastive learning. In our experiments, we use such methods an initial step for creating a reasonably strong bi-encoder.
Cross- and bi-encoder for pairwise sentence tasks. Sentence-BERY (Reimers & Gurevych 2019) points out the speed advantage of bi-encoders and proposes to train BERT with siamese architecture for sentence-pair tasks. Poly-encoder (Humeau et al. 2020) proposes a hybrid architecture in which intermediate encodings are cached and final layer of cross-attention is used exploit inter- sentence interactions. Conceptually similar to our work, Di-pair (Chen et al. 2020), augmented Sentence-BEi BERT (Thakur al. 2021), RocketQA (Qu et al., 202T), and DvBERT (Cheng 2021) distil
knowledge in supervised cross cross-encoders to bi-encoders. We show that it is also possible to distil knowledge from bi- to cross-encoders.
Self-distillation. In standard self eeff-dittillation, models of the same architecture train on predictions by previous checkpoint(s) with the same task formulation (Furlanello et al., 2018 8 Mobahi et al., 2020). However, the standard approach does not alter the training paradigm (task formulation) as we do (bi- and cross-encoder training). In experiments, we have shown that alternating task formulation the key to the success of our method.
7 CONCLUSION
We propose TRANS-ENCODEE ENCODER, an unsupervised approach of training bi- and cross-encoders for sentence-pair tasks. The core idea of TRANS- -AAND-E is self-distillation in smart way: alter- natively training a bi-encoder and a cross-encoder (of the same architecture) with pseudo-labels created from the other. We also propose self-distillation models trained in parallel. On sentence-pal tasks including sentence similarity, ques- tion de-duplication, question-answering entailment, and paraphrase identification, we show strong empirical evidence verifying the effectiveness of TRANS-ENCN ENCODER. We also found the surprisingly strong generalisation capability of our trained cross- encoders across domains and tasks. Finally, conduct thorough ablation studies and analysis to verify our design choices and shed insight on the model mechanism.
mntual-distillation extension to mutually bootstrap two

---

Published as a conference paper at ICLR 2022
8 ACKNOWLEDGEMENTS
We thank Hanchen Wang, Giorgio Giannone, Arushi Goel for providing useful feedback during internal discussions.
REFERENCES
Encko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agire. SemEval-2012 task 6: A pilot on semantic textual similarity. In "SEM 20/2, pp. 385-393, 2012.
Encko Agirre, Daniel Cer, Mona Diab, Aitor CoozzllezAAgrree, and Weiwei Guo. *SEM 2013 shared task: Semantic textual similarity. In *SEM 2013, pp. 32-43,2015,
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. SemEval-201 task 10: Multilingual semantic textual similarity. In SemEval 2014, pp. 81-91,2014,
Encko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot interpretability. In SemEval 2015, pp. 252-263, 2015.
Guo, Iñigo
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval-2016, pp. 497-511, 2016.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In EMNLP 20/5, pp. 632-642--4 2015. 2015.
Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou. Erik Ylipää Hellqvist, and Magnus Sahlgren. Semantic re-tuning with contrastive tension. In ICLR 2021, 2021.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval 20/7, pp- 1-14, 2017.
Jiecao Chen, Liu Yang, Karthik Raman, Michael Bendersky, Jung-Jung Yeh, Yun Zhou, Mare Najork, Danyang Cai, and Ehsan Emadzadeh. DiPair: Fast and accurate distillation for trillion-scale text matching and pair modeling. In EMNLP-findings 2020, pp. 2925-2937, 2020.
Xingyi Cheng. Dual-view distilled bert for sentence embedding. SIGIR 2021, pp. 2151-2155, 2021.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pp. 4171-4186, 2019.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In ICLR 2018, pp. 1607-1616, 2018.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. EMNLP 202/, pp. 6894-6910, 2021.
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. DeCLUTR: Deep contrastive learning for unsupervised textual representations. In ACL-IJCNLP 2021, pp. 879-895, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR 2020, pp. 9726-9735. 2020.
Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In NAACL 2016, pp. 1367-1377, 2016.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Archi- tectures and pre-training strategies for fast and accurate multi-sentence scoring. In ICLR 2020, 2020.
10

---

Published as a conference paper at ICLR 2022
Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. Self-guided contrastive learning for BERT sentence representations. In ACL-JJCNL 2021, pp. 1528-25400 2021.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In NeurlPS 2015, pp. 3294 4302, 2015.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In EMNLP 2020, pp. 9119-9130, 2020.
Fangyu Liu, Ivan Vulić, Anna Korhonen, and Nigel Collier. Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders. In EMNLP 2021. pp. 1442-1459, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representa- tions. In ICLR 2018, 2018.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR 2019, 2019.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In LREC 2014, pp. 216-223, 2014.
Hossein Mobahi, Mehrdad Farajtabar, and Peter L Bartlett. Self-distillation amplifies regularization in hilbert space. arXiv preprint arXiv:2000.05715, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXXiv/807..055 2018.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. Rocketqa: An optimized training approach dense passage retrieval for open-domain question answering. In NAACL 2021, pp. 5835-5847, 2021.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP 2016, pp. 2383-2392, Austin, Texas, 2016.
Nils Reimers and Iryna Gurevych. Sintence-BERT: Sentence embeddings using Siamese BERT- networks. In EMNLP- 2019, pp. 3982-3992, 2019.
Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. In NAACL 202/, pp. 210- 2021.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In ICLR 2019, 2019.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. ConSERT: A contrastive framework for self-supervised sentence representation transfer. ACL-IJCNLP 2021, pp. 5065-5075, 2021. 2021.
Yan Zhang, Ruidan He, Zuozhu Liu, Lidong Bing, and Haizhou Li. Bootstrapped unsupervised sentence representation learning. In ACL-IJCNLP 2021, pp. 5168-5180, 2021.
Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou, and Ke Xu. Scheduled DropHead: A regulariza- tion method for transformer models. In EENLLP-fidding 2020, pp. 1971-1980. 2020.
11

---

Published as a conference paper at ICLR 2022
dataset-+
STS12
STS13
STS14
STSI5
STS16
STS-B
SICK
avg.
Mirror-RoBERTa-basss
64.67
81.53
73.05
79.78
78.16
78.36
70.03
75.08
+ TTEN (bi)
71.99
81.85
75.73
83.32
79.97
81.19
69.47
77.64
+ TENC (cross)
73.10
82.31
76.49
83.71
79.64
81.70
69.70
78.09
Mirror-KiBBBKKTT--aeeedddpppppp
68.15
82.55
73.47
82.26
79.44
79.63
71.58
76.72
+ TENC (bi)
74.95
83.86
77.50
85,80
83.22
83,94
72.56
80.26
TENC (cross)
75.70
84.58
78.35
86.49
83.96
84.26
72.76
80.87
Contrastive-Tennn Cension-BERT--e
61.98
76.82
68.35
77.38
76.56
73.99
69.20
72.04
+ TENC (bi)
69.58
79.38
69.32
77.59
77.96
78.23
70.70
74.68
+ TENC (cross)
70.67
79.27
69.26
76.77
78.78
79.08
70.66
74.93
MeCLLUTR-RoBERRT--bas
45.56
73.38
63.39
72.72
76.15
66.40
68.99
66.66
+ TENC (bi)
62.40
77.55
70.05
80.28
81.81
77.31
72.60
74.57
TENC (cross)
60.68
78.11
69.23
80.48
81.03
78.47
70.07
74.01
Table 4: English STS (models beyond SimCSE). Spearman's scores are reported.
dataset-+
QQP
QNLI
MRPC
avg.
Hirro-RoBBRTa-base
78.89
73.73
75.44
76.02
+ -TENC (bi)
82.34
79.57
77.08
79.66
TENC (cross)
82.00
79.87
78.40
80.09
78.36
75.56
77.18
77.03
TENC (bi)
82.04
82.12
79.63
81.26
TENC (cross)
82.90
82.52
81.38
82.27
Contrastive-Tensiiee Tereeee--niiin-TTT-T--- BERT-base
78.95
69.73
72.86
73.85
+ TENC (bi)
80.96
68.31
72.54
73.94
+ TENC (cross)
81.22
74.17
71.33
75.57
DeCI.UTR--RBBRRTa--ae
78.68
74.77
72.86
75.44
TENC (hi)
83.64
82.87
74.55
80.35
TENC (eross)
84.02
83.41
75.55
80.99
Table 5: Binary classification (models beyond SimCSE). AUC scores are reported.
A APPENDIX
A.1 RESULTS WITH OTHER CONTRASTIVE LEARNING METHODS
In the main text, experimented with SimCSE, but in theory TRANS-ENCODEN ENCODER is compatible with any unsupervised contrastive bi-encoder learning models. this section, we experiment with Mirror-BERT (which is similar to SimCSE) and also two additional unsupervised contrastive models Contrastive- Tension (Carlsson et al.) 2021) and DeCLUTR (Giorgi et al. 2021) to verify the robustness of -ECODDE,
As seen Tab. 4 and Tab. 5] TRANS-ENCODER brings consistent and significant improvements to all base models, similar what have observed SimCSE models. After TRANS-ENCODEE training. the Minror- Birioor-BRT--aae BERT-based models perform even slightly better than SimCSE-bases models, on both the STS and binary classification tasks. We suspect it is due that SimCSE checkpoints are trained for more iterations (with the contrastive learning objective) and thus are more prone to overfit the training corpus.
A.2 MORE DISCUSSIONS AND ANALYSIS
How robust is TRANS-ENCODER random seed. Due to the scale of the experiments, performing multiple runs with different random seeds for all setups is overly expensive. We pick STS and the base variants of BERT and RoBERTa
Tab. 6 As mentioned, main experiments used one fixed
12

---

Published as a conference paper at ICLR 2022
-ataset->
mean S.D.
SimCSE-BERT base
TENC (bi)
78.72
0.16
TENC (cross)
78.21 0.26
SimCSE-RoBEIH RoBERRT--baae
TENC (bi)
78.38
0.26
TENC (cross)
78.90
0.34
Table Mcan and standard deviation (S.D.) of five runs on STS.
Data->
STS
SICK-R
model
off-the-shelf
+TENC (bi)
+TENC (cross)
+TENC (bi)
+TENC (cross)
BinCSE-BERT-bar
72.22
71.84
71.16
74.13
74.43
68.62
67.63
69.51
68.39
70.38
SimCSE-BERT-lare
73.88
71.46
70.90
74.92
74.98
SimCSE-Roo RoBERTa-large 71.23
68.36
67.90
72.63
73.13
Table 7: Compare TRANS- ENCODER models trained with STS data (using STS-B's dev set) and SICK-R data only (using SICK-R's dev set). Large performance gains be obtained when treating SICK-R R as a standalone task.
examples for showing results of five runs (shown in Tab. 6). In general, the approach is quite robust and stable.
STS and SICK-R R as different tasks TTb. Fig. 6). is worth noting that in the STS main results table TTb. 1). SICK-R is the only dataset where TRANS- "TRANN-ENCCDEE models lead to performance worse than the baselines in some settings (e.g., on LLmCSE-BERT-largee that SICK-R is essentially different task from STS2012-2016 and STS-B since SICK-R is labelled with different aim in mind. focuses specifically on compositional distributional semantics T0 To verify our claim, we train SICK-R K as standalone task using its official dev instead of STS-B. B, All sentence pairs from SICK-R are used as raw training data. The results are shown in Tab. 7 and Fig. As shown, around 3-4% gain be obtained by switching training SICK-R only, confirming the different nature of the two tasks. This suggests that generalisation is also dependent on the standard of how relevance is defined between sequences in the target dataset, and training with an in-domain dev set maybe crucial when the domain shift between training and testing data is significant.
We believe it is due to
Gradient derivation of BCE and MSE loss. We show that BCE essentially a normalised version of MSE where the gradient of z; is scaled by the sigmoid function (ar(-)). Here are the gradients of the loss functions:
dLBCE dz,
N (v(o((zs)) -) +(1 +(1-2))o(zz))
N (a(2:) - V/i
aLMSE
2
Iz yí
Or:
N
As can be seen, the only difference lies whether 4; scaled by (.).. This gives greater degree of freedom to LBCE especially the areas in the two ends (close to 0 or 1): large change of z; results
"SICK-R "includes large number of sentence pairs that rich in the lexical, syntactic and semantic phenomena that Compositional Distributional Semantic Models (CDSMs) are expected account for (e.g.. contextual synonymy and other lexical variation phenomena, active/passive and other syntactic alternations, impact of negation, determiners and other grammatical elements), but do not require dealing with other aspects of existing sentential data sets (e.g., STS, RTE) that not within the scope of compositional distributional semantics." (sce http:: rcobareni org/ httpg//omposssss/o s cickk. html
13

---

Published as a conference paper at ICLR 2022
(ofti-tte
-TEn:
"Thoe SICK-R
-Thb SICK-R
Strammatt tho on SISSSCK. Rt
74-
72-
0900
093300 20-
66
Sinn SE-Rolilk Tehes:
SimCSE-Dritt-lere
Sint SmnEB-RRCERR Rec.ee---bbeggg
Figure Graphical presentation of Tab. 7
losS so cross
IoSS
result
BCE
MSE
TRANS TRANS-ENCODES ENCODER configuration (V)
BCE
BCE
cross-to-bi distillation won't I converge (X)
MSE
MSE
bi-to-cross distiltation overfits completely to pscudo labels (x)
bi-to-cross disüllation overfits completely to pscudo labels (X) &
MSE
BCE
cross-to-bi distillation won't converge (X)
Table 8: Compare different loss function configurations.
in small change of (xx)), making the approximation of y; an easier task. In other words, in LACF, in general casier for the model to push the gradient to while in CMSE, only a strict numerical match leads to 0 gradient.
Loss function configurations Tab. 8
We experimented with loss function configurations compre- hensively and found that it requires caution when choosing learning objectives for cross- to bi-encoder and bi- to cross-encoder distillation. As mentioned in the main text and the paragraph above, choosing MSE for bi-to-cross distillation causes severe overfitting to the pseudo scores and choosing BCE for cross-to-bi distillation fail to make the model converge. As result, our configuration of using BCE and MSE for bi-to-cross and cross-to-bi distillations respectively becomes the only plausible solution (see Tab. 8)
For the regression loss used for cross-to-bi distillation, besides MSE, we also experimented with several other regression loss functions such as Log-Cosh CCsh but found no improvement.
Do sentence pairs leak label information? Tab. TRANS-ENCODER has leveraged the sentence pairs given by the task. One concern is that the sentence pairs may be positively correlated and the model may have exploited this naive bias. We highlight that this work does not have this concern the tasks in our study test the relative similarity ranking of sentence pairs under the evaluation metrics (both spearman's 's rank used in STS and AUC used in binary classification are ranking-based), Therefore, the overall correlation distribution of sentence pairs does not reveal what is tested by the tasks in this study. To further verify the model has not learned label distribution from the task, we run controlled setting where we treat all sentence pairs from the tasks (pseudo-)posse pposiivee pairs and apply a standard contrastive finetuning framework (i.e. Mirror-BER' and SimCSE finetuning) with Eq. (I) the loss on these pairs. Performance improvement from this finetuning setup will indicate that the overall positive correlation from the sentence pairs can be exploited by the model. Results are shown in Tab. Contrastive methods on pseudo-positive pairs almost alway underperform the original SimCSE and Mirror-BERY checkpoints significantly while TRANS- -ANN-EENCODRR models improve the checkpoints by large margins.
11 The only exception QNLI We suspect it is because QNLI is from the QA domain and contrastive tuning on QNLI sentence pairs has some domain adaptation effect. According to Liu et al. (2021) contrastive tuning on randomly sampled sentences from the target domain can also achieve domain adaptation, and sentence pains from the task thus not must.
14

---

Published as a conference paper at ICLR 2022
dataset->
STS (avg.)
QQP
QNLI
MRPC
avg.
SimCSI-RoBERTha-bas
76.10
81.82
73.54
75.06
76.63
TENC (bi)
78.36
84.13
77.08
75.29
78.72
TENC (cross)
78.86
85.16
80.49
76.09
80.15
4 conträstive tuning
73.93
774.00
77.56"
75.25
75.19
76.72
78.36
75.56
77.18
77.00
TENC (hi)
80.26
82.04
82.12
79.63
81.01
TENC (cross)
82.90
82.52
81.38
82.27
contrastive tuning
72.27
73.51
77.50
75.65
74.7T
Table 9: Compare TRANS- CAANS-ENCOERR with SinCSSE/Miroo-BERRT-SS contrastive tuning on sentence pairs given by the task.
model size
dataset
GPU required
time
TENC (basc)
STS
1
00.5 hrs
TENC (large)
STS
1
1=1. hrs
TENC-mutual (base)
STS
2
11.25 hrs
TENC-mutual (large)
STS
2
hrs
TENC (base)
QQP
hrs
TENC (large)
QQP
1
22.5 hrs
TENC-mutual (base)
QQP
2
^/.25 hrs
TENC-mutual (large)
QQP
2
=5 hrs
TENC (basc)
QNLI
1
=0.. hrs
TENC (large)
QNLI
1
#1.75 hrs
TENC-mutual (base)
QNLI
2
= 11.25 hrs
TENC-mutual (large)
QNLJ
2
R33.5 hrs
TENC (hase)
MRPC
1
A0.25 hrs
TENC (large)
MRPC
1
00.75 hrs
TENC-mutus (base)
MRPC
2
==0.5 hrs
TENC-mutual (large)
MRPC
2
1= hrs
Table 10: Running time for all models across different datasets/tasks. We experimented with NVIDIA V100, A100 and 3090 and found the estimated time to be similar.
A.3 TRAINING TIME
models different tasks and configu- rations in Tab. 1O) All base models can be trained under 2.5 hours and large models can be trained within 5 hours. TENC-
Overall runtime. We list the runtime of
models usually take twice the amount of time training self-distillation model. Also, TENC-mutual models require two GPUs while self-distillation models need only one. We have tested running TRANS-ENCODES on three types of Nvidia GPUs: V100 (16 GB), A100 (40 GB) and 3090 (24 GB). The runtimes are similar.
Cross-encoder vs. bi-encoder. For training one epoch, bi-encoder takes significantly less time than cross-encoders. However, bi-encoders take much longer time to converge (15 epochs vs. 3 epochs on binary classification tasks and 10 epochs vs. epoch on STS; see Tab. 16). As a result, practice, bi-to-cross and cross-to-bi distillations take similar amount of time.
A.4 FULL TABLES
Here, list the full tables of SimCSE binary classification results Tab. 11); SimCSE domain transfer results (Tb.. 12); and full tables for ablation studies (Tb.. 13] Tab. 14). which are presented figures in the main texts in Fig.
15

---

Published as a conference paper at ICLR 2022
# -
QQP QNLI MRPC
avg.
single-model results
80.38
71.38
75.02
75.59
+ TENC (bi)
82.10
75.30
75.71
77.70
TENC (cross)
82.10
75.61
76.21
77.97
+ T'ENC-mutual (bi)
84.00
76.93
76.62
79.18
TENC-mutual (cross)
84.29 77.11
77.77
79.72
81.82 73.54
75.06
76.81
+ TENC (bi)
84.13 77.08
75.29
78.83
+ TENC (cross)
85.16 80.49
76.09
80.58
+ TENC-mutual (bi)
84.36 77.29
76.90
79.52
+ TENC-mutual (cross)
84.73 77.32
78.47
80.17
SimCSF-BBERT-large
82.42
72.46
75.93
76.94
+ TENC (bi)
84.53
78.22
78.42
80.39
+ TENC (cross)
84.18 79.71
79.43
81.11
+ TENC-mutual (bi)
85.72 81.61
79.59
82.31
+ TENC-mutual (cross)
86.55 81.55
79.69
82.60
B-BCEE-RoBBRTaalargge
82.99 75.76
77.24
78.66
+ TENC (bi)
85.65 84.49
79.34
83.16
+ TENC (cross)
86.16
84.69
81.00
83.95
+ TENC-mutual (bi)
85.65
81.87
79.53
82.35
TENC-mutual (cross)
86.38
83.14
81.77
83.76
ensemble results (average predictions two models)
SimCSE-base ensemble
81.10
72.46
75.04
76.20
TENC-base (bi) ensemble
83.11
76.19
75.45
78.25
TExC-base (cross) ensemble
84.15
78.55
78.90
80.53
TEXC-mutual-bass (bi) ensemble
84.18
77.11
76.76
79.35
TEXx-mutuul--asee (cross) ensemble
84.68
77.92
78.22
80.27
SimCSE-large ensemble
82.47
74.09
76.55
77.70
TENC-large (bi) ensemble
85.09
81.35
78.88
81.77
TENC-large (cross) ensemble
86.22
83.78
81.06
83.69
TEXC-mutual-large (bi) ensemble
86.10
81.74
79.56
82.47
TENC-mutual-aarge (cross) ensemble
86.26
82.86
81.41
83.51
Table 11: Binary elassification task results (SimCSE models; full table). AUC scores reported.
A.5 DATA STATISTICS
complete listing of train/dev/test stats of all used datasets can be found in Tab. 15 Note that for STS 2012-2016, dropped all sentence-pairs without valid score. And the train sets include all sentence pairs (w/o labels) regardless of split in each task.
A.6 HYPERPARAMETERS
The full listing of hyperparameters is shown in Tab. 16 Since performing grid-search for all these hparams is ultra-expensive mainly used the default hparams from the sentence-BBel BERT library [2] The hparams for TENC and TENC-mutual are the same.
A.7 PRE-TRAINED ENCODERS
A complete listing of URLs for all used PLMs is provided in Tab. 17
sbe
16

---

Published as a conference paper at ICLR 2022
QQP
QNLI
MRPC
avg.
BRMCCE-BERTEE
80.38
71.38
75.02
75.59
+ TENC (bi)
80.80
72.35
74.53
75.89
+ TENC (cross)
80.84
78.81
79.42
79.69
TENC-mutual (bi)
83.24
72.88
75.86
77.33
+ T'ENC-mtual (cross)
82.40
78.30
78.72
79.81
SmCCE-RROBERTaa-bae
81.82
73.54
75.06
76.81
+ TENC (bi)
82.56
71.67
74.24
76.16
+ TENC (cross)
83.66
79.38
79.53
80.86
+ TENC-mutual (bi)
81.71
72.78
75.51
76.67
+ TENC-mutual (cross)
83.92
79.79
79.96
81.22
SinCSE-BERTT-arge
82.42
72.46
75.93
76.94
TENC (bi)
81.86
71.99
74.99
76.28
+ TENC (cross)
83.31
79.62
79.93
80.95
TENC-mutual
82.30
72.47
76.74
77.17
+ T'NC-mutual (cross)
83.04
79.58
81.18
81.27
S SinCSt-RRBEERTa-larg
82.99
75.76
77.24
78.66
TENC (bi)
82.34
70.88
76.05
76.42
+ TENC (cross)
85.98
80.07
81.20
82.42
TENC-mutual (bi)
85.28
71.56
76.81
77.88
+ TENC-mutual (cross)
86.31
81.77
81.86
83.31
Table 12: Full table for domain transfer setup: testing TRANS-ENCODER (SimCSE models) trained with STS data directly on binary classification tasks.
dataset
STS (avg.)
SinCSE-BERT-bar
76.21
+ TENC (bi) sequential
78.08
TENC (cross) sequential
77.92
+ TENC (bi) refreshing
78.65
TENC (cross) refreshing
78.32
SinCSE-RRBERTA-bDe
76.10
+ TENC (bi) sequential
78.32
+ TENC (cross) sequential
78.72
+ TENC (bi) refreshing
78.36
+ TENC (cross) refreshing 78.86
Table 13: Ablation: sequential training with the same set of weights vs. refreshing weights for all models.
dataset->
STS (avg.)
QNLI
76.21
71.38
+ standard-self-diitene
77.16
70.95
+ TENC (bi)
78.65
75.30
+ TENC (cross)
78.32
75.61
MonCSE-RoBERTa-beee
76.10
73.54
76.25
73.40
TENC (bi)
78.36
77.04
TENC (cross)
78.86
80.49
Table 14: Ablation: compare TRANS-ENCODER with standard self-distillation.
17

---

Published as a conference paper at ICLR 2022
Dataset
[Trin]
[evv
Test
STS 2012
3,108
STS 2013
1,500
STS 2014
3,750
STS 2015
3,000
STS 2016
1,186
STS-B
1,500
1,379
SICK-R
495
4,906
STS train (full)x
37,081
QQP
21,000
1,000
10,000
QNLI
15,463
1,000
4,463
MRPC
5,801
408
1.725
Table 15: A listing of train/dev/test stats of used datasets. collection all individual sentence- pairs from all STS tasks.
task
direction
learning rate
batch size
epoch
max token length
cycles
base models
STS
bi cross
2c-5
32
64
STS
cross +
5c-5
128
10
32
binary
bi + cross
2c-5
64
binary
cross -
5e-5
128
15
32
large models
STS
bi cross
64
STS
cross + bi
5e-5
64
10
32
binary
bi + cross
2e-5
32
64
binary
cross + bi 5e-5
64
15
32
Table 16: A listing of hyperpamters used for all TRANS-ENCODER models.
model
URL
BERT
https: / http::///hggaaaacccecc co co/bert base uncesed
RoBERTa
sttps: htgps//huggsaatace..
SimCSE. BERT hase
ht /hwgg agrane www.ton-oornnn http://owo.o/.//oooooooaooooeeee
SinCSE-RoBBRTT--ee
TtIps://hugsi hage,, 00/0: Inpeloorooo on unppuusup- smmsee roberta base
SinCCSE-BEKT-lage
Bert uacased
sessuppssmcssee. Foberta- 1arqs
Miroo-RoBERTa-bae
LLpp: (Lpst//huggigge
toberla
Minor-
Mimor-RoBERil RoBERTT- RoBBERE--bss-dddeeeen----- bropedr
htt httpss//nuggian http:///gggaaaacc.cmcmo cttp:/caarraaoooo mmaeer
macrree-teeereeeeeeeeeeee-aee roberta base sertence- dropbead
Contrastive BERT base
https://nuggingtt htt.a//aaaaaacccccoooooooaaaaaaa htw.:aactoooooooooooooooooo. Baao///----
http6://hugg ngtaae.co/ htt.:/ooooooooggggg//ggoo hrgp::/..........ooooooggoaong/o str-eeellttraaaaa
Table 17: A listing of HuggingFace URLs of all PLMs used in this work.
18


Reviews:
[{"review_id": "tpFJS_PRTUI", "reviewer": "Reviewer_ofUo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths (Reasons to Accept)\n- Solid work. What is claimed by the paper is clear and persuasive.\n- This paper points out the convention in the sentence-pair modeling literature that bi- and cross-encoders have been considered separately. From this, it proposes an intuitive combination of the two encoder paradigms.\n- When sticking to the experimental environment reported in the paper, the improvement in performance looks significant.\n\n# Weaknesses (Reasons to reject)\n- It is hard to say that the proposed method is novel, especially considering that all of the technical components used in this paper (e.g., pre-trained language models, contrastive learning for sentence modeling, knowledge distillation between two similar models, pseudo labeling for data augmentation/generation) are quite common in the literature. I believe that the main novelty of this paper only comes from the fact that it proposes for the first time (to the best of my knowledge) to combine bi- and cross-encoders. However, the specific way introduced to merge the encoders is not that special.\n- Sentence modeling using (fine-tuned) pre-trained language models is one of the most active research areas in NLP. Considering this, it is questionable whether the proposed experiments are comprehensive enough to back up the paper's claim. Even though there exist so many other concurrent and similar works for sentence modeling other than SimCSE, this paper only focuses on incrementally refining SimCSE. To claim that the proposed method is universally effective, it would be desired to test the method on top of other similar sentence modeling frameworks in addition to SimCSE.\n- After all, it is still unclear why the iterative and coupled fine-tuning of bi- and cross-encoders results in performance improvement. Assuming that the information contained in the same dataset is ideally equal, what is the specific factor that can be easily captured by cross-encoders but not by bi-encoders (and vice versa)? The reported improvement in performance is entirely due to the combination of two different encoder paradigms? How about using two separate bi-encoders or cross-encoders? What is the main and independent factor that contributes to the improvement most?\n", "summary_of_the_paper": "# Summary\n\n- This paper highlights the fact that bi-encoders (sentence encoders) and cross-encoders (sentence-pair encoders) have been considered somewhat separately in the literature of sentence-pair modeling, and that cyclic knowledge distillation from one to another (and vice versa) can be effective for improving the performance of both the encoder stereotypes.\n- The authors combine the two sentence modeling paradigms into an iterative joint framework, which is called TRANS-ENCODER, to simultaneously learn enhanced bi- and cross-encoders.\n- Specifically, given the original SimCSE or Mirror-BERT checkpoints provided by prior work, the proposed framework first (pseudo-)labels sentence pairs sampled from the task of interest by utilizing the off-the-shelf sentence encoders. And then, the labeled sentence pairs are utilized to improve the quality of cross-encoders. Further, the fine-tuned cross-encoders can also be exploited to label other sentence pairs which are again able to be used as a dataset for tuning the bi-encoders (sentence encoders). The framework repeats this process until both the bi- and cross-encoders become satisfactory in terms of their performance.\n- The proposed method is evaluated on semantic textual similarity (STS) tasks and binary classification datasets such as QQP, QNLI, and MRPC. The results demonstrate that the method improves both the bi- and cross-encoders to be more suitable for sentence-pair modeling tasks.", "summary_of_the_review": "- This paper clearly points out that bi- and cross-encoders have been evaluated and dealt with separately in the literature. From this, it proposes to combine the strengths of the two different paradigms by relying on cyclic knowledge distillation (or data augmentation).\n- The paper is generally well-written and easy to understand. However, it is a little bit doubtful whether (1) the proposed method is novel enough and (2) the reported experiments are comprehensive enough. Furthermore, I feel that this paper does not contain a thorough analysis on \"why\" the proposed method works, which is what I want to know the most.\n- Therefore, my suggstion is a weak accept (or on the borderline to be accepted).\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "p8ID7YaNVFU", "reviewer": "Reviewer_DZEu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The author experimented on stronger models (RoBERTa-large) too, and the results still hold.\n- The writing is very clear. The figures are great. Reproducibility is likely strong. \n\nConcerns/comments\n- The paper will be stronger if the authors can include more hypotheses or justification on why, in some cases (see Table 1 and Table 8), TNec-mutual works much better than TEnc (cross), especially because the bi-encoders do not perform as well.\n- It will be great if the paper includes more details on the training time. How do the training time compare among different approaches, for GLUE tasks (especially larger ones)? How difficult is it to train the mutual distillation? \n- The mutual distillation approach requires doing self-distillation on multiple pretrained language models. What’s the aggregate training overhead of mutual distillation vs. cross-encoder only vs. bi-encoder only?  \n\nMore questions...\n- What's the training stability for self-distillation? Will results have a small standard deviation?\n\nMinor\n- I’m interested in seeing the performance on more difficult tasks that might require more world knowledge (e.g., MNLI), but this is not necessary. \n- Typo on page 2 “distil” -> “distill”\n- Page 3, paragraph 3, should be Liu et al. (2021) instead of \\citep?\n", "summary_of_the_paper": "This paper deals with tasks where we need to do pairwise comparison between two sequences. Specifically, the paper experimented on STS, QQP, QNLI, MRPC. The paper proposes integrating two models together: bi-encoders and cross-encoders. \n\nA bi-encoder encodes each of the two sequences separately and maps to a common embedding space. A cross-encoder first concatenates two sequences, and the concatenation is sent to the model. There’s more computation for the cross-encoder case, but the cross-encoder usually outperforms the bi-encoder. \n\nFigure 1 illustrates the authors’ proposal well. First, start with a pretrained language model. Then, the bi-encoder generates pseudo-labels for the cross-encoder to learn. Next, the cross-encoder generates (more accurate) pseudo-labels for the bi-encdoer to learn. We reiterate this process. This approach is named self-distillation. \n\nThe author proposes an extension called mutual distillation where the authors do self-distillation on multiple pretrained language models in parallel (but the paper only experiments with two--BERT and RoBERTa). \n\nOn STS, QQP, QNLI, MRPC, the mutual distillation approach archives good performances (much better than the bi-encoder baseline SimCSE). \n", "summary_of_the_review": "The approach is straightforward and nicely tied the bi-encdoer approach with the cross-encoder approach. The improvements compared to SimCSE is significant. The writing is clear and the reproducibility is good. However, I would expect more theoretical justification on why TEnc-mutual could be much higher than TEnc-cross. Moreover, training stability (of self distillation) and aggregate training time (for mutual distillation) deserve more discussion in my opinion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gamhc_4C1KC", "reviewer": "Reviewer_53d8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. TRANS-ENCODER is the first completely unsupervised cross-encoder for sentence similarity.\n\n2. TRANS-ENCODER outperforms state-of-the-art unsupervised sentence encoders on the sentence similarity benchmarks.\n\nWeaknesses:\n1. It is counter-intuitive that in the \"bi- to cross-encoder\" phase, the learnt cross-encoder outperforms the bi-encoder which produces the labeled data for the cross-encoder. It is also surprising that the student bi-encoder sometimes outperforms its teacher cross-encoder in the \"cross- to bi-encoder\" phase. The paper lacks the theoretical analysis and empirical investigation of why exactly these happened to TRANS-ENCODER.\n\n2. Although the labels are not used, the test sets of any dataset should not be used for training. They should be considered unseen and held-out in the training phase.\n\n3. What is the rationale for using different training losses for \"bi- to cross-encoder\" and \"cross- to bi-encoder\" (soft BCE vs. MSE)? Is it possible to use the same loss for the two phases?", "summary_of_the_paper": "This paper proposes TRANS-ENCODER, an unsupervised approach to training bi-encoders and cross-encoders for sentence similarity tasks such as information retrieval, natural language inference, semantic textual similarity and clustering. The core idea of TRANS-ENCODER is self-distillation that alternatively trains a bi-encoder and a cross-encoder with pseudo-labels created from the other. The authors also propose a mutual-distillation extension to mutually bootstrap two self-distillation models trained in parallel. The effectiveness of TRANS-ENCODER is verified by empirical evidence.", "summary_of_the_review": "Overall, this paper introduces an unsupervised method that seems to effective in sentence similarity tasks, but it lacks the insights and empirical investigation of why this method works well. The work is not well-supported. So I would consider it marginally below the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bjCrSo-v_JV", "reviewer": "Reviewer_z3e4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "S1. The proposed self-distillation framework is interesting and technically sound. The joint training of bi-encoder and cross-encoder goes beyond the existing self-distillation framework, which I believe is the major novelty of this paper. After going through the presented methods in this paper, I'm quite positive that improvements got to be achieved over baselines like SimCSE. \n\nS2. Comprehensive experiments are performed, which clarify most of the critical aspects about the proposed methods.\n\nW1. My biggest concern. I'm afraid that the proposed framework is not really unsupervised (I would call it pseudo unsupervised instead). In fact, it relies on labeled data: although labels are discarded, the pairwise relationships of the sentences are preserved, in which a large portion of the sentence pairs are positively correlated. The paper has to demonstrate that it may still work on a plain corpus where no such paired data is available; otherwise, it will be unrealistic as an unsupervised method. \n\nW2. A minor concern towards the experiment. In table 1, TENC (cross) underperforms TENC (bi) in many cases, which is counter-intuitive, and kind of goes against the statement made in section 2.2.", "summary_of_the_paper": "This paper focuses on the unsupervised training of bi-encoder and cross-encoder for sentences. A self-distillation framework is proposed, in which the bi-encoder and cross-encoder may iteratively distill knowledge from each other. The experiments are performed with typical sentence pair datasets like STS, QQP, QNLI, MRPC, where improvements are achieved over SimCSE and mirror-BERT. ", "summary_of_the_review": "The paper is interesting and technically sound. The experiments are comprehensive and inspiring. However, the assumption of pair sentences will probably make it unrealistic in reality.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: For pairs of pieces of text, the central idea of this paper is to combine the approaches of using bi-encoders (where a vector is formed from each text then compared), which are easily trained in an unsupervised manner, with cross-encoders (where the two texts are related at the token level), which are normally trained in a supervised manner. The chief contribution of this paper is to train a combined model (as a "trans-encoder") by doing co-training of both model types in an alternating cyclic self-distillation framework. The paper suggests that this allows unsupervised training of a cross-encoder. This claim met some pushback from the reviewers, since the method does require good quality aligned text pairs (much like a traditional MT system does), and so the result is a  task-specific sentence-pair modeling approach rather than a generic unsupervised learning approach. 

In the discussion, downsides included the claims of "unsupervised" being overstated, the genuine remaining need for related sentence pairs, the lack of a more theoretical understanding of why this works, and the feeling that the paper is not yet fully mature. Upsides include solid work building from existing models, big performance improvements over SimCSE, novelty in combining previous ideas in a new way for a new problem, and good experiments. To my mind, while the requirement of related sentence pairs does mean the model is task-specific and less than fully unsupervised, this is still a common and useful scenario, the performance of the model is strong, and, while the proposed model is built from existing components and ideas, they are combined in an interesting new way to achieve an intriguing and strong new way of training models, and the discussion here (and now in Appendix A.2) of what the authors had to do to get the model to work in terms of choosing different losses, etc., convincingly demonstrated that the authors had thought significantly and deeply about the nature of their proposal and how to get it to work well. Moreover the authors were able to work expeditiously during the reviewing period to address other weaknesses, such as now providing results with other methods than SimCSE (DeCLUTR and Contrastive Tension) and on other language models (RoBERTa).

As such, although this paper is clearly somewhat borderline rather than an unambiguous accept, I find myself quite convinced by the novelty, thoroughness, and intriguing nature of this work, and so my vote is to accept it.
Decision: Accept (Poster)