BERT Has More to Offer: BERT Layers Combination Yields Better Sentence Embeddings
Authors: MohammadSaleh Hosseini
Munawara Saiyara Munia
Latifur Khan
Venue: EMNLP 2023 Findings (2023)
Keywords: Sentence Embedding
BERT
BERT-LC
Layers Combination

Abstract: Obtaining sentence representations from BERT-based models as feature extractors is invaluable as it takes much less time to pre-compute a one-time representation of the data and then use it for the downstream tasks, rather than fine-tune the whole BERT. Most previous works acquire a sentence's representation by passing it to BERT and averaging its last layer. In this paper, we propose that the combination of certain layers of a BERT-based model rested on the data set and model 
can achieve substantially better results.
We empirically show the effectiveness of our method for different BERT-based models on different tasks and data sets. Specifically, on seven standard semantic textual similarity data sets, we outperform the baseline BERT 
by improving the Spearman's correlation by up to 25.75\% and on average 16.32\% without any further training. We also achieved state-of-the-art results on eight transfer data sets by reducing the relative error by up to 37.41\% and on average 17.92\%.

BERT Has More to Offer: BERT Layers Combination Yields Better Sentence Embeddings
MohammadSalch Hosseini", Munawara Saiyara Munia', Latifur Khan'
'Department of Computer Science
DDeparmment of Electrical and Computer Engincering
The University of Texas at Dallas
seryyednohammadsaleen. hosseini, munavarasaiyara.mmm munia,
hww.esaille.hsssiini.i.... 1khan | @utdallas. edu
Abstract
Moreover, for sentence-pair tasks, BERT uses cross-encoder; nonetheless, this setup is inappropri- atc for certain pair regression tasks, such as finding the most similar sentence in a data set to a spe- cific sentence due to the large number of possible combinations (Reimers and Gurevych, 2019).
Obtaining sentence representations from BERT- based models as feature extractors invaluable it takes much less time to pre-compute one- time representation of the data and then use it for the downstream tasks, rather than fine- tune the whole BERT. Most previous works acquire sentence's 's representation by passing it to BERT and averaging its last layer. In this paper, we propose that the combination of cer- tain layers
Considering the aforementioned drawbacks, re- searchers have tried to derive fixed-sized sentence embeddings from BERT or proposed new BERTs with the exact same architecture but different ways of training (Reimers and Gurevych, 2019; Li et al., 2020). After training, the resultant BERT is used in feature-based manner by passing the sentence and obtaining its embedding vector in different ways, such as averaging the last layer of BERT.
BERT-based model rested on the data set and model can achieve substan- tially hetter results. We empirically show the effectiveness of method different BERT- based models on different tasks and data sets. Specifically, on seven standard semantic textual similarity data sets, outperform the baseline BERT by improving the Spearman's 's correlation by up to 25.75% and on average 16.32% with- out any further training. We also achieved state- of-the-art results on eight transfer data sets by reducing the relative error by up to 37.41 % and average 17.92%.
In this paper, present simple, yet effective and novel method called BERT-LC (BERT Layers Combination). BERT-LC combines certain layers of BERT order to obtain the representation of a sentence. As we will show, this model significantly outperforms its correspondent BERT baseline with no need of any further training. Our work was inspired by Jawahar et al. (2019), who show that different layers of BERT carry different features, such surface, syntactic, and semantic. We argue that each data set with its unique distribution might need a different set of features for its sentences, which can only be fully exploited by combining different layers of BERT in an unsupervised way. (1) We pro- pose a new method called BERT-LC that is capable of acquiring superior results by combining certain layers of BERT instead of just the last layer, in an unsupervised manner. We also include the embed- ding layer, which was to our knowledge ignored in previous works. (2) We additionally show that our method improves SBERT (Reimers and Gurevych, 2019) and SimCSE (Gao al., 2021). which were specifically designed for obtaining sentence repre- sentations (opposite to BERT and RoBERTa (Liu et al., 2019)). (3) We developed an algorithm that
1 Introduction
Learning sentence vector representations is cru-
cial problem in natural language processing (NLP)
and has been widely studied the literature (Con-
neau et al., 2017; Cer et al., 2018; Li et al., 2020).
Given a sentence, the goal to acquire a vector that semantically and/or syntactically represents it. BERT (Devlin et al., 2019) has set new state-of-th art records on many NLP tasks (Madabushi et al., 2020; Hu al., 2022; Skorupa Parolin al., 2022; Wang and Kuo, 2020). However, this achieved by fine-tuning all of BERT's s layers. The disadvan- tage of fine-tuning is that computationally ex- pensive as even BERT-base has 110M parameters; hence, pre-computing a representation of the data and using it for the downstream task is much less computationally expensive (Devlin et al., 2019).
Our contributions are as follows:
OOur code is available at: https: ttwb. thub. Di www.mmocc//BBeee. /BERT- Layers wwweers--ombiiie Combinationn

---

speeds up the process of finding the best layer com- bination (among 213 layer combinations in base cases) by factor of 189 times. (4) We propose an innovative method that integrates the layer com- bination method with the CLS pooling head, im- proving the performance metrics for certain models. (5) We achieve tate-of-the-art performance on the transfer tasks using layer combination.
BERT, RoBERTa, SBERT, and SimCSE. To the best of our knowledge, no similar work has been done that whether combines different and arbitrary layers of these models or further integrates layer combination and the CLS pooling head.
3 Approach
Given an input sentence S = ($1, $2,.... ..........g sN). the goal sentence embedding model is to output a vector Es E IR which carries the semantic and/or syntactic information of the sentence. In order to obtain a sentence embedding, we first pass the sentence to a BERT-like model, which outputs the tensor H E L'xxxx in which d is the dimension of the token vectors in each layer, and L' L+ 1 is the number of layers (including Layer 0) BERT. We then apply to this tensor a pooling function p. which can be maz or mean, but we choose mean as it yielded better results in our experiments. The pooling is done across all the tokens in all the desired layers set, D. For example, for mean pooling, p is defined as:
We demonstrate the superiority of our approach through conducting extensive experiments on seven standard semantic textual similarity (STS) data sets and eight transfer tasks, On the STS data sets, our method is able to outperform its corresponding baseline by up to 25.75% and average 16.32% for SenT-llare-uncased We also achieve the state- of-the-art performances on transfer tasks, reducing the previous best model's relative error rate by an average of 17.92% and up to 37.41%.
2 Related Work
Learning sentence embeddings is a well-studied realm in NLP. There are mainly two methods used for this purpose: methods that use unlabeled data or labeled data. Although the latter use labeled data, the target data sets and tasks on which they are tested are different from the training data set and task. Early work on sentence embedding utilized the distributional hypothesis by predict- ing surrounding sentences of a sentence (Kiros et al., 2015; Hill et al., 2016; Logeswaran and Lee, 2018). Pagliardini et al. 2018 build on the idea of word2vee (Mikolov et al., 2013) using n- gram embeddings. Some researchers simply use the average of BERT's last layer embeddings as the sentence embedding (Reimers and Gurevych, 2019). Recently, contrastive learning has proven to be very powerful in many domains (Khorram et al., 2022; Munia et al., 2021; Hu et al., 2021); thus, some recent methods exploit contrastive learning Gao et al., 2021; Zhang et al., 2022; Yan et al., 2021) and utilize the same sentence by looking at it from multiple angles. For instance, the popular method SimCSE leverages different outputs of the same sentence from BERT's standard dropout.
p(H, - D
Him
(1)
IED
n=1
where Hin C IRd is the Transformer vector at layer number corresponding to the nth token.
Previous works usually set 1) to {I.} or use CLS pooler, the output of the MLP layer attached to the last layer's first token ((CLS)) obtain the sen- tence embedding. Using CLS pooler was shown to underperform last layer averaging (Reimers and Gurevych, 2019). Further, as we will empirically show, choosing D = {L} leads to significant un- derperformance as well; hence, in this work, we iterate through all possible Ds (D E P(A) 0, where A = {O..... L}), performing 1) our layers to pool from.
and choose the best-
As iterating through all possible Ds is very time- consuming, we propose an algorithm that speeds up the process of finding the best layer combination (213 layer combinations base cases) by a factor of 189 times, which can be found in Appendix A.
We further propose an extension to our method, and that is exploiting the MLP head and layer combination simultancously. The idca is to pass P(H, D) in Eq. the MLP head and use the out- put of the MLP head as the new p. We spot that this method works better than merely using the layer combination on SimC We conjecture that this is because SimCSE's MLP head was trained for learn-
There are some previous works (Ethayarajh, 2019; Jawahar et al., 2019; Bommasani et al., 2020) that investigate the impact of different BERT lay- ers. However, they either investigate each layer individually or are restricted to considering set of consecutive layers strictly starting from layer 1.
We apply our method to different variations of

---

ing sentence embeddings opposed to the other methods, such as BERT, SBERT, and RoBERTa; hence, it carries important information to be uti- lized. Consequently, we use the two-step pipeline of layer combination and MLP for SimCSE mod- cls, and we propose that the two-step pipeline is utilized for any other BERT-baase sentence embed- ding model whose MLP head has been trained for sentence embedding.
combinations with up to four and three layers, re- spectively, as we did not spot much difference when combining more layers.
For the logistic regression, we use SAGA (De- fazio et al., 2014) as the optimizer, 0.01 as the tolerance level, 200 as the maximum number of iterations, and 10 as the regularization parameter.
4.3 Results and Discussion
The results for our proposed method and the base- lines transfer tasks are shown in Table 1. The results are statistically significant (p-value < 0.01). Note that our methods are denoted by an LC at the end of them (e.g.. RoBERTa-L-LC means RoBERTa-L when layer combination is used).
4 Experiments and Results
We carry out our experiments on two different tasks: transfer and STS tasks. In this section, we discuss the tranasfer tasks, while the STS tasks are dis- cussed in Appendix B.
Since all models have accuracies close to 100% on most of the transfer data sets, we believe it is more reasonable to see how much improvement acquiring by considering the relative error reduction. Relative error for a model and its base- line with respective accuracies of A& and AMB is defined as AMr-AMR, From Table 1, we observe that our method significantly outperforms its corre- sponding baselines for all the models on all the data sets, For example, RoBERTa-B-LC, SBERT-L- LC, and UnSupSinCSE-RL--CC improve on their baselines by reducing the relative errors by up to 36. 36.19%, 45.93%, and 40.80% and on average 19.8 19.89%, 14.95% and 27.31 %, respectively.
4.1 Data Sets and Evaluation Setup
For transfer tasks, we use eight data sets from the popular SentEval toolkit (Conneau and Kiela, 2018), which is used for assessing the quality of sentence embeddings: MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST and SSTM (binary and six-class Stanford Sentiment Tree- (anhs) et al., 2013), TREC (Li and Roth, 2002), and MRPC (Dolan et al., 2004).
T-AxB
For each data set, we combine all of its data subsets and randomly split them into training- development (train-dev) and test data with ratios 85% and 15%. This outer cross- velidatiin is done randomly for 10 times, and the average accuracy results on the test data are reported. We further utilize a 10-fold inner cross-validation on train-dev data set, with ratios 82% (train) and 18% (dev).
Furthermore, we spot that our method is more effective on unsupervised versions of SimCSE mod- els. For instance, the relative error improvement over UnSupSimCSSERL is 27.31%, whereas it is 14.61 % for SupSimCSE-RI. However, there is still important information to be extracted from all the layers of the supervised SimCSE models as well. We see that even an unsupervised model such RoBERTa-L when upgraded with LC can achieve 84.58% beating SupSimCSE-RL, a supervised model which held the previous best average re- sult (83.73%). ROBERTA-LL-LC: average accuracy does not fall very short of the new statc-of-the-art model, SupSimCSE-RL--LC (84.68% vs. 85.89%). Finally, we have improved the previous state-of- the-art models on all of the transfer data sets by reducing the relative (absolute) errors by up 37. 17.41% % (3.67%) and on average 17.92% (1.90%), respectively. We achieve a statc-of-the-art average accuracy of 86.36% % on the transfer data sets. We have provided more baselines, which can be found in Appendix D.
The sentence embeddings by our method or the baselines are used feature vectors for a logistic regression classifier. Note that even though there exists a training data set, it is only utilized by the logistic regression classifier and not in our method.
4.2 Baselines and Hyper-parameters
We use the following baselines and apply our layer combination method to them: BERT base/large (un)cased (BERT-(B/L))...) (von) (anceasad) RoBERTA base/large (RoBERTa-B/L), SBERT and SRoBERTa base/large (SBERT/SROBERTa. B/L), (un)supervised SimCSE BERT/ROBERTA base/large (CSS SupSimCCE-RR/RLL,
and which are SimCSE models trained with an additional MLM head. For the base and large models, we consider layer

---

SUBI
SSTM
Avg.
SUJB
MPOA
MRFC
BERT Bunase
80.16
83.17
93.97
94.35
46.06
74.74
70,57
85.01
77:37
S
88.96
93.93
89.55
80.04
74.24
19:19
80.81
BLXT Bu lnst 89881.19
86.17
95.00
85:10
47.53
55.76
73.06
87.08
80.37
SUERE 8 LC
$3.61
89.95
95.17
91.06
49.01
58.56
78.71
89.05
83.22
BEXT-
LC
X2.24
86.60
95.40
90.42
49.22
58..66
77.17
88.17
82.25
SRaBERD B
$4.57
90.12
92.57
89.21
50,59
$1.79
77.13
90,12
$2.05
BLKT
CLS
77.86
83:07
91.33
85.27 29
45.19
09.91
70.64
83,45
15.98
91.75
94,50
90.51
53.65
$7.95
78.92
98.78
84.26
BEXT a
90.55
95.19
94.64
87.63
46.48
53,65
74,30
79.78
SupSinC
89.31
54,60
89:73
50.16
52.75
74.60
58.81
91,48
BEXT
X1.42
86.56
95.11
89.88
47.76
$5.00
75.36
86.86
81.37
SupSinCSE $1.35
89.31
95.68
90.87
50.16
$9.09
77.68
89.67
85,04
Koblekia last avg
N2.58
31.09
94.55
85.85
50,25
$1.90
72.51
81:28
79.95
SupSimCSH: RU
$4.29
91.50
93.12
90.19
52.64
51.12
76.14
90.11
82.40
RoBERTo-B
85.89
95.68
74.85
83.11
SupSinCSE RB-LC
92.35
84.52
UnSupSinn BB LC
UnSupSimt 85-
83.61
35
86.77
95.16
90.03
85.75
45-14
47.67
$9.05
72.M4
77.29
87.49
81.51
79.61
SupSinCSH-RIe
SupSinCSi: Rid LC
96.11
92.38
95/51
90.63
$4.10
$8.40
76:23
78.14
90.13
91.00
82.47
84.57
UnSuySinnCEE:RB
90.85
85,
36
92.40
86.70
47.25
76,.1
72.74
85.67
78.39
SHERT-
54.50
90.67
$4.40
50.25
49.24
79.71
74.99
90.92
81.85
UnSupSimt RL-
84.00
88.99
94,69
88884
86,02
77.59
88.35
82.37
SUERT-L-LC
65.41
91.64
95
91.15
51.54
89.03
79.03
91.46
84.40
Lavoons 43.5
X3.22
95
10.00
90.75
46.73
69.10
76.49
SRORERTOH
90.83
97.44
83.16
BERP Lascone last aug
83.68
88:
95.51
85.51
19.83
K3.81
71,19
88.18
30.88
SRoBERTLLL-LCC
57.95
92.49
95.36
92,06
54.23
58.51
79.56
92.94
85.39
BLKU
LC
N5.21
89.38
96.08
90.17
51,06
$8.89
76.57
89.96
83.48
SupSinCCSE-Bt
85.47
90.69
95.01
90.35
51.16
$4.28
73.70
90.83
82.69
RERT PERT CLS
81.65
62:79
91.41
83.49
45.76
64.21
70:00
84:83
75.53
SupSinCSF-BI-11 65.60
90.69
95.888
91.30
52.30
89.47
77.01
91.39
84.19
Lowwe ang
85.03
49:23
$4.10
72.62
BLKT LC
85.34
90.19
95.41
90.50
50.92
88.62
77.17
90.00
83.52
SupSinCSE-RR: SupSioCCSE-II..C
98.69
97.81
90.69
96.29
92.01
55.39
90.19
79,82
93.49
H5.09
RollERTT---bsst
64.30
85
91.93
87:25
50.59
61.75
67.24
89:11
80.19
KolliKia
88.04
91.48
96.51
54.11
88,06
75,01
92.08
84.58
SupSinCSS-RR..r
67.78
92. 10
94.72
90.51
52.43
63.85
74.30
97.60
83.55
UnSupSimn
84.85
55:15
60
SupSinCSSE.RL....cco
93.44
96.41
91.43
56.10
$8.06
78.97
93.44
85.87
UnSupSimCSE: BL-BLLLC
84.85
89.84
95.77
90.62
50.93
89.07
77.10
90.04
83.53
(bi Seperiiadd Micil
82.20
86.34
92.77
88.12
45.98
k2.02
73.01
88.08
79.03
LiSSigSi(SSE-RL-L.
86.75
91.01
96.72
90.85
52.83
87.93
79.13
91.57
84.47
tiveperried Modak
Table 1: Transfer task results for different sentence embedding models (measured as accuracy * 100)
MAIG
STOLLC
102
111
114
100
Rade
(a) TREC
(b) SUBJ
(c) MR
Figure 1: Varying the size of the training data for transfer tasks,
5
Ablation Studies
6 Conclusion and Future Work
While more ablation studies, such as the effect of the number of layers in layer combination and the effect of excluding a particular layer from layer combination, can be found in Appendix E, in this section, we discuss the performance of layer com- bination in limited data settings. For this, we re- duce the size of training data for MR, SUBJ, and TREC, while keeping the test data untouched. Fig. shows the results for ROBERTA-B (RB), BERT- B Beased (BBC) and SROBERTA-B (SRB) on these data sets. We report the average results of 10 dif- ferent seeds. As we can see, for all the models on all the data sets, the more reduction in size is done, the more the difference is between the LC version and the last layer average. For instance, for RB on SUBJ, on the original training data, the accuracies for RB-LC and RB are 94.77% and 93.73%, re- spectively, while when reducing the size to 1/256th, the respective accuracies are 78.99% and 50.83%. This suggests that even more beneficial to use LC when one has small training data.
In this paper, we proposed a new method called BERT Layer Combination, a simple, yet effee- tive framework, which when applied to various BERT-based models, significantly improves them for the downstream tasks of STS and transfer learn- ing. Further, it achieves the saate-of-the-ar perfor- mances on eight transfer tasks. Our method com- bines certain layers of BERT-based models in an unsupervised manner, which shows that different layers of BERT hold important information which was previously ignored. We demonstrated the effec- tiveness of our approach by conducting comprehen- sive experiments on various BERT-based models and on a host of different tasks and data sets.
As future work, we would like to apply the tech- nique of layer combination to other NLP tasks (e.g.. punctuation insertion (Hosseini and Sameti, 2017) and question answering (Qu et al., 2019)) and also utilize it in other domains where deep learning mod- els are used (e.g.. biosignal analysis (Munia et al., 2020, Munia et al., 2023) and image captioning

---

(Huang et al., 2019)).
2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385- 393, Montréal, Canada. Association for Computa- tional Linguistics.
7 Limitation
One limitation of our work is that though we ap- plied our layer combination technique to raft of different models, all of them are BERT based; ergo, whether or not this technique works on other deep learning NLP models remains unsettled. We leave this experimentation as future work.
Eneko Agirre, Daniel Cor, Mona Diab, Aitor Gonzalez- Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In Second Joint Conference Lexical and Computational Semantics (*SEM), Volume I: Proceedings of the Main Confer- ence and the Shared Task: Semantic Textual Similar- ity, pages 32-43, Atlanta, Georgia, USA. Association for Computational Linguistics.
8 Acknowledgement
Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020. Interpreting Pretrained Contextualized Representa- tions via Reductions to Static Embeddings. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 4758- 4781, Online. Association for Computational Lin- guistics.
This research was supported in part by NSF awards DMS-1737978, DGE-2039542, OAC-1828467, OAC- 1931541, and DGE-1906630, The material presented here is partially based on High Perfor- mance Computing (HPC) resources supported by the University of Arizona TRIF, UITS, and Re- search, Innovation, and Impact (RIT) and main- tained by the UArizona Research Technologies de- partment.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation -SemEval-2011 pages 1 1-14, Vancouver. Canada. Association for Computational Linguistics.
References
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, lñigo GazzzoG Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 task 2: Semantic tex- tual similarity, English, Spanish and pilot inter- pretability. Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 252-263, Denver, Colorado. Association for Computational Linguistics.
Daniel Cer, Yinfei Yang. Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Suajardo-Cespedee
Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations, pages 169-174, Brussels, Belgium. Association for Computational Linguistics.
Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. In Proceedings of the Fleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonvalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th Interna- tional Workshop on Semantic Evaluation (SemEval 2014), pages 81-91, Dublin, Ireland. Association for Computational Linguistics.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Nat- ural Lnguage Processing, pages 670-680, Copen- hagen, Denmark. Association for Computational Lin- guistics.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross- lingull evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemExal-2016), pages 497-511, San Diego, Califor- nia. Association for Computational Linguistics.
Aaron Defazio, Francis Bach, and Simon Lacoste- Julien. 2014. Saga: A fast incremental gradient method with support for non-strongly convex composite objec- tives. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEEval-2012 task pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Compu- tational Semantics Volume I: Proceedings of the main conference and the shared task, and Volume
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of

---

the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186. Minneapolis, Minnesota. Association for Computational Linguistics.
Computational Linguisties: Human Language Tech- nologies, pages 5469-5482. Seattle, United States. Association for Computational Linguisties.
Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yoni Yong Wei. 2019. Attention on attention for image caption- ing. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4634 4643.
Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Un- supervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In COL- ING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 350- 356, Geneva, Switzerland. COLING.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computational Linguistics.
Kawin Ethayarajh. 2019, How contextual are contextu- alized word representations? Comparing the geom- etry of BERT, ML,,,, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing (EMNLP- (JCNLP), pages 55-65. Hong Kong, China. Association for Computational Linguistics.
Soheil Khorram, Jacyoung Kim, Anshuman Tripathi, Han Lu, Qian Zhang. and Hasim Sak. 2022. Con- trastive siamese network for semi-supervised speech recognition. In ICASSP 2022-2022 IEFF Interna- rional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7207-7211. IEEE
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 19140,910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang. Yiming Yang, and Lei Li, 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 9119-9130, Online. Association for Computa- tional Linguistics.
Felix Hill, Kyunghyun Cho. and Anna Korhonen. 2016. Learning distributed representations sen- tences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Hu- man Language Technologies, pages 1367-1377, San Diego California. Association for Computational Linguistics.
Xin Li and Dan Roth. 2002. Learning question classi- fiers. In Proceedings of the 19th International Con- ference Computational Linguistics Volume I, COLING '02. page 1-7, USA. Association for Com- putational Linguistics.
Seyyed MohammadSaleh Hosseini and Hossein Sameti. 2017. Creating a corpus for automatic punctuation prediction in persian texts. In 2017 Iranian Confer- ence on Electrical Engineering (ICEE), pages 1537-
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
1542.
Minqing Hu and Bing Liu. 2004. Mining and sum- marizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '04, page 168-177, New York, NY, USA. Association for Computing Machinery.
Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representa- tions. CoRR, abs//803. 803.02893.
Harish Tayyar Madabushi, Elena Kochkina, and Michael Castelle. 2020. Cost-sensitive bert for gener- alisable sentence classification with imbalanced data. arXiv preprint arXiv:2003. 11563.
Xiang Hu, Teng Li, Tong Zhou, Yu Liu, and Yuanxi Peng. 2021. Contrastive learning based on trans- former for hyperspectral image classification. Ap- plied Sciences, 11(18):8670.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zam- parelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference Language Resources and Evaluation 14), pages 216-223, Reykjavik, Iceland. European Lan- guage Resources Association (ELRA).
Yibo Hu, MohammadSalch Hosseini, Erick Sko- rupa Parolin, Javier Osorio, Latifur Khan, Patrick Brandt, and Vito D' O'razio.. 2022. ConfiBERT: A pre-trained language model for political conffict and violence. In Proceedings of the 2022 Conference of the North American Chapter of the Association for

---

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. In Advances in Neural Information Processing Sys- lems, volume 26. Curran Associates, Inc.
learning in python. Journal of machine learning re- search, 12(Oct):2825 2830.
Chen Qu, Liu Yang, Minghui Qiu, W. Bruce Croft, Yongfeng Zhang, and Mohit lyyer. 2019. Bert with history answer embedding for conversational ques- tion answering. In Proceedings of the 42nd Interna- tional ACM SIGIR Conference on Research and De- velopment in Information Retrieval, SIGIR' 19, page 1133-1136, New York, NY, USA. Association for Computing Machinery.
Munawara Saiyara Munia, Seyyed MohammadSalch Hosseini, Mehrdad Nourani, Jay Harvey, and Hina Dave. 2021. Imbalanced EEG analysis using one- shot learning with siamese neural network. IEEE.
Munawara Saiyara Munia, Mchrdad Nourami, Jay Har- vey, and Hina Dave. 2023. Interictal epileptiform dis- charge detection using multi-head deep convolutional neural network. In 2023 45th Annual International Conference of the IEEF Engineering in Medicine Biology Society (EMBC).
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLL)) pages 3982-3992 Hong Kong, China. Association for Com- putational Linguistics.
Munawara Saiyara Munia, Mehrdad Nourani, and Sammy Houari. 2020. Biosignal oversampling us- ing wasserstein generative adversarial network. In 2020 IEEE International Conference on Healthcare Informatics (ICHI), pages 1-7.
Erick Skorupa Parolin, MohammadSaleh Hosseini. Yibo Hu, Latifur Khan, Patrick T. Brandt, Javier Osorio, and Vito D' D'Orazio. 2022. Multi-coped: A multilingual multi-task approach for coding politi- cal event data on conflict and mediation domain. In Proceedings of the 2022 AAAI/ACM Conference AI, Ethics. and Sociery, AIES '22, page 700-711, New York, NY, USA. Association for Computing Machinery.
Mattco Pagliardini, Prakhar Gupta, and Martin Jaggi. 2018 Unsupervised learning of sentence embed- dings using compositional n- n-gram features. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. Vol- ume I (Long Papers), pages 528-540, New Orlcans, Louisiana. Association for Computational Linguis- tics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642 Seattle, Washington, USA. Association for Computational Linguistics.
Bo Pang and Lillian Lee. 2004. A sentimental educa- tion: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting on Association for Com- putational Linguistics, ACL '04, page 271 1-s, USA. Association for Computational Linguistics.
Bin Wang and C-C Jay Kuo. 2020. Shert-wk: A sen- tence embedding method by dissecting bert-based word models. IEEE/ACM Transactions on Audio, Speech, and Language Procexsing, 28:2146-2157.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploit- ing class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Compu- tational Linguistics (ACI. 005) pages I 115-144 Ann Arbor, Michigan. Association for Computational Lin- guistics.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation. 39:165-210.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Te- jani. Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high- nigh-performance deep learning library. In Advances in Neural Information Process- ing Systems 32, pages 8024-8035. Curran Associates, Inc.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. COnSERT: A con- trastive framework for self-supervised sentence repre- sentation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5065-5075 Online. Association for Computational Linguistics.
Yuhao Zhang, Hongji Zhu, Yongliang Wang, Nan Xu, Xiaobo Li, and Binqiang Zhao. 2022. A contrastive framework for learning sentence representations from pairwise and triple-wise perspective in angular space. In Proceedings of the 60th Annual Meeting of the
Fabian Pedregosa, Gaöl Varoquaux, Alexandre Gram- fort, Vincent Michel, Bertrand Thirion. Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vin- cent Dubourg, et al. 2011. Scikit-learn: Machine

---

Association for Computational Linguisties (Volume Long Papers), pages 4892-4903, Dublin, Ireland. Association for Computational Linguistics.
dividing D into two partitions, and then three par- titions, and so on. If no such partitions exist, the resultant array empty.
Method 4: This method further improves over Method 3 by avoiding certain multiplications (Algo. For instance, if = 4}) and 3 p(H, ( ..8)), then pl H, {1, ..... 8)) can be calculated as a+ø instead 2
A Optimized Algorithm for Finding the Best Layer Combination
2, max SaanOptim=True
For a data set X - ( (r)p)s 3|x() ) where each sample consists of a list of sentences
p H. {1,
24 [S [S!..... ) (n=2 for sentence-pair tasks
of a+4 18+4 reducing two vector multiplications. 8 get P Part Lenssenssaatss in Algo. 2 returns set of the lengths of parts' elements.
such STS) and label Yt, the besf-perforning layers set (D*) is calculated by the following equa- tion: D*
Algorithm 1: Layer Combination Iterator input :'Tessor H. Set A, Bool maxOptim, Int mar
argmax m ((f......fx)) (V), (uI.............. Mx)) (C)+(A)
(2)
where m is the desired metric function, and f& is
output :St D, Arry P
= (p [H) D) .)... .
(H,", D)) (3)
from Algo. use pool
in which H is the tensor of the ith sample's 's jth sentence, and f is function, such cosine simi- larity.
2 Function layer (H, A,
marOptim, marMem): mem hashMap ()
Eq. requires calculating p(H, D) for each sentence in X and for every possible D. which can very time-consuming. In this subsection, we discuss our proposed algorithm for efficiently calculating all p( H, D)s in step-by-step manner. Each method improves over the previous one, with Method 1 being the naive algorithm. The algo- rithms discussed here utilize mean as the pooling function, yet they can be easily modified use other functions such as max.
for i 1 to Alength do
for D in powerset (A, i) do
P P=pool (H, D, mem, mazMem)
7
yield D, P
To show the effectiveness of our algorithm, we carried out an experiment. We randomly seleet 1000 samples from the SICK dataset and try to find the best layer combination among all possible 8192 layer combinations, once with our algorithm and once with the naive algorithm (Method 2). We do this experimentation five times and report the erage results. Our algorithm takes 5.65 seconds to find the best layer combination after the ten- sor of all layer and token vectors are obtained by BERT-base-une Saase-u while Method takes 1067 seconds. The one-time forward pass of BERT takes 10 seconds.
Method 1: For every D Pt (A) = P(A)
calculate p(H,D) D).
Method 2: For every D E 11 < i < L}
calculate Pt = p(H, D) = 7 E HI... Then.
for every other 1) E P! (A), calculate p H, D) as 1 EienPt.
P(H, D)- TT
Method 3: This method is explained in Al- gos. 1 and 2 (mazOOtiim=aalse). The idea is that for calculating every p(H, D), we can ex- ploit other P(H, D')s that we have calculated before. For instance, p(H, (1,3,6, 6, 8, 103) (1366,
B STS Tasks' Experiments and Results
This method uses bottom-up dynamic programming to store the pre- vious p( H,, D)s. powerset (A, i) in Algo. 1 is equal to (DID E P(A) A DD]= i). mem a hash map with key-value pairs as (D,p(H, D)). Part(yPart(( mem) returns an array of parti- tions of D such that all partitions are present in mem, and each partition has the highest length possible. This is done in a greedy manner, by first
B.1 Data Sets and Evaluation Setup
For STS tasks, we use the seven standard STS data sets: STS2012-2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017), and SICK Relatedness (Marelli et al.. 2014). These data sets consist of sentence pairs and a similarity score from to assigned to each one of them. For each data set, we first combine all of its data subsets, shuffle it, and choose a small

---

Algorithm 2: Layer Combination Pooler
our method or the baselines to calculate the co- sine similarity between two sentences and then use Spearman's 's correlation (p) for evaluating the mod- els' performances as suggested by (Reimers and Gurevych, 2019).
input :Tensor H, Set D, MashMap mem,
Bool marOptim, Int MarMeem
output:Array :Array P
Function pool (HI D, mem, maxOptim, marMem):
B.2 Baselines and Hyper-parameters
parts = greedyPart (D, mem)
We use the same baselines that we use for the trans- fer tasks except for the MLM versions of the Sim- CSE models those were proposed in the SimCSE paper strengthen their models for the transfer tasks.
if parts/ then
P=array (parts [O].lenghh, 0) // array (N. v) returns an array of size N filled with us denom = D.length if maxOptim then
For the base models with 13 layers, we iterate through all the possible layer combinations for choosing the best one, yet for the large models with 25 layers, we only iterate through all the layer combinations with up to eight layers since in our experiments, we observed that combining more than eight layers hurts the performance.
lens gettaatIens (parts) foreach len in lens do
curp = array (Plleggth, 0)
10
foreach part of size len in parts do
L cuurp += mem [part]
B.3 Results and Discussion
12
if lens.lentth then
[ curp *= len
The results for STS tasks shown in Table 2. From this table, we observe that our method significantly outperforms its corresponding varia- tions of BERT, RoBERTa, SBERT, and SRoBERTa baselines on all the STS data sets. For instance, for BERT-L-uneaedd RoBERTa-L, and SRoBERTa- B, we obtain improvements of up to 25.75%, 13.94%, and 3.95% and average 16.32%, 9.52%. and 2.75%, respectively, compared to their best baselines (last avg). SimCSE-LC versions also out- perform most of their baselines by decent margin, For example, UnSupSimCSFF-LLL
13
else 14
denom len
16
+= curp
else
foreach part in parts do
18
P += mem [part] ] len
19
P /= denom
21
else
improves its baseline's p by up to 2.30% and on average 0.70%. Finally, the best SimCSE (Sup-RL)'s performances are also improved by up to 1.14%.
22
PP D)
if mem.lenghh mar.emm then
24
mem [part P
25
return
We can also see that our method is more effec- tive unsupervised models such as BERT and RoBERTa, as spot higher improvement com- pared with the supervised models such SBERT or SupSimCSE. This suggests that the unsuper- vised models' layers hold more important infor- mation than the supervised models for STS tasks. However, there is still crucial information to be ex- ploited from all the layers of the supervised models as well.
subset (the first 350 sentence pairs) as our devel- opment data set and the rest as our test data set. We do this splitting times randomly and report the average test performance. Please note that our method does not require training data. Nonetheless, we note that our method still needs a development data set in order for it to be bootstrapped. However, the required data set can be as small as 350 samples. Furthermore, as opposed the other models, this data leveraged for fine-tuning, which is very time-consuming.
We also observe that we have higher improve- ments for uncased BERT versions over their base- line compared to the cased versions. Nonetheless, after applying our method to either one, the final ps are within a 1.25% range of each other. This suggests that a lot BERT uncased 's information BERTmwosee
We use the sentence embeddings obtained by

---

is carried in its layers, and when exploited, it can perform more closely BERT casedd-CC
do not see drastic changes for SRoBERTa-B when moving from one layer to two layers.
Computation Setup
E.2 Effect of Excluding Particular Layer from Layer Combination
To conduct the experiments discussed in this paper. we used computer with 128GB of RAM and one Intel Core 19-9980XE CPU. Our code uses PyTorcl (Paszke et al.. 2019), huggingface 2, and sci-kit learn (Pedregosa et al., 2011).
In this section, we discuss how excluding one par- ticular layer from layer combination affects the performances. For this, we show the results for the models BERT-B cased- RoBERTa-B, and SBERT-B on the data sets STS16, STSB, and SICK in Fig. 4. We can mention these observations:
More Baselines
In this section, we compare the results of the layer combination technique with more baselines both STS and transfer tasks. We use the same baseline models as shown in Table but instead of only comparing with the last layer for BERT/RoBERTa models and original SimCSE and SBERT/SRoBERTa models (which use CLS pooler and last layer average, respectively), we also com- pare the results with these two new baselines: last four layers average and random layers. The re- sults are shown in Table 3 and Table 4. The results show that our method performs better than the new baselines as well.
1. Most of the time, excluding layer 11 hurts the performance, showing that this is an important layer.
2. Excluding either of the layers 4, 5, or 6 does not hurt the performance, suggesting that these layers are not important for these models and data sets.
Last layer (L12) is always important for SBERT-B, is important for BERT-B two cases (STS16 and STSB), and only im- portant for RoBERTa-B in one case (STSB).
4. Layer 0, which the embedding layer, proves to be important in more than half of the cases here, showing that it carries important infor- mation to be considered for STS tasks, and it cannot be ignored.
Ablation studies
E.1 Effect of the Number of Layers
In this subsection, show the effect of combining only N layers of model, N varying from 1 to 13 for the base models. To this end, we show this ef- feet on the data sets STS16, STSB, and SICK, and for the models BERT-B cased, ROBERTa-B, SBERT- B, and SRoBERTa-B B. From Figs. 2 and 3, we can see that (1) For all the data sets and all the models adding more layers shows upward trend in per- formance up some peak point (e.g., N for BERT-Baased on STS16), and adding more layers after that point shows downward trend perfor- mance; however, this point is different for different models and data sets, yet it is always at most (2) For BERT-B RoBERTa-B, and SBERT-B, moving from one layer to two layers leads to huge increase in performances for all but one data set-model pairs, yet combining more than two lay- ers always leads further substantial increase in performance. For instance, on STSB, an absolute p improvement 1.22% can be obtained when mov- ing from two layers to six layers. We, nonetheless,
5. Excluding any particular layer in any of the nine model-data set pairs shown in Fig. 4, Icads to maximum decrease of 0.97% (L12 for SBERT-B STS16) in the performance. However as shown in Table in any of these 9 cases, we get at least an improvement of 1.86% by combining certain layers (consid- ering all the layers). This suggests that layer combination still outperforms the baselines even if one (any) layer is not considered at all.
Fnttps://hugginggfae co/ co/traroforrresss

---

Model
S1812
ST813
ST514
STS15
STS16
STSB
SICK
Model
STS12
S1813
51814
S1515
S1N16
STSB
SICK
Avg.
BERT-B Ba
7.23
78.44
12.48
15.79
78.21
5.16
29.83
18.19
SBERTE
70.78
76.69
75.13
79:08
74.30
76.66
72.75
74.76
BERT Busoood last
31,08
59.58
47.39
60.17
63,04
46.24
57,90
52.20
SBERTB-LC
72.02
79.18
74.50
82,63
76.42
78.52
76.51
77.11
BERT-B B Bancased LC
50.89
64.67
54.94
72.22
67.95
59.42
63.54
61.95
SROBERT--B
70.82
73.06
70.61
78,
17
76.
74.31
73.99
BERT-B. BERT-B
14.16
22,51
16.41
24.11
27.52
12.69
39.22
22.37
BROBERT-B-LC
72.94
76.14
72,83
82.29
77.13
78.99
76.90
76.75
HIRT-B blaat
38,30
62.12
53:25
64.43
63.66
55.86
58.92
56.68
SBEREL
72.15
78.49
74-91
80:93
76.72
78.82
73.63
76.52
BERT B. Bond LC
50.55
66.25
$7.91
72.51
67.90
62.98
61.57
62.81
SBERT-L-LC
72.79
$1.39
76.75
84.14
79.15
80.43
77:23
78.84
RoBERTa B-last
32.18
45.05
61.11
55.13
62.10
53.24
SRoBERTI-L
74.06
77.01
73.07
81.59
76.65
78.21
74.08
76.40
45.58
60.83
51.20
69.23
64.68
60.17
64.00
59.38
SROBFRG-I-LCC
74.96
80.25
75.54
84.71
79.52
W0.78
77.71
79.07
BERT I .CS
18.66
21.47
13.79
11.01
23.29
13.31
25.11
SepSinCSF.RI.
75.51
N6.56
N0.22
86.09
81.45
94.8
90.93
82.06
BIRT Lenconee- last
27.76
55.11
44.37
51.59
61.10
46.52
53.56
4H.57
SupSimCSE BL-
75.51
$6.56
80.23
86.09
82.42
84.86
80.93
82.37
BERT Incouret IC
53.51
68.19
58.45
74.43
71,03
63,97
64.64
64.89
SupSinCSE-RL 77.35
16.366
82.18
86,59
83.92
86.58
81.72
83:67
14.64
8.27
6.04
9.74
25:00
10.69
26.99
14.48
SupSinCSE-RLL CE-LLLL 77.35
$7.82
82.18
87.12
85.05
86.60
81.72
83.98
HFRT- cuseelldd
45,77
63.81
54.41
68.87
64.55
58.25
63.24
19,84
61.04
75.66
70.70
66.16
65.22
2he Sagpaetteddddcee Mocils
ReBERTo-L-ba
33.49
$7.64
45.
62.74
61.40
51.59
$7.86
$2.89
47,43
65.33
55.14
72.44
69.04
62.60
64.88
62.41
UnSopSinnCHH-BL
69.21
83.95
75.58
83.86
7M.9%
77.89
73.47
77.56
UnSupSinCCSE BL-LC
69.21
84.82
75,88
$4.41
79,85
80,19
73,47
78.26
UnSopSincSEER8
72.11
83:41
74.96
84.03
80.79
81.74
70.82
78.27
UhSSpSiinCSF-R11.. RI.LC
72.11
83.66
75.03
84.37
80.68
82.06
72.53
78.63
Uesuponinal Mockss C3.: CLS pretes,
Table 2: Spearman's 's rank correlation & * 100 between the cosine similarity of sentence representations and the gold labels for different STS tasks. The highest numbers among models with the same encoder are boldfaced.
Moddl
S1S12
S1813
S1814
STS15
SIN16
STSB
SICK
Avg.
Model
81812
S1S13
S1N14
S1815
81816
STSB
SICK
31.19
59.58
47.39
60.17
63.04
46.24
57.90
52.20
SBERTE
70.78
76.69
73.13
79.08
74.20
76.66
72.75
74.76
DERT B Baroor 4
36.44
59.08
49.24
64.83
62.01
47.02
58.36
53.85
SBERT B-last 4
69.86
77.50
73.14
78.97
74.27
77:25
72.38
74.77
BERT-R tind
45.23
61.48
51.89
69.54
61.76
56.73
62.17
58.40
SBERT-R-radd
66.91
72.83
66.02
70.65
72.93
70.47
67.60
60.63
BERT-B +LC
50.89
64.67
54.94
72.22
67.95
59.42
63.54
SBERT-B-LC 61.95
72,02
79.18
74.50
82,63
76.42
78.52
76.51
77.11
HERT-B.... clatt
38.30
12
53.25
64.43
63.86
55.86
58.92
56.08
SKaBEKIIBB
70.82
73.06
70.61
79.34
74.17
76.05
74.31
73.99
BERT-B B
B. last
38.20
60.32
$1.75
65:95
62.29
$1.11
59.37
SRoBERT: B Jast 56.00
71.31
74.64
71.76
77.61
73.88
76.92
71.16
74.33
B... rand
45.52
62.42
71.27
57.89
60.34
rand
59.44
71.22
78.89
67.33
75.33
RFRT-R
50.55
66.25
57,91
72.51
67.90
62.08
61.57
62.81
SROBERTS---IC
72,94
76.14
72.83
82.29
77.13
78.99
76.90
76.75
RoBERIa B-last
32.18
56.27
45,05
61.11
60.81
55.13
62.10
53.24
SBERTL
22.15
78.49
74.91
80.93
76.72
78.82
73.63
76.52
RoBERTa-B-lan
35.63
56.30
46.31
61.03
63.33
55.35
62 09
54.72
SBERT-L-Jat
70.03
78,40
74.52
80.05
73.97
77.98
72.54
75.10
RoBERTa-B-aa
43.75
55,88
47.55
68.56
63.81
58.69
62.51
57.68
SBERT-L-AR
63.69
74.69
61.10
83.55
71.19
76.06
73.70
72.01
45.58
60.83
51.30
69.23
64.68
60.17
64.00
50.38
SRERTELLC
72.79
81.39
76.75
84.14
79.15
MIL43
77.22
78.84
BERT Lyncuue last
23.76
55.11
44.37
$1.59
61.10
46.52
53.56
48.57
SRoBERTSL
74.06
77.04
73:07
$1.59
76.69
78:24
74.08
76.10
BERT L.vosee last
34.91
55.09
49.01
59.16
61.30
48.95
52.28
SRoBERTS L-
21.52
73.67
79.28
77.98
72.85
BERTH aea sand
45.09
62.32
53.19
71.18
67.35
60.51
62.02
60.24
SRoBERTI---and
54.05
62.17
66.
75.98
65.35
70.03
76.83
69.00
HERT4
53.51
68.19
74.43
71.03
63,97
64.64
64.89
SKOBERI---LC
74.96
75.54
84.71
80.78
HERT- -...r-.....llls list
45:77
63.81
$4.41
68.87
64.53
58.25
63.24
59.34
SupSimCNE-HL. 75.51
36.56
$0.22
86.09
81.64
84.56
30.93
82.26
DERT DERT-L. last 4
43.52
61.48
52.87
68.32
63.78
54,70
63.15
58.26
SupSimCSE BL-l
71.16
85.17
77.17
85.19
50.80
83.14
80.18
30.10
RERTH tidd
52.30
65.77
$2.46
72.25
68,80
62.70
62.92
62.46
SupSioCSSF-R -nnd
$2.84
71.87
65.08
76.55
60.46
64.43
68.65
66.88
BERTH LL
53.87
70.23
61.04
75.66
70.70
66.16
68.22
66.13
DESSSICSE-BLLLE
75.51
86.56
$0.22
86,09
82.43
$4.36
50.93
82.37
RoBERIb-L-last
33.49
57.64
62.74
51.59
52,89
SupSimCSE:RL
77.35
187.36
86.59
83,92
86.58
81.72
83.67
RoBERT--LAaat 4
34,28
59.01
47.27
61.86
66.23
$7.98
62.21
55.98
SupSimCSE: RL- Rasl
75.64
87..5
$1.41
85.92
83.73
96.20
50.54
83.00
44.52
61:20
70.37
68.14
61.47
63,85
SupSin CSE-R
70.06
85.995
56.54
83.06
75.41
L-LC
47.43
65.33
55.14
72.44
69.04
62.60
64.88
62.41
SupSimCSE RL-LC
77.35
$7.82
82.18
87.12
85.05
$6.60
$1.72
$3.98
UnSupSimCSE BL
69.21
$3.93
75.58
83.86
78.98
77.89
73.47
77.56
Superiiaa Morkss
64.83
83.43
74.94
53.79
78.16
78.14
72.61
76.56
MANupSiinCSEBBBLnaa
52.67
64.63
56.26
75.56
66.91
69.87
66.78
64.67
UnSagSinnCSE:BB...
69.21
$4.82
75.88
84.41
74.85
N0.19
73.47
78.26
UnSopSimCSE RL
72.11
63.41
74.96
$4.03
$0.79
81.74
70.82
78.27
USSimSinn 60.44
83.28
82.00
71.55
77.90
UnSmpSinnCEERR CSESiinnSE-RL-and
59:92
70:44
67.06
81.83
75.26
59.52
67.31
70.05
UnSupšinc
72.11
83.66
75.03
84,37
80.08
82.06
72.53
78.63
Utospervisod Modes lasc loyt' overage.
Table 3: STS Task's Results with more baselines. The same metric in Table 2 is used (p 100). last is the
average of the last 4 layers, and rand is the random layers combination.

---

Mudd
MR
CR
SURJ
MPQ4
SSTM
TREC
MRRC
SST
Ang.
Modd
MR
CR
SURJ
MPQA
SSTM
TREC
MRPC
SST
BERT-8
95,08
87.97
57.20
SBERT-8 SRERR-8--tat-
98.93
80.04
RFRT-B 1..oeees--laa
67.61
BERT-8
80.89
54.99
94,46
85.8
47:09
55.39
76.49
55.74
SBERT
89.20
94.47
89:52
4P:12
85
37.61
83:92
82.15
BERT
B....e LCC
52.24
87.30
95.45
90.13
29.01
98.76
77.08
58.23
82.26
SIBRRT B LC
83.61
90005
95.15
91.02
49.45
85.59
78.40
89.68
53.27
BFRTT-...... -bet
60.55
85. 19
94.64
$7.87
65. 89
63.68
74.90
85.85
74.71
SR-0EEch- n1
64.57
90.12
92.57
69.1
50.64
51
17.45
10.11
82.06
BERT-B
81.38
96.17
94.80
58.55
46.75
$7.10
73.05
56.45
90:66
SRUBERT----ee Blled
65.81
90.97
93.53
89.67
53.13
R5.31
78.64
90.54
83.47
BERT Sind
59.10
55.65
SRUBERT.
BURT Broer LC
51.42
M5.11
91.08
46.99
53.00
75.45
56.85
SRoBLKTT
91.71
99.02
53,69
57.55
19.78
RoBERTo B-last
82.58
94.99
94.55
65.25
50.10
51:90
72:51
57.25
Stp
89.92
51.16
H1:75
74.60
88.81
81.53
RoBliKib B last
55.01
85.04
95.64
57.55
51.30
55:35
71.93
88.83
81.74
StpSinK BB last
K2.67
19414
55, .8
90.74
69.78
15.01
17.35
19.50
82.75
Relifihh-B-mnd
$6.41
157:15
94.78
87 5h
90.00
5510
72.12
68:11
61.19
SupSinC St-Dil--odd
76.70
57.76
94:31
00.20
48.91
15% 118
16.95
53.00
81:73
B.BEXTaBB-LC
$5.89
90.33
05.65
$8.66
52.48
$6.92
74.85
$9.65
R3.04
SFpSinCSF-BB-LCC
81.85
89.56
95.A8
90.77
50.16
89.09
37.52
89.71
63.04
InSopSiiCSn-nnn
50.67
85.29
94.29
56 75
66.
$3.96
72.55
85.86
79.00
SupSinCSt-RR SR-RR
64:20
91.11
95:12
90.19
52.69
13.13
76.
90.15
82.06
InnSpSSinCSE-RBG
SupSinCSS-RRB-le
UnSipSinCSE: BB-rand
76.46
94.47
94.00
46-78
56.62
73:75
86.52
SupSinCSE
91L61
94.07
90.40
53.01
15. 14
72,49
19.32
$2.56
UnSopSintSe BB LC
$1.61
86.77
95.23
90.13
45.25
89.05
77.36
37.55
61.99
SupSinCSL RB LC
86.00
92.13
95.21
98.87
53.48
87.97
78.76
91,02
$4.43
UnSipSiuCSE-00 RB
80.85
K5.36
92.40
56.70
47.25
76.13
72.90
55.67
79.41
SopSinCCE-RR. RB
84.56
91.89
93.33
89.52
52.47
81.
76.23
90.13
$2.50
UnSipSinttS RB last
48309
57.5
14.49
55.15
09.45
54.01
77.06
57.27
61.52
SupSinCSB KB last
485
92.55
25.00
90.43
53.05
N7.
74.39
14.91
83.39
InSopSmmtSn-c Rel-
$2.05
86-70
91.98
58
76.75
69.00
$3.75
75.94
87.38
90:95
SupSin SE- REy and
64.91
90 73
94.49
189 63
53.25
157
57.90
72.79
88.87
82.97
InSSpSienCSE-RRLL
SEp
BERT- praone
99.01
05.05
57.25
49.83
66.72
75.15
89.16
SBERT-L-ba
91.04
90.46
51.05
84-53
78.00
91.33
BERT-L Hindd
83.29
59.04
05.05
59.35
20.54
$6.62
75.65
59:03
K2.08
LBERT
84.31
91.14
94.50
90.59
40.92
85.
7X.55
86.19
$2.54
BURT LC
85.21
89.88
M6.08
91.17
$3.06
$3..9
76.57
89.96
K3.48
SBERT L LC
N5.41
91.64
95.61
V1.15
51.84
19.01
79,03
V1.46
84.40
BERT- clost
$4.76
53.68
94.95
58.03
23
$4. 10
72.67
98.25
81.33
SRGRERI--1
85.85
90.81
93.16
90.69
50.82
14.14
17.36
97.44
83.16
BERT-
SRuGEKT--L-last
91.50
fard
$5.34
90.19
95.41
90.50
50.92
$8.62
77.17
90.00
SRUBERT-1-LCC
92.49
92.06
R3.51
79.36
92.94
L-lst
$4.30
85.22
94,93
57.25
50.59
$1.75
67.24
59.41
80.09
SogSinCCE BL
85.47
90.69
95.01
90.38
51.16
14.25
73.70
90.53
$2.69
RouUXib 4
50.05
M5.61
95.85
59.52
51.91
56.32
69.3
90.74
K2.33
SupSinCSE BL last
64.54
911.2
95,55
10.70
51.35
10.63
75.08
91.00
83:27
$2.85
88:14
95.51
00:27
$7.63
$6. 71
70.31
90.95
67.36
DypSimCST-BL.coo
81.64
39.65
94
90.75
50.86
94
74.70
90.52
82.73
R.BEXT-LALCC
SupSinCCE-BLLLLC
UnSopSinnCSn-
$4.85
SepSinCSF.R1
InSqpSionCSE.BL.n
80.38
03.60
$6.79
75.40
39.70
DeeSinCSE-RLLIee
91.99
91.36
03.30
UnSapSiucSE: BL-cand
$2.90
96.87
54,90
90.30
.9.24
$6.31
75.75
38:51
81.36
StgSinCSE rad
85.29
91.07
95.45
91.47
54.56
15..45
72.65
02,97
$4.21
UnSipSimCS LC
$4.55
89.84
M5.77
91.43
50.93
59.07
77.19
90.14
K3.53
SupSinCSB KL. LC
19.24
92.20
96.29
12,01
55.39
M0.19
79.82
V3.49
36J18
UnSopSinCSE:R RL
82.29
96.34
92.77
58.12
45.94
$2.02
73.91
59.05
79,93
SupSinCSE: KLy
87.78
92.10
94.72
90.51
$2.43
K3.85
74.39
92,60)
83.25
UnSopSintS
19:45
94.61
38.4
78.97
59.8%
SipSinC Maar
93.15
95.91
41.11
W6.79
M3.29
UnnSppSin
SupSin andd
InSqpSinnCSE.RL...
$6.75
91.01
95.72
90.85
52.83
$7.93
79.13
84.47
SupSinCSE.RL C.--L..yLC
89.09
93.44
96.41
91.43
56.10
88.06
78.97
93.44
Uhogggroooo bel Makh. Last led boye ivorgge
Table 4: Transfer Task's Results with more baselines. The same metric in Table 1 is used (accuracy). last 4 is the average of the last 4 layers, and rand is the random layers combination.
STS16
STSB
SICK
693
6.30
$1.02
60X
11.0
5000
(a) STS16
(h) STSB
(c) SICK
Figure 2: Effect of combining N Layers for BERT-B cased and RoBERTa-B

---

STS16
5T58
SICK
10.00
12.50
13.58
12.00
(a) STS16
(b) STSB
(c) SICK
Figure 3: Effect of combining N Layers for SBERT-B and SRoBERTa
STS16 BERT BASE CASED
STS16 ROBERTA BASE
STS16 SBERT BASE
wa degleer! wiver endoone lyym
53
28
63 2E
88 28 58
697 00 118
102
(a) STS16-BERT base cased
(b) STS16-RoBERTa base
(c) STS16-SBERT base
STSB BASE CASED
STSB ROBERTA BASE
STSB SBERT
STANAAY MAX
(d) CTSB-BERT base cased
(e) STSBS-TSEERRRR RoBERTa base
(i) STSB-SBERT base
SICK BERT BASE CASED
SICK ROBERTA BASE
SICK- SBERT BASE
900 00000000 A4 200
(g) SICK-BERT base cased
(b) SICK-RoBERTa base
(i) SICK-SBERT base
Figure 4: Effeec of Excluding a Particular Layer


Reviews:
[{"review_id": "8rlV1m2DMz", "reviewer": "Reviewer_mPVd", "summary": "", "questions": "- in lines 213-215, what do you mean by the training set was not used by your methods? Did you tune your layer combinations on the validation sets?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments and time.\n\n- We agree that this is more computationally expensive. However, please kindly note that this is true only for the transfer tasks and not for STS tasks because there is no training required for STS.\n\nQuestions:\n- We mean that we do not use the training data to obtain sentence embeddings in our models. We obtain the best layer combination from the validation data set. The training data is just used for training a logistic regressor using the feature vector obtained for each of the sentences by our model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the approach is simple to understand and works quite well\n- the experiments are thorough", "reasons_to_reject": "- the approach seems to be a lot more computationally expensive than the baseline as each layer combination will require re-trainining the top classifier. it's unclear how easy the approach can generalize to other tasks, but this is probably beyond the scope of a short paper", "questions_to_authors": "- in lines 213-215, what do you mean by the training set was not used by your methods? Did you tune your layer combinations on the validation sets?", "paper_topic_and_main_contributions": "the paper proposes to use average pooling across different combinations of transformer layers instead of just using it on the last layer. they tune the choices of layers using labeled data from downstream tasks. in experiments, they benchmark this approach on senteval toolkit across many baseline pretrained transformer models, finding that their approach consistently improve the performance significantly.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qqm4coK7m1", "reviewer": "Reviewer_YvLi", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments and time.\n\nResponses to reasons for rejection:\n1. We agree that our proposed method is simple. However, while the idea itself might not be intricate, we argue that its practical implications and advantages are indeed significant. Our proposed method offers the following advantages: \n\n* **Effective and Easily Adaptable**: It is effective and easy to adapt to existing methods which can potentially open up new avenues for further research and application.  \n\n* **No Fine-tuning Required**: It does not require fine-tuning. \n\n* **Proposed Optimized Algorithm**: We introduce a novel algorithm which speeds up finding the best layer combination by a factor of 189 times for base models.  \n\n* **Extensive Array of Experimentations**: We show the effectiveness of our proposed method on two different sentence embedding tasks and 15 different datasets. We conducted an extensive array of experimentations across many different datasets and baseline models, establishing a valuable reference framework for future endeavors in this field.   \n\n* **State-of-the-art Result for Transfer Tasks**: We also show that our method achieves state-of-the-art result for transfer tasks while improving over a large set of BERT-based models for STS tasks. \n\n* **Good Performance in Limited Data Settings**: We show that our method outperforms the baselines even in limited data settings. \n\n* **Comprehensive Ablation Studies**: We also conducted different ablation studies which provide empirical evidence of the impact of different layer combinations on various tasks.  \n\nWe understand the rigorous standards of EMNLP and are committed to ensuring that our work meets the criteria for acceptance. We believe that the novelty of our work along with the above-mentioned contributions is reasonable for being accepted as a short paper in EMNLP.\n\n2. This paper exclusively focuses on the area of sentence embedding utilizing an extensive array of datasets most widely used by other researchers for sentence embedding tasks. We intend to apply our method to additional models and other tasks; however, we believe this falls beyond of the scope of a short paper.\n\n3. We recognize the importance of analyzing the interpretability of the proposed method. However, our current paper primarily focuses on establishing the effectiveness and performance of our proposed method, not interpretability.  We do not claim that our method can be applied to all the tasks; however, we do intend to thoroughly address the interpretability aspect and potential applicability of our method in future work.\n\n4. We have applied our method to a variety of different BERT-based models that include both supervised (SBERT and SimCSE) and unsupervised models, showing the effectiveness of our method in different scenarios. We can extend our work to newer models for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A new method for obtaining sentence embeddings by combining certain layers of a BERT-based model, which outperforms the baseline BERT on various tasks and data sets, achieving state-of-the-art results.\n2. An analysis of the effect of different layers and combinations of layers on the quality of sentence embeddings, which provides insights into the inner workings of the BERT model and how it can be fine-tuned for specific tasks.\n3. A comparison of the proposed method with other state-of-the-art methods for obtaining sentence embeddings, which shows that the proposed method is competitive and often outperforms them.\n", "reasons_to_reject": "1. The proposed method is based on a relatively simple modification of the BERT model, which may not be novel or significant enough to warrant publication in a top-tier conference like EMNLP.\n2. The evaluation of the proposed method is limited to a few standard tasks and data sets, which may not be sufficient to demonstrate its effectiveness in real-world scenarios or to compare it with other state-of-the-art methods, mostly from 2022 and 2023.\n3. The paper does not provide a detailed analysis of the interpretability of the proposed method, which could limit its usefulness in some applications where interpretability is important.\n4. The paper does not compare the proposed method with other recent methods that have achieved state-of-the-art results on similar tasks, which could provide a more comprehensive evaluation of the proposed method's performance.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new method for obtaining sentence embeddings by combining certain layers of a BERT-based model. The authors address the problem of improving the quality of sentence embeddings, which are widely used in various natural language processing tasks such as text classification, information retrieval, and machine translation.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Neelakantan A, Xu T, Puri R, et al. Text and code embeddings by contrastive pre-training[J]. arXiv preprint arXiv:2201.10005, 2022."}, "extra_text": {}}, {"review_id": "L0DBKSoFlj", "reviewer": "Reviewer_ciSn", "summary": "", "questions": "1.\tIn line 147, do you experiment on different pooling function? Can you report the experiments results over different pooling function?\n\n2.\tIn Appendix A, can you provide the complete time comparison of different methods? I would appreciate a thorough analysis of effectiveness of different improvement.\n\n3.\tCan you provide the details about two-step pipeline mentioned in Line 182?\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments and time.\n\nResponses to reasons for rejection:\n\n1. We agree that these concepts can be explained better. However, given the space constraint of a short paper and the substantial content that we aimed to cover, we tried our best to provide the most comprehensive explanation possible. Please find below further explanations regarding the confusion that you mentioned in your comment: \n\n * **Lines 001-006**: They express the purpose of obtaining sentence embedding representations in NLP. For a downstream task, such as sentence classification (transfer tasks), there are two ways to use BERT: (1) fine-tune the whole BERT and the MLP (classifier) head attached to it on the training data or (2) use BERT as a feature extractor by passing each sentence one time to a pre-trained BERT and get a fixed representation for it (by some pooling method), and use this representation to fine-tune only the MLP head. The second method (the purpose of sentence embedding) is favorable since it removes the need for the expensive computation of fine-tuning BERT itself. \n\n* **The proposed method not requiring training data**: We have actually mentioned the fact that our method does not require training data also in lines 059-061. \n\n* **Lines 103-106**:  They explain the different methods used for learning sentence embeddings. There are two types of models: (1) models that do not use any labeled data (2) models that use some labeled data (such as natural language inference data sets), but the sentence embeddings learned by them are used for different target data sets (such as STS data sets). \n\nWe would be happy to clarify if you have any further questions. \n\n2. \n\n* **Lines 217-224**: We have not used any mathematical symbols here. If you mean the models’ names, this is the best way we could find to define our models’ names in such a short space. However, we can add more explanation to the appendix for this if you think it is necessary. \n\n* **Line 163**: We believe that these notations are widely used for powerset and empty set in set theory. Hence, we did not explain them thoroughly. However, if you think explaining these can be beneficial, we can surely add it later.  \n\n* **Lines 664-686**: We are not able to understand what mathematical symbols are not defined here. We have defined everything either in these lines or previously in lines 150-154. Please kindly refer us to the exact symbol(s) so that we can clarify.\n\n3. We tried our best to make the algorithms as readable as possible, however, we can try and explain the algorithm even more in the text. \n\n4. The purpose of this paper is to do sentence embedding which means a fixed vector representation should be obtained for a sentence and then be used for the downstream tasks without fine-tuning the vector again on training data. Therefore, our method should be only compared with models that do not need fine-tuning, not the fine-tuned models. Thus, by state-of-the-art, we mean the state-of-the-art result for these data sets which are designed for the task of sentence embedding. \n\nResponses to questions:\n\nWe did not experiment with all the models and data sets as we saw in our early experiments that the max pooling results are not as good as mean pooling’s. The following table shows the results for using max pooling function for certain models and STS data sets that we have. As you can see, our method still significantly outperforms the baselines; however, the overall Spearman correlations of max-pooling are lower than mean-pooling's.\n\n | Model| STS16 | STSB | SICK    | Avg\n| :---        |    :----:   |          ---: |           ---: |          ---: |\n| Roberta-B-last      | 53.54        | 50.22   | 59.54  | 54.43 \n| Roberta-B-LC      | 62.41        | 60.19  | 62.33  | 61.64 \n|BERT-base-uncased-CLS Pooler      | 27.52        | 11.72   | 39.23  | 26.16 \n| BERT-base-uncased-last       | 59.7        | 54.1 | 56.89   | 56.9 \n| BERT-base-uncased-LC       | 67.63        | 65.11   | 59.92  | 64.22 \n\n2. Please note that since all the base models (such as BERT, RoBERTa, SBERT, etc.) have the exact same architecture and the same number of weights, obtaining the tensor of all layer and token vectors by any one of them takes the same amount of time. It also takes the same amount of time to iterate through all of the layer combinations with the same method for all of these models. Hence, the results that we have reported for BERT-base-uncased are valid for any other base model. \n\n3. All the BERT-based models have an MLP head attached to them. This head is attached to the output of the first token in the BERT’s last layer. BERT originally used this head to perform a binary task of next sentence prediction. However, as seen in the results and explained in lines 155-160, using the output of this MLP head leads to poor performance for the task of sentence embedding. However, SimCSE trains this head towards learning sentence embeddings. Therefore, this head holds important information for our task of sentence embedding. If we just use a mean pooling of the layers in SimCSE, as we are doing with all the other models, we are ignoring this important information. Thus, as explained in line 171-175, we propose an extension to our method. As before, we try to get a sentence embedding by mean-pooling from a particular set of layers, D. However, after obtaining p(H,D), we then pass p(H,D) to the MLP head of SimCSE, and use its outputs as p’(H,d) which as our new and final sentence embedding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is interesting to see some theoretical or empirical explanation for why combining certain layers of BERT works better than others.\n\n1.\tThe proposed method is concise and effective compared to the selected baselines.\n\n2.\tThe author conduct a substantial number of experiments on multiple pre-trained language models to show the effectiveness of the proposed method.\n", "reasons_to_reject": "1.\tThe writing is difficult to follow. For instance, lines 001-006 in the abstract are not clear enough. The author seems to suggest that the proposed method does not require training, but this point is not emphasized throughout the entire paper until line 211. Lines 103-106 are also confusing and hard to understand. The author should carefully examine each sentence in the paper to ensure they are easily understandable.\n\n2.\tThe use of mathematical symbols in this paper is also challenging to read, such as the writing in lines 217-224, which is not very clear. Many mathematical symbols are not explained, for example, in line 163, it is not clear what $\\mathcal{P}(A)$ and $\\emptyset$ refer to (though can be gussed). Similarly, lines 664-686 also have the same issue, and they are difficult to comprehend.\n\n3.\tThe authors should carefully format Algorithm 1 and Algorithm 2 to make them more easily readable.\n\n4.\tThe paper lacks a comparison with fine-tuned models, for instance, on the SST dataset, where fine-tuning bert-base can easily achieve over 93% accuracy. However, the results in this paper only reach 88%, which is not the state-of-the-art as claimed by the authors.\n", "questions_to_authors": "1.\tIn line 147, do you experiment on different pooling function? Can you report the experiments results over different pooling function?\n\n2.\tIn Appendix A, can you provide the complete time comparison of different methods? I would appreciate a thorough analysis of effectiveness of different improvement.\n\n3.\tCan you provide the details about two-step pipeline mentioned in Line 182?\n", "paper_topic_and_main_contributions": "This paper proposes a new method called BERT-LC (BERT Layers Combination) that obtains sentence embeddings by combining certain layers of BERT-based models in an unsupervised way. The authors argue that different layers of BERT carry different features, such as surface, syntactic, and semantic, and that each data set and task might need a different set of features for its sentences. The authors show that their method outperforms the baseline BERT and other models on seven semantic textual similarity (STS) data sets and eight transfer tasks. They also develop an algorithm that speeds up the process of finding the best layer combination and propose an innovative method that integrates the layer combination method with the CLS pooling head.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a very simple approach for obtaining sentence embeddings: combine representations from different layers (and not necessarily from the last layer only, which is the common procedure). Consistent improvements are reported across many datasets and tasks. There is a consensus among the reviewers on the simplicity and effectiveness of the approach. There were some concerns on the writing of the paper and computational complexity; the authors are expected to work on these. The issues raised by the reviewer YvLi on the limited experiments and interpretability studies are not valid, given the focused scope of this short paper. The SentEval benchmark (8 datasets) is a standard benchmark usually used in the domain. The authors also report results on the STS benchmark (6+ datasets). Interpretability analysis is indeed out of scope.
Decision: Accept-Findings