Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings
Authors: Qian Chen
Wen Wang
Qinglin Zhang
Siqi Zheng
Chong Deng
Hai Yu
Jiaqing Liu
Yukun Ma
Chong Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Sentence embeddings
BERT
Transformer
Self-attention
TL;DR: Introducing Ditto: a learning-free approach that uses model-based importance estimations to weight words and compute sentence embeddings from pre-trained model representations.

Abstract: Prior studies diagnose the anisotropy problem in sentence representations from pre-trained language models, e.g., BERT, without fine-tuning. Our analysis reveals that the sentence embeddings from BERT suffer from a bias towards uninformative words, limiting the performance in semantic textual similarity (STS) tasks. To address this bias, we propose a simple and efficient unsupervised approach, Diagonal Attention Pooling (Ditto), which weights words with model-based importance estimations and computes the weighted average of word representations from pre-trained models as sentence embeddings. Ditto can be easily applied to any pre-trained language model as a postprocessing operation. Compared to prior sentence embedding approaches, Ditto does not add parameters nor requires any learning. Empirical evaluations demonstrate that our proposed Ditto can alleviate the anisotropy problem and improve various pre-trained models on the STS benchmarks.

Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings
Qian Chen, Wen Wang, Qinglin Zhang, Siqi Zheng Chong Deng, Hai Yu, Jiaqing Liu, Yukun Ma, Chong Zhang Speech Lab, Alibaba Group
(tanqing. cq, w. wang]ealibbll
com
Abstract
BERT-based sentence embeddings. as sentence em- beddings from the original BERT yield high sim- ilarity between any sentence pair due to the narrow cone of learned embeddings (Li et al., 2020).
Prior studies diagnose the anisotropy problem sentence representations from pre-trae pre-trained language models, e.g., BERT, without fine- tuning. Our analysis reveals that the sentence embeddings from BERT suffer from bias to- wards uninformative words, limiting the per- formance in semantic textual similarity (STS) tasks. To address this bias, propose simple and efficient unsupervised approach, Diagonal Attention Pooling (Ditto), which weights words with model-based importance estimations and computes the weighted average of word representations from pre-trained mod- els as sentence embeddings. Ditto can be eas- ily applied to any pre-trained language model posiprocessing operation. Compared to prior sentence embedding approaches, Ditto does not add parameters nor requires any learn- ing. Empirical evaluations demonstrate that our proposed Ditto can alleviate the anisotropy problem and improve various pre-trained mod- els on the STS benchmarks.
Prior approaches for improving sentence embed- dings from PLMs fall into three categories. The first category of approaches does not require any learning (that is, learning-free). Jiang et al. (2022) argues that the anisotropy problem may be mainly due to the static token embedding bias, such token frequency and case sensitivity. To address these biases, they propose the static remove biases avg. method which removes cop-frequency tokens, subword tokens, uppercase tokens, and punctua- tions, and uses the average of the remaining token embeddings as sentence representation. However, this approach does not use the contextualized rep- resentations of BERT and may not be effective for short sentences as may exclude informative words. The prompt-basee method (last manual prompt) (Jiang et al., 2022) uses a template to gen- erate sentence embeddings. An example template is This sentence: "[X]" means [MASK]- where [X] denotes the original sentence and the last hid- den states in the [MASK] position are taken as sen- tence embeddings. However, this method has sev- eral drawbacks. (1) It increases the input lengths, which raises the computation cost. (2) It relies on using the [MASK] token to obtain the sentence rep- resentation, hence unsuitable for PLMs not using [MASK] tokens (e.g., ELECTRA). (3) The perfor- mance heavily depends on the quality of manual prompts which relies on human expertise (alterna- tively, OptiPrompt (Zhong et al., 2021) requires additional unsupervised contrastive learning).
1 Introduction
Pre-trained language models (PLM) such as BERT (Devlin al.. 2019), ROBERTa (Liu al., 2019), and ELECTRA (Clark al., 2020) have achieved great success in a wide variety of natu- ral language processing tasks. However, Reimers and Gurevych (2019) finds that sentence embed- dings from the original BERT underperform tradi- tional methods such as GloVe (Pennington et al., 2014). Typically, bedded by the BERT embedding layer, which con- sists of token embeddings, segment embeddings, and position embeddings. The output is then en- coded by the Transformer encoder and the hidden states at the last layer are averaged to obtain the sentence embeddings. Prior studies identify the anisotropy problem as critical factor that harms
input sentence is first em-
The second category of approaches fixes the pa- rameters of the PLM and improves sentence em- beddings through post-processing methods that re- quire extra learning. BERT-flow (Li et al., 2020) addresses the anisotropy problem by introducing a flow-based generative model that transforms
'The source code can be found at httpss//ggithh /otm./.co/ttbbbc///// alibaba- damo- academy/ BoaokkennL//ttee//aainnenee

---

the BERT sentence embedding distribution into smooth and isotropic Gaussian distribution. BERT- whitening (Su et al., 2021) uses a whitening opera- tion to enhance the isotropy of sentence represen- tations. Both BERT-flow and BERT-whitening re- quire Natural Language Inference (NLIY/Semaniic Textual Similarity (STS) datasets to train the flow network or estimate the mean values and covari- ance matrices the whitening parameters.
Lak
(a) BERT
(b) SBERT
The third category updates parameters of the PLM by fine-tuning or continually pre-training the PLM using supervised or unsupervised learning, which is computationally intensive. For example, Sentence-BERT (SBERT) (Reimers and Gurevych, 2019) fine-tunes BERT using siamese/triplet net- work on NLI and STS datasets. SimCSE (Gao et al., 2021) explores contrastive learning. Unsu- pervised SimCSE uses the same sentences with different dropouts as positives and other sentences as negatives, and supervised SimCSE explores NLI datasets and treats entailment pairs as positives and contradiction pairs as hard negatives.
Figure 1: sentence "For those who follow social media transitions on Capitol Hill, this will be a little different.". The impact matrices are computed using BERT (bert-base- uncased) and SBERT on Hugging Face, respectively.
The heatmap shows the impact matrix for the
pendix A.1. Prior works use perturbed masking to recover syntactic trees from BERT (Wu et al.. 2020). Different from prior works, we use per- turbed masking to analyze the original BERT and strong sentence embedding model, supervised Sentence-BERT (SBERT) (Reimers and Gurevych, 2019). Figure shows the heatmap representing the impact matrix for an example sentence in the English PUD treebank (Zeman et al., 2017). The y-axis represents x; and the x-axis represents 2t.. A higher value in Fij indicates that word zy has a greater impact predicting another word 2i. Com- paring the impact matrices of BERT and SBERT, we observe that the impact matrix of SBERT ex- hibits prominent vertical lines on informative to- kens such as "social media", "Capitol Hill", and "different", which implics that informative tokens have a high impact on predicting other tokens, hence masking informative tokens could severely affect predictions of other tokens in the sentence. In contrast, BERT does not show this pattern. This observation implies that the compositionality of informative tokens could be a strong indicator of high- hugh-quality sentence embeddings of SBERT. Fur- thermore, we compute the correlations between the impact matrix and TF-IDF (Sparck Jones, 1972) which measures the importance of a word, and report results in Table 3. We find that the im- pact matrix of SBERT has much higher correla- tion with TF-IDF than the impact matrix of BERT, which is consistent with the observation above. No- tably, ELECTRA performs poorly on STS tasks and shows a weak correlation with TF-IDF. Conse- quently, we hypothesize that sentence embeddings of the original BERT and ELECTRA may be bi-
In this work, we first analyze BERT sentence embeddings. (1) We use a parameter-fiee prob- ing method to analyze BERT and Sentence-BEil (SBERT) (Reimers and Gurevych, 2019) and find that the compositionality of informative words is crucial for generating high-quality sentence em- beddings as from SBERT. (2) Visualization of BERT attention weights reveals that certain self- attention heads in BERT are related to informative words, specifically self-attention from a word to itself (that is, the diagonal values of the attention matrix). Based on these findings, we propose simple and efficient approach, Diagonal Attention Pooling (Ditto), to improve sentence embeddings from PLM without requiring any learning (that is, Ditto a learning-free method). We find that Ditto improves various PLMs and strong sentence em- bedding methods on STS benchmarks.
2 Analyze BERT Sentence Embeddings Observation 1:
The compositionality of infor- mative words is crucial for high-quality sen- tence embeddings. Perturbed masking (Wu et al., 2020) is faremeter-free probing technique for analyzing PLMs (c.g., BERT). Given a sentence x [21, (2,,,,,.... J. perturbed masking applies a two-stage perturbation process on each pair of tokens (æ;, to measure the impact that a to- ken zy has on predicting the other token Z;. De-
tails of perturbed masking can be found in Ap-

---

ICLS)
for
for
those
those
those
those
Sollow
follow
Sollow
tollow
social
social
mediaa
social
social
media
0.1 bo 0.1
transliions
tronsitions
transitions.
tronsitions
on
oz
on
capilol
capitol
capstol
capitol
Food Fooward
leyer koad 10
this
tis
tal5
this
bo
Scl--illntion
diferret diflerent
difeerenr
di different
Token& Segtiou Endoedlings
8. will be fiee
[SEP]
[SEP]
[SEP]
[SEP]
"I will be faee
Figure 2: Iulurttaiinn of attention weights for BERT head 1-10 (on the left) and head 11-11 (on the right). The darkness of line represents the value of the atten- tion weights. The top-5 diagonal values of the attention matrix are colored blue.
Figure 3: The diagram of our proposed diagonal atten- tion pooling (Ditto) method.
Diagonal Attention Pooling
Inspired by the two observations in Section 2, we propose novel learning-free method called Diagonal Attention Pooling (Ditto) to improve sen- tence embeddings for PLMs, illustrated in Figure 3. Taking BERT as an example, the input to the Trans- former encoder is denoted as h = [49.... 11 and the hidden states at cach Transformer en- coder layer are denoted as h' = {1...., L}. Typically, the hidden states of the last layer of the PLM averaged to obtain the fixed- size sentence embeddings, as N +i= h4 (denoted as last avg.). Alternatively, we can also average the static word embeddings 1 hy (denoted as static avg.), ), or average the hidden states from the first and last layers 2N D (Aς + h1f) (denoted as first-last) to obtain the sentence embeddings. Ditto weights the hidden states with diagonal at- tention of certain head. For example, to obtain the sentence embeddings from the first-last hidden states of BERT using Ditto, we first obtain the diag- onal values [ ANN] of the attention matrix A for head I-h of BERT, where and are treated as hyperparameters and are optimized on a develop- ment set based on the STS performance. Then, we compute Li=1 Asr + lif) the sentence em- beddings*
ased towards uninformative words, hence limiting their performance on STS tasks.
Observation 2: Certain self-attention heads of BERT correspond to word importance. Al- though SBERT has a higher correlation with TF- IDF than BERT verified in Observation BERT still shows moderate correlation. Thus, we hy- pothesize that the semantic information of infor- mative words is already encoded BERT but has yet to be fully exploited. Prior research (Clark et al., 2019) analyzes the attention mechanisms of BERT by treating each attention head as simple, no-training-require classifier that, given word as input, outputs the other word that it most attends to. Certain attention heads are found to correspond well linguistic notions of syntax and coreference. For instance, heads that attend to the direct objects of verbs, determiners of nouns, objects of prepo- sitions, and coreferent mentions are found to have remarkably high accuracy. We believe that the at- tention information in BERT needs to be further exploited. We denote particular attention head by <layer>-<head number> (I-h), where for BERT-base-sized model, layer ranges from to 12 and head number ranges from to 12. We visualize the attention weights of each head in each layer of BERT and focus on informative words. We then discover that self-attention from a word to itself (that is, the diagonal value of the attention matrix, named diagonal attention) of certain heads may be related to the importance of the word. As shown in Figure 2, the informative words "social media tran- sitions", "hill", and "little" have high diagonal val- ues of the attention matrix of head 1-10. Section will demonstrate that diagonal attentions indeed have a strong correlation with TF-IDF weights.
Nly e
7. Note that the impact matrix (Section 2) correlates well with TF-IDF (as shown in Table 3) and hence may also improve sentence embeddings. However, the learning- Ditto is much more effi- cient than computing the impact matrix, which is computationally expensive.
Experiments and Analysis
Following prior works (Jiang et al., 2022; Li al., 2020: Su al., 2021: Gao et al., 2021), we exper-
"We did not normalize with N since As < I and normal- ization by N may result in very small values.

---

Table 1: Performance different sentence embedding methods STS tasks (as Average Spearman's 's correla- tion). Table 6 in the Appendix reports detailed results.
extra learning. Note that our learning-free BERT first-last Ditto achieves comparable performance to BERT-flow and BERT-whitening in this group. For further analyzing Ditto, we compute TF-IDF weights on 10 sentences randomly sampled from English Wikipedia as token importance weights and use the weighted average of the first-last hidden states as sentence embeddings, denoted as first-last TF-IDF (the 4th row in this group). First-last TF- IDF yields +8.75 absolute gain over the first-last avg. baseline, only slightly better than the +8.07 absolute gain from our learning- free Ditto (with and h searched on only 1500 samples).
Method
Avg.
Learning-fre methods
BERT static avg.
56.02
BERT last avg.
52.57
BERT first-last uvg.
56.70
BERT static remove biases avg.
63.10
BERT last manual prompt
67.85
BERT static Ditto (Ours)
61.77
BERT last Ditto (Ours)
59.07
BERT first-last Ditto (Ours)
64.77
Methods that fix BERT parameters bud require extra learning
BERT-flow
66.55
BERT-whiening
66.28
The third group of Table 1 presents results of strong baselines that update BERT parameters through unsupervised supervised learning, in- cluding unsupervised SimCSE (Unsup. BERT SimCSE), supervised SimCSE (Sup. BERT Sim- CSE), and supervised SBERT. We find that ap- plying Ditto on the highly competitive supervised learning method Sup. SBERT first-last avg. still achieves an absolute gain of 0.17 (84.94-85..1 demonstrating that Ditto could also improve strong supervised sentence embedding methods. Note that since SimCSE uses the [CLS] representation as sentence embeddings instead of average pooling, SimCSE is not suitable for applying Ditto.
BERT last manual and continuous prompt
73.59
BERT first-last TF-IDF (Ours)
65.45
Methods that apdate BERT parameters
Unsup. BERT SimCSE
76.25
Sup. BERT SimCSE
81.57
Sup. SBERT first-lust uvg.
84.94
Sup. SBERT first-last Ditto (Ours)
85.11
Table Sentence embedding performance of Ditto and learning-free free baselines on PLMs, measured by average Spearman's correlation on the test sets of STS tasks.
Method
BERT
RoBERTa
ELECTRA
First-last avg.
56.70
56.57
36.28
Last manual prompt
67.85
61.08
19.44
First-last Ditto (Ours)
64.77
61.96
52.00
Effectiveness of Ditto on Different PLMs Table 2 compares the baselines first-last avg. and last man- ual prompt and our first-last Ditto method on dif- ferent PLMs. Note that last manual prompt does not work for ELECTRA because this method relies on using the [MASK] token as the sentence embed- dings while the ELECTRA discriminator is trained without [MASK] tokens. Ditto consistently works well on ELECTRA and greatly outperforms the two baselines on RoBERTa and ELECTRA, while underperforming last manual prompt on BERT.
iment on the 7 common STS datasets, the widely used benchmark for evaluating sentence embed- dings. Appendix A.2 presents dataset and imple- mentation details.
Main Results The first group of Table presents the results of the three common learning-free meth- ods (Jiang et al., 2022) described in Section 3, in- cluding static avg.. last avg., and first-last avg., and two learning-fiee free baselines described in Sec- tion 1, static remove biases avg. and last manual prompt (Jiang et al., 2022). Although last man- ual prompi achieves 67.85, different templates can have significant impact its performance, rang- ing from 39.34 73.44 on STS-B dev set (Jiang et al., 2022). Applying Ditto to static avg., last avg., and first-last avg. achieves absolute gains +5.75. +6.50, and +8.07 on the Avg. score, respectively." 'he second group of Table presents results from methods that fix BERT parameters but require
Table 3: Correlation of the impact matrix with the TF- IDF weights. Column STS avg. shows the average Spearman's correlation on the test sets of 7 STS tasks. Columns Pear. and Spear. show Pearson's 's correlation and Spearman's 's correlation between the mean values of the impact matrix N D Fig and the TF-IDF weights for 1K sentences randomly sampled from the English PUD treebank.
Method
STS avg.
Pear.
Spear.
'Since static remove biases avg. may remove important tokens and last manual prompt uses the last hidden states of [MASK] as sentence emheddings instcad of average pooling, they are not suitable for applying Ditto.
BERT
$6.70
57.27
57.44
SBERT ELECTRA
84.94
62.90
70.21
36.28
12.97
21.91

---

Correlation with TF-IDF To further analyze cor- relations between diagonal attentions and word importance, we select the heads corresponding to the Top-4 Ditto performance based on Spear- man's correlation on the STS-B development set, and compute correlations between diagonal val- ues of the self-attention matrix of these heads and TF-IDF weights. Table 4 shows all Top-4 heads exhibit moderate strong correlations with TF- IDF weights. We find high-performing heads are usually in the bottom layers (Section A.2), which is consistent with the findings in Clark et al. (2019) that the bottom layer heads broadly attend to the entire sentence.
0.30
64.77
0.30
25.3
0.10
52.32 55.20
52,03
01.0
-23
-20
-15
-1.2
-25
6.
Figure 4: Alignment-uniformity plot of baselines and Ditto on different PLMs. The arrow indicates the changes. For both alignment and uniformity, smaller numbers better.
are directionally uniform, and the average cosine similarity between random samples should be close to zero. Ethayarajh (2019) originally applied this metric to word representations, and we adapt to sentence representations in our study, We sample 1000 sentences from the English Wikipedia dataset and compute the average cosine similarity of their representations. Table 5 shows the results. Lower values indicate better isotropy. Our proposed Ditto method improves the isotropy of all three learning- free baselines: static avg., last avg., and first-last avg. This result is consistent with the uniformity analysis in Figure 4, where Ditto also enhances the uniformity of different sentence embedding mod- els.
Table 4: The sentence embedding performance of BERT first-last Ditto using different attention heads and the correlation with the TF-IDF weights. Dev de- notes Spearman's ' correlation on the STS-B development set. Test denotes the average Spearman's correlation the test sets of 7 STS tasks. Pear. and Spear. de- note Pearson's 's correlation and Spearman's correlation between the diagonal values of the attention matrix for the certain attention head and the TF-IDF weights on sentences the STS-B task.
Method
Dev
Test
Pear.
Spear.
Ditto Head 1-10
74,56
64,77
64.34
63.56
Ditto Head 2-12
73.13
65.00
47.30
44.17
Ditto Head 11-11
70.59
62.46
47.64
44.68
Ditto Head 1-7
69.54
60.65
65.98
64.30
Table 5: The average cosine similarity of sentence rep- resentations for Ditto and learning-free -ree baselines.
Uniformity and Alignment We the analysis tool from prior works (Gao et al., 2021) to eval- uate the quality of sentence embeddings by mea- suring the alignment between semantically related positive pairs and uniformity of the whole repre- sentation space. Gao et al. (2021) finds that sen- tence embedding models with better alignment and uniformity generally achieve better performance. Figure 4 shows the uniformity and alignment of different sentence embedding models along with their averaged STS results. Lower values indicate better aligmment and uniformity. We find that Ditto improves uniformity at the cost of alignment degra- dation for all PLMs, similar to the flow and whiten- ing methods as reported in Gao et al. (2021). Com- pared to Ditto, flow and whitening methods achieve larger improvements in uniformity but also cause larger degradations in alignment.
Method
avg. Cosine Similarity
BERT static avg. BERT static Ditto
0.843
0.768
BERT last avg.
0.508
BERT last Ditto
0,458
BERT first-last avg.
0.566
BERT first-last Ditto
0,403
Conclusions
We propose a simple and learning-free Diagonal At- tention Pooling (Ditto) approach address the bias towards uninformative words in BERT sentence embeddings. Ditto weights words with model- based importance estimations and can be easily applied to various PLMs. Experiments show that Ditto alleviates the anisotropy problem and im- proves strong sentence embedding baselines.
Cosine Similarity We use the cosine similarity metric from Ethayarajh (2019) measure the isotropy of sentence representations. Isotropy means that the word or sentence representations
Limitations
Although our proposed simple and learning free Ditto approach demonstrates effectiveness in allevi-

---

ating the anisotropy problem and improving strong sentence embedding baselines, there are several limitations. Firstly, we conduct our experiments solely on the English sentence embedding models and the English Semantic Textual Similarity (STS) datasets. We hypothesize that the two observations in Section 2 will hold true on pre-trained models for other languages, hence we predict that Ditto, which is based the two observations, will be effective in improving sentence embeddings for languages other than English. We plan to investi- gate the efficacy of Ditto on improving sentence embeddings for other languages in future work. Secondly, while we select the attention head (that is, determining/ / and h) by conducting grid search of all attention heads based on the performance of the STS development set, will explore other approaches for selecting attention heads for Ditto in future studies. Lastly, we focus on using Seman- tic Textual Similarity tasks for evaluating sentence embeddings in this work. We plan to investigate the quality of sentence embeddings in more tasks, such as information retrieval.
shared task: Semantic textual similarity. In *SEM 2013, pages 32-43. Association for Computational Linguisties.
Samuel R. Bowman, Gabor Angeli, Christopher Potts. and Christopher D. Manning. 2015. large an- notated corpus for learning natural language infer- ence. In EMNLP, pages 632-642. The Association for Computational Linguisties.
Daniel M. Cer, Mona T. Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval- 2017 task 1: Semantic textual similarity multilin- gual and crosslingual focused evaluation. In Se- mEval@AAI. 2017, pages 1-14, Association for Com- putational Linguistics.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis bert's attention. In ACL Work- shop BlackboxNLP, pages 276-286. Association for Computational Linguistics.
Kevin Clark, Whan-TThan Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre- training text encoders as diseriminators rather than generators. In ICLR. OpenReview.net.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: pre- pre-training of deep bidirectional transformers for language under- standing. In NAACI-HLT, pages 4171-4186. Associ- ation for Computational Linguistics.
References
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-Agirre, Weiwei Guo, lñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. Semeval-2015 task Semantic textual similarity, english, spanish and pilot on interpretabil- ity. In SemEval NAACL-HIIT 2015, pages 252-263. The Association for Computer Linguistics.
Kawin Ethayarajh. 2019. How contextual are contex- tualized word representations? comparing the ge- ometry of bert, elmo, and GPT-2 embeddings. In EMNLP-IJCNIPP pages 55-65. Association for Com- putational Linguistics.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M. Cer, Mona T. Diab, Aitor Lonzzalez-Aiiree Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. In SemEval COLING, pages 81-91. The Association for Computer Linguis- tics.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence em- beddings. In EMNLP, pages 6894-6910. Association for Computational Linguistics.
Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei. Haizhen Huang, Denvy Deng. and Qi Zhang. 2022. Prompt- bert: Improving BERT sentence embeddings with prompts. In EMNLP, pages 8826-8837. Association for Computational Linguistics.
Eneko Agirre, Carmen Banea, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-Agiree, Rada Mihalcea, Ger- man Rigau, and Janyce Wiebe. 2016. 2206 task I: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval@ NAA.Evaa@NNAAA..... NAACL-HLT 2016, pages 497-511. The Association for Computer Linguistics.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang. Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In EMNLP, pages 9119-9130. Association for Compu- tational Linguistics.
Eneko Agirre, Daniel M. Cer. Mona T. Diab, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task
6: A pilot semantic textual similarity, In SemEval@ NAACL-HLT, pages 385-393. The Asso- ciation for Computer Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized BERT pretraining approach. CoRR, abs/19072 1669..
Eneko Agirre, Daniel M. Cer. Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013, *sem 2013

---

Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zampar- elli. 2014. A SICK cure for the evaluation of com- positional distributional semantic models. LREC, pages 2-222. European Language Resources As- sociation (ELRA).
A.2 Dataset and Implementation Details
We conduct experiments on common STS datasets, namely, STS tasks 2012-2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS-B (Cer et al., 2017), and SICK-R (Marelli et al., 2014), following prior works. These STS datasets are widely used benchmarks for evaluating sentence embeddings. Each dataset consists of sentence pairs scored from similarity. For evaluation, we follow the setting of Reimers and Gurevych (2019) and report the average Spearman's 's correlation on the test sets of all 7 STS tasks (that is, the "all" setting), without using an additional regressor. Our implementation is based the SimCSE GitHub repository and modify it to fit our purposes. We conduct grid search of the attention head I-h for Ditto based on Spearman's 's correlation on the STS-B development set (1500 samples). In this way, we select head 1-10 for BERTS, head 1-5 for RoBERTa head 1-11 for ELECTRA", and head 3-7 for SBERT*, The TF-IDF weights are learned on 10 sentences randomly sampled from English Wikipedia us- ing the gensim tool We also utilize the English Wikipedia dataset and randomly sampled 1000 sen- tences to calculate the average cosine similarity of sentence representations. We conduct experiments using a single Tesla V100 GPU. Note that BERT- flow and BERT-whitening papers use the full target dataset (including all sentences in the train, devel- opment, and test sets, and excluding all labels) and optionally the NLI corpus (SNLI (Bowman et al.. 2015) and MNLI corpus (Williams al., 2018)) for training.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543. ACL.
to indicate the semantic
Nils Reimers and Tryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. EMNLP-IJCNLP, pages 3980-3990. Association for Computational Linguistics.
Karen Sparck Jones. 1972. statistical interpretation of term specificity and its application retrieval. Journal of documentation, 28(1):11-21.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whiteming sentence representations for better semantics and faster retrieval. CoRR, 1b3/2103.15316.
Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACI-IIILT, pages 1112-1122. Association for Computational Linguistics.
Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. 2020. Perturbed masking: Parameter-free probing for ana- lyzing and interpreting BERT. ACI. pages 4 4166- 4176. Association for Computational Linguistics.
Daniel Zeman, Martin Popel, Milan Straka, Jan Hajic. Joakim Nivre, Filip Ginter, Juhani Luotolahti, and al. 2017. Conll 2017 shared task: Multilingual parsing from raw text to universal dependencies. In CoNLL, pages 1-19, Association for Computational Linguistics.
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: learning vs. learning to recall. NAACL-HLT, pages 5017-5033. Associa- tion for Computational Linguistics.
A Appendix
A.1 Details of Perturbed Masking
the first stage, we replace 2; with the [MASK] token, resulting in a new sequence x (x.). The representation of this sequence is denoted as Hxx)(xx;)); In the second stage, we mask out zj in addition z; to obtain the second corrupted sequence x) 2x; The representation of this sequence denoted HI [[x) x Dr));, Thus obtain an impact matrix F IR. N by computing the Euclidean distance between the two representa- tions Fij = d(H(x) (xx 1). H ((x) x,))))
"https://github. "nttos://giitubb.co//rreetttttt htt.://hww..cmmmmrrreeeemon////s
'nttps://huggiingface.
httpt////whggggggnss...bbeeeaaas cc/bert-base-uncase
"https://huggiiggaac.cc http://ggiiigaac..c/rrrrrrrrerer co/roberta-base
"httpss//tuggingffaeccoo// co/ooglle//
electra-baserriieerininttoo
"hetpss//huggingfaacecc
berrt-base-ml-stttt---m---------
phtto:://hugiingacccccooo//
wikiIn_.or_siince..
tfidfmodel.html

---

Table 6: The performance comparison of different sentence embedding methods on STS tasks (Spearman's 's correlation).
Method
ST812 STS13
STS14
ST815 STS16
STS-B
SICK-R
Avg.
Learning free methods
BERT static avg.
42.38
$6.74
50.60
65.08
62.39
$6.82
58.15
56.02
avg.
BERT first-last avg.
39.70
$9.38
49.67
66.03
66. 19
53.87
62.06
$6.70
remove
avg. (Jiang et al., 2022)
53.09
BERT last manual prompt (Jiang al., 2022)
60.96
73.83
62.18
71.54
68.68
70.60
67.16
67.85
BERT static Ditto (Ours)
$2.61
62.72
59.88
70.40
65.60
63.34
57.85
61.77
BERT last Ditto (Ours)
43.58
64.84
53,27
66.06
65.77
58.88
61.11
59.07
BERT first-latt Ditto (Ours)
53.77
67.99
59.78
73.77
69.66
66.76
61.64
64.77
Methods that fix BERT parometery hut require extra learning
BERT-flow (Li et al., 2020)
58.40
67.10
60.85
75.16
71.22
68.
64.47
66.55
BERT-whitening (Su et al., 2021)
57.83
66. 90
60.90
75.08
71.31
68.24
63.73
66.28
BERT last manual and contimaous prompt (Jiang al., 2022)
64.56
79.96
70.05
79.37
75.35
77.25
68.56
73.59
BERT first-last TF-IF (Ours)
53.78
73.03
59.49
71.98
70.16
66.99
62.73
65.45
Methods apdate RERT panameters (supervised unsupervised learming))
Unsup. BERT SimCSE (Gao al., 2021)
68.40
82.41
74.38
80.91
78.56
76.85
72.23
76.25
Sup. BERT SimCSE (Gao el al., 2021)
75.30
84.67
80.19
85.40
80.82
84.25
80.39
81.57
Sup. SBERT fast-last (Reimers and Gurevych, 2019)
80.67
88
89.78
90.28
82.12
85.
78.55
84.94
Sup. SBERT first-last Ditto (Ours)
80.43
88.65 $8.63
89.84
90.39
82.72
85.62
78.15
85.11


Reviews:
[{"review_id": "wLq3fHx6Hy", "reviewer": "Reviewer_rde8", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for your appreciation of our approach and for considering it well-motivated, learning-free, easy to plug in, and with solid performance improvement. In the following, we provide responses to all of your questions.\n\n---\n\n**Q1: TF-IDF itself has certain drawbacks, and simply looking at the linear relationship between sentence representations and TF-IDF may introduce TF-IDF’s issues into the resulting sentence representations.**\n\n**Response:** \nWe acknowledge the limitations of TF-IDF, such as the requirement of a large corpus to learn reliable TF-IDF weights. We have discussed these issues in our paper, and compared the performance of first-last TF-IDF with our learning-free Ditto approach (Line 261-270). With reliable TF-IDF computations (Line 261-266), they can indicate word importance. Thus, our approach of studying the correlations between the impact matrix and TF-IDF weights is reasonable.\n\n---\n\n**Q2: I wonder if it would be better to concatenate the avged embedding of the first layer and that of the last layer. The process of averaging them and then taking the cosine similarity creates dot-product terms between the first layer and the last layer, which may not be desirable.**\n\n**Response:**\nThe average embedding of the first and last layer is a common baseline in previous works [1][2]. We have also experimented with concatenating the averaged embeddings of the first and last layer for BERT, and the average score on STS is 55.35, which is **worse than the average score of 56.70 achieved by averaging the averaged embeddings of the first and last layer as reported in the paper**. Additionally, concatenating the hidden states of the first and last layer will double the size and may not necessarily represent the sentence accurately.\n\n[1] [Prompt-bert: Improving BERT sentence embeddings with prompts] (Jiang et al., EMNLP 2022)\n\n[2] [Simcse: Simple contrastive learning of sentence embeddings] (Gao et al., EMNLP 2021)\n\n---\n\n**Q3: There are several prior work on learning-free or parameter-free postprocessing for improving sentence embeddings, and I recommend the authors discuss them or perhaps compare the proposed approach to them. \n[1] Jiaqi Mu, Suma Bhat, and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. In International Conference on Learning Representations, 2018.\n[2] Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A simple but tough-to-beat baseline for sentence embeddings. In International Conference on Learning Representations, 2017.**\n\n**Response:**\nWe appreciate the references you provided on parameter-free postprocessing for improving sentence embeddings. We have indeed considered these works and found them not directly related to our work in terms of the specific focus and contributions that our work presents.\n\nRegarding \"All-but-the-top: Simple and effective postprocessing for word representations\" by Mu et al., their method is designed for static word embeddings such as WORD2VEC and GLOVE. It aims to eliminate the common mean vector and a few top dominating directions from the embeddings. However, these embeddings are not learning-free since the common mean vector and PCA need to be computed on a corpus. This method shares a similar idea with the BERT-whitening approach, which we discussed in our paper (Line 082-085).\n\nRegarding \"A simple but tough-to-beat baseline for sentence embeddings\" by Arora et al., their method represents the sentence by a weighted average of static word vectors and modifies it using PCA/SVD. Consequently, this method is also designed for static word embeddings and not for BERT-like sentence embeddings. Moreover, it requires calculations of weights (called smooth inverse frequency, SIF) and PCA/SVD, making it not entirely learning-free. Additionally, the SIF reweighting resembles TF-IDF reweighting, which we have already discussed and compared in our paper (Table 1 and Line 261-270).\n\n---\n\nOnce again, we sincerely appreciate your insightful comments and questions.\n\nBest regards,\n\nThe Authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. the approach is well-motivated using the correlation between attention maps and TF-IDF weights.\n\n2. the approach is indeed learning-free, which can be easily plugged on top of any pre-trained language models with transformers.\n\n3. the performance improvement is solid. ", "reasons_to_reject": "1. TF-IDF itself has certain drawbacks, and simply looking at the linear relationship between sentence representations and TF-IDF may introduce TF-IDF's issues into the resulting sentence representations.\n\n2. I wonder if it would be better to concatenate the avged embedding of the first layer and that of the last layer. The process of averaging them and then taking the cosine similarity creates dot-product terms between the first layer and the last layer, which may not be desirable.\n\n3. there are several prior work on learning-free or parameter-free postprocessing for improving sentence embeddings, and I recommend the authors discuss them or perhaps compare the proposed approach to them.\n\n[1] Jiaqi Mu, Suma Bhat, and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. In International Conference on Learning Representations, 2018.\n\n[2] Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A simple but tough-to-beat baseline for sentence\nembeddings. In International Conference on Learning Representations, 2017.", "questions_to_authors": "", "paper_topic_and_main_contributions": "the submission proposed a learning-free approach to obtain sentence representations from a pre-trained encoder-only model. The approach is well-motivated by the correspondence between the diagonal entries of the attention mechanism and TF-IDF weights. Moreover, the proposed approach improves the performance of the baseline on STS tasks. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0IylTzgK69", "reviewer": "Reviewer_rnBv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for considering our analysis with visualized illustrations impressive and inspiring, and for finding our claims with ample empirical evaluation easy to understand. We have carefully considered your questions and concerns, and below are our responses to all of your questions.\n\n---\n\n**Q1: In Ditto, the authors proposed two methods, one utilizing static word embeddings and the other employing diagonal values from the attention matrix. However, the correspondence between the first method and observation 1 needs to be further elucidated.**\n\n**Response:** \nWe would like to clarify that we only proposed the novel learning-free method based on Diagonal Attention Pooling (Line 008-014, Line 220-230). The static avg., which is based on averaging the static word embeddings as sentence embeddings, is the baseline we compare to and was introduced in earlier work [1].\n\n[1] [Prompt-bert: Improving BERT sentence embeddings with prompts] (Jiang et al., EMNLP 2022)\n\n---\n\n**Q2: In the experimental section, for both the \"Learning-Free Methods\" section of Table 1 and the BERT section of Table 2, the authors' approach did not surpass the \"Manual Prompt\" baseline, which need further solutions or explainations.**\n\n**Response:** \nWe have clarified the drawbacks of the manual prompt approach, which we describe in Line 065-072 of our paper. While it is true that in both the \"Learning-Free Methods\" section of Table 1 and the BERT section of Table 2, Ditto did not surpass the manual prompt baseline, we already explicitly mentioned in Line 249-252 that the performance of the manual prompt approach can vary significantly depending on the templates used, with scores ranging from 39.34 to 73.44 on the STS-B dev set [1]. In contrast, as shown in Line 252-255, **when applying Ditto to static avg., last avg., and first-last avg., we achieved absolute gains of +3.05, +9.20, and +8.07 on the Avg. score, respectively**. Additionally, as shown in Line 285-295, our Ditto consistently performs well on different pre-trained language models (PLMs) (including RoBERTa and ELECTRA) and outperforms the last manual prompt approach.\n\n[1] [Prompt-bert: Improving BERT sentence embeddings with prompts] (Jiang et al., EMNLP 2022)\n\n---\n\n**Q3: Since Ditto cannot be applied to methods like SimCSE that use [CLS] for embedding, this may imply a limited applicability scope for Ditto.**\n\n**Response:** \nIt is true that Ditto is not directly applicable to sentence representation models such as SimCSE that use [CLS] as sentence embeddings (Line 281-284). However, many strong unsupervised and supervised sentence representation models are based on weighted averages of word representations. Table 1 shows that supervised SBERT first-last avg. significantly outperforms supervised SimCSE, yet Ditto still achieves an absolute gain of 0.17 (84.94 $\\rightarrow$ 85.11). This demonstrates that Ditto can improve strong supervised sentence embedding methods as well.\n\n---\n\nOnce again, we sincerely appreciate your insightful comments and questions.\n\nBest regards,\n\nThe Authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper analyze the sentence embedding from two perspective with visualized illustration which is impressive and inspiring.\n2.  The proposed solution, Diagonal Attention Pooling (Ditto), is simple to understand and implement. It doesn't require any additional training, fine-tuning, or changes to the original model's parameters. \n3. The paper backs up its claims with ample empirical evaluations. And the paper is easy to understand.", "reasons_to_reject": "1. In Ditto, the authors proposed two methods, one utilizing static word embeddings and the other employing diagonal values from the attention matrix. However, the correspondence between the first method and observation 1 needs to be further elucidated.\n2. In the experimental section, for both the \"Learning-Free Methods\" section of Table 1 and the BERT section of Table 2, the authors' approach did not surpass the \"Manual Prompt\" baseline, which need further solutions or explainations.\n3. Since Ditto cannot be applied to methods like SimCSE that use [CLS] for embedding, this may imply a limited applicability scope for Ditto.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors of the paper observe that sentence embeddings produced by BERT exhibit a bias towards uninformative words, which consequently hinders their performance on semantic textual similarity (STS) tasks. The goal of the paper is to address this bias and enhance the quality of sentence representations. The paper first analyzes BERT sentence embedding between BERT and SBERT, and then the paper proposes an unsupervised approach called \"Diagonal Attention Pooling\" (Ditto) to mitigate the anisotropy problem in sentence embeddings without requiring further fine-tuning or additional parameters.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mjv4VEmSQf", "reviewer": "Reviewer_Z8t9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to the reviewer for finding our summarization of existing methods nice and for considering our proposed method to be clear with solid results. We appreciate the constructive feedback. Below, we address all of your questions.\n\n---\n\n**Q1: The conclusion for \"alleviate the anisotropy problem\" is not supported enough. Uniformity and alignment are just two metrics used by Gao et al. 2021. Some other methods can be seen in [1][2].\n[1] All-but-the-top: Simple and effective postprocessing for word representations. ICLR 2018 [2] A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space. ACL 2021**\n\n**Response:** \nWe appreciate the references you provided regarding metrics for the anisotropy problem. We have taken these works into consideration and found that they are not specifically designed for sentence embeddings.\n\nIn \"All-but-the-top: Simple and effective postprocessing for word representations\" by Mu et al. and \"A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space\" by Rajaee et al., the isotropy score measures the variance of the inner products between word vectors normalized by the square of the mean norm of all the word vectors. The higher the isotropy score, the more isotropic the word representations are. However, calculating the isotropy score requires obtaining the complete set of word vectors from a pre-trained model. This makes it infeasible to extend the isotropy score to evaluate sentence embeddings since enumerating all possible sentences and then computing their corresponding sentence embeddings are infeasible. Therefore, finding better metrics for evaluating the anisotropy problem of sentence embeddings remains an area for future research.\n\nUniformity and alignment are commonly used metrics to analyze the quality of sentence embeddings (Line 296-301). For example, they have also been used in other works [1][2] in addition to (Gao et al., 2021).\n\n[1] [Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings] (Jian et al, NeurIPS 2022)\n\n[2] [MCSE: Multimodal Contrastive Learning of Sentence Embeddings] (Zhang et al, NAACL-HLT 2022)\n\n---\n\nWe sincerely appreciate the reviewer for their insightful comments and questions.\n\nBest regards,\n\nThe Authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The summarization for the category of existing methods for improving sentence embeddings is nice.\n2. The motivation of the proposed method is clear and the results are solid.", "reasons_to_reject": "1. The conclusion for \"alleviate the anisotropy problem\" is not supported enough. Uniformity and alignment are just two metrics used by Gao et al. 2021. Some other methods can be seen in [1][2].\n\n[1] All-but-the-top: Simple and effective postprocessing for word representations. ICLR 2018\n[2] A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space. ACL 2021", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes Diagonal Attention Pooling (Ditto), which weights words with model-based importance estimations and computes the weighted average of word representations from pre-trained models as sentence embeddings. The resulting embeddings show better results on STS benchmark against various baselines.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method called Diagonal Attention Pooling, an unsupervised way to improve sentence embedding quality. The reviewers converge toward excitement, posing only minor issues regarding clarification and suggestions for future work.
Decision: Accept-Main