SlowGuess/NAACL_2022_OCR huggingface dataset:
In this section, we evaluate our model on three language understanding tasks. First, we choose the lexical substitution task to observe the word-level semantic performance. Then we conduct experiments on two sentence representation tasks: the STS task in unsupervised setting and the supervised STS benchmark (STS-B) task.
5.2 Lexical Substitution
Task and Dataset. Lexical substitution aims to replace the target word in a given context sentence by a substitute word that not only is semantically consistent with the original word but also preserves the sentence's meaning. There are two benchmark datasets for this task: the SemEval 2007 dataset (LS07) (McCarthy and Navigli, 2007) with 201 target words, and the CoInCo dataset (LS14) (Kremer et al., 2014) with 4,255 target words, both of which are unsupervised. The task LS07 releases the official evaluation metrics best/best-mode and oot/oot-mode², which evaluate the quality of the best prediction and the best 10 predictions, separately. We also report the metrics precision@1
$(\mathrm{P}@\mathrm{1})$  and  $\mathrm{P}@\mathrm{3}$ . Because the metric best considers the word frequencies in annotated labels, we take it as the main metric in this task.
Candidate Generation. We use the context encoder pre-trained with GR to generate lexical substitutions. Given a target word  $w$  and its context  $s$ , we directly employ the full contextual token distribution  $p(x|\mathbf{s})$  to perform the word prediction, then sort the candidates by their probabilities.
Before evaluating the score of the generated candidates, we filter out the words with the same lemmatization or with different PoS from the target word. All our experiments employ the same candidate filtering process. The detailed process is in Appendix C.
Post-Process. Previous works proposed several effective approaches to improve LS performance. Arefyev et al. (2020) used the input word embedding to inject more target word information (noted +emb). Zhou et al. (2019) utilized a pre-trained model to re-score candidates (noted +valid). We denote these approaches as post-process and adopt them in our experiments. As Arefyev et al. (2020) reported, the result in (Zhou et al., 2019) is hardly reproduced and their code is not available, we then implement the validation process by ourselves.
Result and Analysis. Table 1 shows the comparison of our models with the previous SOTAs in LS07 and LS14 benchmarks. We use publicly released BERT and RoBERTa models as the baseline to generate lexical candidates as illustrated in the candidate generation process. Initialized by the same BERT or RoBERTa parameters, our GR-models are further pre-trained on gloss dataset with GR module. We first compare the model outputs without post-process. Our GR models surpass their MLM baselines by large margins in all metrics: the best value increases more than 3 points, the oot increases about 8 points in LS07. In separate context encoder structure, the best value of BERT increases from 10.1 to 12.4 in LS14, and the metric increases from 11.0 to 13.1 for RoBERTa. Comparing the P@1 with (Arefyev et al., 2020), the SC GR-RoBERTa base model 48.8 even exceeds the large RoBERTa model with emb 46.5.
Results indicate that GR model generates more semantically similar words and preserve the sentence original meaning even though no LS-like training data is used. This is because the gloss regularization plays the key role in modeling contextual
$^{2}$ http://www.dianamccarthy.co.uk/ task10index.html
Method	Backbone	Post-Process	SemEval 2007 (LS07)	CoInCo (LS14)
best/best-m	oot/oot-m	P@1/P@3	best/best-m	oot/oot-m	P@1/P@3
Roller and Erk (2016)	SGNS emb	-	-	-	19.7/14.8	-	-	18.2/13.8
Zhou et al. (2019)	BERTlarge	-	12.1/20.2	40.8/56.9	13.1/-	9.1/19.7	33.5/56.9	14.3/-
+valid	20.3/34.2	55.4/68.4	51.1/-	14.5/33.9	45.9/69.9	56.3/-
Arefyev et al. (2020)	RoBERTlarge	-	-	-	32.0/24.3	-	-	34.8/27.2
+emb	-	-	44.1/31.7	-	-	46.5/36.3
XLNetlarge	+emb	-	-	49.5/34.9	-	-	51.4/39.1
Baselines	BERTbase	-	13.2/22.3	40.8/57.1	33.1/23.7	10.1/21.9	33.0/56.5	38.4/28.7
RoBERTabase	-	16.7/27.8	45.2/62.9	40.8/28.5	11.0/23.6	34.9/59.3	42.2/31.4
Our work	MT GR-BERTbase	-	17.7/30.8	49.8/67.8	42.5/31.1	12.2/26.5	39.2/64.5	46.4/35.3
SC GR-BERTbase	-	18.2/31.2	49.9/67.6	44.1/31.2	12.4/27.1	39.8/65.5	46.6/35.8
MT GR-RoBERTabase	-	19.7/32.9	53.0/72.8	47.9/34.2	12.9/28.3	40.6/66.4	48.6/37.2
SC GR-RoBERTabase	-	19.4/33.2	52.8/71.5	47.4/33.4	13.1/28.8	40.9/66.6	48.8/37.8
+emb	22.4/38.2	56.4/76.0	53.7/37.8	14.5/32.8	43.8/69.9	53.5/41.4
+valid	22.6/38.4	56.0/73.9	54.8/39.0	15.1/33.7	44.1/69.6	56.0/42.7
+both	23.1/39.7	57.6/76.3	55.0/40.3	15.2/34.4	45.3/71.3	55.9/43.5
Table 1: Comparison with previous SOTA on lexical substitution task. Results of the first three works are from the mentioned papers and the results in the baseline are from our experiments with the same word process.
token distribution  $p(x|\mathbf{s})$  by taking both contextual and semantic information into consideration. Given a sentence context, if two words are semantically replaceable, their gloss text descriptions are naturally similar. As the word contextual embedding is aligned with its gloss, the words in semantically similar contexts are gathered closer indirectly, which benefits the LS task.
We further apply post-process on the SC GR-RoBERTa model. Consistent with previous works (Arefyev et al., 2020; Zhou et al., 2019), both processes improve the performance in testset LS14:  $+emb$  increases the best value from 13.1 to 14.5, and it is to 15.1 using  $+valid$ . By applying both post-processes, our SC GR-RoBERTa model achieves the new SOTA 15.2 in best. We also achieve SOTA in the metrics best-m/oot-m and P@3 in LS14 and all metrics in LS07. Appendix C demonstrates random selected examples of the LS task and the model outputs.
5.3 Unsupervised Sentence Representation Task
STS Task and Dataset. STS tasks deal with determining how similar two sentences are. We evaluate our model on 7 STS tasks: STS tasks 2012-2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (STS-B) (Cer et al., 2017) and SICK-Relatedness (SICK-R) (Marelli et al., 2014). Following the work of Gao et al. (2021) and their setting in STS tasks $^3$ , we use Spearman's correlation with "all" aggregation as the evaluation metric, and use no additional regressor in experiments.
Baselines. Since our experiments are unsupervised w.r.t. STS task: neither STS data nor NLI dataset<sup>4</sup> are used for training, we only perform comparison with previous works in unsupervised setting. SOTA works for these tasks are either trained by carefully designed sentence-level loss [e.g. SimCSE (Gao et al., 2021), BERT-flow (Li and Roth, 2002)] or tuned on sentence dataset NLI [e.g. BERT-whitening (Su et al., 2021)]. Therefore, these models are able to generate effective sentence representation. In contrast, our model is not trained with any sentence tasks, and we simply use the average of contextual word embeddings to represent sentence. Thus, it is not very fair to directly compare with the mentioned sentence encoders. We then focus more on the comparison with the original MLM.
Result and Analysis. Table 2 shows the results on STS tasks. We employ the publicly released BERT and RoBERTa pre-trained models as the baselines, and our GR-models are further pretrained with GR-module in the gloss dataset. With gloss regularization in pre-training, the average Spearman's correlation increases from 56.70 to 65.75 in BERT model and from 56.57 to 67.47 for RoBERTa. Though still far below the SimCSE SOTA performance, our model approaches the BERT-whitening and BERT-flow without any deliberately designed sentence-level tasks or transforming word distribution on domain data. Reimers and Gurevych (2019) report the unsupervised BERT embedding is infeasible for STS and performs even
$^{4}$ NLI dataset consists of SNLI and MNLI, both of which are proved to be effective domain data for STS tasks (Gao et al., 2021; Reimers and Gurevych, 2019).
3https://github.com/princeton-nlp/ SimCSE
Model	STS12	STS13	STS14	STS15	STS16	STS-B	SICK-R	Avg.
GloVe embs	55.14	70.66	59.73	68.25	63.66	58.02	53.76	61.32
BERT-flow	58.40	67.10	60.85	75.16	71.22	68.66	64.47	66.55
BERT-whitening(NLI)	57.83	66.90	60.90	75.08	71.31	68.24	63.73	66.28
SimCSE-BERT	68.40	82.41	74.38	80.91	78.56	76.85	72.23	76.25
SimCSE-RoBERTa	70.16	81.77	73.24	81.36	80.65	80.22	68.56	76.57
BERT(first-last avg)	39.70	59.38	49.67	66.03	66.19	53.87	62.06	56.70
MT GR-BERT(first-last avg.)	53.20	69.68	58.81	73.25	72.16	66.65	66.47	65.75
SC GR-BERT(first-last avg.)	53.69	68.66	58.83	71.90	71.64	66.18	66.46	65.34
RoBERTa(first-last avg.)	40.88	58.74	49.07	65.63	61.48	58.55	61.63	56.57
MT GR-RoBERTa(first-last avg.)	53.73	72.57	61.04	75.23	72.86	69.44	67.39	67.47
SC GR-RoBERTa(first-last avg.)	53.69	70.00	59.24	72.38	72.47	70.12	67.02	66.42
worse than GloVe embedding. Li et al. (2020) blame it on the anisotropic distribution of BERT word embeddings. Our experiments show great gains of GR-BERT in sentence embedding, proving the advantage of gloss regularized contextual representation is also valid for sentences. A brief analysis on sentence representation with gloss regularizer is provided in Appendix D.
5.4 Supervised STS
STS-B Task and Dataset. We validate our model in supervised STS Benchmark (STS-B) (Cer et al., 2017). The data consists of 8,628 sentence pairs and is divided into trainset (5,749), devset (1,500) and testset (1,379).
Since supervised STS performance are largely influenced by the training data, we only use the STS trainset in all experiments. Besides, we randomly reduce the data size to simulate the limit data scenarios and compare our model with MLM baselines. Following the sentence-BERT (Reimers and Gurevych, 2019), we use Siamese BERT network with cosine similarity.
Result and Analysis. Tabel 3 shows the comparison on STS-B. In both BERT and RoBERTa backbones, GR models improve the baselines by around 0.9 points. In low-resource scenarios, the advantage of GR-BERT increases. When  $50\%$  data is available, the gain of MT GR-BERT is increased to 1.87 points, and the gain is up to 3.44 points for  $20\%$  data. Results show that in fine-tuning process, the GR model still preserves its advantage over MLM baselines in sentence semantic representation, indicating the contextual representation
Table 2: Sentence embedding performance on unsupervised STS tasks. Results in the first row are from Gao et al. 2021. Notation (first-last avg) means take the average of word embs from the input and output layer.
Data ratio	Models	Spearman
100%	BERT	83.98 ± 0.16
MT GR-BERT	85.13 ± 0.06
SC GR-BERT	85.00 ± 0.16
100%	RoBERTa	85.90 ± 0.57
MT GR-RoBERTa	86.87 ± 0.21
SC GR-RoBERTa	86.25 ± 0.30
50%	BERT	81.60 ± 0.28
MT GR-BERT	83.47 ± 0.15
SC GR-BERT	83.06 ± 0.19
20%	BERT	76.43 ± 0.37
MT GR-BERT	79.87 ± 0.41
SC GR-BERT	79.18 ± 0.21
Table 3: Evaluation on STS-B test set. All experiments are fine-tune for 4 epochs with batch size 16. Results are the average of 4 random seeds.
model	LS14	STS Avg	STS-B
BERT	10.1	56.70	83.98
+MLM	10.9	62.22	84.62
MT GR-BERT	12.2	65.75	85.13
SC GR-BERT	12.4	65.34	85.00
Table 4: Ablation studies of different training loss in three tasks. +MLM means only use MLM loss in training. We use the metric best for LS14 task, the average Spearman's correlation for 7 STS tasks and STS-B.
pre-trained with GR is transferable in further finetuning. The GR pre-training is able to enhance the semantic knowledge in model, especially in the low-resource data scenarios, which ease the hunger for task training data.
5.5 Ablation Analysis
We now investigate the influence of gloss training data and the model structures. Results are shown in Table 4. Gururangan et al. (2020) reports the domain data pre-training can improve model per
5https://www.sbert.net/examples/training/sts/README.html
formance. To evaluate the influence of dictionary corpus, we pre-train BERT by MLM in the same dataset and find that high-quality data improves all three task performances. However, GR still contributes to the large part of the improvement, especially in the LS task. As for the two proposed structures, the SC-GR utilizes individual context encoders that impose less restriction on gloss learning, and achieves better performance in LS word-level task. On the contrary, the MT model provides a better sentence embedding and surpasses SC structure in STS tasks.
In this work, we propose the GR-BERT, a model with gloss regularization to enhance the word contextual information. We first analyze the gap between MLM pre-training and inference, and aim to model the PMI term that characterizes the word semantic similarity given context. Due to the lack of data that labels the word semantic similarities given contexts, we propose to indirectly learn the semantic information in pre-training by aligning contextual word embedding space to a human annotated gloss space. We design two model structures and validate them in three NLP semantic tasks. In the lexical substitution task, we increase the SOTA value from 14.5 to 15.2 in LS14 best metric and many other metrics in LS07 and LS14 are also improved. In the unsupervised STS task, our GR model shows its capacity in sentence representation without any training in sentence task, and it improves the MLM performance from 56.57 to 67.47. In the supervised STS-B task, GR model exceeds the MLM baseline by about 0.9 points, and the gains increase to 3.44 in the low resource scenarios.
Our work provides a new perspective to the MLM pre-training, and show the effectiveness of modeling word semantic similarity. However, one limitation of our work is the lack of large-scale word-gloss matching data. The training data in our work is far less than that in BERT pre-training, which hinders the large-scale pre-training in GR-BERT. Our future works will focus on mining more word-gloss training data and validate GR model in more NLP tasks. We believe there is still a big room for GR model performance improvement and possible gains in more NLP tasks.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 252-263, Denver, Colorado. Association for Computational Linguistics.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 81–91, Dublin, Ireland. Association for Computational Linguistics.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. 2016. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval-2016. 10th International Workshop on Semantic Evaluation; 2016 Jun 16-17; San Diego, CA. Stroudsburg (PA): ACL; 2016. p. 497-511. ACL (Association for Computational Linguistics).
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics–Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385–393.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *sem 2013 shared task: Semantic textual similarity. In Second joint conference on lexical and computational semantics (*SEM), volume 1: proceedings of the Main conference and the shared task: semantic textual similarity, pages 32-43.
Nikolay Arefyev, Boris Sheludko, Alexander Podolskiy, and Alexander Panchenko. 2020. Always keep your target in mind: Studying semantics and improving performance of neural lexical substitution. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1242-1255, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Michele Bevilacqua and Roberto Navigli. 2020. Breaking through the  $80\%$  glass ceiling: Raising the state of the art in word sense disambiguation by incorporating knowledge graph information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2854-2864, Online. Association for Computational Linguistics.
Terra Blevins and Luke Zettlemoyer. 2020. Moving down the long tail of word sense disambiguation with gloss informed bi-encoders. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1006–1017, Online. Association for Computational Linguistics.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Ting-Yun Chang and Yun-Nung Chen. 2019. What does this word mean? explaining contextualized embeddings with natural language definition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6064-6070, Hong Kong, China. Association for Computational Linguistics.
Ting-Yun Chang, Ta-Chung Chi, Shang-Chi Tsai, and Yun-Nung Chen. 2018. xSense: Learning sense-separated sparse representations and textual definitions for explainable word sense networks. arXiv preprint arXiv:1809.03348.
Ting Chen, Yizhou Sun, Yue Shi, and Liangjie Hong. 2017. On sampling strategies for neural network-based collaborative filtering. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 767-776.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pretraining text encoders as discriminators rather than generators. In ICLR.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pretraining with whole word masking for Chinese BERT. arXiv preprint arXiv:1906.08101.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65,
Hong Kong, China. Association for Computational Linguistics.
Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3509-3514, Hong Kong, China. Association for Computational Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Gerhard Kremer, Katrin Erk, Sebastian Padó, and Stefan Thater. 2014. What substitutes tell us - analysis of an "all-words" lexical substitution corpus. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 540-549, Gothenburg, Sweden. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A lite bert for self-supervised learning of language representations. In ICLR.
Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2020. SenseBERT: Driving some sense into BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4656-4667, Online. Association for Computational Linguistics.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing (EMNLP), pages 9119-9130, Online. Association for Computational Linguistics.
Xin Li and Dan Roth. 2002. Learning question classifiers. COLING '02, page 1-7, USA. Association for Computational Linguistics.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A Robustly Optimized BERT Pretraining Approach.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 216-223, Reykjavik, Iceland. European Language Resources Association (ELRA).
Diana McCarthy and Roberto Navigli. 2007. SemEval-2007 task 10: English lexical substitution task. pages 48-53.
Alessio Miaschi and Felice Dell'Orletta. 2020. Contextual and non-contextual word embeddings: an in-depth linguistic investigation. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 110-119, Online. Association for Computational Linguistics.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. 1st International Conference on Learning Representations, ICLR 2013 - Workshop Track Proceedings, pages 1-12.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
Stephen Roller and Katrin Erk. 2016. PIC a different word: A simple model for lexical substitution in context. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1121-1126, San Diego, California. Association for Computational Linguistics.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316.