SlowGuess/NAACL_2021_OCR huggingface dataset:
where  $\operatorname {I}(\boldsymbol {W})\in [0,1]$  . We follow the standard approach and define  $\pmb{x}$  to be the set of eigenvectors of  $W^{\top}W$  (Mu and Viswanath, 2018; Wang et al., 2020). We remove the top principal components using a modified version of the post-processing method proposed by Mu and Viswanath (2018):
$$
\tilde {\boldsymbol {W}} _ {i} = \boldsymbol {W} _ {i} - \frac {1}{| V |} \sum_ {j = 1} ^ {V} \boldsymbol {W} _ {j} \tag {9}
$$
$$
\boldsymbol {U} = \operatorname {P C A} (\tilde {\boldsymbol {W}}) \tag {10}
$$
$$
\hat {\boldsymbol {W}} _ {i} = \tilde {\boldsymbol {W}} _ {i} - \sum_ {j = 1} ^ {\mathrm {D}} \left(\boldsymbol {U} _ {j} ^ {\top} \tilde {\boldsymbol {W}} _ {i}\right) \boldsymbol {U} _ {j}, \tag {11}
$$
where  $\mathbf{W}$  is the embedding matrix,  $\hat{\mathbf{W}}$  is the postprocessed embedding matrix, and D is the number of principal components removed from the original matrix. Mu and Viswanath (2018) use  $\mathbf{W}$  instead of  $\tilde{\mathbf{W}}$  in the term  $(\mathbf{U}_j^\top \tilde{\mathbf{W}}_i)\mathbf{U}_j$  in eq. 11, but we find the centered version of  $\mathbf{W}$  to be more effective. Following Mu and Viswanath (2018), we set  $\mathrm{D} = \lceil d / 100\rceil$ , where  $d$  is the dimensionality of a model.
5.2 Embedding Benchmarks
Setup: We evaluate each model's embedding's performance on common benchmarks for word similarity and relatedness before and after post-processing. We use the following data sets:
- SimLex-999 (Hill et al., 2015) - measures similarity, rather than relatedness or association.
- MEN Test Collection (Bruni et al., 2014) - measures the relatedness of words.
- WordSim353 (Agirre et al., 2009) - consists of two parts, one measures similarity, and the other measures relatedness of words.
Model	I(W)	I(Wc)	I(Wr)	avg(||e(w)||2)	||μ||2	||μ||2/avg(||e(w)||2)
BERT-base-uncased	0.39	0.98	0.998	1.40	0.94	0.67
BERT-base-cased	0.59	0.98	0.996	1.29	0.50	0.39
BERT-large-uncased	0.44	0.97	0.997	1.45	0.80	0.55
BERT-large-cased	0.52	0.96	0.995	1.53	0.65	0.42
RoBERTa-base	0.50	0.87	0.959	3.65	0.57	0.16
RoBERTa-large	0.07	0.64	0.956	4.36	2.53	0.58
GPT-2 (12 layers)	0.12	0.91	0.969	3.96	2.05	0.52
GPT-2 (24 layers)	0.52	0.95	0.981	3.68	2.04	0.55
Table 1: Isotropy,  $\mathrm{I}(\mathbf{W}) \in [0,1]$ , of embeddings from various language models. Centering an embedding matrix yields nearly perfectly isotropic embeddings in most of the tested models.  $\mathbf{W}_c$  stands for a centered matrix,  $\mathbf{W}_r$  stands for an embedding matrix with  $\lceil d / 100 \rceil$  top principal components removed.  $||\pmb{\mu}||_2 / \mathrm{avg}(||\mathrm{e}(\mathrm{w})||_2$  is the ratio of the  $L_{2}$  norm of the mean vector,  $\pmb{\mu}$ , to the average of the  $L_{2}$  norms of word embeddings.
- Stanford Rare Words (RW) (Luong et al., 2013) - measures similarity of words. In this dataset at least one word in each pair is a rare word.
The data sets are designed to measure embeddings' ability to reflect semantic relations. The performance on the data sets is measured by the correlation between the similarities of the representations and the human scores. We filter out samples consisting of subword units. Although this results in different test sets for different models, our goal is not to compare different models' performance but to validate the benefits of increased isotropy of embeddings. We score relations with both cosine similarity and inner product.
Schakel and Wilson (2015) show that vectors of more frequent words tend to have smaller norms, which was confirmed for BERT by Podkorytov et al. (2020). As the longer vectors of rare words are most affected by common enemies effect (see Section 4.2), we evaluate a "scaled-centering" method to account for that.
Specifically, we first compute the mean vector of embeddings normalized to unit length  $\hat{\mu} = \sum_{\mathrm{w}\in W}\frac{e(\mathrm{w})}{||e(\mathrm{w})||_2} / |V|$ . Then we scale the mean vector by the norm of each word embedding before subtracting it,  $e(\mathrm{w})' = e(\mathrm{w}) - ||e(\mathrm{w})||_2\hat{\mu}$ .
Isotropy: We find that merely removing the mean vector is enough for most models to reach nearly perfect isotropy. The results are in Table 1. The only exception is RoBERTa-large, which had the lowest initial isotropy. Interestingly, Schick and Schütze (2020a) show that RoBERTa-large outperforms BERT models on tasks designed explicitly for rare words. Moreover, according to common
leaderboards (Wang et al., 2019b,a), RoBERTa performs best on downstream tasks among the models we analyzed.
We stress that the  $\operatorname{I}(\mathbf{W})$  is an approximation of the degree of isotropy, and should be treated as such when interpreting its relation to downstream performance. The idea of the partition function  $Z(c)$  states that it's value should be constant for any vector  $c$  (Arora et al., 2016; Mu and Viswanath, 2018). As there is no closed-form solution for  $\min_{c\in X}$  and  $\max_{c\in X}$ , a set of eigenvectors of  $\mathbf{W}^{\top}\mathbf{W}$  has been used as  $\mathbf{X}$  in previous studies to approximate the isotropy (e.g., Mu and Viswanath, 2018; Wang et al., 2020). The vectors in  $\mathbf{X}$ , however, cannot be considered principal components of  $\mathbf{W}$ , unless the matrix  $\mathbf{W}$  has been centered. Pearson (1901) states that unless the mean of the data has been subtracted, the best fitting hyperplane would pass through the origin and not through the centroid. Indeed, for RoBERTa-large, the cosine similarity between the top eigenvector of  $\mathbf{W}^{\top}\mathbf{W}$  and the mean vector is 0.99.
Additionally, as the volume of a cube in  $\mathbb{R}^n$  grows exponentially with  $n$ , it may be sufficient for the embeddings to be isotropic around a point lying on a lower dimensional subspace to retain the desired separation. In fact, embeddings from RoBERTa-large have an average pairwise cosine similarity of 0.33 (angle of  $70.7^\circ$ ).
We speculate that a longer pretraining of RoBERTa compared to BERT results in a more significant shift of the embeddings in the dominating directions. Simultaneously, a larger pretraining corpus and a dynamic masking scheme used in RoBERTa may result in a more diverse set of shift directions. We leave this line of research for future studies.
Moreover, Mu and Viswanath (2018) demon
Model	CosSim	{·,·}
BERT-base-cased	62.29 (+0.00)	60.91 (+0.00)
+ Centered	60.44 (-1.85)	60.08 (-0.83)
+ Centered-Scaled	62.32 (+0.03)	62.41 (+1.50)
+ Post-Process	65.57 (+3.28)	66.10 (+5.19)
RoBERTa-base	66.81 (+0.00)	66.01 (+0.00)
+ Centered	66.85 (+0.04)	67.03 (+1.02)
+ Centered-Scaled	67.02 (+0.21)	66.95 (+0.94)
+ Post-Process	66.87 (+0.06)	67.15 (+1.14)
GPT-2-small	64.71 (+0.00)	59.45 (+0.00)
+ Centered	64.95 (+0.24)	66.04 (+6.59)
+ Centered-Scaled	66.57 (+1.86)	67.32 (+7.87)
+ Post-Process	67.85 (+3.14)	67.67 (+8.22)
Model	CosSim	{,}
BERT-large-cased	61.90 (+0.00)	59.42 (+0.00)
+ Centered	58.66 (-3.24)	57.99 (-1.43)
+ Centered-Scaled	61.18 (-0.72)	61.05 (+1.63)
+ Post-Process	65.72 (+3.82)	65.89 (+6.47)
RoBERTa-large	61.29 (+0.00)	44.79 (+0.00)
+ Centered	64.09 (+2.80)	63.78 (+18.99)
+ Centered-Scaled	65.49 (+4.20)	64.16 (+19.37)
+ Post-Process	64.38 (+3.09)	65.17 (+20.38)
GPT-2-medium	65.53 (+0.00)	59.20 (+0.00)
+ Centered	66.83 (+1.30)	67.90 (+8.70)
+ Centered-Scaled	67.95 (+2.42)	68.22 (+9.02)
+ Post-Process	68.05 (+2.52)	67.81 (+8.61)
Table 2: Average performance (Pearson's  $r \times 100$ ) of the models on the non-contextual benchmarks (SimLex-999, MEN, WordSim353, Stanford Rare Words). Centered stands for embedding matrix centered at origin; Centered-Scaled stands for embedding matrix with mean direction, scaled by the norm of each word embedding, subtracted; Post-Process corresponds to the method defined in eq. 11. Results from different models are not directly comparable due to different tokenization. Best results for each model are underlined. In general, increased isotropy results in increased performance. The improvement of RoBERTa-large is more significant as its initial isotropy is lower. Specific results for each benchmark can be found in Appendix C.
strate that neural language models are capable of learning to remove the mean vector. We leave the question whether Transformer-based language models perform an implicit representation centering operation to future research.
**Embedding Benchmarks:** We present our results on common benchmarks for word similarity and relatedness in Table 2. We report average scores from all tasks. The results on individual data sets are available in Appendix C. We observe that removing the mean vector, and consequently increasing the isotropy of embeddings, consistently improves the performance across all models, except for the most isotropic BERT-cased models. Furthermore, results in Table 2 demonstrate that "scaled-centering" is more effective than simple mean subtraction, and nearly as effective as the more expensive post-processing method. The only case in which "scaled-centering" does not improve performance is BERT-large-cased with cosine similarity as a scoring function.
Performance gains are more pronounced when inner-product is used as a scoring function, regardless of the model or processing method used. Although, initially cosine-similarity yields better results, especially for embeddings with greater  $L_{2}$  norms, mean subtraction is sufficient to close the gap in all but two models (BERT-cased models).
There has been a body of literature demonstrating substantial benefits of improved quality of word embeddings on downstream performance (e.g., Mu and Viswanath, 2018; Wang et al., 2019c; Gao et al., 2019; Wang et al., 2020; Schick and Schütze, 2020a). In particular, Gao et al. (2019) propose to add a cosine similarity regularization to the cross-entropy loss to increase the aperture of the cone in which embeddings are distributed, and report improved performance on machine translation and language modeling. It is straightforward to demonstrate that the cosine regularization proposed by Gao et al. (2019) is equivalent to minimizing the squared norm of the mean direction of embeddings, hence constraining the most significant drift direction. We provide the derivation of the equivalence in Appendix A.
Large-margin classification has been studied extensively, both in NLP (Wang et al., 2019c) and machine learning in general (Weston and Watkins, 1999; Tsochantaridis et al., 2005). As substantial shared components of embeddings will lead to a decreased classification margin in the output softmax layer, our work offers explanation for the fragility of pretrained language models reported in the literature (e.g., Schick and Schütze 2019, 2020a).
Our analyses show clearly that shifting of the embeddings in the embedding space is due to the dynamic interactions between the representations and the embedding vectors. As the embeddings
Visualization of distributions of  $L_{2}$  norms of embeddings from the analyzed models is available in Appendix C.
become more similar, the resulting representations become closer, creating a positive feedback mechanism for the representations to drift collectively. In addition, while isotropy of representations is desirable and has an overall positive impact on performance, the relationships between isotropy and performance in Table 1 and Table 2 suggest that the role of isotropy in model performance needs to be further analyzed. The dynamics of the interactions are being further investigated to pinpoint the root cause and their relationship with the model's performance.
Gao et al. (2019) present an insightful derivation of uniformly negative gradients for nonapparent words and formulate the optimization of rare words as an  $\alpha$ -strongly convex problem but make strong assumptions that the embedding matrix is learned after all other parameters of the model are well-optimized and fixed, which is not the case in practice. We do not make such assumptions, providing a more realistic explanation for the learning process. Wang et al. (2020) propose to reparametrize the embedding matrix using SVD and propose directly controlling the decay rate of singular values. Our paper's purpose is inherently different from that of Wang et al. (2020); we recognize that the fundamental understanding of the problem is missing and provide an explanation for the observations made in previous studies. Another line of work focuses on limitations of the softmax. Yang et al. (2018) suggest that softmax does not have sufficient capacity to model the complexity of language. Zhang et al. (2019) analyze the skip-gram model to show that optimization based on cross-entropy loss and softmax resembles competitive learning in which words compete among each other for the context vector. This idea is closely related to the common enemies effect reported in this paper, however, skip-gram seems to mitigate this through negative sampling (Mikolov et al., 2013b) but similar approaches do not seem to help Transformer pretraining (Clark et al., 2020).
A considerable effort has been made to improve performance of language systems on rare words, but the focus has been on either injecting subword information in non-contextual representations (Luong et al., 2013; Lazaridou et al., 2017; Pinter et al., 2017; Bojanowski et al., 2017), replacing rare words' representations through exploiting their
context (Khodak et al., 2018; Liu et al., 2019a), or both (Schick and Schütze, 2019, 2020a). In comparison, we strive to provide an explanation of the underlying problem, which is necessary to render such post-hoc fixes no longer necessary.
We find that the embeddings learned by GPT-2, BERT, and RoBERTa do not degenerate into a narrow cone, as has been suggested in the past, but instead drift in one shared direction. We recognize that target words produce gradients in the same direction for all the non-target words at each training step. Combined with the unbalanced distribution of word frequencies, any two words' embeddings will be repeatedly updated with gradients of the same direction. As such updates accumulate, the embeddings drift and share common components. Our experiments show that simply centering the embeddings restores a nearly perfectly isotropic distribution of tested models' embeddings and simultaneously improves embeddings' ability to reflect semantic relations. This understanding of the learning process dynamics opens exciting avenues for future work, such as improving the most affected embeddings of rare words and formulation of more computationally efficient training objectives.
Eneko Agirre, Enrique Alfonseca, Keith B. Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and wordnet-based approaches. In HLT-NAACL.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016. A latent variable model approach to PMI-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. In J. Mach. Learn. Res.
Michele Bevilacqua and Roberto Navigli. 2020. Breaking through the  $80\%$  glass ceiling: Raising the state of the art in word sense disambiguation by incorporating knowledge graph information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2854-2864, Online. Association for Computational Linguistics.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with
subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. J. Artif. Intell. Res., 49:1-47.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pretraining text encoders as discriminators rather than generators. In ICLR.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In EMNLP/IJCNLP.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693-1701.
Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41:665-695.
Hakan Inan, Khashayar Khosravi, and R. Socher. 2017. Tying word vectors and word classifiers: A loss framework for language modeling. ArXiv, abs/1611.01462.
Mikhail Khodak, Nikunj Saunshi, Yingyu Liang, Tengyu Ma, Brandon Stewart, and Sanjeev Arora. 2018. A la carte embedding: Cheap but effective induction of semantic feature vectors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12-22, Melbourne, Australia. Association for Computational Linguistics.
A. Lazaridou, Marco Marelli, and M. Baroni. 2017. Multimodal word meaning induction from minimal exposure to natural text. Cognitive science, 41 Suppl 4:677-705.
Qianchu Liu, Diana McCarthy, and Anna Korhonen. 2019a. Second-order contexts from lexical substitutes for few-shot learning of word representations. In Proceedings of the Eighth Joint Conference on
Lexical and Computational Semantics (*SEM 2019), pages 61-67, Minneapolis, Minnesota. Association for Computational Linguistics.
Xiaodong Liu, Yelong Shen, Kevin Duh, and Jianfeng Gao. 2018. Stochastic answer networks for machine reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1694-1704, Melbourne, Australia. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. In arXiv:1907.11692.
Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In CoNLL.
Christopher D. Manning and Hinrich Schütze. 2001. Foundations of statistical natural language processing. In SGMD.
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In ICLR.
Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Černocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In INTER-SPEECH.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In NIPS.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746-751, Atlanta, Georgia. Association for Computational Linguistics.
Jiaqi Mu and Pramod Viswanath. 2018. All-but-thetop: Simple and effective postprocessing for word representations. In International Conference on Learning Representations.
Karl Pearson. 1901. LIII. On lines and planes of closest fit to systems of points in space. In Philosophical Magazine.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL-HLT.
S. Piantadosi. 2014. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic Bulletin & Review, 21:1112-1130.
Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017. Mimicking word embeddings using subword RNNs. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 102-112, Copenhagen, Denmark. Association for Computational Linguistics.
Maksim Podkorytov, Daniel Bis, Jinglun Cai, Kobra Amirizirtol, and X. Liu. 2020. Effects of architecture and training on embedding geometry and feature discriminability in bert. 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8.
A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Adriaan MJ Schakel and Benjamin J Wilson. 2015. Measuring word significance using distributed representations of words. In arXiv:1508.02297.
Timo Schick and Hinrich Schütze. 2019. Attentive mimicking: Better word embeddings by attending to informative contexts. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 489-494, Minneapolis, Minnesota. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2020a. BERTRAM: Improved word embeddings have big impact on contextualized model performance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3996-4007, Online. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2020b. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In AAAI.
M. Schuster and K. Nakajima. 2012. Japanese and Korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149-5152.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. CoRR, abs/1704.04368.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
Martin Sundermeyer, Ralf Schluter, and Hermann Ney. 2012. Lstm neural networks for language modeling. In INTERSPEECH.