SlowGuess/NAACL_2021_OCR huggingface dataset:
Daniel Bió
Tallahassee, USA
bis@cs.fsu.edu
Maksim Podkorytov
maksim@cs.fsu.edu
Xiuwen Liu
liux@cs.fsu.edu
The success of language models based on the Transformer architecture appears to be inconsistent with observed anisotropic properties of representations learned by such models. We resolve this by showing, contrary to previous studies, that the representations do not occupy a narrow cone, but rather drift in common directions. At any training step, all of the embeddings except for the ground-truth target embedding are updated with gradient in the same direction. Compounded over the training set, the embeddings drift and share common components, manifested in their shape in all the models we have empirically tested. Our experiments show that isotropy can be restored using a simple transformation.
Word embeddings, both static (Mikolov et al., 2013a; Pennington et al., 2014) and contextualized (Peters et al., 2018), have been instrumental to the progress made in Natural Language Processing over the past decade (Turian et al., 2010; Wu et al., 2016; Liu et al., 2018; Peters et al., 2018; Devlin et al., 2019). In recent years, language models based on Transformer architecture (Vaswani et al., 2017) have led to state-of-the-art performance on problems such as machine translation (Vaswani et al., 2017), question answering (Devlin et al., 2019; Liu et al., 2019b), and Word Sense Disambiguation (Bevilacqua and Navigli, 2020), among others. However, it has been observed that representations from Transformers exhibit undesirable properties, such as anisotropy, that is tend to occupy only a small subspace of the embedding space. The observation has been documented by a number of studies (Gao et al., 2019; Ethayarajh, 2019; Wang et al., 2020). A similar property has been identified in the past in static word embeddings (Mu
and Viswanath, 2018). To address the issues, postprocessing methods (Mu and Viswanath, 2018), and regularization terms have been proposed (Gao et al., 2019; Wang et al., 2019c, 2020). However, the mechanism that leads to undesirable properties remains unclear. Without understanding the mechanism, it is going to be difficult to address the fundamental issue properly.
The deficiencies are most pronounced in the representations of rare words, as we will show in Section 4. Performance of pretrained language models is inconsistent and tends to decrease when input contains rare words (Schick and Schütze, 2020b,a). Schick and Schütze (2020a) observe that replacing a portion of words in the MNLI (Williams et al., 2018) entailment data set with less frequent synonyms leads to decrease in performance of BERT-base and RoBERTa-large by  $30\%$  and  $21.8\%$  respectively. After enriching rare words with surface-form features and additional context, Schick and Schütze (2020a) decrease the performance gap to  $20.7\%$  for BERT and  $17\%$  for RoBERTa, but the gap remains large nonetheless. Why do even the large-scale, pretrained language models struggle to learn good representations of rare words? Consider a language model with an embedding matrix shared between the input and output layers, a standard setup known as weight tying trick (Inan et al., 2017). Intuitively, at any training step  $t$ , optimization of the cross-entropy loss can be characterized as "pulling" the target embedding,  $\boldsymbol{w}_T$ , closer to the model's output vector  $\boldsymbol{h}_t$ , while "pushing" all other embeddings,  $\boldsymbol{W} \setminus \boldsymbol{w}_T$ , in the same direction, away from the output vector  $\boldsymbol{h}_t$ . This leads to what we call common enemies effect - the effect of the target words producing gradients of the same direction for all of the nontarget words. Compounded over the training set, the embeddings drift and share common components, manifested in their shape in all the models
<sup>1</sup>The code and datasets used in this paper are available at https://github.com/danielbis/tooMuchInCommon.
Based on the results reported by authors.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5117-5130 June 6-11, 2021. ©2021 Association for Computational Linguistics
we have empirically tested; see Figure 1.
Although Gao et al. (2019) report a closely related phenomenon and call it representation degeneration, their analysis is based on an assumption that the embedding matrix is learned after all other parameters of the model are well-optimized and fixed, which is not the case in practice. We conduct our analysis in a more realistic setting, and arrive at different conclusions. We show that embeddings do not occupy a narrow cone, but are shifted in one common direction and only appear as a cone when projected to a lower dimensional space (Section 4.1). In fact simply removing the mean vector of all embeddings, thus centering them, shifts the embeddings back onto a more spherical shape. We evaluate embeddings, before and after centering, on four standard benchmarks and observe significant performance improvement across all of them. Why is removing the mean so effective? We find that the common enemies effect applies to most, if not all, words in the vocabulary but in non-uniform manner. As language is known to follow an approximately Zipfian distribution (Zipf, 1949; Manning and Schütze, 2001; Piantadosi, 2014) even common words will not occur frequently in a text corpus, and in result will be often "pushed" by other target words in the same direction as rare words. Consequently, all embeddings share a significant common direction. We will focus on the analysis of auto-regressive GPT-2 (Radford et al., 2019) and two masked language models, BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b). Our contributions can be summarized as follows:
- We show that as word embeddings repeatedly share same direction gradients, they are shifted in one dominant direction in the vector space. The effects are the most evident in representations of rare words, but are also present in representations of frequent words.
- The shift causes the distribution of projected embeddings to appear as a narrow cone; we show that simply removing the mean vector is enough to restore the spherical distribution.
- We provide empirical evidence of our analyses using state-of-the-art pretrained language models and demonstrate that removing the mean dramatically improves isotropy of the representations.
2.1 Distributed Word Representations
Distributed representations induce a rich similarity space, in which semantically similar concepts are close in distance (Goodfellow et al., 2016; Bengio et al., 2003; Mikolov et al., 2013c). In a language model, the regularities of embeddings space facilitate generalization, assigning a high probability to a sequence of words that has never been seen before but consists of words that are similar to words forming an already seen sentence (Bengio et al., 2003; Mikolov et al., 2013c). Although models such as BERT or GPT-2 produce representations from a function of the entire input sequence, the representations are a result of a series of transformations applied to the input vectors. Consider an example sentence: "The building was dilapidated.", and the sentences resulting from replacing "dilapidated" with either "ruined" or "reconditioned". If the distance in the embeddings space between the two rather infrequent, but anonymous, words "dilapidated" and "reconditioned" is not larger than the distance between "dilapidated" and its relatively frequent synonym "ruined", then by the aforementioned generalization principle there is little to no reason to believe that the distance will become larger in the output layer.<sup>3</sup>
2.2 Tokenization
Do the subword tokenization methods (Schuster and Nakajima, 2012; Wu et al., 2016; Sennrich et al., 2016; Radford et al., 2019) preserve the word frequency imbalance? Examination of the common tokenization methods, such as Byte-Pair Encoding (Sennrich et al., 2016) and WordPiece (Schuster and Nakajima, 2012; Wu et al., 2016), suggests that subword units induced by tokenization algorithms exhibit similar frequency imbalance to that of full vocabulary. This can be explained by the greedy nature of the vocabulary induction process. Although different methods use different base vocabulary symbols to begin with (i.e., Unicode code points, or bytes), all of the methods construct the vocabulary through iterative merging of the most frequent symbols. As a result, the most frequent units are preserved as words, while the rare words are segmented into subword units. Moreover, the words which are segmented into subword units are
3In fact, all three sentences are assigned a negative sentiment, with scores between  $97\%$  to  $100\%$  by RoBERTa fine-tuned on SST.
(a)
(d)
Figure 1: Top: RoBERTa-large. Bottom: GPT-2 (12 layers). (1a, 1d): Word embeddings projected onto first two singular vectors. (1b, 1e) Centered word embeddings projected onto first two singular vectors. (1c, 1f) Singular values of embedding matrix before and after centering. Centering the embedding matrix increases isotropy of embeddings.
infrequent to such a degree that even their combined frequency is orders of magnitude lower than frequency of the most common words.
We confirm this empirically by tokenizing the CNN News corpus (See et al., 2017; Hermann et al., 2015) with WordPiece (used in BERT), revealing that over  $30\%$  of the corpus can be accounted for using 13 most frequent tokens, and  $50\%$  of the corpus can be accounted for using just 85 tokens. On the other hand, to cover at least  $98\%$  of the corpus, nearly 15000 tokens are needed. Therefore, we conclude that the tokens follow approximately Zipfian distribution (Zipf, 1949; Manning and Schütze, 2001) similar to that of full vocabulary. We provide a comparison of frequency distributions of tokens and words based on CNN-News corpus in Appendix B. $^{4}$
3 Learning Language Model
3.1 Autoregressive Language Models
Given a sequence of tokens  $\mathbf{w} = [\mathrm{w}_1,\dots ,\mathrm{w}_N]$  as input, autoregressive (AR) language models assign a probability  $p(\mathbf{w})$  to the sequence using factorization  $p(\mathbf{w}) = \prod_{t = 1}^{N}p(\mathrm{w}_{t}|\mathrm{w}_{< t})$ . Consequently, AR language model is trained by maximizing the
likelihood under the forward autoregressive factorization:5
$$
\begin{array}{l} \max  _ {\theta} \log p _ {\theta} (\mathbf {w}) = \sum_ {t = 1} ^ {N} \log p _ {\theta} \left(\mathrm {w} _ {t} \mid \mathbf {w} _ {<   t}\right) \tag {1} \\ = \sum_ {t = 1} ^ {N} \log \frac {\exp \left(\left\langle \mathbf {h} _ {\theta} \left(\mathbf {w} _ {1 : t - 1}\right) ^ {\top} , e \left(\mathbf {w} _ {t}\right) \right\rangle\right)}{\sum_ {\mathbf {w} ^ {\prime}} ^ {V} \exp \left(\left\langle \mathbf {h} _ {\theta} \left(\mathbf {w} _ {1 : t - 1}\right) ^ {\top} , e \left(\mathbf {w} ^ {\prime}\right) \right\rangle\right)} \\ = \sum_ {t = 1} ^ {N} \log \operatorname {s o f t m a x} \left(\boldsymbol {h} _ {\theta} (\mathbf {w} _ {1: t - 1}) \boldsymbol {W} ^ {\top}\right) _ {l a b e l _ {t}}, \\ \end{array}
$$
where  $h_{\theta}(\mathbf{w}_{1:t - 1}) \in \mathbb{R}^d$  is the output vector of a model at position  $t$ ,  $\theta$  are the model's parameters,  $W \in \mathbb{R}^{|V| \times d}$  is the learned embedding matrix,  $e(\mathbf{w})$  is a function mapping a token to its representation from the embedding matrix, and  $label_t$  is the index of the t-th target token in the vocabulary. To estimate the probability,  $W$  maps  $h_{\theta}(\mathbf{w}_{1:t - 1})$  to unnormalized scores for every word in the vocabulary  $V$ ; the scores are subsequently normalized by the softmax to a probability distribution over the vocabulary. In this paper, we focus on neural language models which compute  $h_{\theta}$  using the Transformer architecture, however the mechanisms is generally applicable to other common variants of language models (Mikolov et al., 2010; Sundermeyer et al., 2012; Peters et al., 2018).
4The preserved imbalance does not imply that subword tokenization is not beneficial to performance of language systems on rare words. It may mitigate some of the issues as shown in (Sennrich et al., 2016), however recent work demonstrates that it does not solve the problem (Schick and Schütze, 2020b,a).
<sup>5</sup>We omit the bias term in softmax for clarity.
Masked Language Modeling (MLM) pretraining objective is to maximize the likelihood of masked tokens conditioned on the (noisy) input sequence. Given a sequence of tokens  $\mathbf{w} = [\mathbf{w}_1, \dots, \mathbf{w}_N]$ , a corrupted version  $\hat{\mathbf{w}}$  is constructed by randomly setting a portion of tokens in  $\mathbf{w}$  to a special [MASK] symbol. Although MLM estimates the token probabilities of all masked positions,  $\bar{\mathbf{w}}$ , simultaneously and renders the factorization from Subsection 3.1 no longer applicable, the mechanism used to "unmask" a token differs only slightly from that in AR, specifically:
$$
\begin{array}{l} \max _ {\theta} \log p _ {\theta} (\bar {\mathbf {w}} | \hat {\mathbf {w}}) \approx \sum_ {t = 1} ^ {N} \mathrm {m} _ {t} \log p _ {\theta} (\mathbf {w} _ {t} | \hat {\mathbf {w}}) (2) \\ = \sum_ {t = 1} ^ {N} \mathrm {m} _ {t} \log \frac {\exp \left(\left\langle \boldsymbol {h} _ {\theta} (\hat {\mathbf {w}}) _ {t} ^ {\top} , e (\hat {\mathbf {w}} _ {t} \right\rangle\right)}{\sum_ {\mathbf {w} ^ {\prime}} ^ {V} \exp \left(\left\langle \boldsymbol {h} _ {\theta} (\hat {\mathbf {w}}) _ {t} ^ {\top} , e (\mathbf {w} ^ {\prime}) \right\rangle\right)} \\ = \sum_ {t = 1} ^ {N} \mathrm {m} _ {t} \log \operatorname {s o f t m a x} \big (\pmb {h} _ {\theta} (\hat {\mathbf {w}}) \pmb {W} ^ {\top} \big) _ {l a b e l _ {t}}, \\ \end{array}
$$
where  $\mathrm{m}_t = 1$  indicates  $\mathbf{w}_t$  is masked, and  $h_\theta (\hat{\mathbf{w}})_t$  is the output representations computed as function of the full, noisy, input sequence. Note, that the main difference between the equations 1 and 2 is the context used to condition the estimation. Models trained with MLM objective, like BERT and RoBERTa, compute the output vector utilizing bidirectional context through the self-attention mechanism, while the unidirectional models use only the context to the left of the target token. Moreover, only the probabilities of masked words,  $\mathbf{w}_i$  such that  $\mathbf{w}_i\in \bar{\mathbf{w}}$ , are estimated.
3.3 Learning Rules
Although the two objectives described above differ in terms of the distribution modeled (Yang et al., 2019), both AR and MLM models rely on the softmax function and cross-entropy loss. Using the notation established above, the cross-entropy loss function for an AR model is optimized by minimizing:
$$
J (\theta) = - \mathbb {E} _ {\mathbf {w} \sim \mathrm {d a t a}} \left[ \log p _ {\theta} (\mathbf {w}) \right], \tag {3}
$$
and for a MLM model it takes a form of:
$$
J (\theta) = - \mathbb {E} _ {\mathbf {w} \sim \mathrm {d a t a}} \left[ \log p _ {\theta} (\bar {\mathbf {w}} | \hat {\mathbf {w}}) \right]. \qquad (4)
$$
The gradient of the cross-entropy loss with respect to the embedding matrix  $\mathbf{W}$  is a sum of the gradient flowing through two paths: first one is through
the output layer where the embeddings are used to create the targets for the softmax, the second path flows through the encoder stack to the input layer. The gradient flowing through the embedding stack to the input layer is complex, and depends on minute details of a model. Although its contribution is not irrelevant, it is not necessary to illustrate the main point of this section. Thus, we focus on the update rule resulting from the gradient with respect to embeddings in the top layer of a model. For prediction of a token  $\mathbf{w}_t$ , let  $h_\theta$  be the output vector of either AR model (at index  $t - 1$ ) or MLM model (at index  $t$ ), let  $y = \mathrm{softmax}(f_t)$ , where  $f_t = h_\theta \mathbf{W}^\top$ , and let  $\hat{y}$  be the true probability distribution, then:
$$
\frac {\partial J _ {t}}{\partial \boldsymbol {W}} = h _ {\theta} (\hat {\mathbf {x}}) _ {t} ^ {\top} \cdot (y - \hat {y}). \tag {5}
$$
The resulting update rule for the embedding matrix is:
$$
\begin{array}{l} \boldsymbol {W} ^ {\prime} = \boldsymbol {W} - \eta \cdot \left(\boldsymbol {h} _ {\theta} ^ {\top} \cdot (y - \hat {y})\right) \\ = \boldsymbol {W} - \eta \cdot \boldsymbol {h} _ {\theta} ^ {\top} y + \eta \cdot \boldsymbol {h} _ {\theta} ^ {\top} \hat {y}, \tag {6} \\ \end{array}
$$
where  $\eta$  be the learning rate. Since  $\hat{y}$  is equal to 0 for all the indices except for the index of the target word  $\mathbf{w}_t$ , all the embeddings will become less similar to the representation produced by a model with the exception of the target word embedding. This leads to what we define as the common enemies effect - target words producing gradients of the same direction for all of the non-target words. As the parameters  $\theta$  are updated during the optimization process, the  $h_\theta$  changes even when the model is provided with the same input. Therefore, the direction of the gradient for the non-target words changes accordingly, but at a particular step the direction of the update is the same for all the non-target words. This is fundamentally different from the conclusion of Gao et al. (2019), who states that there exists a uniformly negative direction such that its minimization yields a nearly optimal solution for rare words' embeddings. We find that the common enemies effect is the most pronounced in the representations of rare words, which are less likely to appear as targets, but it is evident in all embeddings nonetheless.
4.1 Geometry of Embeddings
Previous studies (Gao et al., 2019; Wang et al., 2020) suggest that word embeddings learned by
Figure 2: A toy illustration of the effect that updates in one direction have on geometry of the representations and their singular values. The singular values in 2b correspond to the spheres of the same color in 2a. As the sphere moves away from the origin, the gap between the singular values of the points sampled from the sphere increases.
Transformer-based language models degenerate and occupy a narrow cone in the embedding space, but instead we find that embeddings simply drift in a common, dominant direction. The conclusions of Gao et al. (2019) are strongly influenced by a rapid decay of singular values of an embedding matrix, however, a rapid decay of singular values is not a sufficient condition to reach such conclusions.
In fact, points sampled from a 3D sphere satisfy the condition given above. As at first glance this is not entirely obvious, we provide a toy example in Figure 2 that illustrates why embeddings appear as a cone when projected to a low dimensional space. We sample points at random from two spheres, one centered at the origin and one shifted away from the origin (Figure 2a), and perform Singular Value Decomposition on the two sets of samples. When the sphere moves away from the origin, the difference between the two singular values increases (Figure 2b).
Similarly, the projection of uncentered embeddings (see Figures 1a and 1d) appears as a cone, but when embeddings are centered around origin (Figures 1b, 1e), the shape of their projection changes to resemble a sphere more than a cone; that is simply removing the mean vector  $\mu$  of an embedding matrix  $W$ , where  $\mu = \sum_{\mathrm{w} \in W} e(\mathrm{w}) / |V|$ , increases the isotropy of embeddings. Optimization of a neural language model is certainly more complex than our toy example. Most of all, the common enemy effect is not uniform; the amount by which each vector moves in the most dominant direction depends on many factors, among others the size of the training corpus, the diversity of the training corpus, or whether static (BERT) or dynamic (RoBERTa) masking is used. In a more general
sense, the magnitude of the gradient with respect to a word vector depends on the value in the logit corresponding to that word, hence the shift will not be uniform.
4.2 Unused Tokens and Rare Words
We hypothesize that as rare words drift in common direction, their embeddings become less discriminative than embeddings of frequent words. BERT's vocabulary provides a unique opportunity to investigate the contribution of the same direction gradients to embeddings of particular words. There are 994 special unused tokens in BERT's vocabulary that were not used as inputs or targets during pretraining, thus all the updates to their representations were in the directions opposite to output vectors. As shown in Figure 3, we observe that cosine similarity between the unused tokens and other tokens increases as the frequency decreases. The average cosine similarity between unused words and tokens in indices [28500-29500] is 0.63. In comparison the unused tokens have cosine similarity of 0.27 with tokens in indices [2000-3000] (most frequent tokens, i.e., "to") but the similarity goes up rapidly for tokens other than the most frequent ones. Schick and Schütze (2019), evaluate BERT and RoBERTa on a dataset explicitly measuring the ability of MLM models to "unmask" words of different frequencies, and report that both models struggle to "unmask" rare words. Results presented in this section provide an explanation of this behavior and confirm that embeddings of the rare tokens
Although frequency depends on a corpus, in general higher index implies lower frequency due to the way BERT's vocabulary is constructed.
<sup>7</sup>We observe a similar pattern in RoBERTa using the last 1000 words in its vocabulary in place of the unused tokens.
Figure 3: Cosine similarity between the [unused] tokens and words in vocabulary of BERT-base-case grouped into bins of 1000 (i.e., [1000:1999]).
are most affected by the common enemies effect.
We validate our theoretical analysis through a series of experiments on geometric properties of non-contextualized embeddings.
5.1 Isotropy
Although centering an embedding matrix results in a more desirable spectral distribution, tokens of comparable frequency tend to remain clustered in the embedding space, as shown in Fig 1. Therefore, we empirically test how much actual gain in terms of isotropy is obtained in embeddings of the tested models by removing the shared direction. Moreover, Mu and Viswanath (2018) show that the top principal components in skip-gram embeddings (Mikolov et al., 2013a) correspond to frequency of words and demonstrate that such frequency bias can be mitigated by removing the top principal components of an embedding matrix. We evaluate the effectiveness of this approach on embeddings from Transformer-based models. We use BERT, RoBERTa, and GPT-2 in different sizes in our experiments.
Setup: We measure the initial isotropy of embeddings in each of the models, and the isotropy after removing the mean vector  $\mu = \sum_{\boldsymbol{w} \in \boldsymbol{W}} \boldsymbol{w} / |V|$  from each row of an embedding matrix  $\boldsymbol{W}$ , yielding  $\tilde{\boldsymbol{W}} = \boldsymbol{W} - \boldsymbol{\mu}$ . Next, we use a slightly modified approach of Mu and Viswanath (2018), and remove D top principal components from each model's embedding matrix to obtain highly isotropic representations. Finally, we evaluate whether increasing isotropy of embeddings from Transformer-based models can improve performance on standard embedding benchmarks.
Definitions: To measure isotropy, we use the partition function defined in (Arora et al., 2016),
$$
Z (c) = \sum_ {\mathrm {w} \in V} \exp \left(c ^ {\top} e (\mathrm {w})\right), \tag {7}
$$
where  $e(\mathbf{w})$  maps a word w to its embedding and  $c$  is a unit vector. For vectors to be isotropic, the value of  $Z(c)$  should be approximately constant, according to Lemma 2.1 in (Arora et al., 2016). Based on this property, we empirically measure the isotropy of an embedding matrix  $W$  using:
$$
\operatorname {I} (\boldsymbol {W}) = \frac {\min  _ {\boldsymbol {c} \in \boldsymbol {X}} \mathrm {Z} (\boldsymbol {c})}{\max  _ {\boldsymbol {c} \in \boldsymbol {X}} \mathrm {Z} (\boldsymbol {c})}, \tag {8}
$$