Isotropy in the Contextual Embedding Space: Clusters and Manifolds
Authors: Xingyu Cai
Jiaji Huang
Yuchen Bian
Kenneth Church
Venue: ICLR 2021 Poster (2021)
Keywords: Contextual embedding space
Isotropy
Clusters
Manifolds

Abstract: The geometric properties of contextual embedding spaces for deep language models such as BERT and ERNIE, have attracted considerable attention in recent years. Investigations on the contextual embeddings demonstrate a strong anisotropic space such that most of the vectors fall within a narrow cone, leading to high cosine similarities.  It is surprising that these LMs are as successful as they are, given that most of their embedding vectors are as similar to one another as they are. In this paper, we argue that the isotropy indeed exists in the space, from a different but more constructive perspective. We identify isolated clusters and low dimensional manifolds in the contextual embedding space, and introduce tools to both qualitatively and quantitatively analyze them. We hope the study in this paper could provide insights towards a better understanding of the deep language models.

Published as a conference paper at ICLR 2021
ISOTROPY IN THE CONTEXTUAL EMBEDDING SPACE: CLUSTERS AND MANIFOLDS
Xingyu Cai, Jiaji Huang, Yuchen Bian, Kenneth Church Baidu Research, 1195 Bordeaux Dr, Sunnyvale, CA 94089, USA
lxingyucai, huangj jiaji, yuchenbi an, kennethchurcch) @ba du com
ABSTRACT
The geometric properties of contextual embedding spaces for deep language models such as BERT and ERNIE, have attracted considerable attention recent years. Investigations on the contextual embeddings demonstrate a strong anisotropic space such that most the vectors fall within a narrow cone, leading to high cosine similarities. It is surprising that these LMs are successful they are, given that most of their embedding vectors are as similar to one another as they are. In this paper, argue that the isotropy indeed exists in the space, from different but more constructive perspective. We identify isolated clusters and low dimensional manifolds in the contextual embedding space, and introduce tools both qualitatively and quantitatively analyze them. We hope the study in this paper could provide insights towards better understanding of the deep language models.
INTRODUCTION
The polysemous English word "bank" has two common senses: 1. the money sense, a place that people save or borrow money; 2. the river sense, slope of earth that prevents the flooding. In modern usage, the two senses are very different from one another, though interestingly. both senses share similar etymologies (and both be traced back to the same word in Proto-Germanic) In the static embedding, multiple instances of the same word (e.g. "bank"" will be represented using the same vector. On the contrary, the contextual embedding assigns different vectors different instances of the same word, depending on the context. Historically, static embedding models like Word2vec Mikolov et al. 2013b) and GloVe (Pennington al.) 2014). predated contextual embedding models such as ELMo (Peters et al. 2018), GPT (Radford et al. 2018), BERT (Devlin et al.) (2018) and ERNIE (Sun et al.] 2019). Much of the literature on language modeling has moved contextual embeddings recently. largely because of their superior performance on the downstreaming tasks.
1.1 RELATED WORK
The static embeddings are often found to easier to interpret. For example, the Word2 and GloVe papers discuss adding and subtracting vectors, such as: vec(king) vec(man) vec(women) vee(queen). Inspired by this relationship. researchers started to explore geometric properties of static embedding spaces. For example, Mu & Viswanath (2018) proposed very counter-intuitive method that removes the top principle components (the dominating directions in the transformed embedding space), which surprisingly improved the word representations. Rather than completely discarding the principle components, Liu et al. (2019) proposed to use a technique called Conceptor Negation, to softly suppress transformed dimensions with larger variances. Both approaches, simply removing certain principle components as well as Conceptor Negation, produce significant improvements over vanilla embeddings obtained by static language models. In Huang et al. (2020), the authors studied how to effectively transform static word embeddings from one language to another.
Unfortunately, the strong illustrative representation like the king-queen example above, no longer obvious in general contextual embedding space. Arguing that syntax structure indeed exists in the contextual embeddings, Hewitt & Manning (2019) proposed a structural probe to identify the syntax trees buried in the space, and found the evidence of implicit syntax tree in BERT and ELMo. The advantage of contextual embedding over the static counterpart, mainly come from its capability to assign different vectors to the same word, depending the word sense in the context. Researchers in (Reif et al., 2019) found such a geometric representation of word senses in the BERT model. These papers reveal the existence of linguistic features embedded implicitly in the contextual vector spaces.

---

Published as a conference paper at ICLR 2021
The geometric properties of contextual embedding space are also investigated and compared with the static embedding space. Mimno& & Thompson (2017) found anisotropy when negative sampling is used. In (Ethayarajh] 2019), the authors characterize how vectors are distributed in the contextual space. They found that most vectors occupy in a relatively narrow cone in the space. Pairs of vectors within this cone have large cosines. This phenomenon can be found in most state-of-the-art contextual embedding models, In (Gao et al., 2019), the authors named this phenomenon "representation degeneration", and attempted to mitigate the problem by introducing a regularization term that minimizes cosine similarities between vectors. In very recent work, Demeter et al. (2020) suggest there is structure weakness in the space that leads to bias when using soft-max, as is common with deep language models.
1.2 MOTIVATION AND CONTRIBUTIONS
Isotropy often makes the space more effectively utilized and more robust to perturbations (no extreme directions that lead to high condition number). It is counter-intuitive and not clear why those contextual embedding models perform remarkably well on many tasks given their anisotropic embeddings bring all the vectors close together, hard to distinguish one from another. On one hand, it is widely believed that contextual embeddings encode the relevant linguistic information (c.g. (Reif) al., 2019)), but the other hand, it is also widely believed that the contextual space is anisotropic that representations become degenerated (e.g. (Mimno & Thompson 2017). (Gao et al.) (2019).
(Ethayarajh) 2019). These motivate us to find reasonable understanding that bridges this gap.
This paper is similar in spirit to (Mu & Viswanath| 2018), but different in three aspects. First, we generalize their work on traditional static embeddings more modern contextual embeddings. Second, we introduce clustering methods to isolate the space, whereas they used PCA to remove dominant dimensions (that tend dominate the variance). Finally, we identify low dimensional manifolds in the space, and introduce an alternative approach (LID) to characterize local subspaces.
This paper takes a deeper look into the contextual embedding spaces of popular pre-trained models. It identifies the following facts that were misunderstood or not known before: 1) We find isotropy within clusters in the contextual embedding space, in contrast to previous reports of anisotropy (caused by misleading isolated clusters). We introduce clustering and center shifting to reveal the isotropy, and show more consistent layer-wise behavior across models. 2) We find a Swiss-Roll manifold in GPT/GPT2 embeddings. but not in BERT/DistiBERT embeddings. The manifold is related to word frequency, suggesting a difference in how models evolve as they see more data. We use approximate Local Intrinsic Dimension (LIDD characterize the manifold, and find contextual embedding models, including all BERT, GPT families and ELMo, often have small LIDs. The small LIDs can be viewed the local anisotropy of the space. The code for this paper could be found at htthtttt://// http://aa..com//eeeeeeeeeeeeeeee
Key Contributions:
2
ANALYSIS SETTINGS
2.1 MODELS AND DATASETS
In this paper, we consider popular pre-trained contextual embedding models, including BERT, DistiBBRTT (Sanh al. 2019) (or denoted D-BERT in the rest of the paper). GPT, GPT2 Radford et al. 2019) and ELMo For the BERT and GPT families, we perform our evaluations on the pre- trained uncased base models from Huggingface http://////igiggiii..oo////oooo/ ww/rransfoomrrs/ineee. The pre-trained ELMo model is from AllenNLP (htps::/doos.allemmpporryy wtog/vl...00...... BERT and D- BERT are non-causal models because of their attention mechanism, where tokens can attend to any token in the input, regardless of their relative positions. In contrast, GPT and GPT2 are causal models because attention is limited to the tokens previously seen in the input.
Different models achieve contextual embedding in different ways. For instance, BERT adds positional embeddings to the token embeddings, while ELMo performs vector concatenation. Most models start with an initial layer that maps token ids to vectors. This paper is not concerned with that lookup table layer, and only focuses on the layers after that. The base BERT, GPT and GPT2 models have 12 layers of interest, indexed from 0 to 11, while D-BERT has 6 layers and ELMo has two.
We use Penn Tree Bank (PTB) (Mareus et al. 1993) and WikiText-2 Merity et al. 2016) datasets. The PTB has 0.88 million words and Wexi has million. Both of them the standard datasets
2

---

Published as a conference paper at ICLR 2021
for language models. In the rest of the paper, we report PTB since similar results with both datasets. Details on WikiText-2 analysis could be found in Appendix.
2.2 NOTATION
For each position in corpus, we have word. Words converted into tokens, using the appropriate tokenizer for the model. Tokenizers could split some words into subwords, therefore, the number of obtained tokens (denoted as n) could be more than number of words in the corpus. PTB, for example, contains 0.88 million words, but has n = 1.2 million tokens, when processed by BERT's tokenizer. Let V be the vocabulary, a set of distinct tokens. For any element in the vocabulary V, we call type. For example, BERT has vocabulary of roughly 30, 000 types. We may mix using "word" and "type" for ease of reading. We denote the i-th type in V as &. Let ((t+) = (t)( d2 (f.)....) be the set of all embedding instances of f; (note that different contexts in the corpus yield different embeddings of t,). By construction, Σ, [()) n. We define the inter-type cosine similarity as
Sinter - Eizj [cos (o(t.), ((t)))]
(I)
where (t4:) is one random sample from ((+)), and the same for e(tz) ((2)). The expectation is taken over all pairs of different types. Similarly, we define the intra-type cosine similarity as
S A E, [Raget [cos (4)) (z (t.))]
where the inner expectation is over different embeddings (tt.) for the same type t, and the outer expectation is over all types. Both Simer and Sinrra take values between -1 and 1. Note that for i.i.d. Gaussian random samples 2;, y. the expected cosine similarity E[eos(x, v)] = 0. A cosine value closer to 0 often indicates strong isotropy.
Clearly, the inter-type metric describes the similarity between different types, where the intra-type one measures similarity between same type's embedding instances. Our definitions of Sinte and Sunra are similar to the measures used in |Ethayarajhh (2019), but the corpus level. Note that some types are more frequent than others, especially under a Zipfian distribution (Piantadosi) 2014). and therefore, the size of (() varies dramatically with the frequency of type L.
2.3 AN INITIAL LOOK AT ANISOTROPY
Inspired by Ethayarajh (2019). follow their procedure and take first look the anisotropy identified by Mimno & Thompson (017) and Etthyyarjjh (019). in the contextual embedding space.
inter-lype cosine
intra-type cosine
0.8
D-BERT
0.0
GPT
GPT2
cciee
BERT
0.4
0.4
D BERT
GPT
0.2
GP12
ELMo
0.0
6
10
10
layar id
layer id
Figure 1: Siner (left) and Smar (right). The Syner increases as layer goes deeper, especially for GPT2's last layer. The Simm are generally high. This means arbitrary vectors have high cosine similarities.
Figure T shows strong anisotropy effects in a number of models. These findings are consistent with Ethayarajh (019), though
slightly different metrics. The plots show expected cosine (Smmer and Sumra as a function of layer. For efficiency, we approximate Smar by imposing a limit of 1,000 samples for frequent types, I, if [+() 1000. From the figure we can see the following:
Both Sinter and Simm are high C* 0) across almost all the layers and all the models. In particular,
the same as reported in Ethayarajj 2019). GPT2 is relatively more anisotropic.
Sinter tends to increase with layer, in contrast with Smas which in general decreases but with fluctuations. This means that embeddings for different types are moving closer to one another at deeper layers, while embeddings for the same type's 's instances spreading away.
3

---

Published as a conference paper at ICLR 2021
The last layer is often special. Note that the last layer has smaller cosines than the second last in most cases, with the notable exception of GPT2.
summary, we observe large cosines (across layers/models), especially for the GPT2 model. When cosines are close to 1, embeddings lie in subspace defined by very narrow cone (Ethayarajh) 2019). One might expect embeddings be more effective if they took advantage larger subspace. Are these models missing opportunity to have the benefits from isotropy (Mu & Viswanath 2018 We answer this question the following sections.
3 CLUSTERS IN THE EMBEDDING SPACE
3.1 EFFECTIVE DIMENSIONS
There are m 768 embedding dimensions for BERT, D-BERT, GPT and GPT2, and m 1024 dimensions for ELMo. We perform PCA to reduce the number of dimensions from m down to For each layer of each model, we start with the data matrix, M E R"*m where n is the number of imput tokens (n = 1.2.M for PTB dataset), and m is the original number of dimensions. After PCA, we end up with a smaller matrix, M E R Let the explained variance ratio be: Th = Σ 0, Ee os, where o; is the i-th largest eigen value of M's covariance matrix. In this way, we define the refffectivediimeniion be: d(c) - arg min; F 2 c. For example, d(0.8) = means that 2 dimensions capture 80% of the variance. There is direct connection between d and isotropy: the larger d often implies more isotropy, as data spreads in multiple dimensions.
Table I: The effective dimension d(0.8)
Layer
2
3
4
8
9
BERT
262
273
271
273
276
283
288
282
282
282
283
270
D-BERT
244
226
232
227
217
175
GPT
265
141
65
76
173
210
205
217
221
253
269
307
GPT2
114
1
8
26
66
116
ELMo
455
Table 1 reports d(0.8) for different layers and models. is surprising that GPT2 has few effective dimensions, especially, d(0.8) for layer 2 to 6. The surprisingly small effective dimensionality another way of saying that GPT2 vectors fall in a narrow cone, and consequently, their pairwise cosines are large. If all the vectors lie on 1-D line, all the cosines would be 1, and there would be hardly any model capacity. These observations motivate us look deeper into the embedding space.
3.2 ISOLATED CLUSTERS
ratio ~00.220
ratio r-0.147
r-0.142
ratio 1-0.100
60
-20
15
550 paopooooo
-10
10
(a) GPT2 layer
(b) BERT layer 6
(c) D-BERT layer
(d) ELMo layer
Figure 2: Isolated clusters exist in the embedding spaces for all the models. Here only show few representative middle layers for each model. The full visualization can found supplementary.
By performing PCA to project the original data into 3-D view, we can visualize GPT2's layer 6's embedding space in Figure 2a The three axes refer to the first three principle components, which account for 82.8% of the total variance. All the explained variance ratio will be reported throughout the rest the paper. The axes values are raw coordinates after PCA. In Figure 2a] there are two disconnected islands that far away from each other. Note that the first dimension coordinate values
4

---

Published as a conference paper at ICLR 2021
spans from 0 to 3000, significantly wider than the other 2 dimensions. In fact this first principle dimension dominates the total variance. The left island is bigger than the one on the right. The fact that the two islands are well separated by the first principle component suggests that classifying points by island memhership accounts for much of the variance. This two-island property is exhibited in layers 2 through 10 for GPT2. The two islands merge into single large cluster in the last layer.
We observe similar clustering behavior for all the models across all the layers, though the separations less distinct, illustrated in other panels of Figure This is also consistent with Table the less separation, the higher d(e) values. For GPT2, we had hoped to find that some types are associated with one cluster and other types are associated with the other cluster, but that is not verified in our experiments. Please refer to the supplementary for visualizations of all layers in all the models.
3.3 CLUSTERING
Previous literature estimated the space isotropy on pairs of arbitrary tokens, which could reside in two disconnected clusters. But given that the variance is dominated by distances between clusters, such estimation would be biased by the inter-cluster distances. It is more meaningful to consider per-cluster investigation rather than global estimate.
We start by performing clustering the embedding space. There are many methods for clustering. We chose K-Means (https://scikit-lean fast for large inputs (n = 1.2 million vectors) in high (m 2 768) dimensions. DBSCAN algorithm (Ester et al. 1996) could be an alternative as density based, but only works on small dataset. We use the Silhouette method (Rousseeuw 1987) to determine the number of clusters, |C|. After running K-means, each point (one of the n vectors in M) assigned to one of clusters. For a data point p assigned to the cluster C, calculate the following:
http://cckktteeem..rrr/ttteeeee www.g//tabeeeoooulsss/sssssseese htmlitt,, because it reasonably
B bp.
if |c| >
0g
Σ gEc, psf
dist(p, g) by = min EFC
dist(p, g) 8p
otherwise
gEa
where ap is the mean distance between p and other points in the same cluster; by is the minimum (min over 2) mean distance between p to points another cluster ē; and 8, is the Silhouette score for point p E The sp takes value [-1, 1]. The higher Sp. the better assignment of p to its cluster. Better choices of |C| would lead to better values of sp (and better clustering). We define the Maximum- Mcan-Silhouell
(MMS) score for the embedding space as: MMS maXaifierem |C] Ep [sp]. where the maximum is over different |C* values for K-Means. Since it is not feasible to evaluate all choices of [C| [1, n.. consider [C| [1, 15]. The expectation [pp] (the mean Silhouette score), is estimated from 20, 000 sample vectors in M. We select the best [C] that yields MMS.
The MMS values provide systematic way describe how the clusters distributed in the space. If the clusters are very distinct and splitted, this yields higher MMS. On the other hand, if clusters overlapping, blurring together, the MMS score will be low. Note that if MMS 0.1, we [C to be 1, as the Silhouette score does not show significant evidence of more clusters.
Table 2: Number of clusters IC|
1.0
MMS
BERT
D-BERT
Layer
BERT
BERERT
GPT
GPT2
ELMo
GPT
6
7
2
GPT2
6
MMS
ELMo
2
4
15
0.4
3
4
14
2
0.2
3
10
14
6
layer
7
2
2
8
2
2
2
Figure 3: The MMS for all the models. GPT2 has significantly higher MMS scores than other models from layer to layer 11. This means the cluster effects more severe in GPT2.
11
2
9
5

---

Published as a conference paper at ICLR 2021
Table 2 makes it clear that clustering plays important role in most layers of most models. Some models (BERT and D-BERT) have more clusters, and some have fewer (GPT, GPT2, ELMo). This dichotomy of models is also reflected in Figure
Maximum-Mean -ilauuSti scores are shown in Figure for GPT2, starting from the 2nd layer. Recall in Figure 2a we showed that two far-away islands exist in the space and their distance dominates the variance. In Figure the high MMS scores also verifies that. Another interesting observation is, for causal models GPT, GPT2 and ELMo, they all have higher MMS in their middle layers but lower MMS in the end. This means their initial layer and final layers' embeddings tend merge. On the contrary, the BERT and DistilBERT have increasing MMS in deeper layers, meaning that the clusters in their embeddings are becoming clearer in deeper layers.
There are significantly higher MMS values
3.4 ISOTROPY IN CENTERED SPACE WITHIN CLUSTERS
As suggest by Mu & Viswanath (201 2018), the embedding space should measured after shifting the mean to the origin. We subtract the mean for each cluster, and calculate the adjusted Siner- Assuming we have a total of [C] clusters, let (t) = {tf(t), d5 (t).... be the set of type t's embeddings in cluster C, and (t) be one random sample in (t). Define the adjusted similarity:
Sier A E. [Eiz] [cos (4), (4))]] where x²(() (t) Eg [+)
Here E. is the average over different clusters, and (t()) is the original embedding shifted by mean (subtract the mean), where the mean is taken over the samples in cluster c. Similarly we define
Smma
E
Eg
[cos (t), of(+))]]
Inter-type cosine
adjusted intra-type cosine
1.0
BERT
BERT
0.4
GPT2
0.6
ELMo
0.4
D-BERT
0.0
GPT
0.2
-0.2
0.0
luyer
layer
Figure 4: Si (left) and Sym (right). The adjusted Syier are close to zero, meaning that the space is isotropic under the adjusted measure.
The Figure 4 illustrates the adjusted cosine similarities Si and S/ It reveals that:
For the adjusted inter-type cosine (the left plot), all models are having consistent near-zero S This means nearly perfect isotropy exists within each cluster, in each layer of all the models. The last layer of GPT2 and BERT has slightly worse isotropic behavior, nevertheless, general inter-type isotropy stays across all layers. This reveals the distinguishable embedding vectors.
The general decreasing trend of intra-type cosine (the right plot) shows that the multiple instances for the same type/word, is slowly spreading over the layers. This is consistent with the un-centered intra-type cosine shown in Figure
4
LOW-DIMENSIONAL MANIFOLDS
4.1 SWISS ROLL MANIFOLD OF GPT/GPT2
While BERT and D-BERT tend to distribute embeddings along more dimensions, GPT and GPT2 embed tokens in low-dimensional manifolds in their contextual embedding spaces. More specifically, we discover that most of the tokens are embedded on a spiral band, and that band gets thicker in the later layers thereafter form Swiss Roll shaped surface.
Figure 5a and 5b] show the 2-D front view of the manifold in GPT and GPT2. Figure 5a zooms into the large cluster illustrated in Figure 2a (the left one), and discards the smaller (the right one).

---

Published as a conference paper at ICLR 2021
i a
50 3
(a) GPT layer
(b) GPT2 layer 2
(c) GPT layer 2 3-D view (d) GPT2 layer 2 3-D view
Figure 5: The 2-D and 3-D view of low-dimensional manifold in GPT/GPT2's embedding spaces
3-D plots are shown in Figure 5c and 5d to demonstrate two manifolds, band shaped manifold and Swiss Roll shaped manifold. These plots were computed over PTB dataset. Similar results have been obtained from WikiText-2 in supplementary. Figure tracks the progression of a narrow band into Swiss Roll. The Swiss Roll becomes taller and taller with deeper and deeper layers.
ratio 00559
ratio r=0.586
rabio r=0.410
ratio r=0.342
75
X5
5.
10
23
-70
m
19
-L0"
(a) GPT layer
(b) GPT layer 3
(c) GPT layer
(d) GPT layer 5
Figure 6: The evolution from a narrow band into a taller and taller Swiss Roll with deeper layers.
4.2 TOKENS IN THE SPACE
To verify the manifold structure in GPT family, study the token embeddings in the space. It is believed that similar embeddings (c.g. the embeddings for two instances the same word) tend to stay close together in Euclidean space, they should have high cosine similarities. Figure7 down into the embeddings for six frequent words: three punctuation symbols (""(", "&", """) and three common words ('the", "first", "man"". Each panel uses four colors: three colors (black, red, green) for three words of interest, plus gold color for all the other tokens.
var ratio r= r=0.
var ratio r= 147
uar ratio r=0.566
var ratio r=0
Bige dp3
Decwa fistow
-20
menen
20
(a) BERT, symbol tokens
(b) BERT, word tokens
(c) GPT, symbol tokens
(d) GPT, word tokens
Figure 7: Embeddings for symbol tokens and word tokens, in layer of BERT and GPT. This shows that GPT has manifold structure, such that vectors along the spiral band. BERT's space is closer to Euclidean space as similar vectors are in concentrated clusters.
As shown Figure 7b the BERT model indeed group similar embeddings into small regions the space (the red, black and green clusters). However, the GPT models are assigning similar embeddings along the manifold we observed before. In Figure 7c 7 the embeddings for the tokens occupy a spiral band that almost cross the entire space. It does not comply with the Euclidean space geometry as points in such a spiral band would not have high cosine similarity. A Riemannian metric must exist, such that the manifold has larger distance between two spiral bands, but smaller distance the
7

---

Published as a conference paper at ICLR 2021
band. Note that the 3-D plots obtained using PCA, so there is density- based nor non-limen con-linear reduction involved. Therefore, the manifold structures in GPT embedding spaces are verified.
4.3 WORD FREQUENCY
Another key finding is that all the models are trying to map the high frequent words/types to some specific region in the embedding spaces, rather than spreading them out to the entire space. In Figure embeddings (8a ) and corresponding word frequencies (8b Sd] of GPT's layer 8 and are shown. The darker red denoted higher frequency and blue is lower frequency. The numbers at the colorbar show the number of occurrence (of particular word / type).
4000
4000
17-
75
0
24000
02
2400
1030
1000
-3
(a) Layer 8 (front view) (b) Layer by frequency
(c) Layer 9 (side view) (d) Layer by frequency
Figure 8: Word frequency heatmap in GPT layer and 9. Red is high frequency, blue low. High frequency words are at the front end of the Swiss Roll, while low frequency words at the other end.
8b 8d are drawn using trrpoooobbrr so the ring at 8bbs bottom should not be closed.)
The Figure 8a 8c are after PCA, and selecting the two most significant dimensions. From GPT layer to layer 9, as the Swiss Roll becomes taller, more variance accounted for along the height the Swiss Roll. Thus, the perspective switches from front view side view when moving to layer 9.
Figures 8b and Sd show that the most frequent words appear at the head of the Swiss Roll, followed by bands of less and less frequent words. The least frequent words appear the far end of the Swiss Roll. This pattern suggests the model distinguishes more frequent from less frequent words. As the model finds more and more rare words, it appends them at the end of the Swiss Roll.
4.4 MANIFOLD LOCAL INTRINSIC DIMENSION
Although the original space dimension is 768 (1024 for ML,o),, the manifold we observed has a lower intrinsic dimension. means the data point on the manifold has fewer degrees of freedom move around. For example, on Swiss Roll in a 3-D space, any point can only have 2-D freedom thus the intrinsic dimension is only 2. A recent research on the intrinsic dimension for deep networks could be found at (Ansuini et al. 2019). In this section, we adopt the Local Intrinsic Dimension (LID) that estimates dimension locally with respect to a reference point. LID is introduced by Houle 2013). and being used in deep learning model characterization recently, e.g. (Ma et al.) 2018). The LID is often derived using expansion models (Houle et al,, 2012), which tries to obtain the local dimension in the vicinity of reference point from the growth (expansion) characteristics. To illustrate this, borrow an example from Ma et al. (2018). Let y be the radius of an mn- ball in the Euclidean space, denote its volume v, then the volume's growth rate is proportional ymt i.e. 12/v1 (12/m))* from which we can infer the local dimension m by m- = log(u2/1)) 1og(12/7m))
Accurately computing LID is hard problem which requires tremendous amount of data sam- ples and enough density around the reference point. So fewer-sample estimate of LID is being studied in the past decade. One of the efficient estimation is proposed by Amsaleg et al. (2015). This technique relies on K nearest neighbor search (K-NN) For reference point denote the set of its K nearest neighbor points as V {91 .......... gK3 Then the estimate of LID is com-
where the term inside log is the ratio of distance between p to its neighbor, over the maximum distance among them. In our anal- ysis, we use an efficient nearest neighbor computation package FAISS (Johnson et al. 2017) http:/gghhhbboooofffccoooooeeeee
puted as: LiD(p) = - ΣK log dist(p.g:)
max, (dist(p..))))
to perform the K-NN. We set K 100, the same as in (Aumüller & Ceccarello 2019). & distance is used, i.e. dist(p. q) = |p q 9/12- We report the mean LID over all the samples p, as F. (L))())) in Figure
8

---

Published as a conference paper at ICLR 2021
the average UD
Table 3: A comparison of LIDs (using cosine similarity) among contextual and static embed- ding spaces.
12
D-BERT
GPT
moon Up
10
GPT2
ELMe
Model
avg I.ID>
6-
Contxt Embeds
BERT
1.19
768
5.6
D-BERT
1.19
768
7.3
2
GPT
0.96 M
768
6.8
layer
100
GPT2
1.09 M
768
7.0
ELMo
0.88 M
1024
9.1
Figure 9: The average LID using Euclidean distance. ELMo's 's original embdding dimension is 1024, larger than other models' 768.
Static
GloVe
1.18
100
18.0
Embeds
GloVe-2M
2.20 M
300
26.1
GNEWS
3.00 M
300
21.1
As shown Figure 9 the mean LIDs for all the models in all the layers are below 12. The small mean LID values reveals that the manifold's intrinsic dimension is relatively low, especially considering that this is a 768-D (1024 for ELMo) embedding space. Since ELMo's 1024-D is larger than other models 768-D D dimension, its LID is also slightly higher than other models shown in the figure. The existence of a low-dimensional embedding is also suggested in (Reif et al. 2019) when they study the BERT embedding geometry.
In all the contextual embedding layers, there a clear trend of increasing LID values. In Figure 9 we can also see nearly-linear relationship between layer id and LID. With deeper and deeper layers in the net, the manifold is diffusing and slowly loses concentration. This would lead to data samples spreading, consistent with Figure 4 (recall that intra-type cosines decrease with depth). Note that as layer goes deeper, each token embedding is collecting information from context by adding their embeddings (and non-linear transforms concatenated). This could explain the spreading / expanding of the local subspace, and therefore the L.ID increases in deeper layers.
Table compares LIDs for static and contextual embeddings. The table reports results for three static embeddings: GloVe / Glo V-MM (Pennington et al. 2014), and GNEWS (Mikolov et al. 2013a). Results for static embedding LIDs are based on Aumüller & Ceccarello (019). Following Aumüller
& Ceccarello 2019),, we use cosine distance here: dist' (p. g) 1 cos g) that estimates for LTD using cosines very close to the estimates using la distances. Table 3 reports averages of LIDs over each model's 's layers. Even though Glo (Pennington et al. 2014) in Table 3 has much fewer embedding dimensions (100-D compared with BERT's 768-Do D). the LID is still higher than all of the contextual embedding models. From the table we can find that static embedding spaces generally have higher LID than the contextual ones. This means that the data points are more isotropic in the static embeddings, possibly due to their large vocabularies.
Trl Note
(p.0)
5 CONCLUSIONS AND FUTURE WORK
Previous works have reported the strong anisotropy in deep LMs, which is hard to explain the superior performance achieved by these models. We suggest that the anisotropy is global view, being largely misled by distinct clusters resided in the space. Our analysis show that it is more constructive to isolate and transform the space to measure the isotropy. From this view, within the clusters, the spaces of different models all have nearly perfect isotropy that could explain the large model capacity. In addition, we investigate the space geometry for different models. Our visualization demonstrates a now-dimensional Swiss Roll manifold for GPT and GPT2 embeddings, that has not been reported before. The tokens and word frequencies are presented to qualitatively show the manifold structure. We propose use the approximate LID quantitatively measure the local subspace, and compared with static embedding spaces. The results show smaller LID values for the contextual embedding models, which can be seen local anisotropy in the space. We hope this line of research could bring a comprehensive geometric view of contextual embedding space, and gain insights on how the embeddings are affected by attention, compression, multilingualism, etc. Therefore the model performance could be further improved based on the findings.

---

Published as a conference paper at ICLR 2021
REFERENCES
Laurent Amsaleg, Oussama Chelly, Teddy Furon, Stéphane Girard, Michael E Houle, Ken-ichi Kawarabayashi, and Michael Nett. Estimating local intrinsic dimensionality. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 29-38, 2015.
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. In Advances in Neural Information Processing Systems, pp. 6111-6122, 2019.
Martin Aumüller and Matteo Ceccarello. The role of local intrinsic dimensionality in benchmarking nearest neighbor search. In International Conference on Similarity Search and Applications, pp. 113-127. Springer, 2019.
Alexis Conneau and Guillaume Lample. Cross lingull language model pretraining. In Advances in Neural Information Processing Systems, pp. 7059-7069, 2019.
David L Davies and Donald W Bouldin. A cluster separation measure. IEEE transactions on pattern analysis and machine intelligence, (2):224- 227, 1979.
David Demeter, Gregory Kimmel, and Doug Downey, Stolen probability: A structural weakness of neural language models. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0400 04800, 2018.
Martin Ester, Hans- Peter Kriegel, Jörg Sander, Xiaowei Xu, et A density-based algorithm for discovering clusters in large spatial databases with noise. In Kdd, volume 96, pp. 226-232 231, 1996.
Kawin Ethayarajb. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing TEMNI.P-ICCN.PP pp. 55-65, 2019.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations, 2019.
John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers),
pp. 4129-41333 4138, 2019.
Michael E Houle. Dimensionality, discriminability. density and distance distributions. In 2013 IEEE 13th International Conference on Data Mining Workshops, pp. 468 473. IEEE, 2013.
Michael E Houle, Hisashi Kashima, and Michael Nett. Generalized expansion dimension. In 2012 IEEE 12th International Conference on Data Mining Workshops, pp. 587-594. IEEE, 2012.
Jiaji Huang, Xingyu Cai, and Kenneth Church. Improving bilingual lexicon induction for low frequency words. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1310-1314, 2020.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. arXiv preprint arXiv:1702.088:
2017.
Tianlin Liu, Lyle Ungar, and Joao Sedoc. Unsupervised prot-processing of word vectors via conceptor negation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6778- 6785, 2019.
Xingjun Ma, Bo Li, Yisen Wang. Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In International Conference Learning Representations, 2018.
10

---

Published as a conference paper at ICLR 2021
Mitchell Mareus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus of english: The penn treebank. 1993.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1rXiv:1609.0755500 07843, 2016.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. arXiv preprint arXiv: 1301.3711, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111-33 11-1119, 2013b.
David Mimno and Laure Thompson. The strange geometry of skip-gram with negative sampling. In Empirical Methods in Natural Language Processing, 2017.
Jiaqi Mu and Pramod Viswanath. All-but-the-top: Simple and effective post-processing for word representations. 6th International Conference on Learning Representations, ICLR 2018, 2018.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 4 conference empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoye. Deep contextualized word representations. arXiv preprint 180..0180..0536. 2018.
Steven T. Piantadosi. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21(5):1112-1130) 2014.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188)99, 2019.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. In Advances in Neural Information Processing Systems, pp. 8594-8603, 2019.
Peter J Rousseeuw. Silhouettes: graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65 65, 1987.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of 2019.
bert: smaller, faster, cheaper and lighter. arXiv preprint
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu,
Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv: 1eX.vvI904.0922.. 2019.

---

Published as a conference paper at ICLR 2021
SUPPLEMENTARY FULL RESULTS ON PTB AND WIKITEXT-2 DATASETS
A RESULTS ON WIKITEXT-2 DATASET
A.1 THE UNADIUSTED INTER AND INTRA COSINE SIMILARITY
Note that "dist" in the following legends represents DistilBERT model.
inter type cosine
intra type cosine
C.8
6.4
qp42 emo
0.0
0.4
0.4
052
Q0t
10
10
Wyyer 90
layer id
(a) Inter-type cosine similarity. As layers goes deeper, inter-type cosine goes higher. All models' last layer behaves slightly differ- ently.
(b) Intra-type cosine similarity. The intra- type cosine decreases showing the same type's embedding instances are spreading in deeper layers.
A.2 THE CENTER CENTER-SHIFTED SHIFTED AND CLUSTERED COSINE SIMILARITY
The inter-type and intra-type cosines are adjusted using the proposed center-shifting and clustering methods. Now it reflects the isotropy in almost all layers in all models.
inter jype cosine
intra type cosine
02
emo
0.0
0.4
-0.2
0.2
Cmo
20
10
layor 18
layor
(a) The clustered inter-type cosine (center shifted). It shows strong isotropy as the aver-
(b) The clustered intra-type cosine (center shifted). The intra-type cosines are much
age cosine between different types is close to 0 across all layers in all models. The GPT2's last layer still has slighly higher cosine com- pared with others.
more consistent than the unadjusted counter- part (the cosine decreases nearly monotoni- cally layers goes deeper).
A.3 THE APPROXIMATE LOCAL INTRINSIC DIMENSIONS
9
beit
Figure 12: Local Intrinsic Dimensions. The LID increases layer goes deeper, reflecting embeddings spreading out in all models' deeper layers (becoming more locally isotropic).
12

---

Published as a conference paper at ICLR 2021
B FULL VISUALIZATION PTB DATASET B.1 BERT
DET 190333
verralo 1A
(a) BERT layer
(b) BERT layer 1
(c) BERT layer 2
(d) BERT layer 3
ap
454 10
(c) BERT layer
(f) BERT layer 5
g BERT layer
(h) BERT layer 7
TAT INGO 1w0.1
VATTAEO PHO 201
WATT vatratio Pnd 208
(i) BERT layer 8
BERT layer
(k) BERT layer 10
() BERT layer 11
B.2 DISTILBERT AND ELMO
VOI rotD F-0
ror 1-0.142
VOrrato F-0
Varratt r-0.168
a b
12
DistilBERT layer
(b) DistilBERT layer
DistilBERT layer
(d) DistilBERT layer 3
(e) DistilBERT layer 4
( DistilBERT layer S
(g) ELMo layer
(h) ELMo layer
13

---

Published as a conference paper at ICLR 2021
B.3 GPT
FnD 593
(a) GPT layer
(b) GPT layer
(c) GPT layer
(d) GPT layer 3
VAT DATO
VETORTO 257
verratt
(e) GPT layer 4
(I) GPT layer 5
(g) GPT layer
(h) GPT layer 7
291
ri0.17
102
GPT layer
(i) GPT layer
(k) GPT layer
GPT layer
B.4 GPT2
10
(a) GPT2 layer 0
(b) GPT2 layer
(c) GPT2 layer 2
(d) GPT2 layer 3
Oer rato rn 881
ratio r: 857
verratto r:O 823
SET7AN and 7IL
40
Baga Pe
1300 Mal Bea
(e) GPT2 layer 4
GPT2 layer
(g) GPT2 layer
GPT2 layer'
1306 2008
2000 -ap
ii GPT2 layer 8
ii GPT2 layer
(k) GPT2 layer 10
(1) GPT2 layer
14

---

Published as a conference paper at ICLR 2021
C
FULL VISUALIZATION WIKITEXT-2 DATASET
C.I BERT
ONT nelo 1n
15
(a) BERT layer O
(b) BERT layer 1
(c) BERT layer 2
(d) BERT layer 3
12
(c) BERT layer
(f) BERT layer 5
gg) BERT layer
(h) BERT layer 7
TAT HAOO PNO
FIEE PH0.224
b
(i) BERT layer 8
BERT layer
(k) BERT layer 10
() BERT layer 11
C.2 DISTILBERT
VSI FOTD F-0.107
ror T-0.300
VOrrate 111
Varratle T-0.123
DistilBERT layer
(b) DistilBERT layer
DistilBERT layer
(d) DistilBERT layer 3
1000
Mdd
(e) DistilBERT layer 4
( DistilBERT layer S
(g) ELMo layer
(h) ELMo layer
15

---

Published as a conference paper at ICLR 2021
C.3 GPT
61C
S
(a) GPT layer
(b) GPT layer
(c) GPT layer
(d) GPT layer 3
SAS PAOO
VATT PATDO
VETORTN 301
verrate
(e) GPT layer 4
(I) GPT layer 5
(g) GPT layer
(h) GPT layer 7
101
r20.124
GPT layer
ii GPT layer
(k) GPT layer
GPT layer
C.4
GPT2
(a) GPT2 layer 0
(b) GPT2 layer
(c) GPT2 layer 2
(d) GPT2 layer 3
OeT rato rn
ratio r: 253
verratto r:O 823
verratte and 771
b
gaipihee wes
1300 MAl Biac
(e) GPT2 layer 4
GPT2 layer
(g) GPT2 layer
GPT2 layer'
506
ii GPT2 layer 8
ii GPT2 layer
(k) GPT2 layer 10
(1) GPT2 layer
16

---

Published as a conference paper at ICLR 2021
D ADDITIONAL STUDIES
D.1 K-MEANS CLUSTERING ACCURACY
We use K-Means to perform clustering, which raises two issues here. First, K-Means is very sensitive to initialization, different initialization could leads to different clustering results. However, note that in our task, clusters as a big one, is totally fine.
not seeking for optimal clustering. Sub-optimal, e.g. treating two overlapping
To illustrate this, add another metric, Davies-Boulding (DB) index (Davies & Bouldin 1979), to show that slightly different is fine. This DB index the average similarity between each cluster and its closest cluster. The value closer O, the better clustering is done. We still search in [2, 15), and choose K with the minimum DB index (MDB). MDB sometimes gives different K than that by MMS metrie. If MDB is 4, we discard MDB and treat all data one single cluster. We provide the comparison of selecting K using MMS (left) and MDB (right) here in Table We can see that for less-distinct clusters, e.g. in BERT, two metric could yield different К values, due merging or splitting. For very separated clusters, e.g. in GPT2, the two metric agrees. We plot the cosines using MDB's K values, Figure 21 It similar Figure 4 which uses slightly different K from MMS. The values are close to 0 indicating isotropy in the center-shifted clusters. This means that the procedure to reveal isotropy, is not sensitive to in K-Means.
Table 4: K by MMS(left) MDB(right)
inter-type cosine
0.6
Layer
BERT
D-BERT
GPT
GPT2
ELMo
BERT
0.4
D-BERT
0
14
9
8
2
5
1
6
6
10
15
2
2
2
2
2
cosine
GPT2
2
12
5
2
2
ELMo
3
15
11
4
14
10
2
2
2
2
-0.
5
14
13
4
2
2
2
6
14
2
2
-0.4
15
2
2
10
layer
8
7
2
2
9
11
6
2
2
2
Figure 21: The adjusted inter-type cosines, com- puted using from the criteria of minimizing DB index. The values are still close to 0.
10
4
10
2
2
11
3
2
2
Another issue that K-Means implicitly assumes convex clusters, which often does not hold. In fact, it assumes isotropic convex clusters because simply use lz distance. However, density-based clustering such as DBSCAN, is too slow thus cannot handle these datasets (million level). This is trade-off to K-Means, and empirical results above show that it is efficient and very useful to distinguish separated clusters.
D.2 CLUSTERS AND WORDS
We study the tokens and their relationship to the clusters existed in the contextual embedding spaces. We picked some representative tokens to see how they are distributed. We also study the very unique small cluster in GPT2, and how connects to the main cluster that far away. We obtain the following observations:
For BERT, high frequent words (e.g. 'the") stays at one side of the main big cluster, while low-frequent words are at the other side.
For BERT, punctuation are random but occupy distinct islands: "!' is small cluster close to the main island; ... and 'áre distinct islands far away; ??" and some others are on the main cluster.
For GPT2, almost all single letters (a to z) and mid-to-hight frequent would occupy both the left (big) and right (small) islands.
For GPT2, we didn't 't find any token that only appears in the right small island. It seems the token in the small island always has mirrors in the left big cluster.
17

---

Published as a conference paper at ICLR 2021
For word types, e.g. noun, verb, etc, didn't find clear pattern. We suspect word frequency affects more than categories.
We provide few examples. Figure show BERT layer Figure 24
23 show GPT2 layer 3.
var ratio rw0. -.0.33
var ratio rw0.083
var ratio rw3.083
var rabio 0w0.083
30
10
10
20
0
(a) Punctuation are random but less concentrated, They also occupy distinct islands.
Figure 22: BERT Layer 3 Punctuation
var ratio r- 083
var ratio r- ne3
var ratio r-0 06
var ratio r- 063
lhancing thanks
and andarson anderson 1
taching focestops
ha mabiity naccessible naccurate
thaniegrving
that thabchte
andarsson
for praadd
andover
to bee
2p
10
10
20
var ratio r=0 083
ver ratio r= r=0,66
var ratlo r= 063
ver ratlo r= 083
young
Tovoc
stunger
yrungest
afterd
headsche howded
hamzian
your
ahroon
hswt
10
10
10
(a) Frequent words and infrequent words the main cluster, but at two sides. An evidence that words distributed based the frequency.
Figure 23: BERT Layer 3 Words
var ratio rm0.838
var ratio rm0.888
var ratio r-0.88R
var ratio 1m
1997
2000
Syol tybo
(a) Punctuation random. Some occupy both islands, do not.
Figure 24: GPT2 Layer 3 Punctuation
Based on these observations, we have concluded that frequency plays an important role in the token distributions. High frequent words and low frequent words are often taking opposite sides of the space. This is also revealed in Section 4.3. We are yet not clear what causes this, but we suspect it is related to the training process. During training, high frequent words are updated more times. Also, since they used in many many different context, they play a role some shared embedding across context. Similar to the XLM model. the shared embedding are often more isotropic and more concentrated. However, this is early hypothesis and due to future research.
18

---

Published as a conference paper at ICLR 2021
var ratio O-0,
var ratio
button
rosibal
Gbank
Clacoouse coacome
Day.
oearser
obankets
Coecomess
tlburking
Slaccomnig g
Charienuse g
-10
-10
-2p
an ao Boooghon
van
nia
Thants that
Gfist
ma
Cwetfare
ofhand Ohsical 6ftsh
Sthanics
cwell
Owellt
0
beaker
-10
50 go
g 200g
(a) Mid-to to high frequent words often occupy both distinct islands (notice that the right small cluster is also colored), where a roll-s shaped alignment can be observed on the larger island.
Figure 25: GPT2 Layer Words
D.3 EMBEDDING OF TRANSLATION LANGUAGE MODEL XLM
We also perform analysis and visualization on the XLM model (Conneau & Lample 2019). BERT mask language model (MLM), GPT is causal language model (CLM), and XLM translation language model (TLM). We provide visualization of XLM's 6 layers embeddings here. This is on WikiText-2 dataset.
VOl roto F-0.13
rooo T-0
Vorrate 6-0.708
Varratle r-0.999
(a) XLM laMeryyeO 0
(b) XLM layer
XLM layer 2
(d) XLM layer
a 8868
(e) XLM layer 4
( XLM layer 5
We try to establish systematic view of embedding geometry for different types of deep LMs. We have hypothesis and very preliminary results here. BERT (an ML.) show spreading clusters, not very distinct. GPT (an CLM) shows highly separated clusters. XLM (an TLM) does not demonstrate clustering effect, and the embedding are centered.
One possible explanation for XLM's behavior, is that this is multi-lingual model, and the embedding space have to shared between languages. This is forced during the training process of this translation language models. In that case, a single cluster residing in the center, would be good shared embedding across languages. However, this is just hypothesis and requires further study on more models.
19

---

Published as a conference paper at ICLR 2021
D.4 LID ESTIMATION ROBUSTNESS
We follow (Aumüller & Ceccarello 2019) to choose K = 100 for K-Nearest-Neighbor (K-NN) search for LID approximation, and make a direct comparison with them. raises the concern that 100 samples might not be enough to effectively estimate the local dimension. We conduct additional experiments here to select K 200, 500, 1000, and demonstrate that the LID estimation is robust. They provide similar LID estimates across all layers, in all the models. Though using more samples indeed obtain very slightly higher values of LID (in Figure 27 we can see a little bit up-shifting from left-most plot the right-most plot). This is expected, less number of samples often tends to under-estimate and over-smoothing of LID. Nevertheless, the LID is still much smaller than the original dimension 768, so using 100 samples is a good trade-off to efficiently approximate LID.
Byeer
Byee
byeer
GFT
kporid
koardd
(a) 100-NN
(b) 200-NN
(c) 500-NN
(d) 1000-NN
Figure 27: LID estimate using different number of samples for nearest neighbor search.
As layer goes deeper, the LID increases. In other words, the local space dimension expands, at cost of losing density. For example, the spiral band (1-D) in GPT's front layer, becomes a Swiss Roll (2-D), and the roll surface get thickness (3-D), layer increases. But we not clear about the reasons yet, only suspect that data is spreading as more context info is added in later layers (the embedding for a token in deeper layer is based on summation of all embeddings in the context, due to attention). This is due to future study.
D.5 ABLATION ANALYSIS ON CLUSTERING
To better study the clustering effect, we conduct experiment that computes the inter-type cosines, on clusterd-only embeddings and clustered plus center-shifted embeddings. The following figure shows GPT2's cosine on original embeddings without adjustment (blue), the clustering-only embeddings (orange), and full (clustering + centering) adjustment (green).
inter type cosine
inter type cosine
0.0
0.8
0.6
0.6
0.4
0.4
oni
onig
contored
centored
6
10
6
10
layor id
Layor id
(a) GPT2 on PTB
(b) GPT2 on WikiText-2
In the original embedding without adjustment, we see inconsistent behavior in the last layers. However, we perform clustering and measure the Sinter within the clusters (orange), we can see much more consistent behavior across layers (more flat curve). This indicates that the clustering effect exists in all the layers, which is also verified by layer-wise visualization in Appendix B.4 and C.4
Meanwhile, the large values of cosines in the orange curve expected. Now cosines are only computed within clusters, where those clusters are not at the origin. The higher values here, the more concentrated clusters are. These indicate that after clustering, the subspace within each cluster now consistent, across all the layers. Finally, we shift those clusters to the origin, and get the green curve (values near 0), indicating isotropic cluster shapes.
20

---

Published as a conference paper at ICLR 2021
D.6 POSITIONAL ENCODING IN THE GEOMETRY
is very interesting to investigate whether the positional encoding affects the geometry in the contextual embedding spaces. In particular, since GPT/GPT2 have a unique Swiss-Roll shaped manifold, that is not observed in other models, we look at how the manifold is related to the positional encoding GPT2. Note that we truncate the whole PTB text into 512-length segments, and feed those segments into the models. The positional encoding is applied to each 512-length segment. We pick few punctuation and words, and draw them in the space labeled by their relative positions in their corresponding segments. The position ID ranges from 0 to 511.
var ratio r-0 r-0.88F 8BF
100
var ratio r-C.8BR
ratio r-C.888
500
var ratio r-C.988
500
400
400
70
20
200
-20
200
200
200
100
1000
-20
2000
-20
1000
-20
1000
-20
Figure 29: GPT2 Layer Punctuation. The posítion ID is monotonically increasing along the manifold.
var
500
var
othe
OATSE
Gman
400
70
300
300
200
200
-0
-2
200
-2
100
100
1000
1000
1000
1000
2000
2000
2003
2003
Figure 30: GPT2 Layer 3 Words. The position ID) is monotonically increasing along the manifold.
& $", and four words "the first super man", draw them in Figure 29 30 The color bar on the side indicates the relative position ID in their segment. Darker color smaller IDs and lighter color is bigger IDs. Clearly, for both punctuation and words, the center of Swiss-Roll corresponds to lower position IDs, where the other end of the manifold are high IDs. Also, the distribution is monotonic. From the center the far end, the position ID increases. This suggests that the positional encoding is indeed highly correlated with the Swiss-Ren Roll manifold for GPT models. The reason causing this is deferred to future study.
We select 4 punctuation,
Note that this finding consistent with that reported in (Reif et al. 2019). where they found that positions of the token matters (tokens take all neighbors' information indiscriminately. rather than only attending up to their semantic boundaries) in the BERT embedding geometry. We also study the context/semantic influence of the embeddings in the next subsection.
D.7 CONTEXT IN THE GEOMETRY
We also look at how the context information influences the geometry. It is more sophisticated to analyze the context, so we pick a few examples to look at their context and corresponding positions in the embedding spaces. In particular, we choose the common polysemous words "like" and "interest", as two examples. The word "like" often has two different use cases: 1. favor; 2. similar to. There are also some fixed phrases such "would like". The word "interest" has two senses well: 1. like to do something; 2. the money sense.
We identified the target word token ("like" or "interest"), and then print out 5 tokens before and after the target, as the context for illustration in Figure 31 From the figure we are not able to identify a clear pattern that word sense is correlated with the geometric space. However, this is only inspected by manually checking a few samples. A full statistical analysis should be carried out the future work.
21

---

Published as a conference paper at ICLR 2021
ntteresls! Juag 083
var
Mouidher monting
espected
genensly
congreas
initally interest. and trigur apposiion from cally poweriul internnt groups, cuart
manyy whomi varry for big ticker items like coots and su
10
10
10.0
10.0
5.0
-50
-25
0.0
2.5
50
-5.0
(a) BERT layer
(b) BERT layer
var ratto r=0 rm0.888 888
var ratto r-0 888
10
-10
hers, have interal presenving hards
20
We like the Congress
20
CDs interst
What d liu
whole sulject
boin
prablen harlonal lkk
caye
GPT2 layer 3
(d) GPT2 layer
Figure 31: The context and positions in the embedding space.
22


Reviews:
[{"review_id": "vhb-6sq1L2S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Findings:\n- Reproduces various existing findings about anisotropy of contextual representations viewed globally.\n- Contextual representations are highly isotropic within clusters of the representations. \n- GPT representations follow a Swiss Roll manifold, where the most frequent words appear at the head and less frequent words are gradually appended at the bottom.\n- The Swiss Roll manifold is taller in deeper layers.\n- BERT representations fall in a Euclidean space.\n-  The Local Intrinsic Dimension of contextual embeddings is lower than for unigram embeddings.\n\nPros:\n- The manifold analysis of word frequency is intriguing and intuitive.\n- The explanation of experiments was clear in each section.\n- They produce compelling evidence that the global token-level anisotropy of these representations is largely due to membership of large clusters. This is a valuable contribution because it explains previous findings in Ethayarajh 2019 and reconciles them with theoretical expectations.\n\nCons:\n- In Section 2.3 and Section 3.1, the paper gives insufficient credit to Ethayarajh 2019. As far as I could tell, every initial result is a reproduction of a result from Ethayarajh 2019, and the methods are very similar.\n- Though they acknowledge it, the methodology is largely taken from existing unigram embedding analysis.\n- Once they start to identify very well defined clusters, I was very curious about the distinctions between the islands. It would not be difficult to inspect some of the data by hand, so I don't understand why the authors didn't try.\n- The authors offer no analysis for the difference in behavior between different models. I felt like I was reading a taxonomy, and the plots were left for the reader to connect. The authors have presumably spent quite a while thinking about these geometric structures and models, so surely they have conjectures about the behavior they observed or hypotheses they can test.\n\nMinor / style:\n- I didn't realize until the conclusion that your main finding was an explanation of existing claims about anisotropy by considering behavior within the clusters, so that needs more emphasis.\n- The papers you cite do a decent job of explaining why isotropy in the representation space is significant both token- and type-wise. The paper would be a lot more readable if you made a similar effort in explaining background.\n- There is inconsistent use of \"isotropy\" vs \"isotropicity\".\n- Citations needed for claim \"widely believed that the contextual space is so weird\"\n- Needs proofreading for minor grammar and typos (e.g., downstreaming instead of downstream, could resides instead of could reside)\n- Instead of referring variously to high similarity or cosine, silhouette scores, and other measurements of isotropy, it might be clearer to link each concept to isotropy once, and then each subsequent result simply refer to it as isotropy while mentioning the metric. Then the reader doesn't have to constantly remember which metric indicates high isotropy as they read the results.\n\nQuestions:\n- The authors claim to select the clusters that maximize MMS. I read the wording to imply that this optimum is tractable/stable. Is that the case?\n\n(Number rating was updated from 3 to 7 in light of substantial expanded experiments and analysis.)", "extra_scores": {}, "extra_text": {}}, {"review_id": "gEeeZ3w8lbm", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzes the geometry of several contextualized embeddings. The authors show that global anisotropy is caused by strong clustering of word vectors, and that vectors of different word types are isotropically distributed within the cluster.\n\n**Strengths**\n- This work is a nice-to-have extension of [Ethayarajh (2019)](https://www.aclweb.org/anthology/D19-1006.pdf) that dives deeper into the geometric properties of contextualized vectors.\n- The research question is clearly stated (Why doesn't anisotropy hurt performance?) and clearly answered (There's no anisotropy locally).\n- The 3D visualizations provide a better geometric intuition than the flat visualizations that are common in this kind of papers.\n\n**Issues**\n- I don't think that good performance _contradicts_ anisotropy. For example, we already know that the classical static embeddings are also anisotropic [(Mimno and Thompson, 2017)](https://www.aclweb.org/anthology/D17-1308.pdf), and this means that good performance (as measured by downstream tasks) may co-exist with anisotropy. So please consider rewording the beginning of Section 1.2. For example, instead of \"There is an apparent contradiction.\" consider \"It is not clear why ...\"\n- How _representative_ is one random sample from $\\Phi(t_i)$ for measuring $S_\\text{inter}$ in formula (1). You gave an example in the Introduction when the same word type (bank) can have totally different meanings depending on context, and thus (I believe) the corresponding $\\phi_1(\\text{bank})$ and $\\phi_2(\\text{bank})$ may be totally different. Why not taking more samples for polysemous words?\n- Why do you use different distance metrics (Euclidean vs cosine) for estimating LIDs of contextualized vs static embeddings (Table 3)? \n- \"For GPT2, we had hoped to find that some types are associated with one cluster and other types are associated with the other cluster, but that is not verified in our experiments\" -- I think you should look at contexts rather than types (since you're dealing with the contextualized embeddings). It would be interesting to see whether you have the same type in both clusters, and then to look at its contexts. I bet that the contexts will differ.\n\n**Minor issues**\n- \"We find a low-dimensional manifold in GPT/GPT2 embeddings, but not in BERT/DistilBERT embeddings.\" -- but your LIDs are low for BERT/D-BERT layers as well! Why can't you claim the low-dimensionality for BERT/D-BERT embeddigs?\n- I doubt that PTB with 10K vocabulary size gives \"good\" coverage in 2020. You may simply state that this a widely-used dataset.\n- Wiki2 (Merity et al., 2016) is usually referred to as _WikiText-2_.\n- Please consider rephrasing \"experiments\" -> \"analysis\", as you are not conducting _controlled experiments_, but rather performing exploratory analysis of the embeddings.", "extra_scores": {}, "extra_text": {}}, {"review_id": "c2eh9aVicTC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary:\n\nThe authors investigate the token embedding space of a variety of contextual embedding models for natural language. Using techniques based on nearest neighbors, clustering, and PCA, they report a variety of results on local dimensionality / anisotropy / clustering / manifold structure in these embedding models which are of general interest to scientists and practitioners hoping to understand these models. These include findings of (local) isotropy in the embeddings when appropriately clustered and shifted, and an apparent manifold structure in the GPT models.\n\n#### Reasons for score:\n\nThis is a generally thorough and well-executed paper, and a careful examination of these embedding models is of great general interest to the ICLR community.\nHowever, I feel the analysis is a bit shallow at certain points and consists of reporting (interesting) findings without necessarily delving deeper or adequately explaining them.  I think this paper presents a great jumping off point for further research on the subject, as it certainly raised quite a few questions with me. I support its acceptance but would hope to see the authors address some of the questions raised here.\n\n#### Positives:\n\n- Lots of analysis with several different techniques\n\n- Very interesting and relevant subject area\n\n- Thorough use of recent related work\n\n- The paper is quite well written and organized considering the inherent challenge of writing up research of this sprawling nature.\n\n#### Concerns / Comments:\n\n- Noting the very different behavior of GPT from the other representations — could this be due to the learned positional encodings? This might also explain the Swiss-roll style paths seen when examining the different embeddings of the same word types. Could position along those curved paths be correlated to sentence position of the tokens?\n\n- In this spirit, it would be great to get a better characterization of what the different clusters correspond to, especially in the case of GPT. Could there be a better investigation of the relationship between predictive / causal and non-causal models and these clusterings? \n\n- More generally, the “Swiss roll” observation is intriguing, but since it only appears in one family of models that has a very similar (transformer) architecture to models in which it does not appear, what are we to make of it?\n\n- How can the intrinsic dimensionality at each layer increase with depth? Considering each layer as living on a manifold, the transformation at each layer should act as a coordinate chart for the next layer’s manifold, which should only allow a reduction in dimension. Unless I am missing something, that suggests that these estimators do not have enough samples and/or are measuring something different than the dimension of a manifold. Section 4.4 should explain how this could be happening. \n\n- More generally, I find the low LIDs in GPT hard to understand or interpret without more analysis, and would also like to understand the very large first dimension of the GPT models.\n\n- It would be great to see more suggestions / takeaways for practitioners. What, if anything, does local (an)isotropy imply for deep learning researchers doing work in this area?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a broad exploratory analysis of the geometry of token representations in large language models, with a focus on isotropy and manifold structure, and reveals some surprising findings that help explain past observations.

Pros:
- Clear and surprising analytical findings concerning a broad and widely-used family of models.

Cons:
- The paper is a fairly broad exploratory analysis, with no single precise claim that ties together every piece of the work.

I thank both the authors and reviewers for an unusually productive discussion.
Decision: Accept (Poster)