Is Anisotropy Truly Harmful? A Case Study on Text Clustering
Year: 2023

Abstract:
In the last few years, several studies have been devoted to dissecting dense text representations in order to understand their effectiveness and further improve their quality. Particularly, the anisotropy of such representations has been observed, which means that the directions of the word vectors are not evenly distributed across the space but rather concentrated in a narrow cone. This has led to several attempts to counteract this phenomenon both on static and contextualized text representations. However, despite this effort, there is no established relationship between anisotropy and performance. In this paper, we aim to bridge this gap by investigating the impact of different transformations on both the isotropy and the performance in order to assess the true impact of anisotropy. To this end, we rely on the clustering task as a means of evaluating the ability of text representations to produce meaningful groups. Thereby, we empirically show a limited impact of anisotropy on the expressiveness of sentence representations both in terms of directions and L2 closeness.

heading: 1 Introduction
text: Contextualized pre-trained representations are now widely used as input to various tasks such as information retrieval (Lin et al., 2021), anomaly detection (Ait-Saada and Nadif, 2023) and document clustering (Boutalbi et al., 2022). In parallel, several studies have investigated the intrinsic properties of Transformers (Peters et al., 2018; Ait Saada et al., 2021; Ethayarajh, 2019; Kovaleva et al., 2019) in order to demystify these black-box models and the reasons behind their impressive performance levels. Particularly, it has been observed that language models in general (Gao et al., 2019) and Transformer word embedding models in particular (Ethayarajh, 2019; Wang et al., 2020) produce an anisotropic embedding space. This concretely means that the directions of trained dense word
representations do not occupy uniformly the embedding space, which is suspected to limit their expressiveness and thus their expected performance on downstream tasks. The main question addressed in this paper is how harmful this anisotropy really is regarding the quality of text representations.
Several approaches have been proposed to increase the isotropy of dense representations, based on different strategies. In the context of static word embeddings like GloVe and word2vec, both Raunak et al. (2019) and Mu and Viswanath (2018) propose a post-processing method that consists in removing the first principal components before reconstructing the word vectors as opposed to the traditional approach of removing the weakest components. This approach improves the quality of word vectors on several downstream tasks while reducing their anisotropy (Mu and Viswanath, 2018).
As to contextualized representations provided by Transformer models, several approaches have been proposed in order to alleviate the anisotropy problem. For instance, based on the idea that anisotropic representations tend to have high expected pairwise cosine similarity, Wang et al. (2020) propose to apply a cosine similarity regularization term to the embedding matrix. In the same vein, Gao et al. (2019) propose a method named “spectrum control” that allows for increasing the isotropy of Transformer representations and improving the performance of the machine translation task. To this purpose, they propose regularization
1194
terms that hamper the singular value decay of the embedding matrix. However, despite the success of these optimization tricks in lowering the anisotropy of Transformer representations, Ding et al. (2022) have recently shown that they do not bring any improvement, relying on several tasks like summarization and sentence similarity (STS). They even observed a certain deterioration of the performance brought by anisotropy mitigation techniques.
In contrast, Rajaee and Pilehvar (2022, 2021) show that post-processing methods made for increasing isotropy are also responsible for a performance increase in the STS task in both monolingual and cross-lingual settings. Similarly, the whitening operation, which consists in using the principal components normalized by their inertia, has shown an increase in isotropy as well as enhanced performance in STS (Su et al., 2021; Huang et al., 2021) and document clustering (Ait-Saada et al., 2021). However, there is no evidence that the decrease of anisotropy brought by such transformations is directly responsible for the gain of performance, as shown in Figure 1, which gives an initial idea of the question addressed in this paper.
Indeed, despite the great energy devoted to studying and mitigating the anisotropy of dense text representations, there is no clear connection between isotropy and performance, which seems to depend, inter alia, on the sought task. In order to contribute to settling this question, we consider using a task that has never been used for this purpose: document clustering. The rationale behind this choice is to evaluate, under different degrees of isotropy, the capability of text representations to facilitate the clear separation and identification of meaningful groups of documents through clustering algorithms.
The main contributions of this paper are:
• We extend the isotropy study of word embeddings to document representations.
• We investigate the correlation between different isotropy measures.
• We assess the connection between isotropy and quality of representation.

heading: 2 Background

heading: 2.1 Isotropy measures
text: Let X = {xi} be a set of n vector representations, characterizing n words or documents by d features. In Mu and Viswanath (2018), the isotropy is as-
sessed using the partition function ψ as follows:
min∥c∥=1 ψ(c) max∥c∥=1 ψ(c) ; where ψ(c) = n∑
i=1
e⟨xi,c⟩
This approach is inspired by the theoretical findings issued by Arora et al. (2016) who prove that, for isotropic representations X , the partition function ψ can be approximated by a constant for any unit vector c, thus leading to a min/max ratio score of 1. As there is no analytic solution c that maximizes or minimizes ψ(c), Mu and Viswanath propose to use the eigenvectors of the covariance matrix as the set of unit vectors, which leads to:
Ipf (X ) = minwj ψ(wj) maxwj ψ(wj)
where pf stands for partition function, wj is the jth eigenvector of X⊤X (X being the representation matrix). In our experiments, X contains representations of either words or sentences/documents. In addition to this measure, Wang et al. (2020) quantify the anisotropy by the standard deviation of the partition function normalized by the mean:
A(X ) =
√∑d j=1(ψ(wj)− ψ̄))2
d ψ̄2
where ψ̄ is the average value of the partition function. Perfectly isotropic representations lead to A(X ) = 0 and greater values denote a higher anisotropy. For our purpose, we derive the isotropy score as the square root of the precision score τ = 1/σ, which leads to:
Ipf2(X ) = 1√ σ = 1
A(X ) σ being the variance normalized by dψ̄2.
On the other hand, the study of anisotropy provided in (Ethayarajh, 2019) has been applied to word representations and the empirical results have been obtained using a high number of words picked randomly. The authors rely on the assumption that the expected similarity of two words uniformly randomly sampled from an isotropic embedding space is zero and that high similarities characterize an anisotropic embedding space. They hence use the expected pairwise cosine similarity in order to assess the anisotropy level of word representations. The isotropy is thus obtained by:
Icos := Ei ̸=i′(1− cos (xi,xi′)) where the score is computed over m random pairs (xi,xi′) of vector representations.

heading: 2.2 Quality measures
text: In order to assess the quality of text representations X of size n, we rely on the document clustering task, with the aim of estimating the ability of a clustering algorithm to accurately distinguish groups of documents in a corpus represented by X . As the accuracy measure is not reliable when the classes are dramatically unbalanced, this is achieved using two well-known measures: Normalized Mutual Information (NMI, Strehl and Ghosh 2002), and the Adjusted Rand Index (ARI, Hubert and Arabie 1985; Steinley 2004).
Thereby, to compare two partitions A and B into g clusters, the NMI metric takes the following form: NMI(A,B) = MI(A,B)√
H(A)H(B) where MI(A,B)
denotes the mutual information while H(.) denotes the entropy; NMI(A,B) is hence given by:
∑ k,ℓ nkℓ n log
nnkℓ nkn̂ℓ√
( ∑
k nk log nk n )( ∑ ℓ n̂ℓ log n̂ℓ n )
where nk represents the number of samples contained in the class Ak(1 ≤ k ≤ g), n̂ℓ the number of samples belonging to the class Bℓ(1 ≤ ℓ ≤ g), and nkℓ the number of samples that are at the intersection between the class Ak and the class Bℓ.
The ARI metric, is a measure of the similarity between two groups of data. From a mathematical point of view, the ARI(A,B) is related to the precision and is given by:
∑ k,ℓ ( nkℓ 2 )−[ ∑ k ( nk 2 ) ∑ ℓ ( n̂ℓ 2 )] / (n2)
1 2 [ ∑ k ( nk 2 )+ ∑ ℓ ( n̂ℓ 2 )]−[ ∑ k ( nk 2 ) ∑ ℓ ( n̂ℓ 2 )] / (n2)
where the binomial coefficient ( u v ) can be interpreted as the number of ways to choose u elements
from a v-elements set. Intuitively, NMI quantifies how much the estimated clustering is informative about the true clustering, while the ARI measures the degree of agreement between the estimated clustering and the reference partition. Both NMI and ARI are equal to 1 if the resulting clustering partition is identical to the ground truth.

heading: 3 Experiments
text: In this study, we aim to determine to what extent the anisotropy actually affects the quality of the representations and their ability to discriminate data samples through separable clusters. To this end, we use three measures to evaluate the isotropy of the original embedding space before and after post-processing. Then, we compare the changes in isotropy with the corresponding clustering performance in order to establish a potential relationship between the two concepts.
Relying on several isotropy measures allows us to consolidate confidence in our conclusions and, at the same time, verify if the measures agree with each other. In the same spirit, using different clustering methods and performance measures ensures more rigorous assertions.
We make the code and data used publicly available1.

heading: 3.1 Datasets
text: The datasets used for clustering experiments are described in Table 1, where the balance is the ratio between the smallest and largest cluster sizes. We
1https://github.com/miraaitsaada/anisotropy_clustering
used classic3 and classic4 datasets of Cornell University, the BBC news dataset proposed in (Greene and Cunningham, 2006) and random extracts of DBPedia (Lehmann et al., 2015) and AG-news (Zhang et al., 2015) of size 12,000 and 8,000 respectively.
In addition to the datasets used for clustering, we also make use of an external dataset in order to compute an independent score of isotropy. We make use of the dataset used by Rajaee and Pilehvar (2022) which contains sentences extracted from Wikipedia. We use this dataset to evaluate the isotropy measures like Icos, computed between m = 5000 pairs of words and sentences. The 10 000 resulting representations are also used to compute Ipf and Ipf2 .

heading: 3.2 Post-processing
text: In this study, we focus on post-processing operations based on dimension reduction, showing their effectiveness on text clustering and assessing their impact on isotropy. The objective here is to compute a reduced version of X(n×d) called Y(n×d′) that comprises the most useful information present in X while using only d′ dimensions.
As an alternative to removing the dominant principal components (PCs) (Raunak et al., 2019; Mu and Viswanath, 2018), the whitening operation allows to normalize the PCs to unit variance, thus reducing the impact of the first components and producing vectors of better quality. It consists in building a reduced representation Y whereby each value is computed as:
yij = xiwj/ √ δj , ∀i = 1, . . . , n; j = 1, . . . , d′
where wj is the jth eigen vector of X⊤X and δj its jth eigen value. We also compare the classical and whitened version of PCA with a nonlinear dimension technique called UMAP (McInnes et al., 2018), a faster and more robust manifold technique than t-SNE (van der Maaten and Hinton, 2008) that can be used as a post-processing tool with any d′ (while d′ ≤ 3 for t-SNE). UMAP, like t-SNE, is a graph-based method that aims at producing a
reduced space that best preserves the (local) connections of a KNN graph. In order to respect the unsupervised context of text clustering, we avoid all kinds of hyperparameter tuning. We thus set d′ to 10 for all of the post-processing methods. Besides, two strategies are used to leverage Transformer models. The first consists simply in taking the last layer as usually performed in the literature (Reimers and Gurevych, 2019). The second strategy used all of the layers by averaging them together (Ait-Saada et al., 2021).

heading: 3.3 Euclidean vs. cosine
text: As a recall, anisotropic vector directions occupy a narrow cone in the geometrical space. Given this definition, we can expect directional techniques based on the angles between vectors to be particularly sensitive to the alleged lack of expressiveness induced by anisotropy. With this in mind, we use, in addition to k-means (MacQueen et al., 1967), Spherical k-means (Dhillon and Modha, 2001) which is made for directional data and based on the cosine distance instead of the L2 metric. For both algorithms, we use 10 different initializations and keep the partition that yields the best withincluster inertia. For more details about the datasets used, please refer to Appendix 3.1.

heading: 3.4 Correlation estimation
text: In order to assess the linear correlation between two continuous variables, we use the Pearson correlation coefficient ρ (Pearson, 1896) and test its significance. The ρ coefficient between two random variables X and Y indicates how much does one of the variables increase with the growth of the other. It is computed as:
ρX,Y = cov(X,Y )√
σXσY
where X et Y are two random variables of variance σX et σY respectively and cov(X,Y ) is the covariance between X and Y .
In order to test the significance of ρ we rely on the p-value which is a probability that denotes how likely it is that the observed variables have occurred under the null hypothesis which is that the two variables are perfectly correlated (ρX,Y = 0). Thus, high ρX,Y values indicate a stronger linear relationship and the closer the p-value gets to zero, the more we consider significant the correlation between X and Y .

heading: 4 Discussion
text: Figure 2 confronts one quality measure (NMI) and one isotropy measure (Icos) using different postprocessing techniques. We first observe that PCAw produces, by far, the most isotropic representations while increasing the performance of the raw vectors. Indeed, an appealing explanation of the success of the whitening operation is that it considerably alleviates the anisotropy of the embedding space (Su et al., 2021). Applying that reasoning, PCA and UMAP should deteriorate the performance since they both exacerbate the anisotropy (in all cases for PCA and in most cases for UMAP). Nonetheless, the performance of PCA is comparable to that of the raw embeddings and UMAP achieves even better performance than PCAw even though it significantly reduces the isotropy. Overall, averaging the whole set of layer representations achieves better results, even though it clearly decreases the isotropy, compared to using the last layer, as traditionally performed. Also, it is worth noting that even when the directions of the vectors are used (skm), the decrease of isotropy has a negligible impact on the performance. All these observations suggest that, although the anisotropy reduces the spectrum of directions taken by sentence vectors, it does not necessarily alter their expressiveness.
In order to confirm this supposition, we directly compare isotropy and quality measures in a wide range of situations. To this end, we compute the correlation (Table 2) between several isotropy measures and performance scores on 2 models (BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019)) with 2 different strategies (“all layers” and “last”), using 5 datasets and 4 transformations, lead-
2Corresponding p-values are given in Table 3 in the Appendix
ing to a total of 80 occurrences of each measure. We first observe a high correlation (associated with a near-zero p-value in Table 3) between measures within the same family (e.g. Icos and Ipf ). This indicates that the selected measures agree with each other which denotes a certain coherence. However, when looking at the correlation between the two families of measures, it is clear that there is no significant relationship between isotropy and quality measures, since all the values of the correlation coefficient are close to zero, which is corroborated by relatively high p-values, denoting a non-significant correlation. Note that the same observations (not shown in this paper) can be made using the Spearman correlations of ranks (Spearman, 1987).

heading: 5 Conclusion
text: It has been known to happen that transformations that tend to decrease the anisotropy of text representations also improve the performance of downstream tasks. In stark contrast, we observe in the present study that transformations that exacerbate the anisotropy phenomenon may also improve the results, which calls into question the importance of isotropy in text representation. To draw this important conclusion, we relied on the clustering task and several empirical measures to assess the relationship between isotropy and quality of representations, using several datasets. Most importantly, we show that even a directional approach for clustering, which should be primarily affected by anisotropy, does not undergo any performance loss resulting from low-isotropy representations. In addition, we show the advantage of using UMAP as a post-processing step, which provides good-quality representations using only a handful of dimensions, despite a high resulting anisotropy.

heading: 6 Limitations
text: In this study, we focused on the clustering task in order to assess the real impact of anisotropy on the quality of representations. The conclusion is clear regarding Euclidean and directional clustering but investigating other tasks like information retrieval and anomaly detection would further strengthen the present findings. Also, the set of post-processing methods is not limited to the ones used in this study, and it would be interesting to conduct a more comprehensive study, including more transformation functions. Finally, an important future direction is to assess the impact of anisotropy on other languages, especially on embedding models trained on a restrained corpus, which can be the case of low-resource languages.

heading: A Appendix
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
7 A2. Did you discuss any potential risks of your work? We did not identify any risk regarding our work.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 3
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. Left blank.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Finally, an important future direction is to assess the impact of anisotropy on other languages, especially on embedding models trained on a restrained corpus, which can be the case of low-resource languages