Isotropy, Clusters, and Classifiers (2024)

Abstract: Whether embedding spaces use all their dimensions equally, i.e., whether they are isotropic, has been a recent subject of discussion. Evidence has been accrued both for and against enforcing isotropy in embedding spaces. In the present paper, we stress that isotropy imposes requirements on the embedding space that are not compatible with the presence of clusters—which also negatively impacts linear classification objectives. We demonstrate this fact both empirically and mathematically and use it to shed light on previous results from the literature.

heading: 1 Introduction
text: Recently, there has been much discussion centered around whether vector representations used in NLP do and should use all dimensions equally. This characteristic is known as isotropy: In an isotropic embedding model, every direction is equally probable, ensuring uniform data representation without directional bias. At face value, such a characteristic would appear desirable: Naively, one could argue that an anisotropic embedding space would be overparametrized, since it can afford to use some dimensions inefficiently.
The debate surrounding isotropy was initially sparked by Mu and Viswanath (2018), who highlighted that isotropic static representations fared better on common lexical semantics benchmarks, and Ethayarajh (2019), who stressed that contextual embeddings are anisotropic. Since then, evidence has been accrued both for and against enforcing isotropy on embeddings.
In the present paper, we demonstrate that this conflicting evidence can be accounted for once we consider how isotropy relates to embedding space geometry. Strict isotropy, as assessed by IsoScore (Rudman et al., 2022), requires the absence of clusters, and thereby also conflicts with linear classification objectives. This echoes previous empirical
studies connecting isotropy and cluster structures (Ait-Saada and Nadif, 2023, a.o.). In the present paper, we formalize this connection mathematically in Section 2. We then empirically verify our mathematical approach in Section 3, discuss how this relation sheds light on earlier works focusing on anisotropy in Section 4, and conclude with directions for future work in Section 5.

heading: 2 Some conflicting optimization objectives
text: We can show that isotropy—as assessed by IsoScore (Rudman et al., 2022)—impose requirements that conflict with cluster structures—as assessed by silhouette scores (Rousseeuw, 1987)—as well as linear classifier objectives.
Notations. In what follows, let D be a multiset of points in a vector space, Ω a set of labels, and ℓ : D → Ω a labeling function that associates a given data-point in D to the relevant label. Without loss of generality, let us further assume that D is PCA-transformed. Let us also define the following constructs for clarity of exposition:
Dω = {d : ℓ (d) = ω}
sign(ω, ω′) = { −1 if ω = ω′ +1 otherwise
Simply put, Dω is the subset of points in D with label ω, whereas the sign function helps delineate terms that need to be maximized (inter-cluster) vs. terms that need to be minimized (intra-cluster).

heading: 2.1 Silhouette objective for clustering
text: We can consider whether the groups as defined by ℓ are in fact well delineated by the Euclidean distance, i.e., whether they form natural clusters. This is something that can be assessed through silhouette scores, which involve a separation and a cohesion score for each data-point. The cohesion score consists in computing the average distance
75
between the data-point and other members of its group, whereas separation consists in computing the minimum cohesion score the data-point could have received with any other label than the one it was assigned to. More formally, let:
cost(d,S) = 1|S| ∑
d′∈S
√∑
i
(di − d′i)2
then we can define the silhouette for one sample as
coh(d) = cost ( d,Dℓ(d) \ {d} )
sep(d) = min ω′∈Ω\{ℓ(d)}
cost (d,Dω′)
silhouette(d) = sep(d)− coh(d)
max{sep(d), coh(d)} Or in other words, the silhouette score is maximized when separation cost (sep) is maximized and cohesion cost (coh) is minimized. Hence, to maximize the silhouette score across the whole dataset D, one needs to (i) maximize all inter-cluster distances, and (ii) minimize all intra-cluster distances.
We can therefore define a maximization objective for the entire set D: ∑
d∈D
∑
d′∈D sign(ℓ(d), ℓ(d′))
√∑
i
(di − d′i)2
which, due to the monotonicity of the square root in R+, will have the same optimal argument D∗ as the simpler objective OS OS = ∑
d∈D
∑
d′∈D sign(ℓ(d), ℓ(d′))
∑
i
( di − d′i )2
(1)

heading: 2.2 Incompatibility with IsoScore
text: How does the objective in (1) conflict with isotropy requirements? Assessments of isotropy such as IsoScore generally rely on the variance vector. As we assume D to be PCA transformed, the covariance matrix is diagonalized, and we can obtain variance for each individual component through pairwise squared distances (Zhang et al., 2012):
V(D)i = 1 2|D|2 ∑
d∈D
∑
d′∈D
( di − d′i )2
In IsoScore, this variance vector is then normalized to the length of the 1⃗ vector of all ones, before computing the distance between the two:
√√√√∑
i
( ∥⃗1∥2
∥V(D)∥2 V(D)i − 1
)2
This distance is taken as an indicator of isotropy defect, i.e., isotropic spaces will minimize it.
Given the normalization applied to the variance vector, the defect is computed as the distance between two points on a hyper-sphere. Hence it is conceptually simpler to think of this distance as an angle measurement: Remark that as the cosine between V(D) and 1⃗ increases, the isotropy defect decreases. A diagram illustrating this relation is provided in Figure 1: For a given reference point r and two comparison points ca and cb, we can observe that the shortest chord (from r to ca) also corresponds to the smallest angle.
More formally, let ṽ = ∥1⃗∥2∥V(D)∥2V(D) be the renormalized observed variance vector. We can note that both ṽ and the ideal variance vector 1⃗ are points on the hyper-sphere centered at the origin and of radius ∥⃗1∥2. As such, the defect is then equal to the distance between two points on a circle, i.e., the length of the chord between the renormalized observed variance vector and the ideal variance vector—which can be computed by simple trigonometry means, as 2∥⃗1∥2 sin (α/2), with α the angle between ṽ and 1⃗. This can be converted to the more familiar cosine by applying a trigonometry identity (given that 0 ≤ α ≤ π/4):
∥ṽ − 1⃗∥2 = 2∥⃗1∥2 √ 1− cos2(α/2)
1
4d ∥ṽ − 1⃗∥22 − 1 = − cos2(α/2)
where d is the dimension of the vectors in our point cloud. Hence we can exactly relate the isotropic defect (squared) to the cosine (squared) of the angle between ideal and observed variance vectors.
By monotonicity arguments, we can simplify this as follows: To maximize isotropy, we have to maximize the objective OI
OI = cos ( 1⃗,V (D) )
∝ ∑
d∈D
∑
d′∈D
∑
i
( di − d′i )2 (2)
This intuitively makes sense: Ignoring vector norms, we have to maximize all distances between
every pair of data-points to ensure all dimensions are used equally, i.e., spread data-points out evenly on a hyper-sphere. However, in the general case, it is not possible to maximize both the isotropy objective in (2) and the silhouette score objective in (1): Intra-cluster pairwise distances must be minimized for optimal silhouette scores, but must be maximized for optimal isotropy scores. In fact, the two objectives can only be jointly maximized in the degenerate case where no two data-points in D are assigned the same label.1

heading: 2.3 Relation to linear classifiers
text: Informally, latent representations need to form clusters corresponding to the labels in order to optimize a linear classification objective. Consider that in classification problems (i) any data-point d is to be associated with a particular label ℓ(d) = ωi and dissociated from other labels Ω\{ℓ(d)}, and (ii) association scores are computed using a dot product between the latent representation to be classified and the output projection matrix, where each column vector cω corresponds to a different class label ω. As such, for any point d to be associated with its label ℓ(d), one has to maximize
⟨d, cℓ(d)⟩ = 12 ( ∥d∥22 + ∥cℓ(d)∥22 − ∥d− cℓ(d)∥22 )
In other words, one must either augment the norm of d or cℓ(d), or minimize the distance between d and cℓ(d). Note however that this does not factor in the other classes ω′ ∈ Ω \ {ℓ(d)} from which d should be dissociated, i.e., where we must minimize the above quantity. To account for the other classes, the global objective OC to maximize can be defined as
OC =− ∑
d∈D
∑ ω∈Ω sign (ω, ℓ (d)) ⟨d, cω⟩
=− ∑
d∈D
|Ω| − 2 2 ∥d∥22 − ∑
ω∈Ω
|D| − 2|Dω| 2 ∥cω∥22
+ 1
2
∑
d∈D
∑ ω∈Ω sign (ω, ℓ (d)) ∑ i (di − cωi )2
(3) where the weights |Ω| − 2 and |D| − 2|Dω| stem from counting how many other vectors a given data or class vector is associated with or dissociated from: we have one label to associate with any datapoint d, and |Ω| − 1 to dissociate it from; whereas
1Hence some NLP applications and tasks need not be impeded by isotropy constrains, e.g., linear analogies that rely on vector offsets are a prima facie compatible with isotropy.
a class vector cω should be associated with the corresponding subset Dω and dissociated from the rest of the dataset (viz. D \ Dω).2
Focusing on the last line of Equation (3), we find that maximizing classification objectives entails minimizing the distance between a latent representation d and the vector for its label cℓ(d), and maximizing its distance to all other class vectors. It is reminiscent of the silhouette score in Equation (1): In particular any optimum for OC is an optimum for OS, since it entails D∗ such that
∀d,d′ ∈ D∗ ℓ(d) = ℓ(d′) ⇐⇒ d = d′ (4)
Informally: The cluster associated with a label should collapse to a single point. Therefore the isotropic objective OI in Equation (2) is equally incompatible with the learning objective OC of a linear classifier.
In summary, (i) point clouds cannot both contain well-defined clusters and be isotropic; and (ii) linear classifiers should yield clustered and thereby anisotropic representations.

heading: 3 Empirical confirmation
text: To verify the validity of our demonstrations in Section 2, we can optimize a set of data-points for a classification task using a linear classifier: We should observe an increase in silhouette scores, and a decrease in IsoScore. Note that we are therefore evaluating the behavior of parameters as they are optimized; i.e., we do not intend to test whether silhouettes and IsoScore behave as expected on held-out data. This both allows us to precisely test the argument laid out in Section 2 and cuts down computational costs significantly.

heading: 3.1 Methodology
text: We consider four setups: (i) optimizing SBERT sentence embeddings (Reimers and Gurevych, 2019)3 on the binary polarity dataset of Pang and Lee (2004); (ii) optimizing paired SBERT embeddings3 on the validation split of SNLI (Bowman et al., 2015); (iii) optimizing word2vec embeddings4 on
2The corresponding two sums can be understood as probabilistic priors over the data: The objective entails that the norm of a class vector cω should be proportional to the number of data-points with this label ω, whereas one would expect a uniform distribution for vectors d. These terms cancel out for balanced, binary classification tasks.
3all-MiniLM-L6-v2 4http://vectors.nlpl.eu/repository/,
model 222, trained on an English Wikipedia dump of November 2021.
POS-tagging multi-label classification using the English CoDWoE dataset (Mickus et al., 2022b); and (iv) optimizing word2vec embeddings4 for WordNet supersenses multi-label classification (Fellbaum, 1998; pre-processed by Tikhonov et al., 2023). All these datasets and models are in English and CC-BY or CC-BY-SA.5 Relevant information is available in Table 1; remark we do not split the data as we are interested on optimization behavior. We also replicate and extend these experiments on GLUE in Appendix A.
For (i) and (ii), we directly optimize the output embeddings of the SBERT model rather than update the parameters of the SBERT model. In all cases, we compute gradients for the entire dataset, and compute silhouette scores with respect to the target labels and IsoScore over 1000 updates. In multi-label cases (iii) and (iv), we consider distinct label vectors as distinct target assignments when computing silhouette scores. Models are trained using the Adam algorithm (Kingma and Ba, 2014);6 in cases (i) and (ii) we optimize cross-entropy, in cases (iii) and (iv), binary cross-entropy per label. Remark that setups (ii), (iii) and (iv) subtly depart from the strict requirements laid out in Section 2.
Training per model requires between 10 minutes and 1 hour on an RTX3080 GPU; much of which is in fact devoted to CPU computations for IsoScore and silhouette scores values. Hyperparameters listed correspond to default PyTorch values (Paszke et al., 2019), no hyperparameter search was carried out. IsoScore is computed with the pip package IsoScore (Rudman et al., 2022) on unpaired embeddings, silhouette scores with scikit-learn (Pedregosa et al., 2011).
5Our use is consistent with the intended use of these resources. We trust the original creators of these resources that they contain no personally identifying data.
6Learning rate of 0.001, β of (0.9, 0.999).

heading: 3.2 Results
text: Results of this empirical study are displayed in Section 3.1. Performances with five different random initialization reveal negligible standard deviations (maximum at any step < 0.0054, on average < 0.0008). Our demonstration is validated: Across training to optimize classification tasks, the datapoints become less isotropic and better clustered. We can also see a monotonically decreasing relationship between IsoScore and silhouette scores, which is better exemplified in Figure 3: We find correlations with Pearson’s r of −0.808 for the polarity task, −0.878 for SNLI, −0.947 for POStagging and −0.978 for supersense tagging; Spearman’s ρ are always below −0.998.
In summary, we empirically confirm that isotropy requirements conflict with silhouette scores and linear classification objectives.

heading: 4 Related works
text: How does the connection between clusterability and isotropy that we outlined shed light on the growing literature on anisotropy?
While there is currently more evidence in favor of enforcing isotropy in embeddings, the case is not so clear cut that we can discard negative findings, and a vast majority of the positive evidence relies on improper techniques for quantifying isotropy (Rudman et al., 2022). Ethayarajh (2019) stressed that contextual embeddings are effective yet anisotropic. Ding et al. (2022) provides experiments that advise against using isotropy calibration on transformers to enhance performance in specific tasks. Rudman and Eickhoff (2023) finds that anisotropy regularization in fine-tuning appears to be beneficial on a large array of tasks. Lastly, Rajaee and Pilehvar (2021a) find that the contrasts encoded in dominant dimensions can, at times, capture linguistic knowledge.
On the other hand, the original study of Mu and Viswanath (2018) found that enforcing isotropy on static embeddings improved performances on semantic similarity, both at the word and sentence level, as well as word analogy. Subsequently, a large section of the literature has focused on this handful of tasks (e.g., Liang et al., 2021; Timkey and van Schijndel, 2021). Isotropy was also found to be helpful beyond these similarity tasks: Haemmerl et al. (2023) report that isotropic spaces perform much better on cross-lingual tasks, and Jung et al. (2023) stress its benefits for dense retrieval.
These are all applications that require graded ranking judgments, and therefore are generally hindered by the presence of clusters—such clusters would for instance introduce large discontinuities in cosine similarity scores. To take Haemmerl et al. (2023) as an example, note that languagespecific clusters are antithetical to the success of cross-lingual transfer applications. It stands to reason that isotropy can be found beneficial in such cases, although the exact experimental setup will necessarily dictate whether it is boon or bane: For instance Rajaee and Pilehvar (2021b) tested finetuning LLMs as Siamese networks to optimize performance on sentence-level similarity, and found enforcing isotropy to hurt performances—here, we can conjecture that learning to assign inputs to specific clusters is a viable solution in their case.
The literature has previously addressed the topic of isotropy and clustering. Rajaee and Pilehvar
(2021a) advocated for enhancing the isotropy on a cluster-level rather than on a global-level. Cai et al. (2021) confirmed the presence of clusters in the embedding space with local isotropy properties. Ait-Saada and Nadif (2023) investigated the correlation between isotropy and clustering tasks and found that fostering high anisotropy yields highquality clustering representations. The study presented here provides a mathematical explanation for these empirical findings.

heading: 5 Conclusion
text: We argued that isotropy and cluster structures are antithetical (Section 2), verified that this argument holds on real data (Section 3), and used it to shed light on earlier results (Section 4). This result however opens novel and interesting directions of research: If anisotropic spaces implicitly entail cluster structures, then what is the structure we observe in our modern, highly anisotropic large language models? Prior results suggest that this structure is in part linguistic in nature (Rajaee and Pilehvar, 2021a), but further confirmation is required.
Another topic we intend to pursue in future work concerns the relation between non-classification tasks and isotropy: Isotropy constraints have been found to be useful in problems that are not well modeled by linear classification, e.g. word analogy or sentence similarity. Our present work does not yet offer a thorough theoretical explanation why.

heading: Acknowledgments
text: This work is part of the FoTran project, funded by the European Research Council (ERC) under the EU’s Horizon 2020 research and innovation program (agreement № 771113). We also thank the CSC-IT Center for Science Ltd., for computational resources.
Limitations
The present paper leaves a number of important problems open.
Idealized conditions. Our discussion in Section 2 points out optima that are incompatible, but says nothing of the behavior of models trained until convergence on held out data. In fact, enforcing isotropy could be argued to be a reasonable regularization strategy in that it would lead latent representations to not be tied to a specific classification
structure. Relatedly, a natural point of criticism to raise is whether our reasoning will hold for deep classifiers with non-linearities: Most (if not all) modern deep-learning classification approaches rely on nonlinear activation functions across multiple layers of computations. The present demonstration has indeed yet to be expanded to account for such more common cases.
Insofar neural architectures trained on classification objectives are concerned, we strongly conjecture their output embeddings would tend to be anisotropic. The anisotropy of inner representations appears to be a more delicate question: For Transformers, there has been extensive work showcasing that their structure is for the most part additive (Ferrando et al., 2022a,b; Modarressi et al., 2022; Mickus et al., 2022a; Oh and Schuler, 2023; Yang et al., 2023; Mickus and Vázquez, 2023), and we therefore expect anisotropy to spread to bottom layers to some extent. For architectures based on warping random distributions such as normalizing flows (Kobyzev et al., 2021), GANs (Goodfellow et al., 2014), or diffusion models (Ho et al., 2020), the fact that (part of) their input is random and isotropic likely limits how anisotropic their inner representations are.
Thoroughness of the mathematical framework. The mathematical formalism is not thorough. For the sake of clarity and given page limitations, we do not include a formal demonstration that the linear classification optimum necessarily satisfies the clustering objective. Likewise, when discussing isotropy in Equation (2), we ignore the cosine denominator.
Choice of objectives. Our focus on silhouette scores and linear classifier objectives may seem somewhat restrictive. Our use of the silhouette score in the present derivation is motivated by two facts. First, our interest is in how the point cloud will cluster along the provided labels—this rules out any external evaluation metric comparing predicted and gold label, such as ARI (Hubert and Arabie, 1985) or purity scores. Second, we can also connect silhouette scores to a broader family of clustering metrics such as the Dunn index (Dunn, 1974), the Caliński–Harabasz index (Caliński and Harabasz, 1974) or the Davies–Bouldin index (Davies and Bouldin, 1979). Silhouette scores have the added benefit of not relying on
centroids in their formulation, making their relation to the variance vector V(D) more immediate. We conjecture that these other criteria could be accounted for by means of triangular inequalities, as they imply the same optimum layout D∗ as Equation (4).
As for our focus on the linear classifier objective, we stress this objective is a straightforward default approach; but see Appendix B for a discussion of triplet loss within a similar framework as sketched here.

heading: A Supplementary experiments on GLUE
text: We reproduce experiments described in Section 3 on GLUE tasks (Wang et al., 2018).7 We train our models on the provided training sets—hence we only consider tasks for which there is a training set (all but ax) and that correspond to a classification problem (all but stsb, a regression task); we remove all datapoints where no label is provided. Given our earlier results, we limit training to 250 updates; we directly update sentence-bert output embeddings by computing gradients for the entire training set all at once. We compute IsoScore and silhouette scores after every update; to alleviate computational costs, they are evaluated on random samples of 20, 000 items whenever the training set is larger than this (samples are performed separately for each update). We test three different publicly available pretrained SBERT models: all-mpnet-base-v2 (referred to as “mpnet” in what follows), all-distilroberta-v1 (viz. “roberta”) and all-MiniLM-L6-v2 (viz. “miniLM”). Training details otherwise match those of Section 3; see Table 2 for further information on the number of datapoints and parameter counts of all models considered.
Corresponding results are depicted in Figure 4. While there is some variation across models and
7From huggingface.co.
GLUE tasks, all the setups considered display the same trend: Silhouette score increases and IsoScore decreases across training. We can quantify this trend by computing correlation scores between IsoScore and silhouette scores. Corresponding correlations are listed in Table 3: As is obvious, we find consistent and pronounced anticorrelations in all setups, with Pearson’s r always below −0.838 and Spearman’s ρ always below −0.966. This further consolidates our earlier conclusions in Section 3.

heading: B Relation to triplet loss
text: To underscore some of the limitations of our approach, we can highlight a connection with the triplet loss, which is often used to learn clusters.
It is defined for a triple of points da,dp,dn
where ℓ(da) = ℓ(dp) ̸= ℓ(dn) as
Lapn = max (∥da − dp∥2 − ∥da − dn∥2, 0) = max (∥da − dp∥2, ∥da − dn∥2)− ∥da − dn∥2 ≥ ∥da − dp∥2 − ∥da − dn∥2
= ∑
dc∈{dp,dn} −sign (ℓ (da) , ℓ (dc)) ∥da − dc∥2
The objective across the entire dataset D is thus:
OT = ∑
ω∈Ω
∑
da∈Dω
∑
dp∈Dω\{da}
∑
dn∈D\Dω −Lapn
≤ ∑
ω∈Ω
∑
da∈Dω
∑
dp∈Dω\{da}
∑
dn∈D\Dω∑
dc∈{dp,dn} sign (ℓ (da) , ℓ (dc)) ∥da − dc∥2
= ∑
d∈D
∑
d′∈D signwgt
( ℓ (d) , ℓ ( d′ )) ∥d− d′∥2
(5)
using a weighted variant of our original sign function:
signwgt(ω, ω ′) = { |Dω| − |D| if ω = ω′ |Dω| − 1 otherwise
Remark that this is in fact an upper bound on both the silhouette objective as defined in Equation (1) and the triplet objective OT. However, as they are to be maximized, the above does not entail that models trained with a triplet loss will necessarily develop anisotropic representations.

heading: Future Work
text: Another topic we intend to pursue in future work concerns the relation between non-classification tasks and isotropy: Isotropy constraints have been found to be useful in problems that are not well modeled by linear classification, e. g. word analogy or sentence similarity. Our present work does not yet offer a thorough theoretical explanation why