Zipfian Whitening
Authors: Sho Yokoi
Han Bao
Hiroto Kurita
Hidetoshi Shimodaira
Venue: NeurIPS 2024 poster (2024)
Keywords: representation learning
word embeddings
isotropy
natural language processing
Primary area: natural_language_processing
TL;DR: We propose a method to correct anisotropy in the word embedding space by accounting for word frequency; then justify our approach from the perspectives of generative models, word embedding norms, and errors in long-tail distributions.

Abstract: The word embedding space in neural models is skewed, and correcting this can improve task performance.
We point out that most approaches for modeling, correcting, and measuring the symmetry of an embedding space implicitly assume that the word frequencies are *uniform*; in reality, word frequencies follow a highly non-uniform distribution, known as *Zipf's law*.
Surprisingly, simply performing PCA whitening weighted by the empirical word frequency that follows Zipf's law significantly improves task performance, surpassing established baselines.
From a theoretical perspective, both our approach and existing methods can be clearly categorized: word representations are distributed according to an exponential family with either uniform or Zipfian base measures.
By adopting the latter approach, we can naturally emphasize informative low-frequency words in terms of their vector norm, which becomes evident from the information-geometric perspective (Oyama et al., EMNLP 2023), and in terms of the loss functions for imbalanced classification (Menon et al. ICLR 2021).
Additionally, our theory corroborates that popular natural language processing methods, such as skip-gram negative sampling (Mikolov et al., NIPS 2013), WhiteningBERT (Huang et al., Findings of EMNLP 2021), and headless language models (Godey et al., ICLR 2024), work well just because their word embeddings encode the empirical word frequency into the underlying probabilistic model.

Zipfian Whitening
Sho Yokoi Tohoku University RIKEN
Han Bao
Kyoto University
yokoi@tohoku, jpp
bao@i www.koo.oc.......
Hiroto Kurita
Hidetoshi Shimodaira Kyoto University / RIKEN shimo8i kyo.o-u.ac.
Tohoku University
hiroto. kuritadde tohoku. ac.
Abstract
The word embedding space in neural models is skewed, and correcting this can improve task performance. We point out that most approaches for modeling, correcting, and measuring the symmetry of an embedding space implicitly assume that the word frequencies are uniform, in reality, word frequencies follow a highly non-uniform distribution, known as Zipf's s law. Surprisingly, simply performing PCA whitening weighted by the empirical word frequency that follows Zipf's law significantly improves task performance, surpassing established baselines. From theoretical perspective, both our approach and existing methods can be clearly categorized: word representations are distributed according to an exponential family with either uniform or Zipfian base measures. By adopting the latter approach, we can naturally emphasize informative low- wwwffreuency words in terms of their vector norm, which becomes evident from the information geommeric perspective [42], and in terms of the loss functions for imbalanced classification [36]. Additionally, our theory corroborates that popular natural language processing methods, such as skip-gram negative sampling [37], WhiteningBERT [26]. and headless language models [23], work well just because their word embeddings encode the empirical word frequency into the underlying probabilistic model.
? https: /tgp::///ttthinn ///ggtttubbb.. com/ htt/:cltotoocoe tohokuhooouoooooooo zipf an whi weniiinng
Introduction
Representing discrete words by continuous vectors is fundamental and powerful framework of modern ccep-learning-base natural language processing (NLP). Static word embeddings [43, 37], dynamic word embeddings [18, 33), and causal language models [45, 12, 54] have caused a paradigm shift-they have greatly improved the performance of virtually all kinds of NL.P applications and have been actively used in relevant
well. While the embedded units may be characters subwords instead of words, we simply refer to them collectively as word.
Recently, the machine learning and NLP communities have discovered that the word embedding space is "skewed" and that correcting this can lead to better performance in downstream tasks [39, 21, 16, 56). The isotropy of the embedding space would be one factor: vectors dispersing more evenly should be more discriminative than those clustered the same direction [38, 21, 51]. Typically, such spatial symmetry in the embedding space enhanced through centering/whiteningg [39. 16, 26).
Nevertheless, we would like to point out that most existing approaches implicitly assume uniform word frequency to formalize spatial symmetry. Consider the classical centering operation as an example: we first calculate the mean of the word vectors, and then subtract it to ensure they are zero-meaned. This method, however, has an unexpected pitfall. Recall that the definition of the centroid or barycenter of a random vector & ~ assuming has a finite set of distinct realizations,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

is given by [ E, E,p(z)) (a, The classical centering, based on the standard (unweighted) mean, implicitly assumes that all words occur uniformly p(w,,) =.... =((w,) In reality, however, word frequencies are known to follow a highly non-uniform distribution creating a significant gap between the methodology and the actual usage of words. This seemingly obvious issue does not arise when addressing classical statistical estimation problems, as data vectors in our hands are usually representations of observations or instances. In contrast, word vectors used in NLP are representations of types classes; each of them (such the vector for 'the') abstracts the numerous instances (such as the tokens of 'the') appearing in the data. This problem of hidden frequencies becomes apparent in the cases where the type-token distinction [58] is crucial, such when dealing with natural language data ($ 2). The take-home message of this paper can be summarized as follows: use empirical word frequencies when calculating expected values. Following this very simple guideline leads to strong empirical outcomes ($ 3.2, $ 3.3) and opens a rich theoretical landscape ($ 4, $ 5).
wn) denote the vocabulary, i.e., the set of words in interest. Bold-face wy E Rd denotes the row vector of each word type wt, and p(w:) E [0, 1] denotes its frequency.
Notation Let V = (w,,...,
2 Motivation: type-token distinction and expected values
Why have word frequencies been overlooked when considering the geometric properties of em- bedding spaces? This can be explained through the concept of type-token distinction [58], which is a fundamental concept in linguistics and related fields but generally not required in statistical machine learning. Here, type represents a class and token represents an instance. For example, the phrase pperform natural language processing in a natural way' contains eight tokens and seven types. The instances 'natural' appear twice, but as word type. it is counted only once.
With the toke-token distinction in mind, let us take a fresh look at data matrices and their expected values. Typically, each row in data matrix represents one observation, i.e., one instance token. If we want to centralize a set of data vectors, computing the unweighted mean is natural way in the machine learning pipeline. On the other hand, each row of word embedding matrix, i.e., word vector, is a type embedding. Each word vector abstracts the numerous instances appearing repeatedly in corpus, though information on the frequency of instances for each word type is not encoded in it. The unweighted mean of word vectors treats type vectors as token vectors, resulting in the complete omission of word frequency information.
Let us describe the above idea formally. The data ma- trix X Rn the set of data vectors ye C Rd represents a collection of instances, observations, or tokens; then the empirical distribution is Px
the
of
Ex I
n (m;)), where 5 is the Dirac delta function.
Here, the unweighted mean be seen as the expecta- L
isotropy
tion - [m) = Em= " with the empirical distri- bution, On the other hand, the word embedding matrix W E Raxd or the set of word vectors [w,))__, C Rdd represents a collection of types. When describing the empirical distribution, the hidden frequency p of to- kens is necessary. Given p. the empirical distribution Mw = =p(::)o(w; ((w:). From this perspective, the cen- troid of the word vectors should be written as the ex- pectation En [w] Σ, p( ww; over p.
agglutinative
Figure 1: Low-frequeet words {o} and high-frequent words (0) are unevenly dis- tributed in the embedding space [39, 24, 44, 10]. Consequently, the "apparent" mean cal- culated by unweighted averaging * often differs from the actual centroid *.
The distinction is not just "theoretical." First, refer Fig. 1. Word vectors are known cluster by frequency [39, 24, 44, 10]. In this situation, the centroid * weighted by the word frequencies is located near the narrow region where high-frequent words are concentrated (a region with light blue background), and thus differs from the unweighted mean *, Second, see Table 1, which shows
As known as Zipf's law. If we count the frequencies of words in huge English corpora, we find that 'the' has frequency of about 5.89 x 10 and 'isotropy' has a frequency of about 3.47 x 10 difference of a million times greater.

---

50 words sampled from each of types and tokens. Uniform sampling from types, corresponding to an unweighted mean, tends to select mostly rare words from the heavy tail. Sampling from tokens clearly captures more natural representation of language as it typically appears in text.
Table The difference between type-based sampling and token-based sampling.
Words sampled from types
Words sampled from tokens
"scintsllation', 'fanon', 'rubato', "upstanding'. 'collard', "creeks'. "akookum', "unbelievera", 'monocyte', 'nishikawa". 'crusher', 'gerven', 'abrab', 'silverchair', 'hangman', 'unitary'. 'klausen', 'arousal'. 'heat', 'bridgnorth' "mildred'. 'porton'. 'aquasox'. "wylie.. 'hipaa'. 'krimuk'. hheahherron, 'kuei', 'barbera', 'dalvi', 'gilding', "iiaakhapatnam" 'tatauo', 'tarascon', 'bajram'. 'scholes', 'hadad', 'incidental'. 'theodosius', "reichskomisssrraat 'boeheim', 'amal', 'buencamino', 'thrasyvoulos', 'insulated', "discourteay'. 'niera'. 'ycko'. 'luen', 'dooku'
'nine', 'ranked', 'zero', 'the', 'garcia', 'rank', 'atation', 'the', 'for'. 'four', 'williams', 'drunken', 'a', 'one', 'eight', "of', 'were', 'zero', 'debate', 'orchestra", "of". 'wrist', 'points'. 'fractured". 'the". 'to', 'redirect', 'adnan', 'shite', 'car', 'fond", 'concluded', 'under', 'two', 'by". 'five', 'his', 'infection', 'the'. 'the', 'pop', 'in', 'one', 'in', 'one', 'one', 'fram" 'handled', 'battle', 'mutual'
3 Embedding symmetry
3.1 Definition of embedding symmetry
In mathematical science fields, such as high-dimensional probability theory [55] and the volume of convex bodies [27], there are numerous intriguing definitions of spatial symmetry. Among them, we begin with the definition of the symmetry of random vectors with their frequencies [48, 55]. This is suited for dealing with word vectors because they entail word frequencies, unlike usual data instances.
Definition 1 (A random vector v ~ p Rd has zero mean; the /st moment of symmetric random vector).
Definition 2 (A random vector v p on Rd in isotropic position around its barycenter, the 2nd moment of a symmetric random vector).
V Eu = 0
(1)
Cov[u] = Euwp (e-
(2)
From these definitions, we will develop methods to adjust given word vectors to be symmetric in 3.2, and evaluate the symmetry of given word vectors in $ 3.3.
In machine learning and NLP, the spatial symmetry of embedding spaces is hot topic, and numerous theories and algorithms have been proposed [41. 21, 38, 56]. However, the approach in many researches implicitly treats all vectors equally, ignoring word frequency information. the following sections, will detail both the empirical and theoretical issues that uniform approach can cause, especially when applied to NL.P tasks. Furthermore, when embeddings correspond to tokens rather than types-such such in the internal representations of masked causal language models uniform approach tends to be effective. This point will be discussed in 5.1.
3.2 Enhancement of embedding symmetry
This section proposes Zipfian whitening?, which symmetrizes a given set of word vectors with word frequency. At glance, the most natural method to achieve Def. 1 and Def. 2 would be PCA whitening, also known as sphering. Notably, each step of whitening- standardization implicitly involves calculating expected values. Our approach is simple: each time we calculate an expected value, we should weight it by the empirical word frequency. The specific algorithm is shown in Algorithm 1. The only difference from general whitening is that it uses word frequency in the part highlighted in blue. Please refer to Appendix for formal explanation showing that the word vectors obtained by the proposed algorithm actually satisfy Def. I and Def.
decorrelation, and
IIn this paper, "Zipfian" simply used to denote "highly non- uniform" distribution. Our focus on the mismatch between actual word frequencies and uniform distribution, and we have not constructed arguments or experiments that rely specific properties of power laws. Refining experiments and theory based on the degree of tail heaviness is an interesting direction for future work.

---

Algorithm Zipfian whitening: a post-processing algorithm on word embeddings. The part high- lighted blue shows the difference from the typical centering and whitening.
Input: Word embeddings (w, E IRd E VJ. word frequency p: V - [0, 1].
Output: Processed word embeddings.
W E R3 are centered, {w; E ]R are further whitened.
Zipfian centering (Ist moment):
1: Ew,ev p(w;) E Rd
2: for all wy E V/ do
3: W; < w;
Rd
4:
end for
Zipfian decorrelation and standardization (2nd moment):
5: WP
Vp(wi) w.
Vp(wip) w 1
@RMl
6: UEV + sV)(W)))
> Σ diag(ol........ an) E Ruxd consists of the singular values W
7: for all E V do
8:
W. f w, VE-1
> E
(lag(1/o11..... 1/oaa) Raxd
9: end for
Table 2: The empirical performance of Zipfian whitening, which exploits the empirical frequency of words during expectation calculations. Each cell shows the STS-B [15] score x 100. By carefully performing the simple operation of whitening, it consistently outperforms powerful baseline methods.
GloVe
46.17
Word2Vec
56.98
Uniform 45.17
Zipfian 52.25
Uniform 55.85 56.03
Zipfian 58.84
Centering
Centering + Whitening
Whitening
52.21
66.92
66.50
ABTT [39]
54.28
ABTT [39]
$6.98
SIF CCR (7]
58.70
+ SIF + CCR [7]
63.04
We confirm the effectiveness of Zipfian whitening (Algorithm by measuring performance standard sentence-level downstream tasks using post-processed word vectors. We employed the most standard word embeddings Glo [43], word2vee [37], and fastText [11]-and utilized the widely adopted evaluation tasks, including STS-B [15] and related benchmarks. Detailed experimental settings can be found in Appendix B. Table shows the results on the STS-B task. Remarkably, the proposed Zipfian whitening shows significant advantages not only (uniform) centering and whitening but also over the strong baseline method [7] specifically designed to create powerful sentence vectors. Consistent results were obtained with various benchmark datasets, multiple empirical word probabilities, and a language other than English (Appendix C)² In $ 4.2.1, reason for this remarkable performance is clarified from the perspective of information geometry.
Empirical evaluation:
embeedings-Glove
standard
3.3 Evaluation of embedding symmetry
The community is greatly interested not only in making word vector spaces symmetric but also in evaluating how symmetric or asymmetric space is [21, 49]. Here, we return to Def. and Def. 2 and describe metrics for evaluating the symmetry of word embedding spaces with word frequency.
Degree of centrality- 1st moment symmetry: Recall that, the barycenter E[u; is close to 0, then the random vector be considered symmetric in terms of the first moment (Def. 1).
Nottbby, observed improved scores when using word frequencies from the evaluation dataset itself p(w). general, NLP tasks, p(w) refers word frequencies derived from the embedding training data or from a standard large corpus. However, to optimize downstream task performance, it is preferable to base p(w) on word frequencies within the evaluation dataset itself used for those tasks. This adjustment exemplifies "covariate shift" [52] in machine learning, where the distribution of training data differs from that of test data.

---

Thue, examining the value of |[[(]]]| == E[e] 0 appears to be a reasonable way to measure the symmetry of the first moment. However, random vectors v and av (a E >2)) should be considered equivalent in terms of spatial symmetry. Thus, we define the scale-invariant metric (Def. 3), obtained by dividing [[[(]] by the average length [[[[=]]]]
Definition (Degree of centrality for the random vector p; the 1tt moment of symmetry).
Sym = 1
E ump
E
||]|]]]]
(3)
By definition, Sym, (v) takes values in [0, 11, and Sym, (v) = 1 if and only if v is zero mean.
Degree of isotropy-the 2nd moment of symmetry: If the covariance matrix El( [E B[ol)"] is constant multiple of the identity matrix La. i.e., if the random vector v has equal spread in all directions, v is symmetric in terms of the second moment (Def. 2). Following convention, this degree can be confirmed by examining the flatness of the eigenspectrum.
Definition 4 (Degree of isotropy around the barycenter for the random vector v ~ I the 2nd moment of symmetry).
1
Ad "
A1
Sym == log H
(4)
{AI,..., ] are the eigenvalues of the covariance matrix Pa) E log Pi is the Shannon entropy. vrup
E [(v - E [o])(v- E [ol) u~p
v<p
Proposition 1. Sym takes values in [0, 1], and Symz(v)= 1 if and only if v is isotropic around its barycenter (Def. 2).
Proof. Please refer to Appendix D.
Note that the approach of measuring the entropy of the spectrum to evaluate the flatness of a signal can be found in many fields. For example, similar definitions are seen in probability processes [14] and signal processing [17, 47]. We also follow this standard and powerful line.
To compute the evaluation metrics of symmetry (Def. 3, Def. 4) for given word vectors, again, one should just use the empirical word frequency when calculating the expectations. A pseudocode for measuring symmetry is provided in Appendix E.
Algorithm:
Empirical evaluation: To what extent does our symmetry score (an intrinsic evaluation of embedding spaces) correlate with downstream task performance (an extrinsic evaluation of those)? As baselines, we use versions of our symmetry score that do not account for word frequency, calculated in a uniform manner. We also compare with popular symmetry scores in NLP, the average of cosine similarity (Ave. Cos.) [21] and the recently proposed IsoScore [49]. Note that all these baselines implicitly assume uniform word frequency. Additional experimental settings can be found in Appendix B. Fig. 2 shows the results. The right side of Fig. 2 demonstrates the superiority of the Zipfian approach. Moving from the bottom-left to the top-right of the figure-i., i.e. as both the 1st (æ-axis) and 2nd moments (y-axis) of the symmetry score increase performance increases (the color becomes more red). In contrast, in the left-hand plot, which assumes uniform word frequency, there no observed relationship between the symmetry score (r and y-axis) and the downstream task performance (color). Table 3 lists the correlation coefficients between the symmetry scores and downstream task performance in more detail. It can be seen that the symmetry scores considering word frequency can "predict" downstream task performance with remarkably high correlation. On the other hand, the "prediction" performance of other metrics, including Ave. Cos. and IsoScore that implicitly assume uniform word frequency, unsatisfactory. Surprisingly, when the most popular Ave. Cos. metric shows almost no correlation (0.04) with downstream task performance (STS-B), Zipfian symmetry metric has strong positive correlation (0.83) with it.
clearly visible that the downstream task
4 Why is Zipfian whitening better than uniform whitening?
A natural question is why the Zipfian approach empirically dramatically outperforms the uniform approach. We provide theoretical explanation using Table 4. In nutshell, a significant difference arises depending whether the base measure of an exponential family is uniform or Zipfian.
5

---

70
1.00
1.00
(2nd moment, Unif.)
(2nd moment, Zipf.)
(Spearman's p 100)
Deg. of symmetry
Deg. of symmetry
65
STS-B performaat
0.95
0.95
60
0.90
0.90
55
0.85
0.85
50
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
Deg. symmetry (Ist moment, Unif.)
Deg. symmetry
(1st moment, Zipf.)
Figure 2: The relationship between the Ist-order symmetry (Def. 3. z-axis), the 2nd-order symmetry (Def. y-axis), and task performance (color). Each point represents either pre-trained or post- processed word embeddings (GloVe word2 Vec, and fastText). The Zipfian measure well captures the downstream task performance (right), while the uniform isotropic measure cannot (left).
Table 3: Spearman's s p 100 (each cell) between the symmetry scores (each column) and downstream STS-B performance (each row), on pre-trained and post-processed embeddings (GloVe, word2 Vec, and fastText). The scores based on the Zipfian prior show significantly higher correlation with task performance compared to those based on the uniform prior including Ave. Cos. and IsoScore.
Ave. Cos.
IsoScore [49]
Uniform
Zipfian
[21]
1st moment
2nd moment
1st moment 2nd moment
STS-B
-6.5 20.09
0.07
-21.91
221.21
62.13
89.55
SICK-R
18.41
13.26
-8.71
60.04
64.60
4.1 Characterization through generative model, partition function, and whitening
Exponential families: Hereafter, we interpret two salient generative models from the viewpoint of exponential families: one given by Arora et al. [6] and the other generalizing the Levy-Goldbery formula [32, Eq. (7)]. Details of these models will be provided shortly. An exponential family is class of probability distributions of a random variable a parametrized by a parameter 9, written in the following (canonical) form:
p( 0) = π (x:) exp ((x, 0) (0)),, ()) = log ( = llog Σ expl (x. (x)) 01)) (18)
where æ is a sufficient statistic, is called a natural parameter, π the base measure (or "prior") and is the log- ong-paritionn function. Once specify the base measure and the canonical pair (x, 0). the log-partition function determined. That being said, the base measure π is the design choice of an exponential family left for In the following, distributions in the form p(w c), where word w is predicted given context c. Specifically, the context represents co-occurring word (in static word embeddings), cloze sentence (in masked language models), or a sentence prefix (in causal language models). In all of these cases, we predict a word with the logit (w, c). making the exponential family natural probabilistic model. Here, the vector represents the vector expression of the context c, known as the "context vector." Note that, even for the word I, the predicted word vector (()) and the predicting context vector e(l) distinct.
specifically examine exponentyy exponential family of
Uniform prior: Arora et al. firstly considered log linear generative model of word embeddings given a context (6) and demonstrated that when the generative model adopted with normalized context vectors and huge vocabulary, the partition function asymptotically becomes constant (8) [6. Lemma 2. Here, we can regard that this model belongs to the exponential family with the uniform base measure (w) (C)) 1/1V1
"Alhhough Arora et al. [6]'s generative model treats a context vector c as a model parameter drifting by a random walk, we can cast their model into exponential family because they did not specify how the initial context vector is generated. Hence, by regarding observed token with the uniform prior n(c) 1/

---

Table Through the differences in the underlying generative models, the empirical superiority of Zipfian whitening over uniform whitening can be understood.
Generative models behind (whitened) embeddings
p(w c) (w) exp((w,c) e)), Z(c) Σ r(w) expe)xxp((uc))) c)); p(w, c) p(w c)
((c)
(5)
with Uniform prior
with Zipfian prior
P@l (w exp( c))
p(w) exp (w, c))
2 T(c)
(6)
pO(w
Zo(e) =x(c)= - p(c)
Partition functions become constant under certain conditions
Assume |c|| = 1, | + oo, then
At the optimal solution of the corresponding loss,
2 (c) = Σ exp( (w,, c)) const. (8) [6]
Zo(c)= = Σ p(w)cxp(((e exp((w, c)) const. (9) [32]
Whitening coarsely achieves constant partition function
Zo(e)
Zo(e)
= =M++ Σ w
c+ ! A
Σ www
c+
=|V++ Σ Ep(w)unee c E
2
20
1 Ic..... const. (ou)s)urs] 2
|
Ic.... v const. (10][39]
~ 1++ c
Vector norm under generative models
| = 2d p(w) - 22 (12) [6]
|| ((
~ 2KL(p(-)lp(( w)) (13) [42] [Thm. 1J
long vector ++ frequent (wrinformative) word
long vector ++ informative word
Loss and error corresponding generative models p(w c)
softmax cross-entropy loss
logit-adjusted softmax cross-entropy loss
exp( (ww, c))
p(w) exp (w, c)
E
log
(14)
E log
(15) [36]
(w)ee)
70(r))
(w)e)
ZO(c)
misclassification error
balanced error
P P(w.e][w & arg max(w', c)]
(16)
Σ WEV
Paj Paw[w arg max(ww, e)]
(17) [36]
M
Zipfian prior: An exponential family adopted with the Zipfian measure can be written as (7).
This generative model can be naturally derived from the skip-gram model with negative sampling
(SGNS) [37]. By assuming that the linear model c -+ (wc c) is sufficiently capable of discriminating cooccurring words and negative samples (as in the realizable case), model of the word embeddings must comply with the following formula:
that the generative
p(w, c) log k = (w, c).
log p(w)p(c)
(19)
where & the number of negative samples. This optimality formula owes Levy and Goldberg [32], and we call (19) the Goly-Go formula. A more concise derivation is later given by Oyama al. [42]. We can regard the Woy-Goodbeerg formula as an exponential family with the Zipfian base measure, π (w) = p(w) and (c)) = p(c), and the constant log-partition function
Zz(c) = --1
The generative model (7) relaxation of the Levy-Goldeee Goldberg formula since we do not impose the realizability assumption necessary for the derivation of (19).
their model is reduced to (6). The static context prior does not contradict Arora et al. [6]'s model with sufficiently large d, where the random walk drifts extremely slowly.
7

---

What does whitening do? Mu and Viswanath [39] proposed method to approximately make the partition function of the uniform prior model constant by centering the word vectors and removing the top principal components (10). Our Zipfian whitening corresponds to Mu and Viswanath's s post-processing method, in the sense that ours and theirs make the partition function constant up to the second moment (11) and (10), respectively. In summary, Zipfian whitening (11) transforms probabilistic model into an exponential family adopted with the Zipfian base measure (7), making closer to the Levy-Goldberg formula (19).
4.2 Emphasis on rare words by Zipfian prior
Let us explore further why the Zipfian prior results in good performance in downstream tasks ($ 3.2). In summary, the Zipfian prior approach emphasizes low-frequency words, while the uniform prior approach emphasizes high-frequency words, both from perspectives of vector norms and errors/losses. So far in this paper, we have repeatedly discussed weighting each word according to frequency, so it may seem contradictory that Zipfian approach emphasizes low-frequency words as result. To illustrate, let us reconsider centering. In centering, the mean vector is subtracted from each vector. Weighting each word vector by frequency when constructing the mean vector means that signals corresponding to high-frequency words are removed more substantially from each vector. The emphasis low-frequency words has been repeatedly supported throughout the history of NLP and information retrieval, such as Luhn's 's hypothesis [34], inverse document frequency (IDF) [53], and smooth inverse frequency (SIF) [7]. For instance, it is reasonable to emphasize the word "isotropy' when creating a sentence embedding containing both words 'the' and 'isotropy'.
4.2.1 From the perspective of vector norm
Under the Zipfian prior model, words with larger information content have longer (emphasized) vector representations. Conversely, under the uniform prior model, words with smaller information content have longer (emphasized) vector representations.
As representative example of uniform prior models, the norms of word vectors learned by random walk language models are theoretically and empirically proportional to word frequency (12) (see Eq. (2.4) and Fig. 2 in Arora et al. [6)). That is, in such embedding space, words with less information (e.g.. 'the") are emphasized. This tendency is consistently observed in dynamic language models and causal language models that adopt the softmax cross-entropy loss, another typical example of the uniform prior family [28]. By contrast, when training word embeddings with skip-gram negative sampling [37], the word embeddings follow the Zipfian prior family, and their norms become larger with greater information, which we show subsequently [50, 60, 42]. Based on the formulation of the exponential family and following Eq. (12) of Oyama al. [42], we formally describe the norm properties of the word vectors obtained from the Zipfian prior model.
Theorem 1 (The norm of a word vector learned with empirical Zipfian prior models reflect the information amount of the word; a refined version of [42] Eq. (12)). Assume that word embeddings tw;)e follow the Zipfian prior model (7), For the same word f, the vector w on the predicted side and the vector on the predicting side are shared: w(t)= = c(t) (weight rying), and Eeev p(t)w((t)
(centered w.r.t. Zipfian prior), then each word vector w (() satisfy
[|=())) s= 2KL(p(-)llp( [t). G(t) = Σ pit' ())(()))(()
(20)
PEV
where ||l|| A with a positive definite matrix A denote a norm based on a quadratic form / ww' Aw". Proof. Refer Appendix F.
In Fig. 3, we experimentally confirmed that the norms of informative words become larger with Zipfian whitening (shown from center the right in Fig. 3), bringing them closer the ideal Zipfian prior model
'The matrix G(w) takes the form Σ, x.. indeed positive definite, similar a covariance matrix. 'Given these results, some readers may be interested in how the Ziptian whitening process influences the lengths and directions of vectors individually. For details of the ablation study, please refer Appendix G.
8

---

Uniform Waitened
Uniform Centered
Pre-trained GloVe
Ziplian Centeved
Zaplien Whitened
7.5
7.5
7.5
10
5.0
20
2.5
2.5
7.5
0.02.
2.5
5.0
0.05..
2.5
5.0
0.02
5.0
7.5
2.5
5.0
7.5
log p(o)
Figure 3: Relationships between the information content - log pl (w) and the vector norms || for top 500 frequent words w. The figure in the center represents the pre-trained GloVe model. By using Zipfian whitening, the information content gets encoded in the norm (center to right). Conversely, with uniform whitening, this phenomenon does not occur (center to left).
4.2.2 From the perspective of error and loss
The error and loss functions associated with the Zipfian prior model emphasize low-frequency words. In contrast, the error and loss functions of the uniform prior model focus on the average loss across the entire dataset, resulting in a greater emphasis on high-frequency words.
The standard classification loss is the softmax cross-entropy loss (14). By taking its expectation over the dataset (ww, e)), embeddings associated with because the softmax is the uniform inverse link, corresponding to the uniform prior model. By contrast, the logit-adjusted loss (15) has been proposed tackle class imbalance [36]. From our viewpoint, the logit adjustment term p(w) makes the inverse link belong to the Zipfian prior model. The softmax and logit-au adjustad losses are Fisher consistent to the misclassification (16) and balanced (17) error rates, respectively. As the latter tends to stress minor classes, the logit-adjusted loss and Zipfian prior model are suitable for emphasizing fow-frequency words during the learning process.
words receive more updates
Another prominent loss function for representation learning is contrastive loss, with the SGNS loss (word2vec) [37] as a representative example in the context of word representation learning. This loss similarly uses a loss aligned with the Zipfian prior:
E (ar,e)
log (oooo(( c)) +
log logo(((wo(,,)))
(21)
where a is sigmoid function, and k is the number of negative samples. Since high-frequency words are more likely to be sampled as negative examples, the loss has less impact on high-frequency words in positive examples. Consequently, low-frequency positive words are relatively emphasized in representation learning. The Gevy-Goldberg formula in the previous section describes the properties of ideally trained word2vec model, which are essentially the properties of Zipfian prior models.
5 Unified explanation of the efficacy of existing methods
Distinguishing the distribution that the base measure follows helps understand why some existing NLP methods are effective.
5.1 Uniform whitening of token embeddings z Zipfian whitening of type embeddings
Masked language models like BERT [18] and RoBERTa [33] produce dynamic (contextualized) token embeddings. Adding up such token embeddings of constituent tokens to create sentence embeddings often leads to poor empirical performance [46]. However, symmetrizing significantly improves their performance; such methods including "batch centering," "WhiteningBERT." and contrastive learning methods [16, 46, 59, 22, 26, 57]. This improvement can also be explained from the perspective of the Zipfian prior. A dataset or corpus first fed into the model to obtain token embeddings?. Centering/whiteninn token embedding (multi)set has the multiplicity asymptotically proportional to the word frequency.
is then applied to this entire set of embeddings. As this
Heere, the computation of the additive composition s := 1/|s| Σ we can be ignored without major issues in formal discussions of spatial symmetry. This is because the words in sentence generated based word

---

Table 5: The empirical performance difference between "uniform" enforced centering and whitening with uniform prior for dynamic embeddings, and "Zipfian" - conventional uniform centering and whitening over tokens with an implicit Zipfian prior over types. Each cell shows the STS-B [15] score x 100. This comparison reveals that token-level uniform centering/whitening. corresponding to type-level Zipfian centering/whitening. leads to empirically better performance.
BERT-base
63.75
RoBERTa-base
60.75
"Uniform" 64.04
"Zipfian"
"Uniform" 60.34
"Zipfian"
Centering
64.82
Centering
61.30
Whitening 60.53
64.91
Whitening
61.31
65.59
this wniform centering/whitening of token embeddings corresponds to the word-frequenccy-wiighee (Zipfian) sentering/whitening of type embeddings. For a more formal description of the above explanations, please refer to Appendix H. Additionally, recent work has found that contrastive additive sentence encoders implicitly weight words by their information content [30]. This finding is consistent with the previous discussion on vector norms ($ 4.2.1), and can be seen as indirect evidence supporting the idea that these models belong to the Zipfian prior family.
This idea can also be supported by empirical evidence. This idea is also supported by empirical evidence. To establish baseline for centering and whitening token embeddings under uniform prior, we scale each embedding by the reciprocal of its type frequency, ensuring uniform treatment across types. Refer to the Appendix H for the detailed computation of this pseudo uniform approach and a formal explanation of how it achieves type uniformity. Table 5 shows the results. Comparing the pseudo-uniform uniform centering/whitee whitening (which assumes uniform prior over types) with the conventional token-level uniform centering/ centering/whiteniigg whitening (which implicitly assumes a Zipfian prior over types) reveals that the latter approach based on Zipfian prior empirically achieves better performance. Additional experimental settings and results can be found in Appendix B and Appendix I.
5.2 Headless causal language model roughly belongs to Zipfian prior family
The recently proposed headless language model [23] uses only words within the same batch to predict next tokens with a pseudo-softmax function. This method originally aimed to reduce the computational cost of the softmax function in the [V] direction, but an interesting side effect the improvement in the performance. This success can also be explained from the perspective of Zipfian priors. If we repeatedly sample small batches, the sampling frequency of each word will increasingly reflect its true frequency as the batch size approaches 1.
6 Conclusion
Standard methods for adjusting and measuring symmetries in word embedding spaces-Suee as cen- tering and whitening whitening-immlicitty assume uniformly distributed word frequencies. which is unrealistic. We hypothesize that, based on the type-token distinction, using empirical Zipfian word frequencies essential when calculating the expectation ($ 2). Based on the idea and the definitions of first- and second-order symmetry in random vectors, we derived Zipfian whitening, which enhances the symmetry of the word embedding space. Even though is nearly identical to standard PCA whitening, Zipfian whitening significantly outperforms existing methods ($ 3.2). Similarly, we derived metric to evaluate the symmetry of word embedding spaces. Our intrinsic metrics showed a strong correlation with extrinsic task performance, even when popular metrics show almost none ($ 3.3). We then presented a framework explaining the differences effect between whitening based on uniform and Zipfian approaches, by attributing them to differences in the base measure of the exponential family (S 4.1.. By further exploring this viewpoint through information geometry and loss functions, we showed how the Zipfian approach emphasizes the informativeness of low- owwfrequency words ($ 4.2.1). Lastly, through our proposed viewpoint, we found that popular NLP methods perform well because their word embeddings end up encoding Zipfian prior; such models include word2vec [37] 4.2.2), WhiteningBERG [26] (& 5. 5), and headless language models [23] ($ 5.2).
frequency distribution, resulting in the first and second moments (Def. 1, Def. 2) of sentence vectors closely matching those of word vectors.
10

---

Acknowledgements
This work is supported by JST ACT-X Grant Number JPMJAX200S and JSPS KAKENHI Grant Number 22H05 We received numerous constructive and valuable comments from the anonymous reviewers of NeurIPS 2024, which have significantly contributed to the quality improvements from the submission version to the camera-ready version. We would also like to thank Hayato Tsukagoshi of Nagoya University for his insightful comments the handling of dynamic embeddings and on the experimental setup of the SimCSE paper [22], including minor discrepancies between the paper's description and its actual implementation (also see Footnote 16). We also extend gratitude to the organizers and participants of MLSS 2024, the Tohoku NLP group, the Shimodaira lab at Kyoto University, and many members of the Japanese NLP and machine learning community, for their constructive feedback and motivating encouragement throughout our research discussions.
Limitations
How these assumptions might be violated in practice
In theoretical analysis concerning norms, and in the discussion on the relationship between whitening and normalization constants, we have proceeded by ignoring the residual terms beyond the second order. Empirically, focusing only on the first and second order has yielded significant results. However, to accurately identify cases where the proposed method might fail, a detailed theoretical and empirical examination of the asymptotic behavior of higher-order moments might be crucial. This remains an important future work.
The condition that the partition function is constant is only necessary condition from the perspective of both the generative model's optimal solution and whitening. The true logical relationship between whitening and the generative model has not been clarified. In particular, verifying whether the projection through whitening allows us to transition between the two model families (the uniform family and the Zipfian family) is an intriguing and valuable direction for both theoretical exploration and practical application.
The scope of the empirical claims made
Our experiments primarily focused static and dynamic word embeddings, many of their theoretical properties have been understood and they have been central to the rise of isotropization. Admittedly, this paper also advances understanding of causal language models. However, to make a more significant practical impact the era of large language models, employing the proposed method as regularization term for next-token prediction holds great promise for future work.
The experiments utilized typical downstream NLP tasks, particularly popular datasets for sentence- level semantic tasks. By scaling up the task set to include word-level tasks or leveraging a broader range of multilingual data, we can more robustly demonstrate the practical utility of the proposed framework.
The factors that influence the performance of our approach
The proposed method inherently involves numerically unstable calculations, such as multiplying by the inverse of small singular values. Embeddings for low-frequency words are often far from converged even after extensive pre-training, and the eigenvalues of the embedding space are known to decay. Given these situations, the adverse effects of small singular values are plausible. Considering recent advancements in whitening techniques, developing a more numerically stable algorithm an important direction for future work.
Broader Impacts
Potential impacts to AI alignment Dohmatob et al. [19] reported that repeated sampling from generative Als may shift word frequency distributions toward lighter-tailed distributions. This may reduce linguistic diversity and lead to cultural homogenization by diminishing region-specifie or
11

---

culturally unique expressions. Our Zipfian whitening and similar regularization methods could enhance output diversity, enriching the linguistic landscape.
Potential negative societal impacts The sentence similarity tasks used in our evaluation experiments are now considered core technologies for RAG (retrieval-augmented generation), which is essential when large language models leverage external resources. chatbots generate responses tailored to user ideologies or preferred information sources, it may result in negative societal impacts, including political agitation.
References
[1] E. Agirre, D. Cer, M. Diab, and Gonzalez-Agiree SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity. In E. Agirre. J. Bos, M. Diab, S. Manandhar, Y. Marton, and D. Yuret, editors. *SEM 2012: The First Joint Conference on Lexical and Computational Semanties- - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385-393, Montréal, Canada, 7-8 June 2012. Association for Computational Linguisties. URL https: //apps://aaa https////caaaaahloooeoo
org/S12-1055
[2] E. Agirre, D. Cer, M. Diab, A. Gonzalez-Agirre, and W. Guo. *SEM 2013 shared task: Semantic Textual Similarity. In M. Diab, T. Baldwin, and M. Baroni, editors, Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 32 43, Atlanta, Georgia, USA, June 2013. Association for Computational Linguistics. URL https: / acc www.hllnhooooy..oooooooooo Setg//S3-1004.
[3] E. Agine, C. Banea, C. Cardie, Cer, M. Diab, A. Gonzalez-Agirre, W. Guo, R. Mihalcea, G. Rigau, and J. Wiebe. SemEval-2014 Task 10: Multilingual Semantic Textual Similarity. In P. Nakov and T. Zesch, editors. Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 81-91, Dublin, Ireland, Aug. 2014, Association for Computational Linguisties. doi: 10.3155/v1/S1-22100 URL https:/ /a//aaaaaannnoooooyoooooo org/ 00////1120000
[4] F. Agirre, C. Banea, C. Cardie, D. Cer, M. Diah, A. Gonzalez-Agirre, W. Guo, I. Lopez-Gazpio, M. Mar- itxalar, R. Mihalcea, G. Rigau, I.. Uria, and J. Wiebe. SemFval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretabilliy. In Nakov, T. Zesch, D. Cer, and D. Jurgens, editors, Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 252-263, Denver. Colorado, June 2015. Association for Computational Linguistics. doi: 10. 118888/11/11--2220 URL https: battss://aclaaae aalaatooaannt org/S15- 2erg/S15-2455.
[5] E. Agirre, C. Banea, D. Cer, M. Diab, A. Gonzalez-Agirre, R. Mihalcea, G. Rigau, and J. Wiebe. SemEval- 2016 Task 1: Semantic Textual Similarity. Monolingual and Cross-Lingurd Evaluation. In S. Bethard, M. Carpuat, D. Cer, D. Jurgens, P. Nakov, and T. Zesch, editors, Proceedings of the 10th International Workshop Semantic Evaluation (SemEval-200 2016)). pages 497-511, 511, San Diego, California, June 2016. 1081, URL pttttp:://acaanthll..0
Association for Computational Linguistics. doi: 10.18653/v
org/S16- 1081.
[6] S. Arora, Li, Y. Liang. T. Ma, and A. Risteski. A Latent Variable Model Approach to PMI-based Word Embeddings. TACL, 4:385-399, 12 2016. ISSN 2307-387X. doi: (aa1162/ac((_aaa(((ooo) URL
https://aclwee org/ wwtgg//ttoooogypaprgrgg http://pppr/////////ppep
[7] S. Arora, Y. Liang, and T. Ma. A Simple but Tough-to-Beat Baseline for Sentence Embeddings. In ICLR, 2017. URL htt.hhttt///opeeeriiiiiie
net//oorumiid-SyX00555.
[8] A. Bakarov. A survey of word embeddings evaluation methods. arXiv [cs. CL.J, Jan. 2018.
[9] S. Bird and E. Loper. NLTK: The Natural Language Toolkit. Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214-217, 7 2004, URL htttp::::/ww..clllll acwweb..ww........ anthology/ P03-3031.
[10] D. Bis, M. Podkorytov, and X. Liu. Too much in common: Shifting of embeddings in transformer language models and its implications. pages 5117-503,, June 2021.
[11] P. Bojanowski, F. Grave, A. Joulin, and T. Mikolov. Enriching Word Vectors with Subword Information. doi: 10.1 1111166//c___000000
TACL, 5:135-146, 2017.
URL https: http://wwwaalllll..c aclweb. www.ww..oo///oooeeeeooo......lll argggnnholog///
papers/ //ppr////11/017-100000
[12] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herhert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, Sutskever, and D. Amodei. Language Models are Few-Shot Leamers. In NewrIPS, volume 33, pages 1801- 2020. URL https: htttt:::s/pprocedingg nour cc/paper_ Paapaaaeeeeeeaaapaaeaapea
fil files/ http///pr../.00///////////////..
12

---

[13] E. Bruni, G. Boleda, M. Baroni, and N.-K. Tran. Distributional Semantics in Technicolor. In ACL, pages
136-145, 7 2012. URL https://www.. aclweb... org/ heggaanthooloyyP11211000
[14] L. L. Camphell. Minimum coefficient rate for stationary random processes. Information and Control, 3(4): 360-371, Dec. 1960.
[15] D. Cer, M. Diah, F. Agirre, I. Lopez-Gazzio, and I.. Specia. SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. In SemEiaa, pages 1-14, 8 2017. doi: 10.18653/v/S11722000
URL htttps://ww. aclweb. hwaswwe..gg//ee. /ttpa/aooooopppppppe//////////// /t/pappprr/////7/7720001001
[16] X. Chen, N. Ding, T. Levinboim, and R. Soricut. Improving Text Generation Evaluation with Batch Centering and Tempered Word Mover Distance. In First Workshop on Evaluation and Comparison of NLP Syxtems, pages 51-59, Online, 11 2020. Association for Computational Linguistics. doi: 10. 10 2020.eval4nlp- 1.6. URL htwph:ttt//t/..tt //wp:///wwwwwaa/ http:/wwwwwww.b.b/////w/w//ww aclweb. org/ a/taaaaottoll/////////ae wtt.:......1ll0111... eval4nlp-1..6
[17] R. R. Coifman and M. V. Wickerhauser. Entropy-based algorithms for best basis selection. IEFE Trans. Inf. Theory, 38(2):133-718, Mar. 1992.
[18] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL. pages 4171-4186, 2019. doi: 10.18653/v1/N19- 1423. URL
attps://wwet aclweb. org/ http///hooooo//1-19//009
[19] E. Dohmatob, Y. Feng, P. Yang. Charton, and J. Kempe. A tale of tails: Model collapse as change of scaling laws. arXiv [cs.LGG] Feb. 2024.
[20] K. Ethayarajh. Unsupervised Random Walk Sentence Embeddings: Strong hut Simple Base- line. In Rep4NI.P, pages 91-100, 2018. URL https: http://awwleebo..oaaaeeee http//antooogggggprr////// w18-3012/.
[21] K. Ethayarajh. How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. In EMNLP, pages 65,65, 2019. URL http:://rxix org/abs/ 1909. 00512.
[22] T. Gao, X. Yao, and D. Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.
[23] N. Godey, É. V. de La Clergerie, and B. Sagot, Headless language models: Learning without predicting with contrastive weight tying. arXiv [cs. CLJ, Sept. 2023,
[24] C. Gong. D. He, Tan. T. Qin, L. Wang. and T.-Y. Liu. FRAGE: Frequency-agnostic word representation. Adv. Neural Inf. Process. Syst., 31, 2018.
[25] P. He, X. Liu, J. Gao, and W. Chen. (DEBERTA): (DECODING) (hert) (with) (disentan- wtaet//orrnnidd--Ptteee
gled} [attention]. In ICLR, 2021. URL https: https::/ wpwp/evieeeeeeeeeeeeeeeeee
[26] J. Huang, D. Tang. W. Zhong, S. Lu,, I.. Shou, M. Gong, D) Jiang, and N. Duan. WhiteningBERT: An easy unsupervised sentence embedding approach. In M.-F. Moens, X. Huang, L. Specia, and S. W.-T. Yih, editors, Findings of the Association for Computational Linguistics: EMNLP 2021, pages 238-244, Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.
[27] R. Kannan, L. Lovász, and M. Simonovits. Random walks and o*(n5) volume algorithm for convex bodies. Random Struct. Algorithms, 11(1):1-50, Aug. 1997.
[28] G. Kobayashi, T. Kuribayashi, S. Yokni, and K. Inui. Transformer language models handle word frequency in prediction head. pages 4523-4535, July 2023.
[29] K. Kurihara, D. Kawahara, and T. Shibata. JGLUE: Japanose general language understanding evaluation. In N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck. S. Goggi. H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, and S. Piperidis, editors, Proceedings of the Thirteenth Language Re- sources and Evaluation Conference, pages 2957-2966, Marseille, France, June 2022. European Language Resources Association. URL https: http::::///caannholloooo anthology. org/ 2022.022.2222 1.ree-1.311
[30] H. Kurita, G. Kobayashi, S. Yokoi, and K. Inui. Contrastive teaming-based sentence encoders implicitly weight informative words. In Bouamor, J. Pino, and K. Bali, editors. Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10947, Singapore, Dec. 2023, Association for Computational Linguistics.
[31] Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, Eytan Ruppin. Placing search in context: the concept revisited. ACM Trans. Inf. Syst. Secur, 20(1):116-1311 Jan. 2002.
[32] O. Levy and Goldberg. Neural Word Embedding as Implicit Matrix Factoriza- tion. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Wein- berger, editors, NIPS, pages 2177-2188 2014. URL http://papers. www..iinnnn....oeoieoo cc/paper/ 5477-neural woroourr--oodd---e word-enbedding as- wtwpaiiitttmatii-afatttooooooeoe
13

---

[33] Y. Liu, M. Ou, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis. L. Zettlemoyer and V. Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv [cx. CLJ, July 2019.
[34] H. P. Luhn. The automatic creation of literature abstracts. IBM J. Res. Dev. 2(2):199-16) Apr. 1958.
[35] M. Marelli, S. Menini, M. Baroni, L.. Bentivogli, R. Bernardi, and R. Zamparelli. SICK cure for the evaluation of compositional distributional semantic models. In LREC, pages 216-223, Reykjavik, Iceland,
5 2014. URI. http://wwe: -rec-cont org/ wwrggp/occeddnge// wewpeeeddinn///reeeeeee http://reeee111////1 http//dfff//// ///3//333PPPp___ .dff.
[36] A. K. Menon, S. Jayasumana, A. S. Rawat, H. Jain, A. Veit, and S. Kumar. Long-tail learning via logit adjustment. Sept. 2020.
[37] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean. Distributed Representations of Words and Phrases and their Compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, NIPS, pages 3111-3119, 2013. URL http://papers..eeee///////// nips cc/paper/ words www.ooooooo--ddoo-ooeeeede-----e and-phrases-anne and- tonaddthhi--ccee www.eh-ccooositiiooettiioi composttionalityy.....iiiiiia
[38] D. Mimno and L. Thompson. The strange geometry of skip-gram with negative sampling. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pages Copenhagen, Denmark, Sept. 2017. Association for Computational I Linguistics.
[39] J. Mu and P. Viswanath. All-but- the- Top: Simple and Effective Postprocessing for Word Representations. In ICLR, 2018. URL https: http:////ppppeeeeeeeeeee wwwpeeewwwwwweeeeeeeeeeeeeeeee
[40] N. Muennighoff, N. Tazi, L. Magne, and N. Reimers. MTEB: Massive text embedding benchmark. In EACL, pages 2014-20242 2037, Dubrovnik, Croatia, 5 2023. Association for Computational Linguisties. doi: org
(0.18653/v1/2023.oo cac.223.cccmmmmiim www.-maicllaaaaaaaamaaaaaa 148. URL httptttttcclaaaneee /atpa/aaaaaannn...aaaaaaaooollle org/ wrgg/023...o..oooooo eacl-alllll. maal-mmann..... 148.
[41] K. Oono and T. Suzuki. Graph neural networks exponentially lose expressive power for node classification. Sept. 2019.
[42] M. Oyama, S. Yokoi, and H. Shimodaira. Norm of word embedding encodes information gain. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 2108-2130, Singapore, Dec. 2023. Association for Computational Linguistics.
[43] J. Pennington, R. Socher, and C. D. Manning. Glo Global Vectors for Word Representation. In EMNLP, pages 1532-1543, 2014. doi: 10.31 URL https:///ccleee httpe/lwwww.rr/rraeeeeeeeeeeeeee atg//attoooog//ppprss/ D/D14/D14-115 1162/.
[44] I. Provilkov, D. Emelianenko, and F. Voita. BPF-dropout: Simple and effective subword regularization. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882-1892, Online, July 2020. Association for Computational Linguistics.
[45] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language Models are Unsuper- vised Multitask Learners Technical report, 2019. URL httpp::///Amccfpteeeee http:///dducckkkeww...ooddddoort cloudfront net/ better- htw.--aannaagemdmeesss/-eaee Sanellssaaagggeee.mm
[46] N. Reimers and Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In EMNLP, pages 3980-3990, 11 2019. doi: 11181///1/DD11///11 URL https: //:/:///wwww aclueb. org/ anthology/D19
[47] O. Roy and M. Vetterli. The effective rank: A measure of effective dimensionality, pages 606-610 610, Sept. 2007.
[48] M. Rudelson. Random Vectors in the Isotropic Position. Journal of Functional Analysis, 164(1):60-72, 1999. ISSN 0022-1236. doi: https://doi.orr/110 wttp////....///aann.ooommm (006//fan.199..333. URL http://www..cciscceiirc... http//cienee/rrticcoo httpe//ppppp///eeeeeeeee
[49] W. Rudman, N. Gillman, T. Rayne, and C. Eickhoff. IsoScore: Mcasuring the uniformity of embedding space utilization. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Findings of the Association for Computational Linguistics: ACI. 2022, pages 3325-3339, Dublin, Ireland, May 2022. Association for Computational Linguisties.
[50] A. M. J. Schakel and B. J. Wilson. Measuring word significance using distributed representations of words. arXiv [cs. CLJ. Aug. 2015.
[cs.CL].
[51] H. Shi, J. Gao, H. Xu, X. Liang, Z. Li, L. Kong, S. M. S. Lee, and J. Kwok. Revisiting over-smoothing in BERT from the perspective of graph. Sept. 2021.
[52] H. Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):27-224,, 2000.
[53] K. Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. In Document retrieval systems, pages 132-145 142. Taylor Graham Publishing, GBR, Dec. 1988.
14

---

[54] H. Touvton, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro.
F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. LLaMA: Open and Efficient Foundation Language Models, wVXV:2230.1337/wv 2023. URL https://arxiv. htttpe://arxiv..rrr/bbb/////
[55] R. Vershynin. High-Dimgnnn Dimensionnl Probability. Cambridge University Press, 2018
[56] L. Wang, J. Huang, Huang. Hu, G. Wang, Q. Gu. Improving Neural Language Generation with Spectrum Control. In ICLR, 2020. URL htttps://ppnnrevie not./oorumn.dd-ByaYBttlt
[57] L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang. D. Jiang, R. Majumder, and F. Wei. Text Embeddings by Weakly-Supervisee Contrastive Pre-training. arXiv:2212.0353332, 2024, URL https://arxiv.
03533v2.
[58] L. Wetzel. Types and tokens. The Stanford Encyclopedia of Philosophy (Fall 2018 Edition), Apr. 2006.
[59] Y. Yan, R. Li, S. Wang, Zhang, Wu, and W. Xu. ConSERT: contrastive framework for self- supervised sentence representation transfer. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, ACL-IJCNLP, pages 5065-5075, Online, 2021. Association for Computational Linguisties. doi: 10.1866 1/283/11122221. acl-Iong.393. URL pettp::/accaathoo//
org/2021. acl-long..33 393.
[60] S. Yokoi, R. Takahashi, R. Akama, J. Suzuki, and K. Inui. Word Rotator's Distance. In EMNLP, pages 2944- (00/////12020.emm//
2960, Online, 11 2020. Association for Computational Linguistics. doi: 18.18653/v
URL https: //::///w /wwp//wwww.co aclveb. org/ he//anthology/2222222 cmnlp-main.20
15

---

A Explanation fo the Zipfian whitened word vectors will have a zero mean and be in an isotropic position in terms of expectation
The second step of PCA whitening involves decorrelating the dimensions and then normalizing them, which is achieved by transforming the centered random vector uo as follows:
E W
w
1//2w.
(22)
Actually, E. ur " = La holds; i satisfies Def. 2. Computationally. the estimation of E/ w T can be performed efficiently via singular value decomposition (SVD) of the centered "data" matrix W wi ..... w IT. Note again that, this standard method assumes that the frequency of each word (i.e., each row) is uniform, which presents the issues discussed in $ 2. To account for word frequency, SVD should be performed on the matrix
Wr p Vp(wi)w] ..... (
TR IR/
(23)
where each word frequency is multiplied by its square root. In fact, W W, serves as an estimator for IE [ we This can be confirmed by comparing the (j, k)'th elements of each matrix (or w-p
matrix- valued random variable):
E
2 w
[[, k]
E mmp
wr w [A]]]
(24)
pl = Σ (m w wi(ki [k).
(25)
where Alj. k] denotes the (j, k) 'hth element of A, and vīj] denotes the j'th element of v. Finally, the estimator for the desired E| u w 1-1/2 can be computed as
W p UEV
(via SVD)
(26)
E m 1 wmpp
W
W p
-(VV'VT-- VE
-1
(27)
Table 6 shows the correspondence between uniform (normal) whitening and Zipfian whitening. This may be useful for readers familiar with matrix notation.
B Experimental settings
To ensure the reproducibility of the experiments conducted in this paper, we provide detailed configurations below. Additionally, the source code has been made publicly available at https: //github. com/ com/clltohokuu zipfiaipfiaaaaa an-iann-ann whi teitennng.
B.1 Word embeddings
For static embeddings, we used the most standard ones, 300-dim Glo43[43] model trained on Common Craw1 300-dim word2vec[37] model trained Google News". and 300-dim fastText[ subword/non-subbword models trained on Common Cr For the multilingual experiment, we used Festaa [11], fastText model trained Japanese Wikipedia and Common Crawl 11.
300d from Tentennes implementation [46).
from
https:
/ //code. google. cog.lle..cc/arrahavee/////veee//
16

---

Table 6: The correspondence between uniform (normal) whitening and Zipfian whitening.
Uniform whitening for general table data
Zipfian whitening for word embeddings
data matrix, original vectors
X
Zi
W
Wi
E Rxxd
probability
x ~ Uniform dist.
w ~ Power-law dist.
p(ar.)
pl ums = p(w)) (word freq.)
mean vector
P
p(ws)aw
E R
centered matrix, centered vectors
di
Px
w.
wi
Pu
E Radd
centered matrix to create cov. mat.
X
1/(nn
1)
W
VPww)) w
E RRdd
8 = x X
covariance matrix
X X
S= W W
E Rdxd
n 1
SVD of centered matrix
X = UEV
W UEV
Σ diag(o (01, .35)
= diag(oi....,.0
= cigendecompossitin of covariance matrix
8 X X
VAV
S W, W, VAVT
n
A (liag()....... Aa)
A diag(A1..... dd)
diag
= Σ = diag (of,...)
n
A A 1/2
diag
A 1/2 diag
V
diag
diag
of X = XVA
whitened matrix
1/2
W WVA 1/2
whitened vector
x. E. VA 1/2
w.
VA 1/2
For dynamic embeddings, we used three most standard masked language models, BER RoBERTa[(3]*3, and DeBERTa [25]'4, All three models base size. To aggregate the dynamic word embeddings to create sentence embeddings, we follow the first-last average pooling from the prior work [22]. In this setting, first average the hidden states of first and last dynamic layer of
17

---

the model to get the averaged token embeddings then average the token embeddings to get final sentence embeddings
B.2 Empirical word frequency and vocabulary
As the empirical word probability p(w) of English words, we used the enwiki dataset preprocessed by Arora et al. [7]"7. For the Japanese word probability, we used Japanese Wikipedia word frequency from Wiktionary, denoted as jawiki 18, Furthermore, we also used the frequency of words in the evaluation data itself (test set probability). The word frequency in the test set implicitly utilized in [7]'s sentence embedding method and is also natural approach in the context of "covariate shift" [52].
As vocabulary V, we used the overlapping entries between the word frequency list and the pre-trained word embedding model's vocabulary across all settings, including baseline methods.
B.3 Baseline methods
As baselines for post-proces pust-processing of word vectors, we used ABTT (all-but-the-opp) [39], which established the trend of post-processing word vectors; and the strong baseline method by [7], the com- bination of SIF (smoothed inverse frequency) weighting and CCR (common component removal) We followed the hyperparameter choices of the original papers, with the dimensionality reduction parameter for ABTT set to D= = 3, and the weighting parameter for SIF set to == 10
B.4 Extrinsic tasks
As downstream tasks, we used the most commonly utilized ones in the community, STS12-16 [1, 2. 3, 4, 51, STS-B [15] and SICK-R [35]. For the multilingual experiments, we used JSTS (Japanese version of the STS) from JGLUE benchmark [29]. They are sentence-level similarity tasks and are standard for empirically evaluating the performance of word vectors 20 21, These datasets consist of pairs of sentences and their semantic similarity rated by annotators. We first tokenized the dataset
15 Note that, we apply centering/whitennng operations to such token embeddings, not to the final sentence embeddings, in order to match the setting in the theoretical analysis and the static word embedding experiments. 16 Though we followed the experimental setting from the prior work [22], there a slight discrepancy in the experimental results of the baseline setting. We found that this was due to prior work inadvertently taking the average of the hidden states of the zero-th layer (i.e., static word embedding layer) and the final dynamic layer. See the discussion https: //pp:://giie hhtpgttthuuu.... wwtp://innetoooonnplpl.pp..ppp/ http://aaaaatttaaaaaiaiiie/aatte 17 "https:/ http:////aeeeee/////////aaaaaaaa http:///wwwlllaaaa.aaaa////aaaea http///wwiikiiiooieeeeeeeeaeeeea latpimmiin200000 txt
for more details.
https: //ep. wiktiiiaarr htt.a//wwr....egaaatattaaaa/aaea
and
19CR is process applied sentence vectors, but due to its linearity, it can be adapted to word vectors. For more details, please refer to Yokoi et al. [60].
For those outside the field of NLP research or practice, the question, "Why not run word-level evaluation metrics?" is natural and valid one. Our language has property known compositionality allowing infinite semantic content be conveyed through finite vocabulary as building blocks. This principle underlies models like word2vec [37], BERT [18], and the GPT series [12], where the fundamental unit of representation is the word; these models are then applicd to solve tasks with larger components, such as sentences. Our rescarch adheres to this foundational principle of NLP. Also, existing word-level similarity datasets have significant issues that make them less suitable for work (see Bakarov [8. Section 1.). Given that whitening reflects word information content in vector norms, classic tasks like keyword extraction which selects words with high information content coule be good candidates; results from a prior study using methods similar to ours would also be informative [42, Section 7.1].
21 "etttnng aside the criticisms from previous studies for now, we conducted an evaluation using the two most well-know lcxical similarity datasets. Table 7 shows the results. We found that the process of raw + Zipfian centering- + Zipfian whitening consistently improves lexical properties. However, note that the finding "direction: uniform whitening direction: Zipfun whitening" contradicts the experimental results in Appendix G. which showed "direction: wniform whitening, norm: Zipfian whitening direction: Zipfian whitening, norm: Zipfian whitening." " Here, lexical similarity tasks rely solely on vector direction and do not reference vector norms, as only the cosine similarity between word vectors used to predict similarity. This discrepancy likely arises because these datasets are not representative of natural language, as discussed in Bakarov [8, Section 4.1.1]. For example, the widely used dataset WordSim353 [31] includes only about 200 subjective ratings on
18

---

by NLTK [9] with some protessoosssing following [20]"² then lowercased all tokens. The typical experimental protocol we followed sum the word vectors form a "sentence vector" and then check if the angles (cosine similarity) between them correlate well with the gold scores. We reported Spearman's 's rank correlation between the predictions (cosine scores) and human-annotated gold scores
B.5 Computational resources for experiments
We conducted all experiments using single NVIDIA RTX 6000 Ada GPU with 48GB VRAM. Each STS task required 10 seconds per model and whitening method, totaling approximately 10 minutes for the entire experiment, excluding the embedding loading time to the GPU.
For the calculation of the symmetry scores, each setting took minute, resulting in total of minutes, again excluding the embedding loading time and the average cosine similarity (Ave. Cos.) setting. The Ave. Cos. score computation took 10 minutes per model, totaling 20 minutes for the two models.
C Experimental results on all benchmark datasets to evaluate the effects of Zipfian whitening
In $ 3.2, we evaluated the empirical performance of Zipfian whitening on the STS-B dataset. this section, we present experimental results using more comprehensive datasets. Detailed experimental settings can be found in Appendix B. Table 8, Table 9 and Table 10 show the results. Across all datasets, the method incorporating Zipfian prior consistently outperforms the method employing uniform prior.
D Proof of Prop. 1
Proof. We will show the following.
Bi(v- R(())( - [(w)) ] x Ia
(28)
\ = d2 ... Ad
(29)
A
Symg(v)= log E, Σ, 23
(30)
When E[(v - (u B[o)) kI holds, its cigenvalues are [k..... k}.
Since B[(v [l - B[a)) is symmetric positive definite, it can be represented as using diagonal matrix with eigenvalues A = diag(Ai,...., Ad) E dd and an orthogonal
UAU
common nouns, such as (tiger, cat, 7.35) (king, cahbage, 0.23), which may or may not co-occur in the document.
Table 7: Each cell shows the correlation coefficients 100 between the cosine similarity of (corrected) GloVe embeddings and the human-annotated gold score lexical similarity tasks.
WordSim353 [31]
MEN [13]
GloVe
78.70
GloVe
80.49
Uniform
Zipfian
Uniform
Zipfian
Centering
75.39
79.66
Centering
78.07
80.55
+ Whitening
82.31
80.90
Whitening
84.35
83.97
usif wyw1111111111 L126
http://ggtt.b.ccmommmmedddeeeeee the evaluation of the static word embeddings in Table 2. Table 8, and Table 9. For the evaluation the dynamic word embeddings in Table 5 and Table 12. we used the implementation in SimCSE paper [22]: https:// //ge::///iiii fgittuu..... wwwmppinceeonn nlp/ SimCSE, match the experimental setting.
"We used the MTEB [40] implementation:
for
19

---

matrix U. Now we have A1 = X2 _..... Xa = k. then Ei( Blu)) - E(b)) UAU
U = aI
The Shannon entropy (p) of random variable p taking possible values attains its maximum value log d if and only if follows uniform distribution.
E Pseudocode for the evaluation metrics of symmetry
See Algorithm 2 to measure the degree of symmetry of word embeddings.
Algorithm 2 Measure the degree of symmetry of word embeddings
Input: Word embeddings {wi}. word frequency p: V - [0, 1].
Output: Degree of centrality (the 1st moment) Sym, ((w.},p) p) and isotropy (the 2nd moment) Syma(!w:))
Measure the degree of centrality (the 1st moment of symmetry):
1: Ā
Σ
p(w,)w Rd
((w)) will R
WIEV
3: Sym, ((un)(( + 1- 11
Measure the degree of isotropy (the 2nd moment of symmetry):
4: W +
ppwi) (w) AT)
5: UEV SVD(W)
Σ diag(o 1..... Od) Rdxdd consists the singular values of W.
6: (A1 (A1,..., Xd) (op: ..... 0
7: Symy(fw;;,,)) + log E/ A
Σ
d log
x E, A
F Proof of Thm.
Proof. By the assumption, the word and context vectors for the same word V, w(t) and c(t). are obtained through the linear embedding layer with weight tying, namely, w(f) U1, and c(l) U1,, where U E [ddx/V] is the embedding matrix and 1, RPI is the one-hot vector indicating the token I. To derive the KL. divergence for model p(e ww), we need to begin with the generative model p(w) c) (7) and confirm that p(c w) belongs to exponential family.
p(c w) p(w] c)p(c) exp((w, c)) p(c) p(c) exp((U w. 1.) (o)()
p(w)
Zo
where we used the constancy of the partition function (= Z from (9) at the last identity. Hence,
p(c w) is an exponential family parametrized by U w(: E RIV)). and its log- nog-partition function given follows:
Veju ( log Σ p(c) exp( 1c))
The second-order expansion of the KI. divergence can be derived based on the second moment of p(c w), which is given by the Hessian of dlou in the case of exponential families. First, let derive
20

---

the first moment.
Le p(c)
exp 1c)
2
p(c) exp((0. 16)
1. Σ
p(c
w)lc
ple w)
00
pic'l exp( (0, 1c)))
20
CEV
Then, the second moment is derived.
Σ p(c) { a
exp( 10)) 1
Σ Z
(o))
0000
T
2
00
CEV
CEN
Eple | w)1e1e = diagl... p(c w) ...];
CEV
which is the [V] VI diagonal matrix with p(c w) being the (c. c)-th diagonal entry. Now, we are ready to derive the KL divergence. For tokens w, w' E [V/, , we write 0 = Ulw and 0' = Ulw', the KI. divergence of the exponential family can be expanded as the following quadratic
form in their parameters and 0':
@rbelw
2KT.(p(.1 w/ Ilp( w)) (0' - -0)
(0' 00)
0000
(w' w) T
U (w' w)
0000
-ww' un) T
Σ
p(e
(U(. T (1o' - w)
cEIV
Σ p(c w)ec } (wo' (1o'-w) -w)
-ww' w)
(w' w) w) G(w)((()' w)
1|w' w wl(c(u)
We can consider a word wo such that p(-) = p(- wo), that is, uninformative word wp whose presence does not change the marginal distribution at all. Noting from Equation (22) of Oyama et al. [42] that w Eev p(w) s wo, we have
KL(p(-)lp( | w)) = (L (o) lpp( w))
(31)
We w (l2
(32)
~
w (()w
(33)
Assump.
|r ((()))
(34)
n
G Ablation Study: Effects of Zipfian Whitening on Norm and Direction
In the main text, we demonstrated that applying Zipfian whitening to word vectors consistently improves downstream task performance ($ 3.2), and theoretical and empirical evidence that Zipfian whitening has beneficial impact on word vector norms ($ 4.2.1). Building on these findings, some readers may be interested in how the Zipfian whitening process influences the lengths (norms) and directions of word vectors individually. In this section, present an ablation study to analyze the separate effects of Zipfian whitening on the lengths and directions of word vectors.
Settings
The experimental settings here are identical to those used in the experiments presented so far fam-lamely, those for Table (excerpted results, shown $ 3.2) and Table (comprehensive results, shown in Appendix C). For more detailed settings, refer to Appendix B.
21

---

Results
Table 11 presents the results of the ablation study. Here, "w/ Z.F. norm" refers to the process of "correcting word vectors under the uniform prior and then replacing only the norm with that obtained from Zipfian whitening (Z.F.." Similarly, "w/ Z.F. direction" refers to "correcting them under the uniform prior and then replacing only the direction with that obtained from Z.F." These experiments allow to isolate and examine the individual effects of Zipfian whitening vector norms and directions. Here are some interesting findings from the experimental results.
First, replacing either the norm or the direction with its Zipfian whitening (Z.F.) version post hoc generally improves downstream task performance (see the "Ave." columns). This indicates that both the norm, as detailed in 4.2.1, and the direction, which governs all degrees of freedom except for length, transition to a better-conditioned state through Zipfian whitening.
Next, regarding the relative contributions of norm and direction, we observe that the norm despii being mne-dimensional- has greater impact on task performance than the direction. The reasons behind this, as well as theoretical insights related to isotropy and vector direction (or the mathematical understanding of cosine similarity as conventional metric), remain open questions for future research.
Finally, it is worth noting that applying pure Zipfian norm and direction-yielo the best average performance. There no apparent trade-off here, suggesting that, from practical standpoint, simply applying Zipfian whitening is the most effective approach.
its full effect on both
H Formal Explanation of "Uniform whitening of token embeddings Zipfian whitening of type embeddings"
In this section, we provide more formal explanation of "Uniform whitening of token embeddings = Zipfian whitening of type embeddings." as described in $ 5.1. For intuitive explanations and related discussions, please refer to 5.1.
H.1 Uniform whitening of token embeddings x Zipfian whitening of type embeddings
Assume that, when the type word of a token t w. the token embedding t aligns with the shared type embedding
Assumption 1. If typeett) = w, then t w..
Note that this is a rough approximation, token embeddings are dynamic and vary with context. Under this assumption, the unweighted of token emheddings Ro]tt obtained from dataset 'D is asymptotically equivalent Eg w), as |D] + DC:
pend-frequeney-weihhhe (Zipfian) average of type embeddings
t
Assump.
Σ p(w) =: -FOlu))
(35)
D
D
WEV
where Cp denotes the count of type w in D: (D(w) = = # D: ::tpeet)) = =.
H.2 Pseudo-uniform whitening of token embeddings uniform whitening of type embeddings
To establish a baseline for centering/whitening token embeddings under uniform prior, we can apply coefficient 1/1/a1 (tpe(()))) each token embedding for removing type frequencies that implicitly referenced. Here, Vo denotes the vocabulary contained in D: Vo = {w E V : 31 e D, type(t) = w). The "pseudo-uniform" average E [t] calculated in this way is asymptotically equivalent to the uniform average of type embeddings 1 jw), under the previous assumption
22

---

(Assump. 1) that ignores the dynamic nature of token embeddings:
1 ol
Assumpp
Σ
cot PD
100000
1
LED V/D (D((ep((()
R3
V
Cofo
w
V
w
WEVD
WEV
(36)
I Experimental results on all benchmark datasets to evaluate the effects of uniform whitening on token embeddings
In 5.1, we evaluated the empirical performance of uniform whitening of dynamic token embeddings the STS-B dataset. In this section, present experimental results using comprehensive datasets. Table 12 shows the results. Across all datasets, the methods implicitly incorporating Zipfian prior consistently outperforms the method employing uniform prior.
23

---

Table 8: Full results of the empirical performance of Zipfian whitening. Each cell shows Spearman's 's P X 100. As empirical word frequency p(w), we used enwiki. Across all models and tasks. Zipfian whitening outperforms powerful baseline methods.
Method
STS12 STS13
STS14 STSIS STS16 SICK-R STS-B
Avg.
GloVe
(raw)
56.46
50.41
51.13
58.60
49.03
57.01
46.17
52.69
Centering
55.54
46.32
49.67
56.03
46.90
56.44
45.17
50.87
Uniform
Whitening
53.31
62.45
57.93
68.68
58.69
$7.92
52.21
58.74
Centering
54.52
69.20
60.87
69.82
62.61
58.01
52.25
61.04
Zipfian
Whitening
57.76
72.22
67.04
76.80
71.72
61.80
66.92
67.75
ABTT SIF CCR
52.67
67.38
59.40
69.53
60.71
58.56
54.28
60.36
60.23
68.78
62.39
67.26
61.85
56.91
58.70
62.30
Word2 Vec
(raw)
58.57
68.64
63.65
71.73
61.79
61.77
56.98
63.30
Centering
58.17
67.34
62.19
70.15
59.60
61.39
55.85
62.10
Uniform
Whitening
56.53
66.95
62.77
72.42
61.05
62.74
56.03
62.64
Centering
56.89
69.95
65.08
73.91
65.71
62.18
58.84
64.65
Zipfian
Whitening
56.16
70.33
67.20
76.60
70.99
62.52
66.50
67.19
ABTT SIF CCR
55.53
69.32
63.13
72.25
60.98
62.02
56.98
62.89
60.05
73.26
66.87
74.32
67.64
59.22
63.04
66.34
fastText
(raw)
$7.94
68.97
62.37
72.26
63.59
$9.82
63.56
Centering
59.73
55.02
55.16
64.22
53.39
58.85
$2.46
56.98
Uniform
Whitening
52.47
59.01
53.90
65.33
52.61
58.34
48.60
55.75
Centering
58.30
71.69
64.57
74.10
67.59
60.75
59.40
65.20
Zipfian
Whitening
58.86
73.85
68.43
78.07
74.00
62.85
69.55
69.37
ABTT SIF + CCR
58.35
69.09
60.82
71.99
60.76
60.34
$7.02
62.62
61.54
76.95
68.39
76.98
70.27
59.52
67.08
68.67
fastText-
(raw)
49.10
47.34
51.94
61.99
51.54
53.60
50.43
$2.28
Centering
49.21
43.13
49.89
62.03
49.70
54.56
46.91
50.78
Uniform
Whitening
45.12
41.00
47.30
62.08
48.85
54.80
43.55
48.96
Centering
48.68
55.03
54.07
60.23
58.41
54.64
50.38
54.49
Zipfian
Whitening
61.22
60.68
63.18
73.59
69.87
68.20
65.22
ABTT
49.64
41.79
48.81
60.84
47.57
$5.09
44.23
49.71
SIF CCR
57.28
54.50
60.77
68.82
61.63
56.83
60.36
60.03
24

---

Table 9: Full results of the empirical performance of Zipfian whitening, test set frequency setting. Each cell shows Spearman's 's p x 100, As empirical word frequency p(w), we used test set frequency. Across all models and tasks, Zipfian whitening outperforms powerful baseline methods. Besides, the test set frequency setting consistently outperforms the enwiki setting in Table 8, demonstrating that the models benefit from using task-specific statisties in line with covariate shift approach [52].
Method
STS12 STS13
STS14 STS15
STS16 SICK-R STS-B
Avg.
GloVe
(raw)
57.71
50.29
50.61
58.38
48.76
56.76
46.22
52.67
Centering
56.32
61.17
52.68
64.80
55.80
57.98
47.94
56.67
Uniform
Whitening
51.67
60.94
57.14
70.09
63.08
55.14
53.16
58.74
Centering
50.69
70.66
61.59
70.19
68.25
60.03
56.64
62.58
Zipfian
Whitening
61.63
78.36
69.48
76.83
74.08
60.11
71.60
70.30
ABTT
52.93
66.93
60.10
71.93
63.12
58.23
53.72
60.99
Word2
(raw)
59.00
68.92
63.99
72.51
62.25
61.87
$7.15
63.67
Centering 57.88
70.34
64.24
74.71
65.57
62.47
58.09
64.76
Uniform
Whitening
58.45
69.42
65,46
76.43
67.78
62.87
60.85
65.89
Centering
55.02
71.47
65,81
74.36
69.52
62.92
61.02
65.73
Zipfian
Whitening
59.37
76.92
76.42
73.56
60.07
70.42
69.46
ABTT
56.33
70.42
64.71
74.74
65.19
62.55
58.21
64.59
fasiText
(raw)
58.23
69.36
62.89
73.09
64.25
60.22
60.27
64.04
Centering
60.60
69.51
61.09
73.92
64.49
61.14
57.42
64.02
Uniform
Whitening
55.56
63.51
57.73
70.68
62.40
57.93
54.65
60.35
Centering
55.92
73.36
65.72
74.12
72.18
62.30
62.95
66.65
Zipfian
Whitening
62.20
79.35
71.03
77.95
76.28
60.66
73.56
71.58
ABTT
59.13
71.00
63.30
74.80
65.96
61.69
58.23
64.87
fastText-subworl
(raw)
51.37
51.49
54.57
62.75
52.97
53.53
52.41
54.16
Centering
51.31
44.80
49.66
62.27
47.43
$4.86
43.12
50.49
Uniform
Whitening
51.52
49.33
53.51
68.28
58.34
56.94
51.69
55.66
Centering
43.15
53.40
53.67
63.05
59.09
56.57
47.16
53.73
Zipfian
Whitening
60.87
72.21
67.79
75.86
73.88
60.52
70.99
68.87
ABTT
49.06
45.16
49.57
62.14
50.75
55.49
44.53
50.96
Table 10: Evaluation results using Japanese fastText. Each cell shows the JSTS [29] score x 1000 Even in the multilingual setting, Zipfian whitening outperforms powerful baseline methods.
(a) With jawiki p(w)
(b) With test set frequency as p(w)
fastText-ja
55.81
fastText-ja
59.94
Uniform 56.05 55.53
Zipfian
Uniform 59.89
Zipfian 63.05
+ Centering
57.55
Centering
+ Whitening
65.56
+ Whitening
61.75
69.86
+ ABTT
57.14
+ ABTT
63.02
+ SIF + CCR
61.03
25

---

Table 11: Ablation study on the effects of Zipfian whitening on norm and direction. Here, "w/ Z.F. norm" refers to the process of "correcting word vectors under the uniform prior and then replacing only the norm with that obtained from Zipfian whitening (Z.F.)." Similarly, "w/ Z.F. direction" refers to "correcting them under the uniform prior and then replacing only the direction with that obtained from Z.F." Each cell shows Spearman's 's 100. Except for the "w/ Z.F. norm" and "w/ Z.F. direction" settings, the values are simply copied from Table 8. The results indicate that both the norm and the direction transition to better-conditioned state through Zipfian whitening.
Method
STS12 STS13 STS14 STS15 STS16 SICK-R STS-B Avg.
GloVe
(raw)
56.46
50.41
51.13
58.60 38.60
49.03
57.01
46.17
52.69
w/ Z.F. norm
62.08
66.62
59.34
65.93
58.57
57.52
55.43
60.78
w/ Z.F. direction
49.66
64.15
61.73
71.89
66.37
59.52
57.20
61.50
Centering
55.54
46.32
49.67
56.03
46.90
56.44
45.17
50.87
Uniform
Whitening
53.31
62.45
57.93
68.68
58.69
57.92
52.21
58.74
Centering w/ Z.F. norm
61.41
63.94
58.33
63.88
56.90
56.39
54.62
59.35
Uniform
Whitening w/ Z.F. norm
60.20
73.63
64.98
74.21
66.67
60.53
61.55
65.97
Centering w/ 7.F. direction
47.12
61.14
59.42
70.10
63.46
58.58
53.87
59.10
Uniform
Whitening w/ 7.F. direction
49.57
65.00
62.17
72.24
67.83
58.22
57.73
61.82
Centering Whitening
54.52
69.20
60.87
69.82
62.61
58.01
52.25
61.04
Zipfian
57.76
72.22
67.04
76.80
71.72
61.80
66.92
67.75
Word2 Vec
(raw)
58.57
68.64
63.65
71.73
61.79
61.77
56.98
63.30
w/ Z.F. norm
58.87
71.96
65.52
73.55
66.61
61.05
61.59
65.59
w/ Z.F. direction
55.45
68.07
66.49
75.98
68.46
63.41
63.25
65.87
Centering
58.17
67.34
62.19
70.15
59.60
61.39
$5.85
62.10
Uniform
Whitening
56.53
66.95
62.77
72.42
61.05
62.74
56.03
62.64
Centering w/ Z.F. norm
59.01
71.91
65.28
73.26
65.71
61.34
61.27
65.40
Uniform
Whitening w/ Z.F. norm
58.49
71.53
66.22
75.62
67.82
61.95
62.87
66.35
Centering w/ 7.F. direction
54.94
67.87
66.18
75.62
68.27
63.11
62.83
65.55
Uniform
Whitening w/ 7.F. direction
53.40
66.91
65.15
75.11
66.90
63.24
60.95
64.52
Centering Whitening
56.89
69.95
65.08
73.91
65.71
62.18
58.84
64.65
Zipfian
56.16
70.33
67.20
76.60
70.99
62.52
66.50
67.19
fastText
(raw)
57.94
68.97
62.37
72.26
63.59
59.99
59.82
63.56
w/ Z.F. norm
61.35
75.69
66.77
75.69
69.41
61.24
65.31
67.92
w/ Z.F. direction
55.12
70.20
66.23
75.92
71.95
61.57
65.35
66.62
Centering
59.73
55.02
55.16
64.22
$3.39
58.85
$2.46
56.98
Uniform
Whitening
52.47
59.01
53.90
65.33
52.61
58.34
48.60
55.75
Centering w/ Z.F. norm
65.26
70.32
63.63
71.24
62.99
60.34
61.24
65.00
Uniform
Whitening w/ Z.F. norm
62,48
76.02
66.73
76.55
68.65
61.56
64.31
68.04
Centering w/ Z.F. direction
52.86
68.40
64.60
74.39
70.23
60.92
62.70
64.87
Uniform
Whitening w/ Z.F. direction
46.45
64.51
60.10
70.59
64.80
58.58
$4.87
59.99
Centering Whitening
58.30
71.69
64.57
74.10
67.59
60.75
59.40
65.20
Ziptian
58.86
73.85
68.43
78.07
74.00
62.85
69.55
69.37
fastText- subword
(raw)
49.10
47.34
51.94
61.99
51.54
53.60
50.43
52.28
w/ Z.F. norm
58.03
60.35
60.76
67.99
61.74
56.35
61.47
60.96
w/ Z.F. direction
47.65
49.71
55.26
67.96
61.81
56.62
54.72
56.25
Centering
49,21
43.13
49.89
62.03
49.70
54.56
46.91
50.78
Uniform
Whitening
45.12
41.00
47.30
62.08
48.85
54.80
43.55
48.96
Centering w/ Z.F. norm
60.29
58,99
61.11
70.23
65.36
57.78
63.06
62.40
Uniform
Whitening w/ Z.F. norm
61.06
60.35
63.07
74.29
69.19
59.56
65.09
64.66
Centering w/ Z.F. direction
46.17
49.23
54.49
67.33
61.11
55.95
$3.01
55.33
Uniform
Whitening w/ Z.F. direction
43.20
46.68
52.15
65.46
59.01
54.07
49.26
52.83
Centering
48.68
55.03
54.07
60.23
58.41
54.64
50.38
54.49
Ziptian
Whitening
61.22
60.68
63.18
73.59
69.87
59.82
68.20
65.22
26

---

Table 12: Full results of the whitening on dynamic embeddings. Each cell shows the STS score x 100. Token-level uniform centering/whieeingg ("Zipfian" settings), which corresponds to center- ing/whitening the word type level under Zipfian prior, consistently outperforms the "Uniform" setting across all STS tasks.
Method
STS12
STS13
STS14
STSI5
STS16 SICK-R STS-B
Avg.
BERT- uncased
First-last avg.
45.09
64.30
54.56
70.52
67.87
59.05
63.75
60.73
"Uniform" Centering
47.51
64.53
54.68
72.19
69.28
59.77
64.04
61.71
Whitening
40.31
56.11
47.02
68.35
64.53
48.59
60.53
55.06
Centering
47.58
66.26
57.32
73.18
71.09
63.27
64.82
63.36
"Zipfian"
Whitening
53.75
74.07
64.21
73.88
72.83
69.71
64.91
67.62
RoBERTa-basse
First-last avg.
44,00
59.02
49.31
66.63
59.62
57.56
60.75
56.70
Centering
46.07
55.50
46.27
66.06
60.06
51.33
60.34
55.09
"Uniform"
Whitening
37.67
54.64
47.71
66.31
62.85
50.13
61.31
54.37
Centering
44.97
61.19
53.73
69.57
67.88
58.60
61.30
59.61
"Zipfian"
Whitening
52.80
73.39
64.18
72.64
72.02
71.07
65.69
67.40
DeBERTa base
First-last avg.
45.03
61.99
52.39
68.90
64.83
56.54
61.66
58.76
Centering
45.20
61.25
50.84
68.56
63.87
53.18
62.01
57.84
"Uniform"
Whitening
38.12
50.46
45.30
63.52
62.29
46.99
58.19
52.12
"Zipfian"
Centering
45.87
63.24
55.07
70.53
68.88
58.50
63.18
60.75
Whitening
52.97
73.54
63.25
72.60
71.97
69.79
64.63
66.96
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Details are provided Limitation ($ 6).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
28

---

Justification:
See e.g. Appendix D, Appendix F, and Appendix H.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: (Appendix B), we have comprehensively described the experimental settings necessary for reproducing the empirical validations.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification: Source code has been made publicly available at https://github. com/ cl- wt-tthoku/zipfione tono
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ cublic/guides/ http://ddeeummmiiiinneeeeeee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ htapss.ccpublic//gee. gewiiiigggue/////eeeeeeeee Soteae//ideSSSSeseeeeeeeeea wwriiisssiinpiiinyn for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: In (Appendix B), have comprehensively described the experimental settings necessary for reproducing the empirical validations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: We used pre-trained word embeddings and applied PCA-like transformations in all experiments, eliminating the need for training hyperparameters such as seed values. Additionally, we utilized existing data splits for downstream task evaluations using the standard sentence embedding evaluation tool [40]. Consequently, our experiments did not include error bars or statistical significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
30

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The details on the compute resources used on the experiments are shown in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https///c/pc..tt.iiccceeeeeeeee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Details are provided in Broader Impacts section ($ 6)
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
31

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. paperswithcode. com/ datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
32

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provided sufficient documents for using the implementations for the proposed method in the README file in the supplemental material.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
33


Reviews:
[{"review_id": "ruD33wvDoy", "reviewer": "Reviewer_VABi", "summary": "This paper proposes \"Zipfian whitening\" of word vectors, that is, taking\nword probability in consideration when taking averages for whitening them.\nIn addition to presenting the proposed simple algorithm and experimentally\nevaluating in suitable NLP tasks, this paper also introduces measure of\nisotropy of word vectors, and theoretically explains why uniform averaging\ndoes not work well in practice.", "questions": "Nothing.", "limitations": "No problems.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Basically this is a good paper, and matches well with NeurIPS because the\nproposed whitening and the geometric discussion also applies to other fields\nthan natural language processing.\nI would like the authors to include some actual words uniformly sampled from \nthe vocabulary, which clearly shows averaging uniformly with such (mostly rare) \nwords is a bad idea.", "weaknesses": "My only concern is the title and assumption: why \"Zipf\"? Zipfian distribution\nmeans that the probability of each item decays inversely proportional with\nthe rank, yielding a heavy-tailed distribution. However, this kind of Zipfian\ncharacteristics does not seem to be used in the theory: it is just a\n\"Expected whitening\" rather than \"Zipfian whitening\". \n\nActually many distributions, including words, have Zipfian property, thus it\nis interesting to see, empirically and/or theoretically, if the proposed method\nworks for non-uniform, but non-Zipfian distributions. \nWithout such considerations, the title of \"Zipfian whitening\" might be \nmisleading as a scientific research.\n\nMinor\n- Figure 2: performance of each configuration could be displayed as the size of\neach disk. This does not need color printing, and humans generally have more\nsenses over the difference of sizes over difference of intensities.\n- p9: frequency We -> frequency. We", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're pleased to receive your positive evaluation! We intend to address all the points you've raised.\n\n### 1. Qualitative demo of the unnaturalness of uniform word distribution\n\n> I would like the authors to include some actual words uniformly sampled from the vocabulary, which clearly shows averaging uniformly with such (mostly rare) words is a bad idea.\n\nThank you for this excellent suggestion! Given that NLP is inherently driven by real data, incorporating concrete examples as a qualitative evaluation can likely provide readers with a more intuitive understanding of our paper's idea. We've quickly performed a sampling using the word frequencies employed in our paper:\n\n- uniform sampling: `['scintillation', 'fanon', 'rubato', 'upstanding', 'collard', 'creeks', 'skookum', 'unbelievers', 'monocyte', 'nishikawa', 'crusher', 'gerwen', 'abrah', 'silverchair', 'hangman', 'unitary', 'klausen', 'arousal', 'heat', 'bridgnorth', 'mildred', 'porton', 'aquasox', 'wylie', 'hipaa', 'krimuk', 'hexahedron', 'kuei', 'barbera', 'dalvi', 'gilding', 'visakhapatnam', 'tatsuo', 'tarascon', 'bajram', 'scholes', 'hadad', 'incidental', 'theodosius', 'reichskommissariat', 'boeheim', 'amsl', 'buencamino', 'thrasyvoulos', 'insulated', 'discourtesy', 'nisra', 'ycko', 'luen', 'dooku']`\n- Zipfian (frequency-aware) sampling: `['nine', 'ranked', 'zero', 'the', 'garcia', 'rank', 'station', 'the', 'for', 'four', 'williams', 'drunken', 'a', 'one', 'eight', 'of', 'were', 'zero', 'debate', 'orchestra', 'of', 'wrist', 'points', 'fractured', 'the', 'to', 'redirect', 'adnan', 'white', 'car', 'fond', 'concluded', 'under', 'two', 'by', 'five', 'his', 'infection', 'the', 'the', 'pop', 'in', 'one', 'in', 'one', 'one', 'fram', 'handled', 'battle', 'mutual']`\n\nThe latter clearly seems to capture a more \"natural\" representation of language as we typically encounter it in text, while the former uniform sampling is likely to give an impression quite detached from human language. We will include this comparison in the section where we propose calculating expectations weighted by frequency.\n\n### 2. The naming of \"Zipfian\"\n\n> My only concern is the title and assumption: why \"Zipf\"? Zipfian distribution means that the probability of each item decays inversely proportional with the rank, yielding a heavy-tailed distribution. However, this kind of Zipfian characteristics does not seem to be used in the theory\n\nYour point is well taken. Thank you for bringing this to our attention. Our focus is on the mismatch between word frequency distribution and uniform distribution, and we haven't developed an argument dependent on the degree of tail heaviness. We used the well-known example of \"Zipf\" because power-law distributions are common in the real world, and we're specifically dealing with word frequencies. You're right that it would be better to make the method name scientifically accurate. We appreciate your suggestion of \"expected whitening,\" and alternatives like \"frequency-aware whitening\" or \"distribution-aware whitening\" could also be considered. In any case, we will aim for a name that truly reflects the essence of the method.\n\n---\n\nThank you also for your comments on the presentation and typo! We'll address these as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FcpxA8pve4", "reviewer": "Reviewer_fUiC", "summary": "Prior work in natural language processing has shown that word embeddings are sometimes concentrated in a small cone of the embedding space.\nPrior work has also shown that correcting this can lead to better performance in some downstream tasks.\nThese prior work, however, do not typically consider a word’s frequency when correcting this issue.\nThis paper proposes Zipfian whitening: to zero-centre and standardise embeddings while considering their words’ Zipfian frequency.\nThey show this improves results in a downstream task and then make theoretical arguments for why.", "questions": "> To evaluate the pre-trained word encoder models and post-processed ones, we used the most commonly utilized downstream tasks in the community, STS-B [7] and SICK-R [23]. \n\n1. What are these tasks exactly? It would be helpful if you described them here.\n2. Why are you doing sentence-level similarity with word-level embeddings? Why not either: (i) word-level similarity task; (ii) sentence-level embeddings. (The latter should at least be present as a baseline for comparison.)\n\n>  Table 2 lists the correlation coefficients between the symmetry scores and downstream task performance in more detail\n\nHow do you compute a single symmetry score? Before (e.g., in Fig 2) you had two separate scores, the first and second moments.\n\n> $∥w∥_{G_w} ≈ 2KL(p(·)∥p(· | w))$\n\n\nWhat is $∥w∥_{G_w}$? Did you introduce this already? Also, in the appendix, the proof shows $∥w∥ ≈ 2KL(p(·\\mid w’)∥p(· | w))$. Is this a typo, or does the proof work for both cases?\n\n> Another benefit (but slightly more technical) of Zipfian whitening is that we can eventually regard the generative models of a word vector w (given a context c) and a context vector c (given a word w) as being symmetric. This (p(w | c) = p(c | w) can easily be seen from the generative model p(w | c) and the Bayes’ rule, given that the partition function Z z (c) is irrespective of a context c. This symmetry is essential to justify our practice regarding context embeddings being the same as word embeddings.\n\n1. Is this true for Zipfian whitening? Or only for a uniform prior?\n2. Why is this beneficial?\n3. To the best of my knowledge, word2vec and glove use different embeddings for words and contexts, so this is not exactly true in practice. Do the authors mean that embedding and un-embeddings layers sometimes have shared parameters in language models? That's quite a different setting then what's being analysed here.\n4. What does it mean for a context (multi-word) embedding to be the same as a word’s? Maybe making it explicit earlier in this paper that only skipgram- and glove-like models will be analysed (both of whose contexts are assumed to be individual words) would be useful.", "limitations": "The authors addressed the paper's limitations.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The proposed solution is quite simple, and seems to improve results.\n\nThe paper provides both empirical support to their proposal, and theoretical arguments in favour of it.", "weaknesses": "This paper’s contributions and correctness are hard to assess, in my opinion: \n* downstream tasks where experiments are performed are not described in the paper.\n* theoretical results are described at a relatively high level (without step-by-step explanations) which make it hard to evaluate its correctness.\n* other key information is missing, such as how a single symmetry score is extracted from the symmetry moments to get correlations in Table 2.\n\nSome examples can be found below under “Questions”.\n\nBesides that, as I understand them, experiments are run on a sentence-level similarity tasks. But the proposed methods are at the word level. Why not run word-level evaluation metrics? \n\nFinally, the paper makes the argument that Zipfian whitening makes embeddings’ norms proportional to a word’s information content, while uniform whitening does not. As experiments are performed on sentence-level similarity tasks, this could be an important source of its advantage. Adding experiments with a baseline in which uniform whitening  is performed, but then norms are rescaled based on information content, would be interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough reading and your critical and constructive comments. We especially appreciate your feedback on the clarity and self-contained nature.\n\n### 1. How to compute symmetry scores in Table 2\n\nWe made a typo! Thanks for pointing it out. The labels along the x-axis in Table 2 represent symmetry scores as you may guess. The correct labels for proposed measures are:\n- Incorrect — centering, whitening\n- Correct — 1st moment, 2nd moment\n\nThe numbers in Table 2 represent *both* the 1st and 2nd moments of the spatial symmetry, measured with uniform and Zipfian prior. Only when using the Zipfian prior (highlighted in light blue in Table 2), do both the 1st and 2nd moments strongly correlate with the task performance (Line 155–160).\n\n### 2. The choice of task\n\n> downstream tasks where experiments are performed are not described in the paper.\n\n> experiments are run on a sentence-level similarity tasks. But the proposed methods are at the word level. Why not run word-level evaluation metrics?\n\nThis is an important point! Let us address it in detail.\n\n**Datasets we used**: STS-B and SICK-R are both *sentence*-level similarity tasks, which are standard for empirically evaluating the performance of *word* vectors. These datasets consist of pairs of sentences and their semantic similarity rated by annotators. The typical experimental protocol we followed is to sum the word vectors to form a \"sentence vector\" and then check if the angles between them correlate well with the gold scores.\n\n**Why evaluate word vectors at the sentence-level tasks**: The question \"Why not run word-level evaluation metrics?\" is a natural and valid inquiry. Our language has a property known as compositionality, which allows infinite semantic content to be conveyed through a finite vocabulary as building blocks. This perspective underlies models like word2vec, BERT, and GPT series, where the fundamental unit of representation is the word; and these models are used to solve tasks with larger components e.g. sentences. Our research adheres to this basic principle of NLP.\n\n**Word-level evaluation**: Your suggestion to evaluate post-processing effects on word vectors using word-level tasks is reasonable! However, existing word-level similarity datasets have significant issues making them less suitable for our work (see Bakarov \"A Survey of Word Embeddings Evaluation Methods\" 2018, Section 4.1.1). Given that whitening reflects word information content in vector norms, tasks like keyword extraction (which selects words with high information content)  could be good candidates; we'll include such tasks in the camera-ready version.\n\n### 3. Experiments with a mix of uniform and Zipfian settings\n\n> the paper makes the argument that Zipfian whitening makes embeddings’ norms proportional to a word’s information content, while uniform whitening does not.\n(...)\nAdding experiments with a baseline in which uniform whitening is performed, but then norms are rescaled based on information content, would be interesting.\n\nIt's a really interesting idea to isolate the effect of the norm and empirically verify its impact.\nWe promptly conducted an experiment similar to Table 1 using GloVe and STS-B.\n\n||Uniform|Uniform + $\\alpha$|Zipfian|\n|:-|:-:|:-:|:-:|\n|+ Centering|41.27|53.66|55.15|\n|+ Whitening|53.22|64.83|70.22|\n\n\"Uniform + $\\alpha$\" refers to the process of \"correcting word vectors using a uniform prior, then replacing only the norm with that obtained from Zipfian whitening\". We found that appropriate weighting by norm has a critical effect on task performance. It's also interesting that pure Zipfian centering/whitening performs even better. This implies that Zipfian correction has two effects: (i) the *norm* becomes representative of information content (Section 3.1), and (ii) vectors disperse more evenly (isotropic), leading to appropriate positioning w.r.t. *direction* as well. We will incorporate comprehensive results into the manuscript!\n\n### 4. Notation of norm\n\n> What is $||w||_{G_w}$?\n\nWe used $||x||_A$ to denote a norm based on a quadratic form $\\sqrt{x^\\top Ax}$. We will clarify this point in the manuscript.\n\n> in the appendix, the proof shows $||w|| \\approx 2 KL(p(・|w’) || p(・|w))$. Is this a topo, or does the proof work for both cases?\n\nIs Reviewer fUiC referring to Line 435? We also use the notation $||w||_{G_w}$ here rather than $||w||$. If there is a typo elsewhere, we would appreciate it if you could let us know.\n\n### 5. Symmetry of w and c induced by whitening (Page 7, Footnote 11)\n\n> 1. Is this true for Zipfian whitening? Or only for a uniform prior?\n\nIt's true regardless of the prior. You can see this by Bayes' rule: $p(c|w) = p(c)p(w|c)/p(w) = p(c)\\exp(\\langle w,c\\rangle) / Z(w)$, where Eq. (7) and $Z(w)=Z(c)$ is used.\n\n> 2. Why is this beneficial? / 3. To the best of my knowledge, word2vec and glove use different embeddings for words and contexts\n\nExactly. In standard learning algorithms for static embeddings, asymmetric embeddings are learned although the target co-occurrence distribution is symmetric. What's interesting here is that whitening as a post-processing step can restore this symmetry, which benefits us in approaching the desirable inherent symmetry originally present in the data.\n\n> 4. What does it mean for a context (multi-word) embedding to be the same as a word’s?\n\nIn both static and masked/causal models, word prediction is performed through inner products. The \"context\" can refer to a single word, a cloze sentence, or a prefix of a sentence; abstractly, c encompasses all of these. However, when discussing symmetry as in Footnote 11 for example, it would be clearer to restrict c to co-occurring words.\n\n---\n\nThank you once again for your critical reading. Your feedback will help us refine the manuscript to ensure clarity and accuracy. If any remaining points are unclear, please feel free to share your candid feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IdPoYb8Riq", "reviewer": "Reviewer_jUzu", "summary": "This paper considers the problem of post-processing static word embedding spaces based on the observation that the distribution is spatially skewed based on the occurrence frequency of corresponding words. The authors propose Zipfian whitening, an approach to symmetrize the embedding space using the empirical frequency distribution to weight the influence of each embedding. The authors present empirical evidence for their proposed approach on standard datasets and metrics. Theoretical connections to exponential family distributions provide intuition for why Zipfian whitening is better than uniform whitening.", "questions": "* Could you more explicitly explain how uniform whitening of contextual word embeddings is related to Zipfian whitening?\n* How would the proposed approach be used to regularize large language models trained on next token prediction?\n* What is $c$ in Section 3? Is it a class label?\n* If a word is very infrequent, is there a concern that it's embedding is of poor quality? In this case, would we be concerned that its poor quality would skew the centroid of the space?\n* Could this approach be useful for understanding large language model embeddings? \n* How do the insights provided in this paper inform proposed approaches in safety and alignment? (connections to alignment are mentioned in the Broader Impacts, but not explained) Could this work have impact on aspects of mechanistic interpretability?", "limitations": "The authors have thoroughly addressed the limitations and potential societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*  The potential downstream impact of this paper is well beyond the immediate superficial contributions. The paper presents limited (yet profound!) results on small, unrealistic, and impractical datasets, but these results and observations, along with the theoretical connections, could influence how we address the long-tail in many aspects of training, evaluation, safety, and alignment. The technical contributions might not be directly applicable to all of these problems, but making the community aware of such an approach could inspire many other researchers to solve problems of this sort.\n* The paper is written in a refreshingly unorthodox fashion. It does not follow the standard template of machine learning papers making it much more enjoyable to read and more likely to have a bigger impact.\n* The paper addresses all aspects of the problem. The authors not only show that their proposed approach works, but also present empirical and theoretical evidence for why it works.", "weaknesses": "* The empirical results are quite impractical. It would be better if there was empirical evidence on some sort of dynamic or causal embeddings. \n* Section 4 seems to be a bit of a stretch. It is understandable that there is limited space, but it is a quite hand-wavy explanation. I'm not sure how much this particular section contributes to the paper.\n* It is not immediately obvious how this work could be applied to causal and dynamic large language models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive evaluation of our paper, including its potential impact on future research! We also appreciate your many insightful and constructive questions. We'll do our best to provide honest answers below. If any remaining discrepancies in our understanding or points need clarification, we'd be grateful for your further input.\n\n### 1. Correction of contextual/causal embeddings\n\n> The empirical results are quite impractical. It would be better if there was **empirical evidence on some sort of dynamic or causal embeddings**.\n\n> Could you more explicitly explain **how uniform whitening of contextual word embeddings is related to Zipfian whitening**?\n\nWe agree that the ML/NLP community's focus is on contextual and causal language models, and acknowledge our Section 4 lacks detail.\n\nIn the revision, we'll add a section discussing our research's relationship with these models, starting with the **type-token distinction**. The type-token distinction is a well-known concept in linguistics and related fields, where *type* represents a class and *token* represents an instance. For example, the phrase \"perform natural language processing in a natural way\" contains eight tokens and *seven* types; the instances \"natural\" appear twice, but as a word type, it is counted only once.\n\nThis distinction clarifies the relation between uniform whitening of contextual word embeddings and Zipfian whitening. Addition-based sentence embeddings are obtained by summing up *token* embeddings in a sentence. Given that a sufficiently long sentence may represent the underlying Zipfian word frequency, the uniform sampling on these tokens is approximately nothing else but the Zipfian sampling on *types*. That's why we claim that the uniform centering/whitening of sentence embeddings corresponds to the Zipfian centering/whitening of type embeddings. Our contribution provides a new explanation for the empirical success of existing methods like uniform centering (*1: Chen et al. 2020) and whitening (*2: Huang et al. 2021) of token vectors. To strengthen our argument, we'll include an experiment applying a *pseudo*-uniform prior by multiplying token embeddings by $1/p(w)$. We'll compare this with existing methods (*1, *2) that implicitly use a Zipfian prior!\n\n### 2. Connection with other aspects of contextual/causal LMs\n\n> How would the proposed approach **be used to regularize large language models trained on next token prediction**?\n\nA promising direction is the orthogonalization of embedding matrices. Previous studies added regularizers to increase the effective rank or impose orthogonality of word (un)embedding matrices (e.g. Wang et al. \"Improving Neural Language Generation With Spectrum Control\" 2020). However, they treat the word (un)embedding matrix as a standard data matrix, thus implicitly assuming a uniform prior. For future work, we’d like to try to adopt a Zipfian prior to make vectors in the (un)embedding matrix effectively isotropic, thereby maximizing expressive power while accounting for word frequency.\n\n> How do the insights provided in this paper inform proposed approaches in **safety and alignment**? (connections to alignment are mentioned in the Broader Impacts, but not explained)\n\nDohmatob et al. \"A Tale of Tails: Model Collapse as a Change of Scaling Laws\" (2024) reported that repeated sampling from generative AIs may shift word frequency distributions towards light-tailed ones. This could reduce linguistic diversity and cause cultural homogenization by decreasing region-specific or culturally unique expressions. Our Zipfian whitening and similar regularization methods can be used to enhance output diversity, thereby enriching the resulting linguistic landscape.\n\n> Could this work have impact on aspects of **mechanistic interpretability**?\n\nRegarding the type-token distinction, embedding and unembedding matrices in causal LMs primarily retain word *type* information. Thus, improving the logit lens approach, which analyzes hidden vectors by projecting them onto the unembedding matrix, could contribute to the mech. interp. community; the softmax function used after projection implicitly assumes a uniform prior.\n\n### 3. Other points of discussion\n\n> What is $c$ in Section 3? Is it a class label?\n\nThank you for pointing this out! Our description was lacking. $c$ represents context, generalizing the information used to predict a word $w$ in static/masked/causal LMs. Specifically, it represents a co-occurring word, a cloze sentence, or a prefix of a sentence. In all cases, prediction involves calculating $\\langle \\boldsymbol w, \\boldsymbol c\\rangle$. Furthermore, for a rigorous discussion, particularly when addressing Thm. 1 for example, we should restrict $c$ to co-occurring words. We will revise the manuscript to clarify this point.\n\n> If a word is very infrequent, is there a concern that it's embedding is of poor quality? In this case, would we be concerned that its poor quality would skew the centroid of the space?\n\nWe appreciate your insightful comment. Our algorithm removes information from high-frequent words, enhancing low-frequent word embeddings. Whitening parameters mainly depend on high-frequent word vectors (Algorithm 1). Thus, this suggests the concern may not materialize empirically; however, extremely low-quality embeddings for rare words might still not improve. Subword-based embeddings could address this. Our response to Reviewer 9YCm under \"1. Embedding models\" includes results from subword-based fastText embeddings, showing higher performance than non-subword models like GloVe. This indicates the potential value of subword approaches for low-frequent words. We'll incorporate such experiments in our revision, considering the quality aspect you mentioned.\n\n---\n\nWe've attempted to answer within the character limit, but there might be some information gaps. Please feel free to ask for clarification on any points you find unclear.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NJydvDXmcb", "reviewer": "Reviewer_9YCm", "summary": "This paper proposes new Zipfian whitening for static word embeddings inspired by Zipf's law. The main idea is to use empirical word frequencies as a prior rather than using a uniform prior. The authors show the superiority of their method compared to previous widely-used baselines. The paper also presents the metric for measuring symmetry in word embeddings.", "questions": "- In the paper, you propose a method that relies heavily on the empirical frequency. Thus, it would be interesting to look at how changing the empirical distribution will affect the downstream task performance\n- Also, as far as I understand, you discard lots of infrequent tokens (frequency less than 200 according to the enwiki vocab), while in my opinion, the most interesting effect can be seen on low-frequency tokens. What is your opinion on this matter? And how should you deal with the OOV embeddings (i.e., missing in the frequency distribution)? It is especially crucial for low-resource languages.\n- While it's a minor point, I'm curious why you haven't explored token-level embeddings, given that it's a de-facto standard in the field today. For instance, there are static bpe level embeddings available (fasttext). Incorporating these could significantly bolster your claims.", "limitations": "I would encourage authors to add the limitations section addressing their empirical analysis scope.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Novel approach for inducing symmetry into the embedding space inspired by Zipf's law\n- Easy to follow paper; nicely written\n- Empirical results on downstream tasks show the efficiency of the proposed Zipfian whitening\n- Analysis from different prospectives is presented\n- Paper draws an interesting connection to the prior research", "weaknesses": "- Limited evaluation on word-level static embeddings: one language studied, one dataset/vocabulary and its frequencies", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review! We're delighted to hear it. We especially appreciate your constructive feedback on the various aspects of our experimental setup. Below, we provide our responses.\n\n### 1. Embedding models\n\n> While it's a minor point, I'm curious why you haven't explored token-level embeddings, given that it's a de-facto standard in the field today. For instance, there are static bpe level embeddings available (fasttext). Incorporating these could significantly bolster your claims.\n\nAbsolutely! Using fastText is indeed a very natural and convincing experimental setup. Thank you for the suggestion. We quickly conducted similar experiments to those in Table 1 using fastText embeddings trained with Common Crawl and STS-B dataset.\n\n|fastText|Uniform|Zipfian|\n|:-|:-:|:-:|\n|raw|60.46|60.46|\n|+ Centering|51.39|61.64|\n|+ Whitening|48.55|72.26|\n\nThe stable effects of Zipfian whitening were confirmed. We will include the results applied across the entire paper in the camera-ready version.\n\n### 2. Word frequency\n\n> Also, as far as I understand, you discard lots of infrequent tokens (frequency less than 200 according to the enwiki vocab), while in my opinion, the most interesting effect can be seen on low-frequency tokens. What is your opinion on this matter? And how should you deal with the OOV embeddings (i.e., missing in the frequency distribution)? It is especially crucial for low-resource languages.\n\nThank you for your very interesting question, which touches on important issues, the heavy tail of word frequency and low-resource languages. We agree that OOV words are inevitable since word frequencies follow a long-tail distribution. We will include several considerations below in the revised version.\n\n- **Algorithm** — Since whitening is a global affine transformation on the entire word embedding space (Algorithm 1), our post-processing is possible/applicable even if the frequency of a target word is unknown or extremely low.\n- **Empirical Results** — Based on your comment, we found it interesting to conduct experiments specifically designed to create cases with intentional OOV words (words with unknown frequency). We conducted experiments using only the top 1,000 word frequencies (0.5%) from `enwiki_vocab_min200`, similar to Table 1 with GloVe and STS-B. Results were largely maintained, suggesting the method's robustness to some OOV words.\n\n||Zipfian|\n|:-|:-:|\n|raw|43.65|\n|+ Centering|55.15|\n|+ Centering (OOV settings)|57.59|\n|+ Whitening|70.22|\n|+ Whitening (OOV settings)|71.54|\n\n- **Implications** — The mean vector used in post-processing ($\\hat\\mu$ in Algorithm 1) and the weighted data matrix ($W_p$ in Algorithm 1) are not significantly influenced by extremely low-frequency words with small $p(w)$. Instead, the process primarily *removes signals from high-frequency words*. This suggests that even for low-resource languages, as long as the \"head\" is well-observed, the process can be applied and might preserve signals from the \"tail\" side. we will experimentally test this hypothesis using multilingual vectors (e.g. fastText) and the multilingual evaluation dataset including low-resource languages (Ousidhoum et al. \"SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages\" 2024), and include these results in the camera-ready version.\n\nAdditionally, to clarify, the experiments conducted on English text, as reported in the paper, were carried out in a setting where OOV occurrences were minimal. Specifically, `the enwiki_vocab_min200` contains 188,033 words. For comparison, one of the current largest models, LLaMA 3.1 by Meta, has a vocab size of 128,000 (h/t \"The Llama 3 Herd of Models\" arXiv 2407.21783; though this is subword tokenized, so it's not a perfect one-to-one comparison). For reference, the lowest-frequency words in `enwiki_vocab_min200` with a frequency of 200 include minor proper nouns such as \"abbi\", \"aberto\", \"abgrallaspis\", \"abhilasha\", and \"acurate\" (which is not \"accurate\"). We have provided more detailed data in our response to Reviewer VABi. Also, there were no OOV issues when solving STS and SICK-R tasks in our settings. We will include a detailed explanation of this in the manuscript.\n\n---\n\nIf there are any discrepancies between our proposed experimental settings and your expectations, we would appreciate your further comments. Once again, thank you for your wide-ranging suggestions to make our empirical results more convincing!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes using the information of word frequency in in post-processing word embeddings, and provides both theoretical and empirical justifications of its benefits. The reviewers liked the simplicity and the theory components of the work, though some remain a bit unconvinced of certain parts of the paper and its experimental settings (still leaning positive). My only real concern is its relatively limited scope of correcting the word embedding space, but the paper seems to make sufficiently strong contributions in novelty and rigor in this scope to justify an accept.
Decision: Accept (poster)