On the Emergence of Linear Analogies in Word Embeddings
Authors: Daniel James Korchinski
Dhruva Karkada
Yasaman Bahri
Matthieu Wyart
Venue: NeurIPS 2025 poster ()
Keywords: word embedding
analogy structure
random matrix theory
Primary area: theory
TL;DR: We introduce a theoretical model for the co-occurrence statistics of words that quantitatively explains the emergence of linear analogies of word embedding in models like Word2Vec and GloVe.

Abstract: Models such as Word2Vec and GloVe construct word embeddings based on the co-occurrence probability $P(i,j)$ of words $i$ and $j$ in text corpora. The resulting vectors $W_i$ not only group semantically similar words but also exhibit a striking linear analogy structure---for example, $W_{\text{king}} - W_{\text{man}} + W_{\text{woman}} \approx W_{\text{queen}}$---whose theoretical origin remains unclear.
Previous observations indicate that this analogy structure:
(i) already emerges in the top eigenvectors of the matrix $M(i,j) = P(i,j)/P(i)P(j)$,
(ii) strengthens and then saturates as more eigenvectors of $M (i, j)$, which controls the dimension of the embeddings, are included,
(iii) is enhanced when using $\log M(i,j)$ rather than $M(i,j)$, and
(iv) persists even when all word pairs involved in a specific analogy relation (e.g., king--queen, man--woman) are removed from the corpus.
To explain these phenomena, we introduce a theoretical generative model in which words are defined by binary semantic attributes, and co-occurrence probabilities are derived from attribute-based interactions. This model analytically reproduces the emergence of linear analogy structure and naturally accounts for properties (i)--(iv). It can be viewed as giving fine-grained resolution into the role of each additional embedding dimension. It is robust to various forms of noise and agrees well with co-occurrence statistics measured on Wikipedia and the analogy benchmark introduced by Mikolov et al.

On the Emergence of Linear Analogies in Word Embeddings
Daniel J. Korchinski"
Dhruva Karkada
Department of Physics
Department of Physics UC Berkeley Berkeley, CA, USA
Ecole Polytechnique Fédérale de Lausanne (EPFL)
Lausanne, VD Switzerland
daniel. horiiiske.r
dkarkada@berkkeley edu
Yasaman Bahri
Matthieu Wyart
Google DeepMind Mountain View, CA, USA yasamanbahri@@maill.. com
Johns Hopkins & EPFL
Baltimore, MD, USA & Lausanne, VD Switzerland
mwyart10jh.edi
Abstract
Models such as Word2Vec and GloVe construct word embeddings based on the co-occurrence probability P(i.j) of words and j in text corpora. The resulting vectors W; not only group semantically similar words but also exhibit strik- ing linear analogy structure-foe example, Waing Wman + W Wooman W Wqueen whose theoretical origin remains unclear. Previous observations indicate that this analogy structure: (i) already emerges in the top eigenvectors of matrix M(i,j)
(ii) strengthens and then saturates as more eigen- vectors of M (i.j), which controls the dimension of the embeddings, are included, (iii) is enhanced when using log M (i,j) rather than (i.j), and (iv) persists even when all word pairs involved in specific analogy relation (e.g.. king queenn man-woman) are removed from the corpus. To explain these phenomena, introduce a theoretical generative model in which words are defined by binary se- mantic attributes, and co-occurrence probabilities are derived from suribute-based interactions. This model analytically reproduces the emergence of linear analogy structure and naturally accounts for properties (i)-(iv). It can be viewed giving fine-grained resolution into the role of each additional embedding dimension. It is robust various forms of noise and agrees well with co-occurrence statisties measured on Wikipedia and the analogy benchmark introduced by Mikolov et al.
1. Introduction and Motivation
Vector representations words have become cornerstone of modern natural language processing. Models such Word2Vec 2 and GloVe 3 map any word
continuous vector space W: E I& based on their co-occurrence statistics in large text corpora. These embeddings capture rich semantic relationships, including remarkable form of analogical structure: for instance, the arithmetic expression
Kiing
W man + W Wwoman& W Wquson
(I)
often holds in embedding space [ (2) While widely observed, understanding the origin this geomet- ric regularity remains challenge. Several works (5.3 indicate that this property is already contained
in the co-occurrence matrix M(i,j) P(i.j)/ P()) or PMT().) = log( M (i, called
hatpp:///djoorchiinktt gith.b.thhhbb... io/
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the pointwise mutual information or PMI matrix. Here, 5( is the empirical probability that words i and appear together within context window, and P(i) the marginal probability of word i.
Key observations to explain are that (i) the top eigenvectors of M already separate semantic aspects of words [6; (ii) If the word embedding is built by approximating M W W, where W is K X m matrix whose columns W; are the embeddings of each word = 1...m and rows W; T are the (scaled) eigenvectors of M, then one finds that linear analogies initially becomes more accurate as K increases, such that more of the top eigenvectors of . are retained [6] (iii) linear analogies improve further when the PMI is used instead of M itself, i.e. by approximating the PMI = W W 7 Moreover, (iv) linear analogies persists even when the word pairs exemplifying the target analogy are removed from the dataset [8] (e.g. removing from the corpus all word pairs including a masculine word and its feminine counterpart, such as man-woman, king-queen, etc.).
2. Our Contributions
In this work, we propose theoretical generative model for word co-occurrences, which rationalizes the observations above. The model enables compute the properties of M and of the PMI analytically and elucidates how linear analogies emerge.
(I). Our theory posits that each word is defined by a vector of d discrete semantic attributes, and that each attribute a word affects its context in an independent manner. This leads to an exact Kronecker product structure in the co-occurrence matrix j)i,j) and derived quantities such M (i, which allows us to compute eigenvectors and eigenvalues of M and the PMI analytically.
(2). We show that analogies naturally emerge as a result of dominant eigenvectors when the matrix M or (M)) is used to create embeddings: vector arithmetic on the d-dimensional space of attributes gives rise to the same arithmetic relations for word embeddings. We emphasize that this property arises naturally in the eigendecomposition as a result of the factorization assumption. We show that this linearity however breaks down K increases if M is considered instead of the PMI.
(3). We find that our conclusions are remarkably robust to the presence of noise or to perturbations of the co-occurrence matrix, including the addition of i.idd. noise, the pruning of most words, as well as the removal of co-occurrence probabilities that compose an analogy (such as "king" " "mmn"" and
and correlations between attributes, showing how the analoyy-supporiig subspace remains robust.
(4). Throughout this work, we test specific predictions of our theoretical model numerically through comparison with Wikipedia text, demonstrating the predictive power of our model.
3. Related Work
Extensive empirical evidence suggests that many models of natural language 12 3 including those trained on non-english corpora 1911, and contemporary large language models 12.15,, exhibit striking linear structure in their latent space. This observation motivates contemporary research in modern language models. including mechanistic interpretability [16 wn-context-lcariing [19 21 and LLM alignment [4]. Linear analogy structure in word embedding models is the natural precursor to these phenomena; thus, to understand linear representations in general, it is important to develop a theoretical understanding of linear analogies in simple models.
The origin of the linear analogy structure in word embedding models has been the subject of intense study 29) Prior works [3] (00 have based their reasoning on the insight that ratios of conditional probabilities, such as p(x man) (ammmm(Xp //(x/ (ooom//))) for word are relevant for discriminating
its content. led the postulate [27 30 that for arbitrary words x in the corpus,
p(x (ixlking))/(((lquen))
p(x(man)///mmmmm / pxx (woman).
If this approximation is equality, and if the embedding is built from the full rank of the PMI, then can be derived [30]. Yet, how good of an approximation Eq and how large the embedding dimension should be for this argument to work? Will some analogies be learned before others?
To make progress on these questions, further assumptions have been proposed, particular the existence of a 'true' Euclidean semantic space with some dimension d in which words are associated

---

with latent vectors. In that space. text generation corresponds to a random walk, such that closer words co-occur more often [2] 27 29 Using the assumption that this space perfectly spherically symmetric, [27 deduces that Wi W P)) and proves that deviations from Eq 2 are tamed, such that holds.
We argue that such views are problematic for two reasons. The proposed symmetry of word embed- dings would imply that the spectrum of the PMI corresponds in fact the spectrum of the PMI is broadly distributed, as we shall recall. More fundamentally, the assumption that the true Euclidean semantic space has some geometry unconstrained by relations of the kind of Eq 2 appears unlikely. It is inconsistent with the observation that the top eigenvectors of the PMI have semantic content that correlates with analogy relationships (6].
identical eigenvalues, whereas
Instead, we base our approach on human psychology experiments describing how words can be characterized by list of features or attributes 31 32 We propose that the co-oceurrence of two words is governed by the relationship between their two lists. In that view and contrast to previous approaches, the semantic space corresponds the vector of attributes, and is thus organized by relations of the kind of Eq In a simplified setting of binary attributes, the geometry of the semantic space is that of an hypercube (as already mentioned in (II), yet co-occurrence does simply depend on the Euclidean distance between word representations. As we shall see, this view naturally explains points (i-iv) above.
4. A model for words co-occurrence
Our main assumption is that the occurrence statistics of word i is governed by the set of attributes that define it [32], such as feminine v.s. masculine, royal u.s. non-royal, adjective v.s. noun, etc. For simplicity, consider that there are d binary attributes (extending the model to attributes with more than two choices is straightforward, as discussed in Appendix (A3). The word i is thus represented as vector a E {-1,+1 )ad of the d-dimensional hypercube. As a first step, we assume that all the 2d possible words exist. We will see below that even if the vocabulary is much smaller than the total number possible words, our conclusions hold.
We further assume that different attributes affect the statistics of words in an independent manner (this assumption is relaxed in Appendix A5). As result, the probability P(i) of word i follows from the set {Pk I 1/2, k = 1...d,, where pş indicates the probability that attribute & +1:
P())
p (1) .1) + (1 pp))o(;; (0 -1) 1=
II
(k) where ji, denotes the Kronecker delta (6(i,)) = iff = j and zero otherwise), and o; denotes the 1tth entry of ca,, Likewise, the probability P(i.j) that words i and co-occur follows from the probability (a
(k) ).) that two words with given attributes ag and aj co-occur:
II
pil (k) (ag .af
(2))
The symmetric matrices p(k) e 2x must be such that P(i,)) P(i), imposing that they can be parametrized by single scalar spt
p(t)_
1 + sp
1 1 g2.kk
1 + Lh
where 8 E [0. 1] characterizes the strength of the "signal" associated to attribute, and Ø =
Pa/(1 pp) < 1 captures the asymmetry in incidence between positive and negative instances of the attribute.
Note that in this noiseless version of the model, Eq holds exactly. Indeed, if and differ by single attribute (say the first one) such that af = -t.. 1, and is any word, then ((ali)/((ali")) pl (a + (1) -1), a result which does not depend on the choice of Below we will add
noise to the model to study how linear analogies persist.

---

5. Word embedding directly from the co-occurrence matrix
Denote by W the K x 2d matrix of word embeddings of dimension K, whose columns W; R correspond to the embedding of word i. Embeddings can be constructed by demanding that the rescaled co-occurrnnce (i.j) P( is approximated by W W,, equivalently M = WTW.
P(),f)
In L2 sense, the optimal embedding corresponds to:
W = Σ S=1...
VAs us (i)us
(5)
where is the rank of the eigenvector us of eigenvalue As, such that A Ao z ...Ag, and the {us, S k...K} is any orthonormal basis. Thus, solving for the word embeddings corresponds to diagonalizing M, as we now proceed.
Theorem: The matrix ji = P().I)(((())G))) defined by Eq 3 has eigenvectons
U5 = 2 (1) 8 0z 2 ... 8 (d) with ag E (+,-}
(6)
where @ indicates Kronecker product. Its component for word of attributes o; is thus ( (aa))
(i) & (a (e (a
(7)
Its associated eigenvalue follows:
As
(k)
(8)
where the _ are eigenvalues of the 2 x 2 matrices p(k) defined in Eq and o are the two eigenvectors of these matrices.
Proof sketch: This theorem follows from the fact that the matrix M a Kronecker product of the matrices p(k), and from standard results on the eigen-decompositionn of these products [33]. The block structure of M defining a Kronecker product is most apparent if we order each word based on the lexicographic order of its attributes a;, That is, the first coordinate that differs between or and or; will determine which word appears first, with i appearing first if its coordinate is +1. See Appendix AT for the detailed proof, review of Kronecker products, and an explicit construction for the case d dd=2 2.
Diagonalization of the matrices p(k); This symmetric matrix is diagonalizable, with eigenvalues reported in Appendix A. To study the regime of weak semantic signal, we expand the eigenvalues and eigenvectors to first order in sk. Let = Sy and g == qu for brevity. We find in that regime:
2 + s(1 2 (s²),
(k) = s(1 + 0(s²)
(9)
associated to unnormalized eigenvectors (A) and (K)
(a)
22 (1)²²
1 s99 -1/- (-1) (1) = V2 (
s -1)/4 (1 +0(5))
11+ v2
V2
(10)
Note that when y = 1, i.e., Pk = 2. these vectors reduce for all s to (1, 1)T and (1, - -) with the exact eigenvalues 2 and 2s.
Eigensystem of M: The spectrum of M can be obtained using Equation 9 in Equation and the corresponding cigenvectors by using Equation (IO) in Equation 7 In the limit where the signals (st. I..dd are small, the top eigenbands are:
A top eigenvalue Ån 2d and corresponding eigenvector 1 (with higher order corrections linear the (st, l: = 1...d),, corresponding to all as = +.
d "attribute' eigenvectors vi with ag/4 = + and ag = -. giving eigenvalue ss
2 (d 2) Sy + ) and implying (i) x a
d eigenvectors of eigenvalues x 848K1 of order (2) d eigenvectors of order Xx 8k, and so on. At dominant order, the corresponding eigenvectors encode the product of several attributes.

---

berlin
athens iiess
thailand
germany
grea
preece
hans
rats rats
eagid
(c)
(a)
/
5
log M
10-1
bangkok
Density
berlin
9
athens
tokyo
Log-normal
10-2
thailand
10a
germany
Wikipedia
10-3
Japan
greece
Density
10
hand
10-1
10
10a
rats
eagles
Eigenvalue
hahds
(b)
10-²
10
10
10"
10º
102
10
Eigenvalue
Figure 1: (a) A subset of the co-occurrence matrix for Wikipedia data, with labels drawn from the cafegories "country-capitals" and "noun-plural". (b) A subset of the co-occurrence matrix M generated by our model (d = 8, 8 ~ ((1/2,0x) with = 10-$). Colors indicate value of M.j on log-scale. (c) Averaged eigenvalue spectrum of the co-occurrence matrix M for d 8, obtained from 50 random realizations of the semantic strength for c = 10 2 x 10-2, and 10 10-1, the uniform 8y (0, 1), and empirical co-occurrence data. The inset reveals that the spectrum of the PMI is peaked with density of nearly identical cigenvalues, as assumed in some previous works.
As result, the spectrum of M depends on the distribution of the 8y. representative M with minimal variance in that distribution is shown in Figure [bb. in the symmetric case where 1 for all k, and can be contrasted with the empirical Wikipedia derived co-occurrences reported in Figure[p Figure D shows the density of eigenvalues, averaged over 50 realizations. When the variance the distribution in the 8g 's is small, the eigenvalue spectrum is resolved into discrete bands, while at higher values of the variance in the sg, these bands merge and the distribution is well-fitted by a log-normal distribution.
Emergence of linear analogy: Nearly perfect linear analogies appear under the restrictive conditions that the {sk, 1...d} are small, (ii) they are narrowly distributed and (iii) the dimension of the word embedding satisfies K < d + 1. Indeed in that case, the first K eigenvectors into which words are embedded belong the span of the 2k and the constant vector Embeddings are thus affine in the attributes. It implies that if four words A, B, C. D satisfy:
= Q + ac
Then it must be that:
WA WB + We WD
(11)
We next show that this property emerges much more robustly if the matrix of elements PMI(i, is considered.
6. Word embedding from the pointwise mutual information (PMI) matrix
Successful algorithms such Glove focus on the PMI matrix, our model justifies. Given word pairs i,j with semantic vectors aj E { (-1,+11 the log co-occurrence matrix or PMI is:
log M(ijj)
D log plk) a (K)).
1=1
5

---

100
Analogy Accuracy Analogy Accuracy
1.00
curbycadal
Noise 00
0.75
chyildt
intherdence
10-1
0.50
10
0.25
(a)
Wikipedia
(c)
10.1)
(e)
(0,
0.00
10-3
1.00
Mi
M
0.75
0.50
0.25
(b)
Wikipedia
(d)
E (0.1)
(f)
(0.1)
0.00
10"
10a
10a
10a
10
10a
10a
10a
10²
Embedding dimension,
Embedding dimension, K
Embedding dimension, K
Figure 2: Analogy completion accuracy emerges at low embedding dimension. (a): Wikipedia analogy completion accuracy by analogy category for representations constructed from occurrences Mij with a vocabulary different matrix targets M, and log( Mij + ER) (regularizer = 10-2). Shaded area indicates the sample standard deviation across analogy categories. (c): Analogy completion accuracy for single realization of the model for = 8, for matrix target M,y. (d): Analogy accuracy for the model with different matrix targets, averaged across all analogy tasks and 50 realizations of the $Ą. Shading indicates standard deviation between realizations. (e): Analogy accuracy under the introduction of a multiplicative noise each entry of the co-oceurrence matrix with P. P.y exp(Ei)) for symmetric Eij ~ N (0, og) averaged over 10 realizations of 8 and noise &. (f): Analogy accuracy after both
10,000 words. (b): Analogy accuracy for Wikipedia text, with
sparsifying the vocabulary and including a multiplicative noise (ōş = 10-1), retaining only a fraction 0.15 of words in 12.
For each attribute k, the scalar log p(k) (a, b), with a,b {-1, can be written bilinear expansion:
log (()) (a, b) = 6% + ma(a + b) + nnab,
where the values of these coefficients are indicated in Appendix A.
The total matrix takes the form log (i.j) + 17 a; + n a, + a. Doy, where 5 = =Ex Sc.
77 .,,...., na), and D diag(71...... Ta).
Matrix Form, Rank and Eigenvectors:
E 222 the matrix with components A(i,k) =
af Then:
log M 8- 11 ADA An1 + 1n A
(12)
Result 1: The row space of A is of dimension d, thus rank(log M) < d + 1.
Result 2: Note that the eigenvectors
A. V22
=
/2d
Uk
=
for k = =1, l....,d
span the row space of (log M). The vector Uks previously introduced, simply indicates the value of attribute l: for any words. Thus eigenvectors must be affine functions of the attributes.
Result 3: Consequently, the analogy relation of Eq holds exactly in this model, independently of the magnitude or distribution of the {sk, k 1...d.
Result The eigenvalues of PMI are (22)) follows from the fact that of (log M) = (24) for k = O,..., d. When 7 is small (n7 0 if q 1 for all k), the matrix log M has simple spectral structure, with a top eigenvalue An 248 whose eigenvector approximately uo; and d attribute eigenvalues d 2d which correspond approximately to the Uy 's. All other eigenvalues zero.

---

Numerical validation: The inset of Figure D confirms that the spectrum of the PMI matrix lacks the higher-order modes present in the spectrum Myj. For each realization of the PMI matrix in the symmetric case simulated here, there are exactly d semantically relevant eigenvalues in the spectrum.
Next, we turn to the emergence of linear analogies in model. We define the top-1 analogy accu- racy by considering how frequently analogy parallelograms defined by equation I1 satisfied, as
are approximately
top-1 accuracy
1
Σ (A.E,
8 (D, argmin WB Wc Worlz)
(WD,
(13)
ITI
D'EV
where T is a set of analogies consisting of quadruples of words satisfying A:B::DC,, j(i denotes the Kronecker delta ()( = iff and zero otherwise), and V denotes the vocabulary of all possible words. That is, we check how often, for a given analogy family TT, the analogies in that family, such as (King, Man, Woman, Queen), are satisfied in embedding space, with WQueen being the closest vector in the vocabulary V to WKing W + W woman.
In Figure Za we report analogy accuracy using embeddings derived from Wikipedia text co-occurence matrices. The Mikolov et al. analogy task set consists of 19.544 sets of four words analogies, e.g. divided among 13 familics c.g. dnjeetive-superlative verb- curb-paticcill,, country- nationality, etc. In Figure 2p report the average performance for embeddings obtained with different Ma and log( M,y). The log(M) target performs strictly better than the raw M target, and saturates at high embedding dimension.
haand:hands:ratttinn
We compute analogy tasks in the symmetric (q = 1, i.e. Ps: = 1/2) binary semantic model (in d 8) as in the Wikpedia text data, by constructing sets of words T(k,, k2) that differ in two semantic dimensions j and l2 and satisfy the parallelogram relation. Each analogy is defined by a base word,
(ab (A) +1 with og (k1) = -1 and ag²) = -1 fixed. There are 2d-2 such base words (and therefore analogy tuples) in the vocabulary. The analogy is constructed in the obvious way, defining 0 ao + 2ê,, Q2 ak + 2êkr, and a = ag + 2ê,, + 2ê,, (where ég denotes the 1-th standard basis vectors in semantic space). Denoting wo the representation of ar,, W the representation of a etc., the analogy for base word ogo has score of if
W12 = argmin ll(w1 Wo + W2) -w/12
(14)
and zero otherwise.
In Figure 2 show the emergence of linear analogical reasoning for single realization of this model for different fixed k attributes, averaged over the liz * k1 'hee broadly distributed 85 se's give rise analogy performances that vary considerably with embedding dimension. In Figure 21, show the performance for the same targets that are used in the text data. Validating our results, the log(AM) target achieves 100% accuracy, regardless of the 5& distribution for K d. This is better than the Mij target, because performance there is only good when entire eigenbands of the spectrum are included (as show in the appendix). For broadly distributed 8y (as occurs in real text data, cf. Figure- Tb. the bands mix and linear analogical reasoning is lost at increasing K.
7. Additive Noise Perturbation to the PMI matrix
We now establish the robustness of the spectral structure of log M under additive noise. Let us consider a perturbed matrix:
log (i, = log (i. E(.
where ((i,j) are independent, zero-mean random variables with bounded variance (i) of and {(..j) {(j, i) preserve symmetry. We have: log M' log M + A with A the symmetric noise matrix. The spectral norm of the random matrix A is asymptotically || |2 ~ 20 20 22 (see theorem 2.1 of (34). At fixed Øg in the limit of large d, this norm is negligible in comparison with distance between the semantic eigenvalues, which is (22) /d) in the non-degenerate case. We can invoke the eigenvalue stability inequality, AA (log M') >u (log M)l S ||||| 2d/2 (a consequence of Weyl's inequality, see se)) to justify that the eigen-decomposittinn of M thus not affected the limit of large d. The linear analogies of Eq II is thus approximately preserved in that limit.
7

---

Numerical validation: Analogy performance under this perturbation is shown in Figure- 2e). We confirm a strong robustness noise: even with og z (1), excellent analogy accuracy possible for the PMI case. The performance of linear analogies degrades at high K when enough "bad" eigenvectors from the noise are introduced. This occurs at an embedding dimension of order K 12/2 as shown in appendix.
8. Spectral Stability under Vocabulary Subsampling
Our model assumes that all the possible 24 combinations of attributes are incarnated into existing words, an assumption that is clearly unrealistic. We now show that even if we randomly prune an immense fraction f of the words, the spectral properties of the PMI matrix are remarkably robust, as long as the number of words m = f2 >> d.
We analyze the impact of this sampling procedure the eigendecomposition of log M in the symmetric case Ps = 1/2 for all k, and focus on the interesting, non-constant part of the PMI matrix. From Eq 12 obtain:
(15) where Ā = AD1/2, After pruning, we restrict Å to the retained vocabulary, yielding a matrix As E Rmxdd Our goal is to study the eigen-decomposition of AsAz. Its positive spectrum is identical that of the x d Gram matrix:
log - 6. 11 ADA ÄÄ
G== As 11/2 A3
Let af denote row of As. Each a is drawn independently from a uniform distribution on the hypereube. We have G 1 Ei=t... a,a? Da/2, Thus, EGG] mD where Eet = (0())
The convergence of Σ to the identity is described by the Marchenko- Pastur theory for sample covariance matrices of the form Σi=I.. a, /m, stating that eigenvalues converge unity as m/d + oo [36]. Thus for m > d, the eigenvalues of log Mg follow that of the un-pruned case, except for a trivial rescaling of magnitude m/24..
Acting with the operator Ag eigenvectors of the Gram matrix, one obtains the desired eigenvectors of ÁAT_ As m/d + oo, the Gram matrix eigenvectors are simply the set eą R of basis vectors associated with a single attribute, and Ase: x vs is the attribute vector introduced above. The limit m >> d thus recovers the decomoiccooooiiioonn of the un un-pruned matrix, and analogy must be recovered.
Numerical validation: in Figure- 2 D. we report results on a sparsified variation of the model in d 12 for retention probability f 0.15 (see Appendix for a representative sparsified co-occurrence matrix). Despite both multiplicative noise and the removal of > 97% of the co-occurrence matrix, excellent analogical reasoning performance can be obtained for K 2 d.
9. Analogy structure after removal of all pairs of a given relationship
Remarkably, it is found that linear analogical structure persists even when all direct word pairs of given attribute are removed from the training set [8]. Considering for example the massuline-femiint relationship, even if all sentences where all the pairs of the type (king, queen). (man, woman), (actor, actress), are removed, one still obtains that
This observation is naturally explained in our model. Fix attribute index k: = 1. Consider all the 2d-1 pairs (i.)) such that words and j differ only in the value of the first attribute k; i.e., a (1) = a for all ب 1, and af = -a!. Next, define modified oo-occurrence matrix log(M'(i,, where we set log ( j)
for these 2d-1 entries, while leaving all other entries identical those of M. Let = M M be the resulting perturbation.
The operator norm of A bounded by trace(ATA)//2 ~ V22 (since is a sparse matrix which consists of 2d elements of order one), which negligible with respect to the eigenvalues M, of order 2d as discussed in the previous section. Again using the eigenvalue stability inequality,
8

---

Analogy Accuracy
1.00
(a)
(b)
(c)
0.75
0.50
0.25
0.00
100
10a
102
10a
10
10
102
10a
10"
10
102
Embedding dimension, K
Embedding dimension, K
Embedding dimension, K
capitay-capital
male-female adj-adverb
adj-s.periative verb-partiicpll
noun-plaral verb-plural
capttry-capita22
No pruning
Prune xus excl. k-nd
cumrecy
ad) antorym
country-rationaliy verb preteribe
Average
excl.
city state
adi comparative
Prune incl.
Figure 3: (a) Performance of the log(M) Wikipedia text co-oceurrence matrix for analogy tasks. (b) As in (a), but having pruned from the co-occurrence matrix all co-occurrences of pairs matching the indicated analogy. The average, in black, reports the analogy accuracy when all analogies are pruned from the corpus. (c) Pruning of analogies from the co-occurrence matrix in the symmetric synthetic model over ten realizations st E (0, 1) disorder for d = Solid curves represent the average analogy performance analogies involving the unpruned dimension, while the dashed curve reports performance on analogies that involve the dimension affected by pruning.
obtain that if eigenvalues of M are not degenerate, its eigenvectors and eigenvalues are not affected in the limit of large d. Consequently, the embedding of each word is unaffected in this limit, and I still holds.
Numerical validation: In Figure Bl show the performance of the PMI matrix for different analogy families. For each family, we construct pruned PMI matrix, with co-occurrences of pairs matching the analogy set zero, and find minimal performance degradation in analogy accuracy in Figure 3.. We try this experiment the model, pruning either the strongest semantic direction (WLOG, t 1) or the weakest semantic direction (WI.G) k d in Figure 3 In both cases, linear analogies survive this perturbation, with perfect accuracy appearing at = d. Analogies that include the pruned dimension perform similarly analogies that do include the pruned dimension. As with the real data (Figure 3b), this pruning introduces additional noisy eigenvectors the representation high embedding dimension which eventually leads a breakdown of linear analogies.
10. Discussion
Word embeddings are central to the interpretation of large language models. Surprisingly, linear subspaces characterizing semantically meaningful concepts, originally found in classical word embedding methods, are also realized in large language models [12 13
37
38 where they can
enable control of model behavior [17 and are crucial for fact retrieval 33 While these linear subspaces are sometimes encoded in context-dependent manner [155] is possible to obtain context- free LLM embeddings (as [40]) to test our views in modern LLMs. The prevalence of these lincar subspaces suggest that the statistics of language plays fundamental role, and revisiting classical word embedding methods allows us to develop a sharper understanding this question.
We have shown that linear analogies in the word embeddings in such algorithms naturally arise words are characterized by a list of attributes, and each attribute affects the context of their associated word in an independent manner. We formulated simple, analytically tractable model of co-oceurrence statistics that captures this view, where attributes are binary and combinations of attributes correspond to word. This model rationalizes various observations associated with the emergence of línear analogies, and provides a fine-grained description of how they depend on embedding dimension and the co-occurrence matrix considered. Remarkably, the model is extremely

---

robust to perturbations including noise, the sparsification of the word vocabulary, the introduction of correlations between semantic attributes, or the removal of all pairs associated to a specific relation.
Limitations
Our model obviously a great simplification of actual word statistics. For example, polysemantic words such as bank (a bank, river bank, to bank) complicate co-occurrence statistics. Furthermore, some attributes may be hierarchically organized [38]; this property can be captured by random hierarchy models [4T] and revealed by diffusion models 42 k The possibility that such properties may be studied from co-occurrence alone is an intriguing question for future work. It calls for the development improved analytically tractable models that capture these effects.
Data, code availability, and compute budget
The code used to produce the model results, Wikipedia co-occurence statis- tics, and figures is available analagiallls...aaa word-wod---oe wordd-embddding reproduction and in the supplementary files. All
GitHub hhtts://gittin: com/DJKorchinski.
simulations together in under 150 minutes Nvidia H100.
Acknowledgments and Disclosure of Funding
D.J. Korchinski acknowledges financial support from the Natural Sciences and Engineering Research Council of Canada (NSERC PDF 587940 2024). D. Karkada thanks Google DeepMind and BAIR for funding support. This work was supported by the Simons Foundation through the Simons Collaboration the Physics of Learning and Neural Computation (Award ID: SFI-MPS-POL- 00012574-05), PIs Bahri and Wyart.
References
[1] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In ICLR, 2013.
[2] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
[3] Jeffrey Pennington, Richard Socher, and Christopher D Manning, Glove: Global vectors for word representation. In EMNLP, 2014.
[4] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In NAACL, 2013.
[5] Omer Levy and Yoav Goldberg. Neural word embedding implicit matrix factorization. In NeurlPS, 2014.
[6] Dhruva Karkada, James Simon, Yasaman Bahri, and Michael R DeWeese. Solvable dynamics of self-supervised word embeddings and the emergence of analogical reasoning. arXiv preprint arXiv:2502.00886 098000 2025.
[7] Takuma Torii, Akihiro Maeda, and Shohei Hidaka. Distributional hypothesis as isomorphism between word-word co-occurrence and analogical parallelograms. PloS one, (9(10):e0312151 2024.
[8] Hsiao-Ya Chiang, Jose Camacho-Collados, and Zachary Pardos. Understanding the source of semantic regularities in word embeddings. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 119-131, 2020.
[9] Viljami Venekoski and Jouko Vankka. Finnish resources for evaluating language model se- mantics. In Jörg Tiedemann and Nina Tahmasebi, editors, Proceedings of the 2 Nordic Conference on Computational Linguistics, pages 231-236, Gothenburg, Sweden, May 2017. Association for Computational Linguistics.
10

---

[10] Tzu-Ray Su and Hung- Yi Lee. Learning chinese word representations from glyphs of char- acters. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 264-273, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 1...863//1//D11-1222
[11] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguisties, 135-146, 2017. doi: 10.1 162/tacl a 00051.
[12] Yibo Jiang, Goutham Rajendran, Pradeep Kumar Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 21879-21911. PMLR, 21-27 Jul 2024.
[13] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 39643-39666. PMLR, 21-27 Jul 2024. URL https //proceddingsm.. mlr. www.:mmwr.....ee.emmm matsss/v35//prreee htmmll
[14] Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text-controlled generative models. Advances in Neural Information Processing Systems, 36, 2024.
[15] Evan Hernandez, Arna Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. In The Twelfth International Conference on Learning Representations, 2024.
URL httl:tttt///eeeeeiiieee www.eeoorrmdddw/1122222
[16] Yuchen Li, Yuanzhi Li, and Andrej Risteski. How do transformers learn topic structure: Towards mechanistic understanding. In International Conference on Machine Learning, pages 19689-19729. PMLR, 2023.
[17] Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Na- joung Kim, Arya McCarthy, and Hosein Mohebbi, editors, Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 16-30, Singapore, De- cember 2023, Association for Computational Linguistics. doi: 10.18653/v http://1/2222.bakbbba 2. URL https: acl www.anollantoooo.ooooooooooooole .wrpegg..rgg//// org/2023 blackboxnlp-1.2 1.
[18] Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. arXiv preprint arXiv:2401. 011967 2024.
[19] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04088 2022.
[20] Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXiv:2310. 2023.
[21] Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec-style vector arithmetic. arXiv preprint arXiv:2305.1611 16330 2023.
[22] Anne Lauscher, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulić. A general framework for implicit and explicit debiasing of distributional word vector spaces. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8131-8138 2020.
[23] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024.

---

[24] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, al. Representation engineering: top-down approach to ai transparency. arXiv preprint arXiv:2310.014400 2023.
[25] Tatsunori B Hashimoto, David Alvarez-Melis, and Tommi S Jaakkola. Word embeddings
as metric recovery in semantic spaces. Transactions of the Association for Computational Linguistics, 4:273-286, 2016.
[26] Omer Levy and Yoav Goldberg. Linguistic regularities in sparse and explicit word representa- tions. In Proceedings of the eighteenth conference on computational natural language learning, pages 171-180, 2014.
[27] Sanjeev Arora, Yingyu Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings. In TACL. 2016.
[28] Kevin Allen, Timothy Hospedales, and David Amos. Analogies explained: Towards understand- ing word embeddings. In ICML, 2019.
[29] Alex Gittens, Dimitris Achlioptas, and Michael W Mahoney. Skip- Skip-gram- zipf+ uniform= vector additivity. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 69-76, 2017.
[30] Kawin Ethayarajh, Dan Jurafsky, and Ranjay Krishna. Towards understanding linear word analogies. In EMNLP. 2019.
[31] David E Rumelhart and Adele A Abrahamson. A model for analogical reasoning. Cognitive Psychology, 5()) 1973.
[32] Ken McRac, George S. Cree, Mark S. Seidenberg, and Chris Menorgan. Semantic feature production norms for large set of living and nonliving things. Behavior Research Methods, 37 (4):547-559, November 2005. ISSN 1554-3528. doi: N.37558BBF03192722
[33] Charles F Van Loan. The ubiquitous kronecker product. Journal of computational and applied mathematics, 123(1-2):85-100, 2000.
[34] Mark Rudelson and Roman Vershynin. Non-asymptotic theory of random matrices: Extreme singular values, April 2010.
[35] Terence Tao. Topics in Random Matrix Theory. American Mathematical Society, 2011. ISBN 978- 17744-74555.
[36] Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv: 1.3022, 2010.
[37] Wes Gurnee and Max Tegmark. Language models represent space and time. In The Twelfth International Conference on Learning Representations, 2024. URL ltttps://pperrvviee
net/forumiidd=
[38] Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. In The Thirteenth International Conference on Learning Representations, 2025. httt:::t/ttp/ttreeeet
[39] Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple Word2Vec-style vector arithmetic. In Kevin Duh, Helena Gomez, and Steven Bethard, ed- itors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 1: Long Papers), pages 5030-5047, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v 1/25/3//1/022.... 1ttp:///22222..mlloogo.... lon...1
[40] Rishi Bommasani, Kelly Davis, and Claire Cardie. Interpreting pretrained contextualized repre- sentations via reductions to static embeddings. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pages 4758-4781, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v 1/20/..acl-mmm mc1-mann.431
12

---

[41] Francesco Cagnetta, Leonardo Petrini, Umberto M. Tomasini, Alessandro Favero, and Matthieu Wyart. How deep neural networks learn compositional data: The random hierarchy model. Phys. Rev. X, 14:031001, Jul 2024. doi: 10. 1 100. 14.003X011 htl..httss://n1m
aps. oos.oo///oi/10.. www.://ee.../////.e....... 14. 031001
[42] Antonio Sclocchi, Alessandro Favero, Noam Itzhak Levi, and Matthieu Wyart. Probing the latent hicrarchical structure of data via diffusion models. arXiv preprint a7X11:2410.13770 2024.
[43] Antonio Sclocchi, Alessandro Favero, and Matthieu Wyart. A phase transition in diffusion models reveals the hierarchical nature of data, arXiv preprint 1rX1v22402.16991, 2024.
13

---

Appendices
A1. Eigenspectrum of the p(k) matrices. Recalling the definition of the p(k) matrix,
plk)
1 + Sp
1 - 9r5k
(16)
1 9k
1 + 5K
the eigenvalues are given by:
(k) 1 + o +
(2 (1 22)
8k +
(17)
and the corresponding eigenvectors are
1 v (0 1) -895
=
2(1 90)
(18)
In the special (symmetric) case that q = 1, the eigenvalues are simply 2 and 2s for the and + cases respectively with cigenvectors U1 = [1, 1]
Thereom proof
Theorem:
(k) (h) indexed by word of attributes a, and word j of attributes a2 defined by Equation (16) has eigenvectors
The matrix - ((:)))))) (a
ys
(
with as E {+1,-11
(19)
where & indicates a Kronecker product. Its component for word of attributes a; is thus
us(()
(a, ye (a
bdl (a.).
(20)
with associated eigenvalue:
As
(k) ag
(21)
where the _ are eigenvalues of the x 2 matrices p(k) defined in Equation (16), and (k) are the
two eigenvectors of these matrices given Equation (88)
Proof. This theorem follows from the fact that the matrix M is Kronecker product of the d matrices p(k), and from standard results on the eigen-deconposition of these products (33] It suffices to show that M matrix admits Kronecker representation. We show this by induction. Let
M (1.))
II (()) (k) .aa
(8))
(22)
denote scaled co-occurrence matrix up semantic dimension d'. Clearly M Mf(d) We will show that (d') P(M)
To do so, we consider a canonical basis whose basis vectors correspond to words. Word is the vector of zero everywhere, except at position n(aa,, d) that follows:
n(as, d) Σ 2d (6)
(23)
a +1
Note that the binary representation of n(a,, d) is simply given by a (k) with the change a (k)
-1 + and a + 1. A useful consequence of this definition is that
n(a, d 1) 2n(a, d) 8 2n(a, d) + +n(a d+1 1).
(24)
14

---

Base case: For a single semantic dimension, we trivially have that
M(d=1) p())
which is the Kronecker product of one term.
Induction step: Assume that M(d') = ox p(h) for some d' 2 1. We must show that M (d'
M & p(d
By the definition in Equation (22)
have:
(d'+1)
d' 11
Mld'+Dn +1 (1,1)
II
(()) (a
plk) plal
(a."
.aft)) ple (a a (
Using the row-indexing notation Equation 23 this expression can be written as:
Mld++1 m((dd"+)))..id++++)
M(d)) n(o, mila, d') nia (d)) ())aa" 1),
pld'+1)
The Kronecker product between an arbitrary matrix A and matrix B E IRPx4 is defined as
(A & B Di aj+s Ag
With B = 1(d R2x we then have
1l1++1) "it.d' m() (M((') & p(d'+1) dn(as,d) tala! (1) (1 (f) (f
and using the indexing identity Equation 24 this simplifies to
M
m(i.d'+)),,(dd'34 (Mr(d") p(d'+1)
Thus
M(d++1) = Mddd)) @ p(d'+1)
By the assumption of our inductive step, we thus get:
M +1) p(k).
Properties of Kronecker products
may be instructive to review a few standard results on the eigen-decomposition Kronecker products. For product of matrices A Rmxx with
Au
A
(A 2 B)
(25)
Am B
Amn B
X and u are a eigen(value/vector)) of A so Xu = Au and X, and v similarly satisfy A = Bv, then the vector
u1 V VgV
=
(26)
Un V
15

---

is an eigenvector of A & B with eigenvalue Audo [33.
This can be seen schematically, with:
A114 Bv
+
+ UIn Bv
(A @ 8)
Amiul u1 Bv
+
+
Un Bv
N E, Anau
Av E, Amius
11
Andu
un
Aodu(u @ v)
Explicit construction for d=2
In this section, we provide explicit example of the construction of the p(k) matrices and the cigenvectors and eigenvalues for the case 2, which the smallest case to support linear analogy.
Consider the symmetric q case for d with two semantic strengths $1 and 82- The p(k) matrices are:
81 +1 1- $1 p(1)
(27)
1- 81
81 11+1
and
(2)_
82 +1 1 82
(28)
1-82
+1
with eigenvectors 01 = (1, and eigenvalues A1 = 22(171)/2 The matrix M is given by:
(1 (1++)) 8()
(1 - s (22)
M
(1 - 81 ((2)
(1 + 81 ((
[1 +1+811 s1)+
(1 (1+ 82)
(1 (1- 82)
(1 s1) 82
(1++))11-22
(1+8)) 82)
(1 (1-81) 82)
[1 $1)
(1 (1++
(1-s))(1- 82)
(1+s))
(1 $1) 52)
(1 (1) (1-s))(++s)
(1+5))(1-5) (1))1-5)) 82 (1 s1) (1))))
The eigenvalues are An = -o 282, A
= = =22318 with corresponding eigenvectors: vo 11), 0 = (-1,-,,,,1 01,--1,1111 -1,-1,1,-11) and U12 (1, 11) These
are precisely the eigenvectors and eigenvalues guaranteed by the theorem.
In d 2, there is exactly one analogy possible. It can of course be made explicit, by using the following mapping of words to semantic attributes
OtMan (-1,-1)
O'mmaan = (-1,1)
OrKing (1,
QQueeen = 11,
where the attribute corresponds to "Royalty" and to "Gender". The word embeddings for
K and K 3 are given Table These embeddings are generated by using the embedding matrix W, [vo, v1 and W; = [vo, V2] for K = 2 and K = respectively. The representation of each word the corresponding column of diag(An, A2, For example: WQueen, [diag(v Ag Vdr, V²2) W]..quenn
The only possible analogy (up to trivial permutations) in d = is then given by the equation:
WQeen WWoman + WMan WKing
(29)
which is satisfied uniquely when K 3, but not when =
2 as is depicted in fig. and fig. 5 As
the example here shows, each successive eigenvector (after the first trivial vector) included in the
16

---

Word
K 2 Embedding
K = 3 Embedding
Man
2(1, -V81
2(1, -V81. 52
Woman
2(1,
2(1, /81. +/
King
2(1, +
2(1, + /111. -V
Queen +v.
2(1, 11:1s$1++ +vS2)
Table I: The explicitly constructed embeddings for the words "Man", "Woman", "King" and "Queen"
the = case. The K 3 embeddings are obtained by adding third dimension corresponding to the semantic strength $2-
M7
Womma
WWomaan
WMan 2L,
O UuQueen
Wing
(1, v
Figure 4: A degenerate analogy when K 2, since both WQeeen and WKing satisfy x = WQueen
(WMan WWoman)
WWomran
2 21, V8I. /82)
Wiqueen = (L, / /82)
WWWommm
O
WMMa 2 (L, /s1, $2)
WKing 2 (1, /51, /s2)
Figure 5: The analogy no-longer degenerate when
representation allows the embedding to disambiguate one additional semantic direction with K only the royalty axis present in the representation, while in K 3, both the royalty and gender axes are present.
A2. Coefficients characterizing the PMI matrix
The definitions for the three parameters 6g. 7e and "/ are as follows:
8% 4
[oog(1 + sx) + log(1 + sx) + 2 log - gesx)],
1
1 [log[ + sk) logg1111111 + 9f sx)
7k 4 log(1 + sx) + log(1 + 9M) log(11 gusu)]
A3. Tertiary model with neutral attribute
We now generalize our model to the case where each semantic attribute (4) can take one of three values: 1,, 0, or +1. The value 0 corresponds to a neutral setting, indicating that the word does not express this semantic dimension. This extension allows to model richer vocabularies where many attributes may be inactive for given word. For simplicity of notations, we consider the case where the three values of each attribute are equally likely.
17

---

We define the co- co-occurrence matrix p(k) for attribute & as symmetric 3 x matrix, where the rows and columns correspond to {-1, +1,0 We assume:
1+ 5k + /
1 - 5k
ft fa
p(k) 1 5k
fx
1 +
8k
+
fx
1- fi
fk.
1 2fx
Here: S/. E [0, 1] encodes the strength of similarity between matching vs. opposite polarities (+11 1)) f& < is the frequency of non-neutral settings for attribute k.
This matrix can be decomposed as:
p(M) ( + faB( + +0(2
where the unperturbed component is:
1 + Sk 3k
1). B(K)
(k)
Sk
+ Sk
1
2
We now analyze the eigenvalues of p(k) using first-order perturbation theory in fk-
Eigenvectors of A(k This symmetric matrix has three orthonormal eigenvectors:
V3 (1, 11,
(constant mode)
V1 V2 (1, -1
(contrast between and +1)
1/ v6
(1, 21)
(contrast between neutral and polar values)
Resulting eigenvalues of p(k)
Up first order in Sk. perturbation theory gives:
1 3 - O(f))
2 (k) 2sk + O(f2)
2 (k) 6. O(fi)
Interpretation. order in fE.
The top eigenvalue corresponds to the constant mode and is unaffected at linear
The contrast direction between -1 1 and +1 preserves the same eigenvalue as in the binary case: 28k.
A new third direction, orthogonal to both, emerges with small but nonzero eigenvalue 6fk- reflecting the semantic impact neutral attribute values.
This shows that even in the presence of neutral words, the dominant structure of the embedding space necessary for analogy remains intact up to small corrections.
A4. Additional numerical study of the model
Here, we report several additional experiments on the symmetric binary semantics model.
Dependence of analogy accuracy on disribution of the
In Figure we measure the analogy accuracy for narrowly distributed semantic strengths 8 ~ (1112, 2, 10 10-3). Accuracy with the M, matrix target is perfect whenever complete eigenband is included in the representation. Perfect analogy reconstruction possible for the symmetric model, but is very fragile and requires narrow distribution of sj.
18

---

1.0
Analogy Accuracy
log(M)
0.6
0.4
0.2
M
0.555 10
0.0
10"
10a
10a
Embedding dimension, K
Figure 6: Analogy performance for narrowly distributed 8p for different matrix targets in d = 8.
Shading represents the standard deviation across 50 replicates. Vertical lines at K₁ 1 + (9)
K₂ Ki + (0) and K3 - K2 +
mark the complete inclusion of the different eigenbands.
1.0
d-5
Analogy Accuracy
0.8
dm6
log(/)
d-8
0.6
M
0.4
0.2
(0.
0.0
10"
10a
10a
Embedding dimension, K
Figure 7: Analogy performance for uniformly distributed są in different dimensions. Shading represents the standard deviation across 50 replicates. Vertical lines mark K = 5, 6, K = and K 8, corresponding to the dimension of the semantic embedding space for the main curves.
When there is greater variance in the sæ distribution, e.g. E (0, 1). then the higher-order eigen- values (corresponding s2.w) begin be included, without first capturing all of the first-order cigenvalues. In this case, perfect analogy accuracy is typically not attained and the peak in accuracy happens higher d, included at K = d, as is indicated by the vertical lines in Figure 7
be seen Figure In the log(M case, the semantic eigenvalues all
Vocabulary sparsification
In Figure show the effect of sparsification the co-occurrence matrix M at fraction f 0.15. Each row M corresponds to co-occurrences including word i. By removing row and column i, effectively remove word i from the vocabulary. Despite removing 98% of the co-occurrence matrix, the overall hierarchical structure is still readily apparent. We test the effect of sparsification the top d semantic eigenvalues of a log(M) target in Figure As is predicted top d eigenvalues This breaks down only when the number retained words approaches m - d, meaning that tremendously sparsified vocabulary still retains the same eigenvalue structure.
the theory, the minimally perturbed by sparsification beyond simple rescaling of m/24.
Breakdown analogy accuracy with increasing in the presence noise.
Here, we study the effect introducing noise perturbation log(M',)) log( +Gy onto the PMI matrix, with ειj ~ (0, og). The spectral density has two prominant features: a set "semantic" eigenvalues, with Dg 2d and a set of small, noisy eigenvalues starting at a scale set by as as can be seen in Figure 10A.
19

---

Original
Sparsified
4096
599
4096
599
Figure 8: The effect of sparsification (f = 0.15) on a realization of the co-occurrence matrix with 12 and Sp E (0, 1). Colours represent the value of log(M).
300
10-1
m
250
10-2
0.11
Eigenvalue,
200
101
10a
150
100
1000
2000
3000
4000
Number of retained words m = f fx2"
Figure 9: The value of the top d 12 eigenvalues for log(M) matrix with 8 (0, 1) function of sparsification. Shaded area reflects the standard deviation across 20 realizations of sparsification, for the same fixed Sk. Inset is the same data, but rescaling by the asymptotic value of each eigenvalue so as to effect a collapse for m >
The 2d 2d dimensional matrix has spectral norm converging to ||||| 2a, v2 as d oc. This scale collapses the noise floor of (M') across both dimension and a as can be seen in Figure [Op.
As the the embedding dimension is increased, additional noisy eigenvectors representation. The representation vectors are constructed from the w eigenvectors as
included in the
(30)
We expect these noisy eigenvectors interfere with analogical reasoning when their combined magni- tude is of the same order magnitude as semantic eigenvectors, i.e. when
K
~ |1112 ~ 2d
Wh
A=d+1
Since the uksd are orthogonal, in the limit K expect linear analogies to break down when have an embedding dimension order
> d. we have that Σ
Ek=d+1 W6
VK0220/2
2/2 We
K ~
24/2
0
20

---

(a)1o²
(b)
(c)
1.0
101
Analogy Accuracy
10a
10a
Density, p())
noise floor
10 1
Noise Scale, oc
pl (a:2
10
0.6
10-2
semantic Au
0.4
10
10-3
10
00.1)
10-1
8x.F (0.1)
0.0
(0.1)
10
10-1
10"²
10
10a
10-1
109
10
10-1
109
10.
Eigenvalue,
24/2)
K(ag/24/2)
Figure 10: (a) Eigenvalue spectrum of the PMI matrix with the addition of an elementwise inde- pendently and identically distributed gaussian noise, for d and N 10 replicates. (b) As in a rescaled to collapse the noise floor across both noise scale and dimension. (c) The accuracy of analogy tasks for different dimensions and noise scales, rescaling the embedding dimension to collapse the breakdown in analogy accuracy at high d.
This is confirmed by the collapse in Figure TOc.
A5. Extension with correlated attributes
In so far we assumed that different attributes affected co-occurrence in an independent fashion. Here we show that our main result holds, even when is not true. We consider a more generic co-occurrence model with P(i.j) Z(i)Z() II (i,j) where p(k.k') (i, = 1 + S aj and the Zi) are normalization factors chosen such that E, P(i.j) P(i). To maintain the symmetry of the co-occurrence matrix requires 8k, _k
(k) (k),
Consider, to simplify notations, the symmetric pk 1/2 case. We have that E, II p(.,k)(..)) 1 + (s) so at this order of approximation henceforth considered, we have: P(i,)) z P())P()))TA p(M,k((i.)) where P(i) = 2 2 As result, the PMI matrix for pair of words, is is
PMI(i,)) Σ log( (K) k,
(31)
Each of the constituent log (p(ak?) can be re-expressed as
log (P(RAA) b)) log(1 + ab) = 6ū,N* + na
(32)
where 5A,A' 1/2 (log(1 + SK,K) + log(1 Sk,K() and Yk.k = (log(1 + 8k,K)) log(1 8E,K)))
Identifying ai (k) and aj (K)) with a and b, we have that
PMI 5 + AyAT
(33)
where = 11 -1 1 6R,4' the rows of A E 2² xd correspond to ā; as before, and the elements of the 7 matrix are just the "k.k".
Thus the main conclusions of Section 6 hold: The rank of the PMI is at most d+ 1 and its eigenvectors are linear in the attributes, implying that linear analogies hold exactly. This is the central result. However, because is not diagonal, eigenvectors of the PMI will linearly combine attributes.
21

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] is generally preferable to "[No]" it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction accurately describe the paper's main contribution: a theoretical generative model that explains the emergence of linear analogies in word embeddings and accounts for previously observed empirical phenomena. This reflected in the theoretical derivations (Sections 5-9) and numerical validations (Figures 1-3) compared results on empirical data.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
22

---

Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes dedicated "I "Limitations" subsection within Section 11 (Discussion). This section acknowledges simplifications in the model, such as handling of homographs and hierarchical attribute structures areas for future study.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well sell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Theoretical results, such as the eigendecomposition of M (Theorem 1, Section 5) and the rank and eigenvector properties of log M (Section 6), presented with their assumptions. Proofs or derivations are provided the main text (e.g.. proof sketch for Theorem 1) and supplemented by details in Appendix Al and A2.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
23

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper describes the generative model (Section 4), its parameters used in each figure (e.g., d = 8 or d 12 for simulations, distributions for 8k, noise parameters ag. pruning fraction O). and the analogy evaluation metric (Equation 12). The analogy tasks for the empirical data are descried and their source cited.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code and Conda environment necessary to reproduce the numerical simulations and the figures included in the supplementary. The full Wikipedia co- occurrence statistics can not be included due to size constraints.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not
24

---

including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: For the model- based experiments, parameters like the number of attributes (d), distributions for semantic strengths (sg), noise levels (ot), and vocabulary sparsification (f) are specified in the text and figure captions (e.g., Figure 1, Figure 2). For Wikipedia
experiments, the vocabulary size and analogy co-occurrence statistics are used with vocabulary of 10,000 words and the Mikolov et al. analogy task set.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Figures 2b, 2d, 2e, 2f, 3c, Figure (Appendix), and Figure (Appendix) include shaded areas representing the standard deviation across analogy categories model realizations. This indicates variability the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
25

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Most of the experiments run in few minutes H100.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research is theoretical and analytical, aiming to understand existing phenomena in word embeddings. It does not involve human subjects, direct data collection from individuals, or applications with immediate ethical risks outlined in the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper focuses on a foundational, theoretical understanding of word embedding models. While the authors believe in the positive social impact of scientific understanding, we do not explicitly detail broader positive or negative socictal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate
26

---

to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper introduces theoretical model and does not release new data or pretrained models that would carry high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper properly cites foundational models like Word2Vec and GloVe (e.g.. references [1, 3)) and refers to the Wikipedia corpus and the Mikolov et al. analogy benchmark, which are standard and publicly known assets the field. Specific licenses for these general resources are not typically detailed in individual research papers focused on theoretical modeling based on them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce new datasets or software assets for release. The primary contribution a theoretical model.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The research presented in this paper does not involve crowdsoureing experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research presented in this paper does not involve crowdsoureing or experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core methodology of this research involves a theoretical generative model and mathematical derivations. LLMs were not used as important, original, or non-standard component these core methods.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy for what should or should be described.
29


Reviews:
[{"review_id": "cnepZb6T0y", "reviewer": "Reviewer_Zp5k", "summary": "This paper proposes a theoretical generative model for word co-occurrences. By defining each word as a vector of d binary semantic attributes and assuming that co-occurrence probabilities result from independent attribute interactions, the paper analytically derives the PMI and M matrices and systematically explains the mechanism behind the emergence of linear analogy relationships in word embeddings.\nBeyond its core theoretical contribution, the paper also validates its predictions via extensive numerical simulations and confirms strong alignment with empirical statistics from real corpora like Wikipedia. Notably, the model is shown to be robust to noise, vocabulary sparsification, and even the removal of specific analogy word pairs.", "questions": "1. Could correlated semantic attributes bias the model’s predictions or its spectral structure?\n2. Has the model been evaluated beyond English corpora (e.g., on morphologically rich or logographic languages such as Chinese or Finnish)?\n3. In the era of contextualized word representations (e.g., BERT, GPT), what is the broader significance of studying static word embeddings?\nHow sensitive is the model to the choice of the PMI transformation (e.g., using shifted PMI, SVD truncation, or subsampling strategies)?\n4. Could you provide insights or explanations regarding the five weaknesses previously identified?\n5. The model assumes attribute independence. Could you provide examples illustrating that such independence is satisfied within the Wikipedia dataset?\n6. The physical meaning of some notations in Section 4 is confusing. Why does $p_k \\leq 1/2$? And what is the significance of The symmetric matrices $P^{(k)}$, are they denoting probabilities, since the text states $\\sum_j P(i, j) = P(j)$? However, the values in Equation (4) do not all lie within the [0, 1] interval—for instance, $1 + s_k > 0$—which is perplexing. Could the meaning of the formulas and symbols in Section 4 be clarified in greater detail?", "limitations": "Yes.", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. This paper proposes a theoretical generative model for words based on binary semantic properties, directly linking the structure of co-occurrence or PMI matrices to the emergence of linear analogies, thus effectively explaining phenomena observed in prior work.\n2. This paper not only provides a conceptual explanation, but also derives the spectral structure of the co-occurrence matrix through closed-form resolution. Moreover, by comparing the spectral properties of the model-generated matrix with empirical data from Wikipedia, the theoretical predictions are validated.\n3. Code and data are provided to ensure reproducibility of the paper.\n\nWeaknesses\n1. The mathematical notation and statements used in the paper lack clarity. Some sections, particularly section 4, 5 and 6, use a lot of mathematical notation, the logical transitions between mathematical statements and their explanations are abrupt, and even some notations are not explained, e.g. $\\delta_{a^{(k)}_i, 1}$. These paragraphs may be difficult to understand for researchers unfamiliar with these theories. The use of simpler and clearer language with appropriate examples may give the reader a better understanding.\n2. The modeling assumptions have limitations. The theoretical model assumes that semantic properties are strictly binary and mutually independent. However, the continuous nature and context-dependent relevance of semantics in the real world mean this assumption diverges significantly from reality, which significantly limits the model's broader applicability.\n3. This paper empirically validates on Wikipedia data and standard analogy benchmarks, demonstrating validity only on single corpus and synthetic data.  However, it was not validated on more complex corpora, such as social media text or multilingual data, and the results may not generalize to morphologically rich languages.\n4. The conclusions focus on the classical embedding model (Word2Vec/GloVe), but do not discuss the connection with the Large Language Model (LLM). Do the latter produce linear analogy due to the same mechanisms?\n5. The limited discussion of existing methods. Section III identifies the reasons why existing methods are problematic and whether more detailed elaboration or empirical results can be provided to quantify these problems.\n6. Binary attributes may oversimplify continuous/multi-valued semantics.\n7. The author assumes that  each attribute of a word affects its context in an independent manner. But real-world attributes may correlate, potentially underestimating complex interactions.\n8. The paper lacks empirical comparison with alternative models to contextualize the novelty and practical benefit of the proposed framework.\n9. Despite its theoretical depth, the model may have limited applicability to modern LLM embeddings.", "was_revised": true, "final_justification": "I appreciate the valuable efforts of the authors in the response, which have addressed most of my concerns. Considering the overall quality of this work, I would like to keep the BR rating.", "rebuttal": "Referee Zp5k \n\nWe thank the referee for their comments and criticism, they are useful to clarify the manuscript. \n\nWeaknesses: \n\nWe respond to the distinct points raised by the referee below:\n\n\n1. Summary of referee comment: The mathematical notation and statements used in the paper lack clarity, particularly in sections 4, 5, and 6. \n\nWe will improve the flow and clarity, particularly in the highlighted sections. The $\\delta$ refers to the standard Kronecker delta, so $\\delta_{a_i^{(k)},1}$ evaluates to 1 whenever $a_i^{(k)}$ is 1, and is otherwise zero. We note that the appendix also offers a more pedagogical treatment of several of the calculations, a review of Kroenecker products, and an explicit worked example for $d=2$. We will draw the reader’s attention to this earlier in the manuscript. \n\n\n2+6+7. Summary of referee comment: The model makes simplifying assumptions: (i) binary variables don’t capture multi-valued (or continuous) semantics, (ii) there may be correlations between semantic attributes. \n\nPoints 2+6 criticize the assumption of binary variables. Please note that our model can be generalized to multi-valued semantics. In fact, we had already performed an extension to tertiary variables in the appendix, and our findings carry through largely unchanged. We will highlight this result earlier in the paper. The extension to n-valued variables is straightforward, and may represent an approach to the continuous limit. We hope our contribution inspires such lines of inquiry. \n\nPoint 2 also points to the fact that semantic variables might be correlated.  This is an excellent comment, which led us to generalize our model to correlated attributes. We  briefly sketch this generalization below in answering Q1.\n\n\n3. Validation on a single test set (Wikipedia), and limitations of applicability more broadly to other languages or corpora. \n\nThe presence of linear analogies our theory seeks to explain has been demonstrated in numerous other languages, including more morphologically complex ones such as Finnish (Venekoski and Vankka, ACL 2017) and Chinese (Su and Lee, ACL 2017) when using the appropriate scale of morphologically aware tokenization, such as FastText (Bojanowski et al., ACL 2017). We will make sure to cite these references in our introduction, to highlight the generality of the problem we are addressing. \n\nThe objective of our contribution is to generate a robust, minimal theoretical framework that explains several key aspects of word embeddings. Other work in the realm of theory building (e.g. Arora et al. TACL 2016) also uses a single dataset and language (English Wikipedia). Comparing quantitatively multiple languages is certainly an interesting endeavour, but it presents a work on its own. It should not obscure the presentation of the new formalism introduced here in its purest form.  \n\n\n\n4+9. Summary of referee comments: This work addresses primarily a feature of context-free word-embedding models. To what extent does this generalize to LLMs? \n\nThe semantic content of language exhibits both elements that are independent of context, and others that depend on it (HP Grice 1975, Pustejovsky, J., 1995). Strictly speaking, our theory is making statements about the former, since it studies context-free embeddings like word2vec. Concerning LLMs, they must necessarily also capture this context-free element in their internal representations, and so our work has some bearing on them. Bommasani et al. 2020 showed that such context-free embeddings can be obtained by pooling the representation of words across many contexts. This approach could be used to test our views in LLMs, as we will discuss in the conclusion. \n\nThe work of Hernandez et al. 2024 ICML is also very pertinent: The authors find that the internal representations of LLMs obey linear analogy (up to a context-dependent linear projection), satisfying relations like $W_r (\\text{Miles Davis} - \\text{Billy Joel}) = \\text{Trumpet} - \\text{Piano}$, where the relational matrix $W_r$ is induced by the words “plays the instrument” (and thus depend on context). Thus, linear analogy structure survives in the inner representations of LLMs in a contextual way. Other work by Merullo et al., ACL 2024, indicate that these linear analogies are important for fact retrieval by LLMs. Understanding the emergence of contextual linear analogies remains a theoretical challenge; the ideas presented in our work are possible handles to tackle it.  We will discuss these works in conclusion.\n\n\n\n5+8. Limited comparison between our work and existing methods. \n\nThe limited comparison with existing methods that the reviewer points out reflects the novelty of our work: we are making predictions where other approaches did not, including on how the quality of linear analogies depends on the dimension of the embedding considered. To our knowledge, there are no alternative predictions in the literature on that. We simply emphasize in the text that some key assumptions of previous methods do not hold (such as the isotropy of the PMI matrix). \n\n\n\nQuestions:\n\n1. Could correlated semantic attributes bias the model’s predictions or its spectral structure?\n\nWe thank the referee for this interesting point. It has prompted us to consider an extension to the model that includes correlated attributes, which we will include in the appendix of the paper. The PMI matrix is robust to correlations.\n\nIn brief, in this extension equation 3 takes the modified form:\n$$ P(i,j) = P(i)P(j) \\left( \\prod_k P^{(k)}(i,j) \\right) \\left( \\prod_{k, k’} P^{(k, k’)}(i,j) \\right) $$\nwhere the $P^{(k,k’)}(i,j) = 1 + s_{k,k’} \\alpha_i^{(k)}\\alpha_j^{(k’)}$ introduces correlation between attributes $k$ and $k’$ of strength $s_{k,k’}$. This formulation approximately preserves normalization when the $s_{k,k’}$. The subsequent analysis of the PMI matrix carries through in the same manner, resulting in an identical factorization with equation 12, except that the matrix $D$ is no longer diagonal. All of the following results (1,2,3,4) hold, along with the emergence of linear analogies at dimension $d$. The main difference is that the eigenvectors of the PMI matrix in this case do not map one-to-one onto semantic attributes, but this has no impact on the linear analogies. \n\n\n2. Has the model been evaluated beyond English corpora (e.g., on morphologically rich or logographic languages such as Chinese or Finnish)?\n\nSee our response to weakness 3 above. \n\n\n3. In the era of contextualized word representations (e.g., BERT, GPT), what is the broader significance of studying static word embeddings? \n\nSee our response to weakness 4+9 above.\n\n\nHow sensitive is the model to the choice of the PMI transformation (e.g., using shifted PMI, SVD truncation, or subsampling strategies)?\n\n(i) Shifted PMI adds a constant offset to the PMI matrix. This is a rank-1 perturbation to the PMI matrix and  therefore has minimal effects on its spectral properties, as we checked.\n(ii) The choice of the embedding space dimension $K$ already implements SVD truncation. \n(iii) Regarding subsampling: we test a subsampled (sparsified) vocabulary in figure 2, panel f in the model. We have checked that subsampling by limiting the vocabulary also does not affect our results on Wikipedia, as we will mention in the text. \n\n\n4. Could you provide insights or explanations regarding the five weaknesses previously identified?\n\nSee above. \n\n\n5. The model assumes attribute independence. Could you provide examples illustrating that such independence is satisfied within the Wikipedia dataset?\n\nAs we highlight in our response to question 2 and will add to the paper, our theoretical model is robust to including a correlation between attributes."}, {"review_id": "Uojk7kKENJ", "reviewer": "Reviewer_vqG4", "summary": "This paper presents a theoretical generative model aimed at explaining why linear analogies (such as $W_{\\text{king}} - W_{\\text{man}} + W_{\\text{woman}} \\approx W_{\\text{queen}}$) naturally arise in word embeddings like Word2Vec and GloVe. Key contributions include:\n\n1. **Generative Model:** Proposes that words can be represented through binary semantic attributes, with co-occurrence probabilities determined by independent interactions between these attributes. This creates a structured co-occurrence matrix enabling analytic computation of eigenvectors and eigenvalues.\n\n2. **Emergence of Analogies:** Demonstrates analytically that linear analogies naturally emerge due to dominant eigenvectors of the co-occurrence (or PMI) matrix, directly reflecting arithmetic relationships between word attributes.\n\n3. **Robustness Analysis:** Validates the theory numerically, showing that the linear analogy structure is robust against noise, sparsity (subsampling vocabulary), and even the removal of analogy-specific word pairs.\n\n4. **Empirical Validation:** Tests predictions using actual co-occurrence data from Wikipedia, showing strong agreement between theoretical predictions and empirical observations.\n\nOverall, this paper provides a rigorous theoretical explanation for the presence and robustness of linear analogy structures observed empirically in widely used word embeddings.", "questions": "N/A", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n1. Theoretical Clarity and Rigor\n2. Explanatory Power\n3. Robustness Analysis\n4. Empirical Validation\n\n### Weaknesses:\n1. Simplified Assumptions: The model makes strong simplifying assumptions, including binary semantic attributes and independence of attribute effects.\n2. Generalizability and Realism: The assumption of all possible combinations of attributes existing in the vocabulary is unrealistic.\n3. Potential Degeneracies in Parameter Choices: The model’s predictive accuracy and analogy results are sensitive to the choice and distribution of parameters (e.g., semantic strength distributions $s_k$).", "was_revised": true, "final_justification": "The authors clarified my concerns.", "rebuttal": "Referee vqG4\n\nWe thank the referee for their useful comments. \n\nWeaknesses\n\n1. Simplified Assumptions: The model makes strong simplifying assumptions, including binary semantic attributes and independence of attribute effects. \n\nSee response to referee Zp5k:  we had already provided an extension to tertiary semantic attributes in the appendix. Extension to n-valued attributes is straightforward. In response to both referees, we have extended our theory to the case of correlated attributes, see a sketch of the formalism above, which will appear in Appendix in the final paper. \n\n \n2. Generalizability and Realism: The assumption of all possible combinations of attributes existing in the vocabulary is unrealistic.\n\n(i) Our model is robust to sparsification  (i.e. subsampling the vocabulary). In that case, all possible combinations of attributes do not exist in the vocabulary. Thus, we have already tested that this assumption can be relaxed. \n(ii) Clearly, some attributes do not enter in the semantic meaning of a given word. For example, a word can be neither masculine nor feminine. This can be readily modeled in our framework by considering tertiary attributes with a “neutral” category, each time a given attribute does not apply to a given word. \nWe will add a discussion on these points.\n\n3. Potential Degeneracies in Parameter Choices: The model’s predictive accuracy and analogy results are sensitive to the choice and distribution of parameters (e.g., semantic strength distributions ). \n\nThe notion that there exists a distribution of semantic strength is not a bug in our opinion, but a nice feature of our approach. It explains why distinct linear analogies do not appear at the same dimension of embedding, which is not captured by previous models."}, {"review_id": "5j7wDpMDhn", "reviewer": "Reviewer_imGd", "summary": "Summary: This paper proposes a latent factor generative model to explain the emergence of linear analogies in word embeddings. Specifically, it hypothesizes that each word is defined by a vector of latent binary attributes (eg: feminine v masculine, adj vs noun, etc), and that word co-occurrence probability depends on the shared attributes. It then follows this through to study how word embeddings and analogies would arise under such an assumption, and compares the predictions of this model to empirical observations.  The results show various ways in which this model makes better predictions than prior theoretical models for word embeddings. For example, in terms of the PMI spectrum, the embedding dimension required for analogies to arise, robustness to pruning specific co-occurrence examples from the training set, etc.", "questions": "- What can this tell us about popular contemporary LLMs?\n- Does this analysis give us any insight into how we can develop better word embeddings or improve LLM design today?\n- Is it possible to do some kind of binary latent factor analysis to discover the actual attributes that might underpin the existing words?", "limitations": "Limitations are OK.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ This paper provides an elegant and intuitive generative model that could explain word embeddings and analogies. \n+ This is more elegant than previous theoretical studies that attempted to explain analogies, but didn’t provide a complete generative model. \n+ Predictions by the proposed model match empirical data quite well. \n\nWeaknesses:\n- Polysemous words, hierarchy, etc are not so neatly explained by this model (but this is a common limitation in prior work as well).\n- The simple co-occurence based models mentioned (glove, w2v, etc) are rather old by today’s standards.", "was_revised": true, "final_justification": "Thanks to the authors for the rebuttal comments. I have read all the reviews and author feedback. I don't see any dealbreakers for this work, it seems worth acceptance.", "rebuttal": "Referee imGd\n\nWe thank the referee for their comments and questions. \n\nWeaknesses\n\n1. Polysemous words, hierarchy, etc are not so neatly explained by this model (but this is a common limitation in prior work as well). \n\nPolysemous words could be studied theoretically in our framework by ‘merging’ pairs of semantically distinct words (i,j) into a polysemous word k, by summing their corresponding rows and columns in the PMI matrix, such that P(ell)= P(i)+P(j) and P(w, ell)= P(w,i)+P(w,j) for any word w. It deserves a study of its own, but we expect our results - which are extremely robust to perturbations - to hold if the fraction of polysemous words is small.\n\nModeling the presence of hierarchical attributes is a nice problem for the future; we hope our contribution will stimulate future studies in that direction. \n\n\n2. The simple co-occurence based models mentioned (glove, w2v, etc) are rather old by today’s standards. \n\nSee below.\n\n\nQuestions\n\n1. What can this tell us about popular contemporary LLMs? \n\nSee reply to referee Zp5k to weaknesses 4+9: linear analogies are also observed in LLMs. They are argued to be functionally important, as they can mechanistically explain LLM fact retrieval in some cases (Merullo et al., ACL 2024).  How they emerge is not fully understood, and our work is a first step in seeking to build this understanding. We will discuss this literature in the introduction.\n\n\n2. Does this analysis give us any insight into how we can develop better word embeddings or improve LLM design today? \n\nIt was shown with SENSEBERT that if a machine (BERT in that case) is trained by predicting the meaning (a linguistic notion; e.g. cat belongs to the “animal” category) of a word and not only the word itself, performance is improved (Levine et al., ACL 2020).  It would be very interesting to check if LLMs perform better if instead of predicting the next token, they are trained to also predict the attributes of the associated word (see below for a practical proxy to these attributes). \n\n\n3. Is it possible to do some kind of binary latent factor analysis to discover the actual attributes that might underpin the existing words? \n\nAs a proxy for attributes, we could simply consider the projections of word embeddings into the top eigenvectors of the PMI matrix. As discussed in the text and above, attributes are somewhat mixed in these eigenvectors. However, linear combinations of attributes may be enough for practical purposes, such as the extension of SENSEBERT discussed just above. We will discuss these interesting points in the conclusion."}]

Author rebuttal:

Meta-review: The paper proposes a generative model to explain linear analogies in word embeddings.  This is done by hypothesizing that each word is a vector of binary attributes and that word co-occurrence probability depends on the shared attributes.  Under this assumption the authors study the emergence of embeddings and analogies and their predictions are compared with practice.  The results of the authors show various ways in which the proposed model makes better predictions than prior theoretical models for word embeddings.

It is unfortunate but only three out of the four reviewers returned a review.  Furthermore, the reviewer who is recommending borderline reject, probably busy with other things, was also slow in responding to the authors' rebuttal and explain what other parts required elaboration from the point of view of the authors.  This way the authors never had a chance to make an additional comment to the reviewer's comment on the authors' rebuttal.  Admittedly, most of the concerns of Reviewer Zp5k were addressed by the authors, but the main issues that remained for the reviewer were: 

(1) the empirical validation was limited to English Wikipedia, and 

(2) the connection to modern contextualized models is largely theoretical and untested.  

I think that Issue (1) about the use of the English language is minor.  In my opinion the authors give a good answer to defend this issue in Point (3) of their rebuttal.  Regarding Issue (2), I believe there can be two "camps" given the wild enthusiasm that LLMs enjoy in recent years.  In my opinion the authors result have merit even if they do not directly touch LLMs and moreover, the authors are willing to better frame their work in the final manuscript and clarify differences and similarities along this path.

Hence, at this point I have looked into the discussions, questions, arguments, and responses, and I have also looked into the paper.  I am more in favor of suggesting acceptance of this paper, rather than rejection.  So, this will be my recommendation, albeit I must stress that I am not entirely confident for my recommendation.
Decision: Accept (poster)