The Geometry of Categorical and Hierarchical Concepts in Large Language Models
Authors: Kiho Park
Yo Joong Choe
Yibo Jiang
Victor Veitch
Venue: ICLR 2025 Oral (2025)
Keywords: categorical concepts
hierarchical concepts
linear representation hypothesis
causal inner product
interpretability
Primary area: interpretability and explainable AI
TL;DR: We extend the linear representation hypothesis to general concepts and show that hierarchical relationships are encoded as orthogonality.

Abstract: The linear representation hypothesis is the informal idea that semantic concepts are encoded as linear directions in the representation spaces of large language models (LLMs). Previous work has shown how to make this notion precise for representing binary concepts that have natural contrasts (e.g., {male, female}) as _directions_ in representation space. However, many natural concepts do not have natural contrasts (e.g., whether the output is about an animal). In this work, we show how to extend the formalization of the linear representation hypothesis to represent features (e.g., is_animal) as _vectors_. This allows us to immediately formalize the representation of categorical concepts as polytopes in the representation space. Further, we use the formalization to prove a relationship between the hierarchical structure of concepts and the geometry of their representations. We validate these theoretical results on the Gemma and LLaMA-3 large language models, estimating representations for 900+ hierarchically related concepts using data from WordNet.

Published as a conference paper at ICLR 2025
THE GEOMETRY OF CATEGORICAL AND HIERARCHI- CAL CONCEPTS IN LARGE LANGUAGE MODELS
Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch University of Chicago
ABSTRACT
The linear representation hypothesis is the informal idea that semantic concepts are encoded linear directions in the representation spaces of large language models (LLMs). Previous work has shown how to make this notion precise for representing binary concepts that have natural contrasts (e.g., {male, female)) as directions in representation space. However, many natural concepts do not have natural contrasts (e.g., whether the output about an animal). this work, we show how to extend the formalization of the linear representation hypothesis to represent features (e.g., is.animall as vectors. This allows us to immediately formalize the representation of categorical concepts as polytopes in the represen- tation space. Further, we use the formalization to prove a relationship between the hierarchical structure of concepts and the geometry of their representations. We validate these theoretical results on the Gemma and LLaMA-3 large language models, estimating representations for 900+ hierarchically related concepts using data from WordNet
INTRODUCTION
Understanding how high-level semantic meaning is encoded in the representation spaces of large language models (LLMs) is a fundamental problem in interpretability. A particularly promising avenue is the linear representation hypothesis (e.g.. Mikolov et al.) 2013 Elhage al., [022] Nanda et al. 2023 Gurnee & Tegmark, 2024; Park et al. 2024). This is the informal hypothesis that semantic concepts are represented linearly in the representation spaces of LLMs. To assess the validity of this hypothesis, and systematically build tools on top of it, we must make precise what it means for a concept to be linearly represented, and understand how semantics are encoded in (the geometry of) the representation spaces.
Focusing on the final softmax layer, Park et al. (2024) give a formalization for the case of binary concepts that can be defined by counterfactual pairs of words. For example, the concept male female is formalized using the counterfactual pairs {("man", "woman")), ("king", "queen"")... They prove that such binary concepts have well-defined linear representation as direction in the representation space. They further connect semantic structure and representation geometry by show- ing that, under suitable inner product, causally separable concepts that can be freely manipulated (e.g., ma le + fema le and french = engl sh) are represented by orthogonal directions.
However, this formalization is limited: many natural concepts cannot be defined by counterfactual pairs of words. For example, simple binary features (e.g.. is_an is_animal) or categorical concepts (e.g., {,ammal, reptile, fish)) do not admit such formalizations. Additionally, it is not clear how semantic relationships beyond causal separability are encoded in the representation space. In particular, we are interested in this paper in understanding how hierarchical relationships between concepts are encoded in the representation space. That is, what is the relationship between the representations of animal, mammal, and dog?
In this paper, we extend the linear representation hypothesis as follows:
We show how to move from representations of binary concepts as directions to representa- tions vectors. As straightforward consequence, this allows us represent categorical
Cooe available at Ligithhh.eomKKiihPakkl.... Categorical Hicrarchical Representations

---

Published as a conference paper at ICLR 2025
Bird
Animal
Elephant
Fish
Organism
Tiger
Plant
Mammal
Dog
Reptile
Reptile
Elephant
Animal
Organism
Bird
Mammal
Plant
Fish
Tiger
Dog
(a) Pictorial depiction of the representation of hierarchically related concepts.
14
5.0
10
mammal
-25
-3.0
animal
(b) Hicrarchy is eneoded as orthogonality in Gemma. (c) Categorical concepts are represented as polytopes in Gemma.
Figure I:
In the representation spaces of LLMs, hierarchically related concepts (such plant animal and mammal = bird adrd) live in orthogonal subspaces, while categorical concepts are rep- resented as polytopes. The top panel illustrates the structure; the bottom panels show the measured representation structure in the Gemma LLM. Section 5 and Appendix A for details.
concepts (e.g., (mamma 1, bird, reptile, fish)) as polytopes where each vertex the vector representation of one of the elements of the concept (e.g., is-bird).
2. Using this result, we show that semantic hierarchy between concepts is encoded geometri- cally as orthogonality between representations,
manner we make precise.
3. Finally, empirically validate these theoretical results the Gemma (Mesnard et al., (024) and LLaMA-3 (Dubey et al.] (024) LLMs. To that end, we extract concepts from the WordNet hierarchy (Miller 1995). estimate their representations. and show that the geometric structure of the representations aligns with the semantic hierarchy of WordNet.
The final structure is remarkably simple, and is summarized
Figure
PRELIMINARIES
We begin with some necessary background.
2.1 LARGE LANGUAGE MODELS
For the purposes of this paper, we consider large language model consist of two parts. The first part a function À that maps an input text x to a vector A(x) in a representation space < Rd This is the function given by the stacked transformer blocks. We take A(x) to be the output of the final layer at the final token position. The second part is an unembedding layer that assigns a vector r(y) in unembedding space I Rd to each token y in the vocabulary. Together, these define
2

---

Published as a conference paper at ICLR 2025
sampling distribution tokens via the softmax distribution:
exp(A(=))
P(y x))
(2.1)
Eocaab exp(A(=)'<(x'))
We note that the results that follow rely on the duality between the embedding and unembedding spaces, and the softmax link between them. Accordingly, we do not address the "internal" structure of the LLMs. However, we are optimistic that clear understanding of the softmax geometry will shed light this well.
2.2 CONCEPTS
We formalize a concept as a latent variable W that is caused by the context X and causes the output Y. That is, a concept thing that could-in principle-be be manipulated to affect the output of the language model. In the particular case where a concept is a binary variable with a word-level coun- terfactual, we can identify the variable W with the counterfactual pair of outputs (Y) Y (1)). Concretely, we can identify ma = female with (Y) Y(1) En {("man", "woman")) ("king", "queen"), ("he", "her"),... ...). }. We emphasize that the notion of a concept as a latent variable that affects the output is more general than the counterfactual binary case.
Given a pair concept variables W and Z, we say that W is causally separable with Z the potential outcome Y(W w; Z = =) is well-defined for all w. That is, two variables are causally separable if they can be freely manipulated-e-a we can change the output language and the sex of the subject freely. these concepts causally separable.
2.3 CAUSAL INNER PRODUCT AND LINEAR REPRESENTATIONS
We are trying to understand how concepts are represented. At this stage, there are two distinct representation spaces: and I. The former is the space of context embeddings, and the latter is the space of token unembeddings. We would like to unify these spaces so that there is just a single notion of representation.
Park et al. 2024) show how to achieve this unification via "Causal Inner Product". This is par- ticular choice inner product that respects the semantics of language in the sense that the linear representations of (binary, counterfactual) causally separable concepts are orthogonal under the in- ner product. Their result can understood as saying that there some invertible matrix and constant vector to such that, if we transform the embedding and unembedding spaces
g + A(7(2) 50), ((x) A A X(x)
(2.2)
then the Euclidean inner product in the transformed spaces is the causal inner product, and the Riesz isomorphism between the embedding and unembedding spaces simply the usual vector transpose operation. We can estimate A as the whitening operation for the unembedding matrix. Following this transformation, we can think of the embedding and unembedding spaces as the same space, equipped with the Euclidean inner product
Notice that the softmax probabilities (eq. (2.1)) are unchanged for any A and %o. so this transfor- mation does not affect the model's behavior. The vector To defines an origin for the unembedding space, and can chosen arbitrarily. We give a particularly convenient choice below.
In this unified space, the linear representation of a binary concept W En (0, is defined as:
Definition 1. A vector Ew is linear representation of binary concept W if for all contexts E, and all concept variables Z that causally separable with W, we have, for all >> 0,
P(W - 1 11 f alw) > P(W = 1).,, and
(2.3)
P(Z ( alw) = (Z E).
(2.4)
That is, the linear representation is a direction in the representation space that, when added to the context, increases the probability of the concept, but does not affect the probability of any off-target
We are glossing some technical details here; see Park et al. 2024 for details.

---

Published as a conference paper at ICLR 2025
concept. The representation is merely direction because alw is also a linear representation for any a > 0 (i.e., there is no notion of magnitude). In the case of concepts corresponding to counterfactual pairs of words, this direction be shown to be proportional to the "linear probing" direction, and proportional to g(Y(1)) - (()( for any counterfactual pair (1),, (0) that differ on W.
3 GENERAL CONCEPTS AND HIERARCHICAL STRUCTURE
Our high-level strategy will be to build up from binary concepts to more complex structure. We begin by defining the basic building blocks.
Binary and Categorical Concepts The most general concept we address in this paper is cat- egorical concept, which refers to any concept corresponding to a categorical latent variable. This includes binary concepts as a special case. We consider two kinds of binary concept: binary features and binary contrasts. A binary feature W En {not_w, is_ww is an indicator of whether the output has the attribute w. For example, if the feature ss_animal true, then the output will be about an animal. A binary contrast + b En {a, b} is binary variable that contrasts two specific attribute values. For example, the variable mammal = bird binary contrast. In the particular case where the binary contrast can correspond to counterfactual pairs of words (e.g., ma le = females the concept matches the definition used in Park et al. (2024).
Hierarchical Structure The next step is to define what mean by a hierarchical relation between concepts. To that end, to each attribute w, we associate a set of tokens y(w) that have the attribute. For example, (mammal) (" dog", cats". TTiee", ,..... Then,
Definition 2. value subordinate
value (denoted by < w) if J(z) S M(w). We say categorical concept Z ER {zo..... - is subordinate to categorical concept W ER {wo..... 1m if there exists a value wz of W such that each value 2, of Z is subordinate to wz.
For example, the binary contrast dog + cat subordinate to the binary feature { {is mam.mama.,, not not_mammal), and the binary contrast parrot + eagle is subordinate to the categorical concept fmammal, bird, fish]. On the other hand, dog = eagle is not subordinate to bird = mammal, and bird = mammal and live in house live in water not subordinate to each other.
Linear Representations of Binary Concepts Now return to the question of how binary con- cepts are represented. A key desideratum is that if Ew is a linear representation. then moving the context embedding in this direction should modify the probability of the target concept in isolation. If adding w also modified off-target concepts, it would not be natural to identify it with the tar- get concept W. In Definitionn I, this idea is formalized by the requirement that the probability of causally separable concepts is unchanged when the representation is added to the context.
We now observe that, when there hierarchical structure, this requirement not strong enough to capture 'off-target' behavior. For example, if Lontmal captures the concept of animal vs not-animal, then moving in this direction should not affect the relative probability of the output being about a mammal versus bird. it did, then the representation would actually capture some amalgamation of the animal and mammal concepts. Accordingly, we must strengthen our definition:
Definition 3. A vector En is linear representation of a binary concept W if
P(W 1| e el P(W 1| E). and
(3.1)
P(Z e = ( E),
(3.2)
for all contexts E, all > O, and all concept variables Z that are either subordinate to or causally separable with W. Here, W a binary feature for an attribute w, then W = denotes W is.w.
Notice that, in the case of binary contrasts defined by counterfactual pairs, this definition is equiva- lent Definition because such variables have no subordinate concepts.
4

---

Published as a conference paper at ICLR 2025
4 REPRESENTATIONS OF COMPLEX CONCEPTS
We now turn to how complex concepts are represented. The high-level strategy is to show how to represent binary features as vectors, show how geometry encodes semantic composition, and then use this to construct representations of complex concepts.
4.1 VECTOR REPRESENTATIONS OF BINARY AND CATEGORICAL CONCEPTS
To build up complex concepts, we need understand how compose linear representations of binary features. At this stage, the representations are only directions in the representation space- they do not have a natural notion of magnitude. In particular, this means we cannot use vector operations (such as addition) to compose representations. To overcome this, we now show how to associate magnitude the linear representation of a binary feature. The key is the following result, which connects binary feature representations and word unembeddings:
Theorem 4 (Magnitudes of Linear Representations), Suppose there exists linear representation (normalized direction) fw of finaryfeature W for an attribute w. Then, there a constant b. and a choice of unembedding space origin 70 in eq. (2.2)such that
(ww = bw i E (ww) ((v) = y & (ww).
(4.1)
Further, if there exist attributes {wo..... Wd-1) } such that the linear representations of the bi- nary features for these attributes are linearly independent, we can choose canonical origin fto in eq. (2.2)
All proofs are given in Appendix B
The theorem says that, if (perfect) linear representation of the animal feature exists, then every token having the animal attribute has the same dot product with the representation vector; i.e., "cat" exactly as much animal "dog" is. If this weren't 't true, then increasing the probability that the output is about an animal would also increase the relative probability that the output is about a dog rather than cat. In practice, such exact representations are unlikely be found by gradient descent in LLM training. Rather, we expect (w)(v) to be isotropically distributed around b and 0, with variances that are small compared to b (so that animal and non-animal words are sell-separated.)
With this result in hand, we can define notion of vector representation for binary features:
Definition 5. We sav that a binary feature W for an attribute w has a vector representation E. Rd fw satisfies Definition 3 and || b in Theorem 4 If the vector representation of binary feature is not unique, we say l,,. is the vector representation that maximizes 6,..
We have now moved from representations as directions to representations as vectors. Definition and Theorem ggive simple way of composing binary features into binary contrasts:
Corollary 6 (Binary Contrasts Are Vector Differences of Binary Features). Let wo wi be binary contrast, and suppose there exist vector representations Ewo and Lw, for each attribute wn and W1. Then, the difference fw,, - is a linear representation Ewg in the sense of Definition 3
We can also apply ordinary vector space operations to construct representation categorical con- cepts, c.g., {mammal, reptiles ri, fish). There is now a straightforward way to define the representation of such concepts:
Definition 7.
The polytope representation of categorical concept W convex hull of the vector representations of the elements of the concept
{wo..... Wk-1 is the
In Appendix D. we state and prove a generalization of Corollary 6 to categorical concepts (which proves Corollary iftts...)
'In Appendix C
we prove that the polytope representation for "natural" categorical concept simplex.
5

---

Published as a conference paper at ICLR 2025
131
INTRA maamIN
310
astrtal mennna bid
103
plrte antmra mareaa bld
7.
-25
-25
-32
-10
Jut
-25
Figure 2: Hierarchical semantics encoded orthogonality in the representation space The- orem 8) The plots show the projection of the unembedding vectors onto 2D subspaces: f (left; (a)), span(fanimal- fuira mammal (middle; (b),. and span sppanfaanmma span aanimai: (gammal) plannt bbird - (mammaa) } (right:(() Gray points indicate all 256K tokens in the vocabulary. and the colored points the tokens V(w). The blue and red vectors are used to span the 2D subspaces.
4.2 HIERARCHICAL ORTHOGONALITY
Now, we turn to the question of how hierarchical relationships between concepts are encoded in the representation space. The core intuition that manipulating the "animal" concept should not affect relative probabilities of the "mammal" and "bird" concepts, we might expect the representations of animal and mammal + bird to be orthogonal. The following result formalizes this intuition by connecting the vector and semantic structures. The result is illustrated in Figure 2
Theorem (Hierarchical Orthogonality). Suppose there exist the vector representations for all the following binary features. Then, have that
(a) En Es Ew for 2 x w:
(b) En Ez - for 7. En {20, 21} subordinate to W En {not_w, is_w];
(c) Euo - two E2 20 for 7 En 1=0, 2 subonlinate to W En {wo, wi }: and
(d) Eu - Ewo 1 Euo - kw,, for W2 X wi < W..
Together, [Dfinition 7 and Theorem 8 give the simple structure illustrated in Figure hierarchical concepts are represented as direct sums of polytopes. This direct sum structure is immediate from Theorem We emphasize that all of the results-involving differences of representations-aa are only possible because we have vector representations (mere directions would not suffice).
5 EXPERIMENTS
We now turn to empirically testing the theoretical results. Here, we present our empirical results on the Gemma-2B model (Mesnard et al.) (2024)
4 In Appendix F further present empirical results on the LLaMA-3-8B model (Dubey et al. 2024), for which our findings are largely analogous.
5.1 SETUP
Canonical Representation The results in this paper rely on transforming the representation spaces so that the Euclidean inner product is a causal inner product, aligning the embedding and unembed- ding representations. Following Park et al. (2024), we estimate the required transformation as:
gly) = Cov (1) -E())
(5.1)
where r is the unemhedding vector of word sampled uniformly from the vocabulary. Centering by Eh}] is reasonable approximation of centering by % defined in Theorem because this makes the projection of random g on an arbitrary direction close to 0. This matches the requirement that the projection word onto concept the word does not belong to should be close
*Code available at Piithah.eomKKKKioaarll/L... Categorical Hicrarchical Representations

---

Published as a conference paper at ICLR 2025
Original Unembeddings
Shuffled Unembeddings
1.5
1,0
1.0
0.5
0.5
0.0
0.0
-0.5
-0.5
-1.0
100
200 300
400
500
100
200
300
400 500 600
Binary Features in Hilranchy
train
best
ranciom
Figure 3: Vector representations exist for most binary features in the WordNet noun hierarchy. For each synset w (indexed on the r-axii) we estimate the vector representation ly using a train subset of the vocabulary (w). The plot shows the projections (g(y) / I| ll2 of train (green), test (blue), and random (orange) words on estimated vector representations for each WordNet feature, using either the original (left) shuffled (right) unembeddings. Our theory predicts that this value should be close to 1 when has the target feature, and close to 0 when does not. The thick lines present the mean of the projections for each feature and the error bars indicate the standard deviation. As predicted, the projections of test words are near I, and random words near O (left plot). Further, this structure does not hold when using the shuffled control without natural semantics (right plot).
WordNet We define large collection of binary concepts using WordNet (Miller 1995). WordNet organizes English words into a hierarchy of synsets, where each synset a set of synonyms. The WordNet hierarchy is based interest in this paper. We take each synset as an attribute w and define )'w)) as the collection of all words belonging to any synset that is descendant of w. For example, the synset mammal.. n. 01 is a descendant of animal.n... 01, so both (mmammal , n. 01) and ('animall .n.maan.... n. 01) contain the word "dog". We collect all noun and verb synsets, and augment the word collections by including plural forms of the nouns, multiple tenses of each verb, and capital and lower case versions of each word. We filter to include only those synsets with least 50 words in the Gemma vocabulary. This leaves us with 593 noun and 364 verb synsets, each defining an attribute.
word hyponym relations, and reflects the semantic hierarchy of
For space, we report results on the noun hierarchy here and defer the verb hierarchy to Appendix F
Estimation via Linear Discriminant Analysis Now, we want to estimate the vector representa- tion l. for each attribute w. To do this, we make use of vocabulary sets Y(w). Following Theorem 4 the vector associated to the concept w should have two properties. First, when the full vocabulary is projected onto this vector, the words in V(w) should be well-separated from the rest of the vocabu- lary. Second, the projection of the unembedding vectors for y E V(w) should be approximately the same value. Equivalently, the variance of the projection of the unembedding vectors for y E (w) should be small. To capture these requirements, Discriminant Analysis (LDA), which finds a projection minimizing within-class variance and max- imizing between-class variance. Formally, we estimate the vector representation of binary feature W for an attribute w as
estimate the directions using variant of Linear
Cov(gre) ((( ||Cov Elg
L ( B((w)) :w: with Fw
(5.2)
where Fo is the unembedding vector of a word sampled uniformly from ((w) and Cov(gw) is a pseudo-inverse of the covariance matrix. We estimate the covariance matrix Cov (ov(gw) using the Ledoit Wolf shrinkage estimator (Ldooi & Wolf. 2004), 2004). because the dimension of the representation spaces is much higher than the number of samples.
5.2 WORDNET HIERARCHY IS LINEARLY REPRESENTED
Existence of Vector Representations for Binary Features The first question is whether vector representations of binary features exist. To evaluate this, for each synset in WordNet split )ww) into train words (70%) and test words (30%), fit the LDA estimator to the train words, and examine the projection of the unembedding vectors for the test and random words onto the estimated vector
7

---

Published as a conference paper at ICLR 2025
Table Adding the linear representation of the parent concept to context embeddings does not affect the logit differences between token pairs in a child concept, and it substantially affects those in the parent concept. For each parent-child pair of binary concepts (wp = w1, 2u = 21), we first add the normalized linear representation lw = Eui, wwo to the context embeddings. Then, we show the P(y. 1 between the pairs (yo. 91) E {(wo)) x)(ww) (parent; top row) and (yo. E )/(z)) x V(z1) (child; bottom row). Notice that, for any context x and any tokens y/ and 1/1 adding a linear representation lw for binary contrast W wg + Wi manipulates the logit difference between the tokens from ((x) (9(x1) - ((y)) to (f(x) + fw) (g(y)) - g(o)), which implies the change in logit difference between the tokens is w (g(y)) (Vo)), irrespective of {()). We show the mean and standard deviation of the change in logit differences over all token pairs.
change logit difference log Plav [2]]
W parent, > parent, & V child, child,
Change in Logit Differences
W = plant.n..222 + animal 01
5.1265 + 1.1731
Z wwm.mml..n.00. reptile.n.
-0.0600 1.2190
W fluid.n.02 solid.
9.8296 + 1.1099
0.3770 1.5410
W = sclentist.nc0 contcsaat..ccccttttttttttttt.t
14.4222 = 0.9458
Z wwtaattee..c... = playor
-0.1545 1.1426
representation. The left plot in Figure 3 shows the mean and standard deviation of the projections, divided by the magnitude of each estimated l Following Theorem 4, if a vector representation would expect the values on the test set to be close 1, and the values
exists for an concept, for random words to be close to O. We see that this is indeed the case, giving evidence that vector representations do exist for these features.
As a baseline, the right plot in Figure 3) shows the same analysis but with the unembedding vectors randomly shuffled. In this case, the test projections are close to O, which indicates that there are no linear representations. Thus, the existence of the linear representations in the original unembeddings relies on the underlying semantic structure.
Intervention Next, we validate that adding the estimated linear representations of a binary contrast changes the target concept without changing other tfr-et concepts, as required by Detinition 3) Table shows the mean and standard deviation of the changes in the logit differences between the pairs from parent or child binary contrasts, after adding the normalized linear representations of the parent binary contrast to context embeddings. The results show that the logit differences change significantly for the target concept, while the off-target concept changes very little.
Relationship Between the Vector Representations of Binary Features We now turn to exam- ining whether the similarity between the vector representations of binary features reflects their se- mantic relation. The direct structure predicts that concepts that close in semantic hierarchy should have similar vector representations. For example, mammal and bird are close in the hyponym graph because they share a common parent (animal). Our theory predicts that Epied and Enammaz share a common component iani (the representation for "bird" is the representation for "animal" plus an orthogonal component, and similarly for mammal). This implies that the cosine similarity between Enamma: and foira should be substantial.
we show the shortest distance between each feature in the (undirected) WordNet noun hyponym graph in the left panel and the cosine similarity between the estimated vector representa- tions Eyp in the middle panel. is clear that, as predicted, the cosine similarity reflects the WordNet structure. As control, we apply the same analysis after randomly shuffling the embeddings and show the resulting cosine similarities in the right panel. Here, direct parent-child (or grandparent- grandchild) relationships are still reflected in the cosine similarity because the shuffled embeddings still respect set inclusion (the words assigned to "mammal" are still a subset of those assigned to "animal"). ). However, sibling relationships are not reflected once the semantic structure is removed by the shuffling. In this case, the representations of siblings are effectively pairs of random vectors, and are nearly orthogonal
In Figure 4
would expect in high dimensional space.
8

---

Published as a conference paper at ICLR 2025
Distance
cost/.. E. (Original Unembeddingg)
cosieu..J. Unembeddings)
100
0.75
300
300
300
0.00
-025
400
4.0
400
-075
-100
Figure 4: Hierarchical semantics in WordNet are linearly represented in Gemma-28 The left heatmap shows pairwise shortest distance matrix between features in the noun hierarchy graph as (1 + min dis.diiceee-- (higher values indicate closeness, such as in child-parent or sibling relation- ships). The middle heatmap shows the cosine similarity between the vector representations lyo. As predicted, this similarity reflects the WordNet structure. The right heatmap is a control where the embeddings are randomly shuffled (removing semantic structure). In this case, nearly everything is orthogonal, expected in high-dimensional space (set inclusion relationships remain due to the estimation procedure). In Appendix we include zoomed-in versions of these heatmaps.
cosli Eowrest Whole
costly lorrrt Training set
0.25
0.25
0.00
0.00
-0.25
-0.25
-0.50
-0.50
0.75
0.75
-1.00
-1.00
100
200
300
400
500
600
0
1000
200
300
400
500
600
Featurees WordNet Hierarchy
Original Unambeddings
Shuffed Unambeddings
Unammeediings Randam
Figure 5: WordNet noun hierarchy is encoded in the orthogonal structure predicted by statement (a) in Theorem 8 We plot the cosine similarity between child-parent vector and a parent vector for each feature in the hierarchy (blue). As predicted, this value is close O. The left plot uses all data for representation estimation, and the right plot uses only 70% independently selected for each synset. We include baselines where a randomly selected feature used the parent (orange) and where the embeddings are shuffled (green) controls for the possibility that the orthogonality simple byproduct of high-dimensional geometry, or the set inclusion relationships used in estimation see main text for details. See Appendix F for an analogous plot for statement (d)
Hierarchical Orthogonality Finally, we evaluate the prediction that hierarchical relations are en- coded orthogonally as predicted in Theorem 8 Figure shows the cosine similarity Éparens and fohtid Earret for the WordNet features. As predicted, this value is close to 0 (blue curve).
Now, challenge here that in high dimensional spaces even random vectors are nearly orthogonal. So, it may be difficult to differentiate whether orthogonality reflects semantic structure is merely consequence of nearly everything being orthogonal high dimensions. a control, we also show the cosine similarity when the parent vector is the representation of randomly selected feature (orange). this case, the cosine similarity is far from O, suggesting that the observed orthogonality not merely byproduct of the high-dimensional geometry.
We also include another baseline that estimates the cosine similarity between the parent vector and the parent vector for each feature using shuffled unembeddings (green). In the left plot, that we still have orthogonality, which could suggest that the orthogonality is a consequence of the set inclusion relations in WordNet (rather than actual semantic structure) [1 To test this, in the right plot we estimate the representations using only 70% of the tokens, independently selected for each
'In Appendix G we explain why set inclusion (hefore train/test split) leads orthogonality.

---

Published as a conference paper at ICLR 2025
synset. This breaks the set inclusion. In this case, we see the orthogonality is preserved for the original unembeddings, but is broken for the shuffled unembeddings. Note that a cosine of -0.2 is highly nontrivial in digediimensional space. This suggests the orthogonality does indeed reflect the semantic structure.
6 DISCUSSION AND RELATED WORK
We set out to understand how semantic structure is encoded in the geometry of representation space. We have arrived at an astonishingly simple structure, summarized in Figure I] The key contributions are moving from representing concepts as directions to representing them as vectors (and polytopes), and connecting semantic hierarchy to orthogonality.
Related Work
The results here connect closely to the study of linear representations in language
models (e.g., Mikolov et al., 2013 Pennington et al.] 2014
Arora et al. 2016 Elhage et al. 2022
Burns et al.] 2022; Tigges et al. 22033; Nanda et al. 2023; Moschella et al. 2022 Li et al. 2023
Gurnee et al. [2023] Wang et al. 2023 liang al. [0224] Park et al.] 2024). In particular, Park al.
(2024) formalize the linear representation hypothesis by unifying three distinet notions of linearity:
word2vec-like embedding differences, logistic probing, and steering vectors. Our work relies on this unification, and just focuses on the steering vector notion. Our work also connects to work aimed at theoretically understanding the existence of linear representations. These include early work on word2vec-stee style embedding models (Arora et al., 2016; Gittens et al. 2017; Arora et al,, 2018 EEthayarahh al., [2018] Frandsen & Ge 2019 Allen Hospedales 2019) as well as dynamic topic models (Blei & Lafferty, [2066] Rudolph et al. 2016). Jiiagg et al. (2024) connect the existence of linear representations in L.MMs to the implicit bias of gradient descent. In this paper, we do not seek to justify the existence of linear representations, but rather to understand their structure if they do exist. Though, by empirically estimating vector representations for thousands of concepts, we add to the body of evidence supporting the existence of linear representations, Elhage et al. (2022) also empirically observe the formation of polytopes in the representation space of toy model, and the present work can be viewed in part as giving an explanation for this phenomenon.
There is also a growing literature studying the representation geometry of natural language (Mimno & Thompson 2017
Reif et al.] 2019 Volpi & Malagø 2021 Li et al.
2020
Chen et al. 2021
Chang et al.] 2022; Liang et al. [022] Tliag et al.] 2023 Wang et al.
2023
Park et al.
2024
Valeriani et al.] 2024). In terms of hicrarchical structures, existing work focuses on connections
to hyperbolic geometry (Nickel & Kiela, 2017 Ganea et al. 2018 Chen et al. 202 He et al., 2024 We do not find such a connection LLMs, but it is an interesting direction for future work to determine if more efficient LLM representations could be constructed in hyperbolic space. [Tang al. (2023) hypothesize that very general "independence structures" are naturally represented by partial orthogonality in vector spaces (Amini et al., 2022). The results here confirm and expand on this hypothesis the case of hierarchical structure in language models.
Implications and Future Work The results in this paper give foundational understanding the structure of representation space in language models. Of course, the ultimate purpose of foundations to build upon them. One immediate direction is refine the attempts interpret LLM structure to explicitly account for hierarchical semantics. As an example, there is currently significant in- terest using sparse autoencoders to extract interpretable features from LLMs (e.g.. (Cunningham et al. 2023] Bricken et al. 2023 Kissane et al. 2024)] Braun et al., 2024). This work searches for representations in terms of distinct binary features. Concretely, it hopes to find features for. e.g., animal, mammal, bird, etc. Based on the results here, these representations are strongly co-linear, and potentially difficult to disentangle. On the other hand, a representation in terms of Cantmal, Cnammal Laninal, foird laninal. etc., will be cleanly separated and equally interpretable. Fun- damentally, semantic meaning has hierarchical structure, so interpretability methods should respect this structure. Understanding the geometric representation makes it possible to design such methods.
In a separate, foundational, direction: the results in this paper rely on using the canonical repre- sentation space, and we estimate this using the whitening transformation of the unembedding layer. However, this technique only works for the final layer representation. It is an important open ques- tion how to make sense of the geometry of internal layers.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work supported by ONR grant N00014-2-112551 and Open Philanthropy.
REFERENCES
Carl Allen and Timothy Hospedales. Analogies explained: Towards understanding word embed- dings. In International Conference on Machine Learning. pp. 223-231. PMLR, 2019.
Arash A Amini, Bryon Aragam, and Qing Zhou. A non-graphical representation of conditional independence via the neighbourhood lattice. arXiv preprint arXiv:2206.0 05829, 2022.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-hased word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399, 2016.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang. Tengyu Ma, and Andrej Risteski. Linear algebraic struc- ture of word senses, with applications to polysemy. Transactions of the Association for Compu- tational Linguistics, 6:483-495, 2018.
David M Blei and John D Lafferty. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning, pp. 113-120, 2006.
Dan Braun, Jordan Taylor, Nicholas Goldowsky- and Lee Sharkey. Identifying functionally important features with end-to-end sparse dictionary learning. arXiv preprint 2rXiv:2405.1224/ 2024.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Tumeer Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph. Zac Hatfield-Dodke Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https//transformmrr
Collin Burns, Haotian Yc, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. arXiv preprint arXiv:2212.03827, 2022.
Tyler Chang, Zhuowen Tu, and Benjamin K Bergen. The geometry of multilingual language model representations. arXiv preprint 1.Xi::2205.1096 2022.
Boli Chen, Yao Fu, Guangwei Xu, Pengjun Xie, Chuanqi Tan, Mosha Chen, and Liping Jing. Prob- ing bert in hyperbolic spaces. arXiv preprint 2 2021.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint 200::223000080000 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 27X 2024.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfiel-DDodee Robert Lasenby, Dawn Drain, Carol Chen, al. Toy models of superposi- tion. arXiv preprini arXiv:2209. 02220090006522 10652, 2022.
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Towards understanding linear word analo- gies. arXiv preprint arXiv:1810.04882. 2018.
Abraham Frandsen and Rong Ge. Understanding composition of word embeddings via tensor de- composition. arXiv preprint arXiv: 100.:0190..000/3. 2019.

---

Published as a conference paper at ICLR 2025
Octavian Ganea, Gary Bécigneul, and Thomas Hofmaann. Hyperbolic entailment cones for learning hierarchical embeddings. In International Conference on Machine Learning. pp. 1646-1655. PMLR. 2018.
Alex Gittens, Dimitris Achlioptas, and Michael W. Mahoney. Skip-gram zipf uniform vector additivity. In Annual Meeting of the Association for Compulational Linguistics, 2017.
Wes Gurnee and Max Tegmark. Language models represent space and time. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL httpss://ppenrvveee.. nett/
Forump1d==18xxmmv
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bert- simas. Finding neurons in haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023.
Yuan He, Zhangdie Yuan, Jiaoyan Chen, and Ian Horrocks. Language models as hierarchy encoders. arXiv preprint arXiv:2401.11377 2024.
Yibo Jiang, Bryon Aragam, and Victor Veitch. Uncovering meanings of embeddings via partial orthogonality. arXiv preprint arXiv:2300.16111 2023.
Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, and Victor Veitch. On the ori- gins of linear representations in large language models. In International Conference on Machine Learning, 2024.
Connor Kissane, Robert Krzyzanowski, Arthur Commy, and Neel Nanda. Sparse autoen- coders work on attention layer outputs. Aligmment Forum, 2024. URL https::/www. //www.
ilignmentforum. wttp:////.ootttt/ttt///eeeeeeee.
Olivier Ledoit and Michael Wolf. well-conditioned estimator for large-dimensional covariance matrices. Journal of multivariate analysis, 88(2):365-411, 2004.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. arXiv preprint Stti::2011.08866 2020.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Plister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. arXiv preprint arXiv:2306.03341, 2023.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Ad- vances Neural Information Processing Systems, 35:17612-17625, 2022.
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint wrXi:22403.08295, 2024.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp. 746-751, 2013.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
David Mimno and Laure Thompson. The strange geometry skip-gram with negative sampling. In Conference on Empirical Methods in Natural Language Processing, 2017.
Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodola. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.1544 2022.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint 11Xiv:2309.00941,, 2023.
12

---

Published as a conference paper at ICLR 2025
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representa- tions. Advances in neural information processing systems, 30, 2017.
OpenAl. GPT-4 technical report. arXiv preprint arXiv:2303008744 2023.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. International Conference on Machine Learning, 2024.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove:
Global vectors for word representation. In Proceedings the 2014 conference empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. Advances in Neural Information Processing Systems, 32, 2019.
Maja Rudolph, Francisco Ruiz, Stephan Mandt, and David Blei. Exponential family embeddings. Advances in Neural Information Processing Systems, 29, 2016.
Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models. arXiv preprint 2rX1::2310.15154 2023.
Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and AI- berto Cazzaniga. The geometry of hidden representations of large transformer models. Advances in Neural Information Processing Systems, 36, 2024.
Riccardo Volpi and Luigi Malagò. Natural alpha embeddings. Information Geometry, 4(1):3-29, 2021.
Zihao Wang. Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text- controlled generatiee models. Advances in Neural Information Processing Systems, 2023, URL httts:://ppenree ew net/ vet/fooumm forum d= rCuwdaB
13

---

Published as a conference paper at ICLR 2025
repöle
10
12
9
10
maminal
mammal
birg
fish
bird
10
12
Figure 6: Categorical concepts are represented as polytopes. The plots show the projection of the unembedding vectors on the 3D subspaces: span marmmal Ebira, Erish] (left) and spanf fmarma } (right). The gray points indicate all 256K tokens
Cnamaal frih Emamma rept the vocabulary, and the colored points are the tokens in J(()). The left plot further shows orthogonality between the triangle and the projection of annimel (black arrow).
A VISUALIZATION OF animal
As a concrete example for visualizations, we examine the theoretical predictions for the famil- iar concept animal. However, the categories in WordNet are not the most intuitive; for ex- ample, the subcategories under textttanimal.n.01 include terms like "chordate", "aquatic verte- brate", "invertebrate" instead of more familiar categories such as "fish", "amphibian", or "in- gather tokens corresponding clear, high-level concepts
sect". This makes it challenging (mammal, bi rd, fish, reptile, amphibian. insect]. Therefore, we generated two sets of tokens M(animal) and ((plant) using ChatGPT-4 (OpenAl, (023). and manually in- spected them. V(animal) is further divided into six sets of tokens for each subcategory (manmal,bird. fish, reptile, amphibian, insect].
Figure 2 illustrates the geometric relationships between various representation vectors. The main takeaway is that the semantic hierarchy encoded as orthogonality the manner predicted by Theorem 8 The figure also illustrates Theorem 4) showing that the projection of the unembedding vectors for y E (ww) is approximately constant, while the projection of y g ((w) is zero.
Figure ójillustrates that the representation of categorical concept is polytope. and the projection of unembedding vectors onto the subspace for the polytope are concentrated on each vertex as predicted in Corollary 10 The left plot also shows that, predicted, the polytope for {fish, mammal, bird is orthogonal to the vector representation animal.
B PROOFS
B.1 PROOF OF THEOREM 4
Theorem 4 (Magnitudes of Linear Representations), Suppose there exists linear representation (normalized direction) fw of a binary feature W for an attribute w. Then, there is a constant bw. > and a choice of unembedding space origin 70" in lnle. (2.2) that
(wg(y) = by if E V(w) (w) = if & ((w).
(4.1)
Further, if there exist d attributes {wo..... wa-1) } such that the linear representations of the bi- nary features for these attributes are linearly independent, we can choose a canonical origin 5o in eq. (2.2)
14

---

Published as a conference paper at ICLR 2025
Proof. For any y1, E (w) yı, yo & (w)) let Z be binary concept where J/( 0) {y0}
and y(Z = 1) = {. Since is subordinate to W, eq. (3.2) implies that
logit. IP(Y y1 Y (yo, y1
), v,
(w) logit P(Y y1 Y E (vo, v)),))
(B.1)
(g(n) ((vo)) = E/ (()(1)) ((6)) =
(B.2)
where A is the invertible matrix inleq. (2.2) This means that fy Ay(y) the same for all E (we),
and it is also the same for all y E [(w).
Furthermore, for any 11 E ((w) and yo & ((w), eq. (3.1) implies that
logit P(Y y1 E y1 v1).e+ (w) logit P(Y y1 (yo. 313.6)
(B.3)
(g(v)) ((30)) = A(r(n) ((60)) > O.
(B.4)
Thus, by setting by = Ey (n)) for any y $ (w)). and b = An((n)) An(36) > for any y (ww)) and yo $ (w),, we get
T Ary) by + d An(y) = be y & y(w).
y E V(w)
(B.5)
Then, we can choose an origin as
7o A Ew
(B.6)
satisfying eq. eq. (4.1)
On the other hand, if there exist d attributes (wo, Wa- } such that the linear representations .w...... Ewa.. of the binary features Wo.... Wa-1 for these attributes are linearly independent, then the linear system
W. 47 by for 0...., 1
(B.7)
has unique solution "oo. We can choose this vector as the canonical origin 7o ineq. (2.2) ensuring that eq. (4.1) is satisfied.
B.2 PROOF OF THEOREM g
Theorem 8 (Hierarchical Orthogonality). Suppose there exist the vector representations for all the following binary features. Then, we have that
(a) Eno Ez for 2 + w:;
(b) En E2 -
-2x- for En { 21 subordinate to W En {not_ w, issw];
(c) Eu Ewo Es for Z En {zos 21 subordinate to W En {wo, w1}: and
(d) Er - -ww, Ewng -
Eu,, for w2 x wh R Wg-
Proof.
(a) For Ew and Eg where x w, by Theorem 4 we have
(E= tuo) g(y) b. bw
if E V(z)
(E2 (..) g(y) = 0 - bw = -b
y E V(w) (2)2)
(B.8)
(Eg Ew))" ((() = 0=
if y & (ww).
When w \ 2 denotes an attribute detined by )'w) \ V(2), L - t, can change the target concept w ( * z without changing any other concept subordinate or causally separable to the target concept. Thus, E- P. is the linear representation e This concept means not_2 + is_2 conditioned on and hence it is subordinate to w.
Therefore, E,, orthogonal the linear representation Empr, = i E,, by the property the causal inner product. they are not orthogonal, adding E.. can change the other concept w 2, and is contradiction.
(b) By the above result (a) FT (Az, F.o)) E (w) = 0. Therefore, [T E2o 0.
15

---

Published as a conference paper at ICLR 2025
W2
W
W
W/
W/
W1
Figure 7: Illustration of the case k 3 in the proof of Proposition 9
(c) Let's say that ui is wz defined in Definition The binary contrast 2g = 21 subordinate the binary feature for the attribute wp. By the property of the causal inner product, Euo is orthogonal to the linear representation E = Es En (by Corollary 10). Then, with
the above result (b) have (Ews -ww)) (l2) 20
(d) By the above result a)) we have
|| Ran,
2m 12
1 11
|| w.y
Cuzz
E ur
11
(B.9)
2
E
11
Then,
mwo
12 Luz
Log 11
(B.10)
[wu, 12
|| 10 12 + 1
|1 12
(B.11)
| 1 || E 112
(B.12)
= l - Euv 12.
(B.13)
Therefore, En,, Ewo is orthogonal to Eug Ew
C NATURAL CATEGORICAL CONCEPTS AS SIMPLICES
Polytopes are quite general objects. Detiiiion also includes representations of categorical vari- ables that are semantically unnatural, e.g., {dog, sandwi.c ch, running). We would like to make more precise statement about the representation of "natural" concepts. One possible notion of "natural" concept is one where the model can freely manipulate the output values. The next propo- sition shows such concepts have particularly simple structure:
Proposition 9 (Categorical Concepts are Represented Simplices). Suppose that (wo...., Wk-1 collection of mutually exclusive attributes such that for every joint distribution Q(wo...Wwk. wh-1) there is some E, such that P(W w; (,) Q(W = wi) for every i. Then, the vector representations Eug..... E. form a (k-1] (k-1)-simplex the representation space. this case, lake the simplex to be the representation of the categorical concept W {wo, wk-1
Proof. If we can represent arbitrary joint distributions, this means, in particular, that we can change the probability of one attribute without changing the relative probability between a pair of other attributes. Consider the case where & = 3, as illustrated Figure 7 If : Ewy lwv are on a line,
then there is direction in that line (to change the value in the categorical concept) such that adding the direction can change the probability of W2 without changing the relative probabilities between we and W1- However, if E Ewo Ewy Euz are not line, they form triangle. Then, there exists line
16

---

Published as a conference paper at ICLR 2025
that is toward wuy and perpendicular to the opposite side of the triangle. Now adding the direction é can manipulate the probability of u2 without changing the relative probabilities between Wp and w1. That is, for any a > and context embedding e
PPW 12 l. al) > P(W = =2 (), and
P(W=w; (tal) P(W== 1
(C.1)
P(W= étaē)
P(W=
Therefore, the vectors Evo fue Ewz form 2-simplex.
This argument extends immediately to higher & by induction. For each {0....,k - 1), there should exist direction that is toward lwi, and orthogonal to the opposite hyperplane ((k 2)- simplex) formed by the other m., Then, the vectors Eu, form a (k >--simplex. 0
D SUBSPACES FOR CATEGORICAL CONCEPTS
By Theorem polytope representations have the following property:
Corollary 10 (Polytope Representations). Let W (wo, ..... Wk-1) be a categorical concept, and suppose there exist vector representations Ew,, for the binary features of each attribute (their con-
vex hull is the polytope representation of W.. Let L = -leu, ..n. P. Lwo E Rdxx (k-1) and let C(T) be its column space. Then, there exist some vectors Two C(r) euch i, such that
[[, Two if V(w)) Hi =0 if & U.
(D.1)
where 11 inverse of the matrix.
T. is the projection matrix onto C(T.), and (L. denotes a pseudo-
Proof. By Theorem 4 for i 1,...,k 1, we have
Il gív) L(LTL)TE L g(v) L(I [O....,0, 0, b. 0, .,....o.''' _0 = Twn
(D.2)
L
for any y e V(w)). Similarly, we have
II
((() LIL g(v)) L)Z [bb [
-b2
Two
(D.3)
F.
for any y E ((wo), while
Il ((L)=L(LTD) g(y) L(L (O...,) =0
(D.4)
L
for any y & u Therefore, we have eq. (D.1) with some constant vectors Twy
In words, the unembeddings of tokens for the same attribute share the same projections the subspace spanned by the differences between the vector representations. Therefore, adding any vector in the subspace C(L) to the context embedding changes the probability of the target concept W = {wo, ... wk-1} withous changing any other concept subordinate to or causally separable with the target concept. Finally, note that Corollary 10 implies Corollary 6
E EXPERIMENT DETAILS
We employ the Gemma-22 -23 -2B version of the Gemma model (Mesnard et al. 2024 online via the huggi buggingfan ace ace library. Its two billion parameters are pra-ted on three trillion tokens. This model utilizes 256K tokens and 2,048 dimensions for the representation space.
which is accessible
We always use tokens that start with a space (u(22581') )') in front of the word, as they are used for next- word generation with full meaning. Additionally, like WordNet data use, include plural
17

---

Published as a conference paper at ICLR 2025
animal
young
invertebrate
domestic animal
bird
mammal
aquatic aquatic_vertebrast vertebrate
reptle
gallinacedds.bing bird
passerine
ungulate
camivore
even-toed ungulate
odd-toed ungulate
Figure 8: Subtree in WordNet noun hierarchy for descendants of animal.
costd (Original Unembedding)
coott IStuiffled Unambeddings)
arimal
driea
0.75
yourg
choraae
invettetrate
bid
pasoel
gall racoue
canvooe
ungulate
oddcted unguate
-1.00
Figure 9: Zoomed-in Heatmaps of the subtree for animal in Figure
forms, and both capital and lowercase versions of the words in ((animml) and ('pl ((plant) for visualization Apppeedi A
In the WordNet synset data, each content of the synset mammal nn indicates that "mammal" word, "n" denotes "noun," and "01" signifies the first meaning of the word. In the WordNet hierarchy, if parent has only one child, we combine the two features into one. Additionally, since the WordNet hierarchy is not perfect tree, child can have more than parent. We use of the parents when computing the L.. Epprent
Lastly, the vector representations estimated by the non-split collection of tokens for Figure 2 Table
Figure
Figure 6 Figure 14] and Figure 18
F ADDITIONAL RESULTS
Zooming in on a Subtree of Noun Hierarchy As it difficult to understand the entire WordNet hierarchy at once from the heatmaps in Figure 4. we present zoomed-in heatmap for the subtree (Figure 8) for the feature animal 1. in Figure 9 The left heatmap displays the shortest distance between the nodes of the subtree in Figure 8. The middle heatmap shows that the cosine simi- larities between the vector representations E,, reflect the child-parent or sibling relationships. The right heatmap demonstrates that the vector representations estimated by shuffled unembeddings only reflect the set inclusion relationships, and other pairs have cosine similarities close to O.
Additional Results on Hierarchical Orthogonality Analogous to Figure 5 we validate the state- ment (d) in Theorem 8 Figure 10 shows that child-parent vector and trrent-grandparent vector for each feature estimated by the original unembedidngs (blue) orthogonal. The random par- ent and grandparent vectors (orange) make the cosine similarity not close to O, suggesting that the orthogonality not merely byproduct of the high-dimensional space.
18

---

Published as a conference paper at ICLR 2025
cosile Tesmont Toser of ofnduneettte
loveec leance de Feaeggane ote). Training
0.25
0.25
0.25
0.25
Ginery Features
Wordfeet Hierarchy
Original Unembeddings
Shuffidd Unembeddings
Oriiinll Unembeddings Random Parent and Grandparent.
Figure 10: WordNet noun hierarchy is encoded as the orthogonal structure predicted by statement in Theorem 8 The cosine similarity between child-parent vector and srrenttgrandparent vector
for each feature in the hierarchy estimated by original (blue) and shuffled (green) unembedding vectors with the whole (left) and training (right) set of tokens. Another baseline (orange) is the cosine similarity with random parent and grandparent vectors.
Original Unembeddings
Unamboddings
2.0
2.0
1.5
1.5
1.0
1.0
100
200
300
400
500
600
100
200
300
400
500
600
Binary in WordNet Hieerarchyy
Figure 11: Comparison of projection of train (green), test (blue), and random (orange) words estimated mean vector for each WordNet feature, from the original unembeddings (left) and the shuffled unembeddings (right). The value for each word y is (g(y) (o(g))(()/l1 [l 11 that expect to be when has the target feature. The r-axis indices denote all features in the noun hierarchy. The thick lines present the mean of the projections for each feature and the error bars indicate the standard deviation. The features are ordered by the hierarchy.
Similar to the findings in Figure 5 the shuffled unembedding vectors (green) reflect the set inclusion relationships, and the cosine similarities are close to O. When we violate the set inclusion by using only the 70% training of tokens, the right plot in/Fiuue 10]shows that the cosine similarities from shuffled (green) unembeddings are much smaller, while those from original (blue) unembeddings are still close to O. This indicates that the hierarchical orthogonality is not trivial consequence of the set inclusion, but rather reflects the semantic hierarchy.
The Mean Estimator for the Vector Representation Has High Variance The mean vector E(gw) can serve as an estimator for the vector representation Ey. However, Figure 11 which is analogous to Figure 3) shows that the projections of train and test words onto the mean vector have larger variances. Therefore, the LDA direction (eq. (5.2)) would be a more appropriate estimator for the vector representation.
Geometry of Hierarchical relations on the Fuclidean Inner Product In this paper, we trans- form the representation spaces to use the causal inner product. However, what if we employ the naive Euclidean inner product instead? To address this, we estimate the vector representations for the WordNet noun hierarchy estimated from the original Gemma unembedding vectors through cen- tering alone, without applying whitening (which necessary for the causal inner product). This approach still preserves the Euclidean inner product. Then, Figure 12 shows that the hierarchi- cal orthogonality not satisfied in the Euclidean inner product space, as evidenced by the cosine similarities not being close to O.
19

---

Published as a conference paper at ICLR 2025
fpueeee of feamer oruh
0.4
-0.2
Modaaap a wll M uot Mll Mwwn Ma wnl Amd/
-0.4
-1.
100
Binary Eudidean Inner Product
Hierarchy
Causal Inter Producct
Figure 12: WordNet noun hierarchy is not encoded as the orthogonal structure when we use the naive Euclidean inner product, whereas it is encoded as orthogonality in the causal inner product.
Original Unembeddings
Shuffled Unembeddings
0.0
0.0
-0.5
-0.5
-1.0
-1.0
350
Binary Features in WordNet
train
test
rancoo
Figure 13: Vector representations exist for most binary features the WordNet verb hierarchy.
WordNet Verb Hierarchy In the same way as for the noun hierarchy, we estimate the vector representations for the WordNet verb hierarchy. Analogous to Figure Figure shows that the vector representations exist for most binary features in the WordNet verb hierarchy. Analogous to Figure 4 Figure 14 shows that the hierarchical semanties in WordNet representation space. Analogous to Figure 3) and Figuure 10 Figure 15) and Figure 16) show that the hierarchical orthogonality encoded in the Gemma representation space for the WordNet verb hierarchy.
encoded in the Gemma
WordNet Noun Hierarchy encoded in LLaMA-3 Model We also validate the theoretical pre- dictions in the LLaMA-3-8B model (Dubey et al. (024) in the same way as for the Gemma model. The vector representations for most binary features the WordNet noun hierarchy exist in the LLaMA-3 model (Figure 17p. The hierarchical semantics in WordNet are encoded in the LLaMA-3 representation space (Figure 18). Lastly, the hierarchical orthogonality is encoded in the LLaMA-3 representation space for the WordNet noun hierarchy (Figure 19 and Figure 20).
G WHY DOES SET INCLUSION GIVE THE ORTHOGONALITY?
The left panel of Figure 3 shows that the cosine similarity between the child-parent vector and the parent vector also close to 0 when they estimated by the shuffled unembedding vectors. One of the possible explanations for this phenomenon is that the set inclusion rela- tionships (and the estimating process) give the orthogonality For example, the set inclusion between the collections ()=) = (dog, sasdwiinihi ch, running) for a child and ((w) = (dog, sandwict ch, running, France, scientist. st, diamond] for parent w make the child-parent vector and the parent vector orthogonal.
Since the LDA-based vector representation estimated by jeq (5.2) is similar the vector E(gw). we briefly derive the orthogonality using the mean vector as the vector representation. Suppose that a aj,..., a,.. by..... bN, i.idd (0,, which may correspond to the (centered and whitened) shuffled unembeddings. Here, az's for the child and both az's 's and by 's are for the par-
20

---

Published as a conference paper at ICLR 2025
cost/.... E (Original Unembeddings)
cosieu..J) Unembeddings)
100
0.75
150
150
150
0.00
-025
300
300
300
-075
350
-100
Figure 14: WordNet verb hicrarchy is encoded in Gemma representation space.
costi forn fo a). Whole
costly fuuee oaanecaal Training set
0.25
0.25
0.00
0.00
0.25
0..5
-0.50
-0.50
0.75
0.75
-1.00
-1.00
100
350
200
50
100
150
200
Binary WordNet Hierarchy
Original Unembedkings
Shuffee Unambedddigs
Unembediings
Randomm
Figure 15: WordNet verb hierarchy is encoded as the orthogonal structure predicted by statement (a)
in Theorem 8
ent w. Then, the parent vector Vw No (Σ a, +E)+ and the child vector is v, = N.
+ Σ a.
We know that .i...., CN,,, probability. Therefore, we have
bi...., bN, are mutually orthogonal and ||a:|]2 ||b: 1|2 d with high
(u= -Uw) Cw
(G.1)
No
Σo: a
1
Σb,)
No + Nob Σ
0a
No + No Eb,
(G.2)
Na + No)
Na + No
No
1
Na(Na No)2
01)
T
Σ
a
(Na + No)
1 bi) ΣE b;
(G.3)
No
1
N..N. No)2 (Nad)
(Na No)2 (Nod)
(G.4)
(G.5)
with high probability.
In words, the orthogonality between the child-paren vector and the parent vector can be derived from the set inclusion relationships. We can say that the inclusion is the hierarchical relation as defined in Definition 2, but natural semantic hierarchy is our main focus in this paper. In this context, the right panel of Figure 5) shows that the violation of the set inclusion relationships makes the cosine similarity not close to O, suggesting that semantic hicrarchy is encoded orthogonality.
21

---

Published as a conference paper at ICLR 2025
cos Ferrett otnn fomeectt
fyoer of fesncrate
fyargaaee otal Training
0.25
0.25
0.00
0.00
M My M M w WMM
-0.25
-0.50
-0.50
0.75
Binary Fealtures
WordNet Hierarchy
Original Unembeeddngg
Shuffled Urembarddings
Original Unambaddiings Randomm and Graz dpanent
Figure 16: WordNet verb hierarchy encoded as the orthogonal structure predicted by statement (d)
in Theorem 8
Original Unembeddings
Shuffled Unembeddings
0.0
0.0
005
-0.5
1.0
1.0
Binary Features in WordNel Hiieranchy
train
test
rancom
Figure 17: Vector representations for most binary features the WordNet noun hierarchy exist in the LLaMA-3 model.
costi.. (Original Unembeddings)
costd.. (Shuffled Unembeddings)
0.75
0.50
0.00
-025
0.50
400
400
1.00
400
Figure WordNet noun hierarchy is encoded in LLaMA-3 representation space.
costi Lewrest of loaret or a). Whole set
costir- Eoveert fowren al. Training set
0.25
Binary Features WordNit Hierarchy
Original Unembeddings
Shuffed Unambeddings
Original Unembeddingg
Random Farent
Figure 19: In LLLMA-3 representation space, WordNet noun hierarchy is encoded as the orthogonal structure predicted by statement (a) in Theorem
22

---

Published as a conference paper at ICLR 2025
coslfu- Inarectate: aoverrat fyoodpureeettdd. Wht
cosit, frover of Training
0.25
0.25
0.00
0.00
0.25
-0.25
-0.50
-0.50
-0.75
-0.75
-1.00
-1.00
100
200
300
400
1:00
2000
300
4000
Bimary Fealures
WordNet Hierarchy
Original Unembeddings
Shuffled lrrembedding
Original Unembeddings
Randomm Parent and Grardparent
Figure 20: In LLaMA-3 representation space, WordNet noun hierarchy is encoded as the orthogonal structure predicted by statement (d) in Theorem
23


Reviews:
[{"review_id": "2DrG25onBD", "reviewer": "Reviewer_yqwm", "summary": "This paper presents a theoretical and empirical analysis of how concept hierarchies are encoded in the geometry of a pretrained LM's unembedding layer. The theoretical part develops a rigorous formalization of several notions related to the linear representation hypothesis, such as linear representations, vector representations of binary concepts, polytope representations of categorical concepts, and the notion of hierarchy being represented via orthogonality. The empirical part uses this formalism to identify how the conceptual structure of the WordNet hierarchy is reflected in the linear representational geometry of two LMs (Gemma-2B and Llama-3-8B). The empirical results, along with clear visualizations, constitute convincing evidence that the developed theoretical formalism closely tracks the actual representation of a conceptual hierarchy in the (un)embedding layer.", "questions": "n/a", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "**Originality:** The sequence of definitions (linear representations, vector representations of binary concepts, polytope representations of categorical concepts) culminating in proposed structure of hierarchical orthogonality (Theorem 8) is a novel perspective while at the same time very intuitively appealing. Among the related work I'm familiar with, hierarchical orthogonality is reminiscent of partial orthogonality (Jiang et al., 2023; https://arxiv.org/abs/2310.17611, cited in the submission), but the two notions are distinct enough\n\n**Quality:** The paper is of exemplary quality. The empirical analysis validates all theoretical predictions.\n\n**Clarity:** The paper is exceptionally clear, well-written, and easy to read.\n\n**Significance:** The paper significantly advances our understanding of the unembedding layer, revealing a highly intuitive linear structure in the representational geometry of a conceptual hierarchy. While the analysis in its current form is limited to concepts/features that are expressed as single tokens in the vocabulary of the LM's tokenizer, the formalism of hierarchical orthogonality can potentially provide a theoretical foundation for the analysis of more complex concepts in early-middle layers, which has recently become the focus of work on detokenization/retokenization (https://transformer-circuits.pub/2022/solu/index.html., https://arxiv.org/abs/2305.01610), stages of inference (https://arxiv.org/abs/2406.19384) and the LM's \"inner vocabulary\" (https://arxiv.org/abs/2410.05864).", "weaknesses": "I don't see any substantive weaknesses. \n\nAn obvious point that might be made here is the limitation (also acknowledged by the authors) that the presented formalism and empirical results are only applicable to the unembedding layer, but in my view the unembedding layer is sufficiently complex and interesting to serve as an object of study in its own right.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7kFJiwJh7u", "reviewer": "Reviewer_GuHR", "summary": "The paper generalises the idea that binary concepts can be represented as directions of their vectors in a vector space. The authors argue that this idea fails when concepts are not binary, e.g. we have mammals versus birds and each of these has many elements therein. They they construct a new theory that for such manifold concepts one needs a polytope in a vector space. They devise a nice theory,  generalising the notion of linear combinations over binary concepts to these polytopes and experiment with these findings using the vector spaces formed by Gemma and LLaMA.", "questions": "can you please answer to the points in weaknesses? I would appreciate it.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1- Introducing a nice mathematically solid theory into the vector representations learnt by large language models. \n2- Finding a novel way to represent ontological hierarchies of concepts in the spaces learnt by LLMs. \n3- Solid maths and theory", "weaknesses": "Certain concepts are not well defined:\n\n1- Please provide a clear definition of the unembedding space early in the paper, explaining its role in their theoretical framework.\n2- Please provide concrete examples of concepts that are and are not causally separable, and explain how this distinction is crucial for their theoretical framework. The notion of causally separable introduced in the paper at the moment is vague, many concepts can be imagined to be causally separable. On the other hand, it is not clear at all what is not causally separable? I think the authors want to use this dichotomy to categorize entities under different general labels, but the definition is not helping. \n3-  Do please provide a brief explanation of the causal inner product within their paper, highlighting its relevance to their work, rather than relying solely on the external reference. At the moment, the notion of a causal inner product is not clear. The authors refer to the work of Park et al, but I am not sure why and I am not sure what is taken from it. \n4- The dataset on which the theory is testes is rather small (less than 600 nouns and less than 400 verbs). It does provide a proof of concept, but cannot be used as a large scale validation.  I unfortunately do not know what is mainstream in the field. I suggest the authors find a dataset used in knowledge graph literature and test on it. \n5- I wonder why the theory has only been tested on nouns and verbs. In other words, it is not clear what exactly are concepts. I would like to suggest the authors explicitly define what they consider to be a \"concept\" in the context of their work, and discuss whether their theory could be applied to other parts of speech or more abstract concepts.\n6- Please discuss how your approach compares to or could be integrated with existing work on knowledge graphs and ontologies, and how this might enhance the applicability of their theory. the paper needs a better theory of concepts to start from and to base itself on. I feel there is a good argument to be made by relating it to knowledge graphs and ontologies rather than working with a subset of WordNet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o7LIfQV7Jq", "reviewer": "Reviewer_yBgV", "summary": "This paper investigates the representation of semantic concepts in large language models (LLMs), specifically testing and extending the linear representation hypothesis. This hypothesis proposes that semantic features, like gender or categorical distinctions, can be represented linearly in the embedding space. However, existing representations primarily address binary concepts with natural contrasts. This paper broadens this to include hierarchical and categorical concepts, proposing that these can be represented as vectors and polytopes within the representation space. Additionally, the paper formalizes hierarchical relationships as orthogonality between representations and validates these claims on the Gemma and LLaMA-3 models using WordNet-derived hierarchies.", "questions": "1. On the Selection of Parameters for the Causal Inner Product: Can you elaborate on how you select the matrix A and the vector gamma in the causal inner product, and whether this choice generalizes across models? Would different values of A significantly alter the representation space, potentially affecting concept separability?\n\n2. Approximation in Magnitude and Orthogonality of Representations: In cases where the representations for binary and hierarchical concepts are not exactly separable, do you have methods to measure or improve the alignment with the proposed theoretical structure (e.g., adjusting vector magnitudes or enforcing orthogonality)?\n\n3. Impact of Hierarchical Orthogonality on Model Behavior: Could you provide more examples or case studies on how hierarchical orthogonality impacts the model’s downstream tasks? For instance, does this structure improve classification accuracy or clustering quality in hierarchical classification tasks?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative Extension of the Linear Representation Hypothesis: The authors rigorously extend the hypothesis by introducing vector and polytope representations for non-binary, hierarchical concepts. This is particularly valuable because it addresses concepts lacking binary contrasts and presents a way to capture categorical hierarchies in language model embeddings.\n\n2. Mathematical Rigor and Clear Formalization: The introduction of the causal inner product is mathematically grounded, ensuring that causally separable concepts are orthogonal in the transformed space (Equation 2.2). This rigor adds depth to the standard linear representation hypothesis by linking embedding and unembedding spaces without affecting model probabilities (Equation 2.1).\nTheorem 4 (on magnitudes of binary representations) and Theorem 8 (on hierarchical orthogonality) are presented with precise definitions and assumptions. The mathematical derivation ensures that vector magnitudes and orthogonal properties align with the semantics of hierarchical structures.\n\n3. Clear and Well-Defined Experimental Setup: The experiments on the WordNet hierarchy using Linear Discriminant Analysis (LDA) are robust. By estimating vector representations for over 900 concepts, the paper empirically tests whether hierarchical concepts are represented orthogonally in the embedding space. Results showing the preservation of hierarchical orthogonality even after shuffling embeddings further validate their claims.\n\n4. Practical Implications: Extending concept representation to polytopes and using orthogonality for hierarchy offers a promising direction for LLM interpretability. The authors’ approach could pave the way for more structured and interpretable embeddings, potentially enhancing downstream tasks such as knowledge extraction and structured generation.", "weaknesses": "1. Clarity on the Choice of Transformation in Causal Inner Product: While the causal inner product transformation (Equation 2.2) is mathematically sound, further clarification on how the invertible matrix and constant vector are determined would strengthen the paper’s rigor. Specifically, elaborating on the practical implications of choosing these parameters and their influence on orthogonal representations in various models could provide more insight for researchers replicating or extending this approach.\n\n2. More Explicit Connection to Hierarchical Concepts: The hierarchical orthogonality introduced in Theorem 8 is foundational to the paper’s approach, yet its practical consequences could be explored more thoroughly. For instance, how does this orthogonality affect the interpretability or utility of representations in hierarchical tasks? Including a discussion of possible applications, such as taxonomy creation or hierarchical clustering, would enrich the paper’s contribution and situate it within the broader field of LLM interpretability.\n\n3. Expanding on Limitations and Approximation Issues: The paper notes that exact linear representations may not always emerge during LLM training due to stochasticity in optimization. While this limitation is understandable, the authors could discuss how close approximations might affect the reliability of hierarchical representations, especially under more complex, multi-level hierarchies. This would give readers a clearer view of the limitations of the proposed framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qEs7NcmZjV", "reviewer": "Reviewer_zPtr", "summary": "As I understand it, the core value proposition of this paper is the report of an empirical observation: that, with respect to a particular inner-product, conceptual representations in transformer-based LLMs are vectorial, and moreover that hierarchical relationships between concepts correspond to orthogonality, while co-hyponymic categorical concepts are organised as simplices, altogether resulting in a direct-sum of polytopes.\n\nThe particular (sort/kind of) inner-product used is called \"causal\" (CIPs), which established in (apparently immediately) prior work by Park et al. CIPs are a piece of conceptual technology, which I understand as follows:\n> Assuming an operational/counterfactual notion of concept as a latent variable in a causal chain <Input -> Latent -> Output>...\n> Defining causal separability of two concept-variables to be well-definedness of the output on their independent joint distributions...\n> An inner product is \"causal\" when it treats causally separated concept-variables as orthogonal.\n> Park et al. gives methods to obtain and compute with such CIPs, and shows how CIPs admit a duality theorem that permits embedding and unembedding spaces to be identified and viewed as isomorphic to some Euclidean space equipped with the usual inner product (in contexts where there are embedding, softmax, and unembedding maps to speak of.)", "questions": "I am currently borderline because I have questions, but I am happy to raise my score if my major questions are addressed: First is what CIPs have to do with LLMs and with the particular empirical observation. Second is how the <Word-Token-Concept> gap was crossed. I'm going to write a little adversarially now, but in hopefully a productive way; I like the paper and would like to strengthen it.\n\n**************\n\n1. Regarding CIPs, their robustness, and their conceptual relevance for the empirical claim at hand.\n\nThe content of CIPs was difficult to glean from the materials presented in this paper alone, and I had to refer to the prior work for details. Given the closeness of presentation (down to the choice of variables), I would recommend explicit attribution of credit to the prior work. As a presentational note, just reproducing Thm 3.2 would have gone some way to answering my questions as a reader about why \"duality\" is used as a term, and why the embedding and unembedding spaces are assumed to be Euclidean and of matching dimension, etc. It is perhaps clearer to indicate that having a CIP is desirable because it enables dual presentation of embeddings and unembeddings, it renders computation of a semantic character as easy as regular Euclidean inner products, and so on.\n\nHere are several cases of conceptual difficulty I had that made it difficult for me to understand what the role of CIP in this story is, and I hope you can help sort me out.\n\n> The definition of CIP from causal separation is potentially too stringent; consider the counterexample case where two co-hyponymic concepts are only defined in the domain of a common hypernym, e.g. \"is_bird\" and \"is_nocturnal\" are causally separable properties but only hold in the domain of \"is_animal\", whereas the current statement requires well-definition of birdness and nocturnality everywhere. So ok, let's say that as a reader I don't trust your first-principles approximation of what a concept is; is it worth spending the time and space to convince me or is that not really important for what needs to be shown here?\n\n> In the mathematical setup of CIP, it is assumed that there is a single euclidean embedding space for text, a matching unembedding space, and a softmax. This is for just the \"final token position\" (line 107), so I'm assuming an autoregressive transformer, but then there's no further talk of the \"internal structure of LLMs\" (line 113). This is suspicious to me, because if there's absolutely nothing structurally relevant about the architecture of LLMs to the subsequent analysis of concepts and causal separation and then CIPs, then I can freely swap \"LLMs\" for anything else? If you're doing something so general-purpose, then I would like to be convinced that CIPs are actually necessary, rather than a contrived thing that the empirical data doesn't really depend upon.\n\n> Where exactly are CIPs used or useful? My best guess so far is that CIPs and their assumptions are required in order to prove Theorem 8, which yields the analytic presentation in Figure 1 as a direct sum of polytopes. However, the polytope-simplices never have their convexity property explored or exploited [cf. Gardenfors \"Geometry of Meaning\"], and the particular example presented in Figure 1 is a special case where at most one node at every level has children, so I believe a fair equivalent characterisation would also be: \"different levels are orthogonal, and co-hyponymic nodes form clusters.\" I'm not convinced that this is really worth all the baggage of CIP, so where else is it being employed?\n\n2. The Word-Token-Concept gap.\n\nWordNet and vectorial embeddings of words are perfectly good and commonplace. What I was excited to learn about was how you managed to extend this analysis over to the tokens of LLMs. I must have missed something, because the only thing I can see that gives some kind of methological hint as to how this gap was crossed is a sentence in line 955, tucked in the appendix: \"Lastly, we use the vector representations estimated by the non-split collection of tokens for Figure 2, (etc.)\" I feel like I must have missed it somehow because the omission of this crucial methodology would be such an inconsistency with the attention to detail for the rest of the paper, but I just can't figure out what is being done here to cross-validate findings on WordNet with Gemma and Llama. The mathematical assumptions made only raised more questions (assume final token only for an autoregressive... predictor? autoencoder?).\n\n**************\n\nThese two questions are crucial for me because they would make the difference between an uncharitable reading (e.g. CIP is a just-so construct and the empirical work is mostly WordNet with unclear methodology for LLMs), and a very happy one (e.g. CIP is a general-purpose information geometry, and here is a clear and well-motivated demonstration of how this conceptual lens can be applied to find conceptual structure even at the token level of LLMs).", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "To me, there are two noteworthy aspects of the empirical claim.\n\n<Emergence> First is that this structure arises emergently and naturally in LLM representations. This would be a scientific contribution to ongoing discourse concerning how and whether LLMs understand [cf. Bender & Koller's \"Climbing towards NLU\" which casts LLMs as Searle-type non-understanders vs. Sogaard's \"Grounding the Vector Space of an Octopus\" which surveys evidence that language models obtain geometric models of concepts in their vectorial representations, as the current paper does.]\n\n<Word-Token-Concept> Second is that the claim holds for transformer-based LLMs, where (to my understanding) tokenisation decouples the relationship between words and semantic vectors. Demonstrating empirically that semantic representations not only exist but satisfy geometric constraints according to their natural-language conceptual organisation *at the token level* is big-if-true, even just methodologically.", "weaknesses": "This is a simple and compelling observation, especially in the broader context of approaches to vectorial semantics downstream of Firth's distributional hypothesis, where there are many rhyming ideas in the literature. For example, the usage of an inner-product in a high-dimensional space to test for membership in a vectorial data structure is prefigured by Kanerva's hyperdimensional computing. Hyponymic structure alongside convex simplicial structure is treated mathematically in [Interacting Conceptual Spaces I: Grammatical Composition of Concepts; https://arxiv.org/pdf/1703.08314], and the idea of looking for hypo/hypernymic structural representations in vector spaces is its own cottage industry [as a google search of the terms \"hyponymy\" and \"density matrix\" will demonstrate.]\n\nWhile the authors have taken pains to point out (lines 51, 124, 225, 308) that the current approach is a conceptual advance *due to the consideration of vectors* rather than, e.g. directional cones in prior work, this could be read as a strawman that misrepresents a broader research context of vectorial semantics, and it just doesn't do justice to the empirical claims of the paper. Look, even if I totally discount any claim to conceptual novelty regarding the use of vectorial semantics, I still would consider the paper to be strong on the basis of the empirical observation alone, provided the observation is adequately situated.\n\nIntegrating some comparison to approaches in vectorial semantics could be an opportunity to strengthen the conceptual dimension of the paper. As far as I can tell, other vectorial semantics worok typically has a \"feature-engineering\" rather than exploratory nature, and using the geometry of causal inner products seems to be an effective and relatively novel idea; having these as literature supported claims could potentially broaden the appeal and reach of this work at little cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)