Not All Language Model Features Are One-Dimensionally Linear
Authors: Joshua Engels
Eric J Michaud
Isaac Liao
Wes Gurnee
Max Tegmark
Venue: ICLR 2025 Poster (2025)
Keywords: Mechanistic Interpretability
Learned Representations
Dictionary Learning
Sparse Autoencoders
Features
Circuits
Primary area: interpretability and explainable AI
TL;DR: We define multi-dimensional language model features and automatically find and intervene on multi-dimensional circular features.

Abstract: Recent work has proposed that language models perform computation by manipulating one-dimensional representations of concepts ("features") in activation space. In contrast, we explore whether some language model representations may be inherently multi-dimensional. We begin by developing a rigorous definition of irreducible multi-dimensional features based on whether they can be decomposed into either independent or non-co-occurring lower-dimensional features. Motivated by these definitions, we design a scalable method that uses sparse autoencoders to automatically find multi-dimensional features in GPT-2 and Mistral 7B. These auto-discovered features include strikingly interpretable examples, e.g. $\textit{circular}$ features representing days of the week and months of the year. We identify tasks where these exact circles are used to solve computational problems involving modular arithmetic in days of the week and months of the year. Next, we provide evidence that these circular features are indeed the fundamental unit of computation in these tasks with intervention experiments on Mistral 7B and Llama 3 8B, and we examine the continuity of the days of the week feature in Mistral 7B. Overall, our work argues that understanding multi-dimensional features is necessary to mechanistically decompose some model behaviors.

Published as a conference paper at ICLR 2025
NOT ALL LANGUAGE MODEL FEATURES ARE ONE-DIMENSIONALLY LINEAR
Joshua Engels
Eric J. Michaud
Isaac Liao MIT il Aaiiammmi..e.m
MIT
MIT & IAIFI
Jengel 1 s@mit.
er 1 jm@mit. cimmmiiteed
Wes Gurnee
Max Tegmark MIT & IAIFI
MIT
wesgêmit. edu
tegmarkemit.en edu
ABSTRACT
Recent work has proposed that language models perform computation by manipu- lating one-dimensional representations of concepts ("features") in activation space. In contrast, we explore whether some language model representations may be inherently multi-dimensional. We begin by developing a rigorous definition of irreducible Iulli-diimensionl features based on whether they can be decomposed into either independent or con-co-occurring lower-dimensional features. Motivated by these definitions, we design scalable method that uses sparse autoencoders to automatically find multi-diimnsional features GPT-2 and Mistral 7B. These auto-discovered features include strikingly interpretable examples, e.g. circular features representing days of the week and months of the year. We identify tasks where these exact circles are used to solve computational problems involving modular arithmetic in days of the week and months of the year. Next, we provide evidence that these circular features are indeed the fundamental unit of computation in these tasks with intervention experiments on Mistral 7B and Llama 3 8B, and we examine the continuity of the days of the week feature in Mistral 7B. Overall, our work argues that understanding muuti-ddmmensionl features is necessary to mechanistically decompose some model behaviors.
INTRODUCTION
I Language models trained for next-token prediction on large text corpora have demonstrated remark- able capabilities, including coding, reasoning, and in-context learning (Buheck al., 2023; Achiam et al.. 2023; Anthropic, 2024; Team et al., 2023). However, the specific algorithms models learn to achieve these capabilities remain largely mystery to researchers; we do understand how language models write poetry. Mechanistic interpretability is field that seeks to address this gap by eererseengineniing trained models from the ground up into variables (features) and the programs (circuits) that process these variables (Olah et al., 2020).
One mechanistic interpretability research direction has focused on understanding toy models in detail. This work has found multi-dimensionn representations of inputs such as lattices (Michaud et al., 2024) and circles (Liu et al., 2022; Nanda al., 2023a), and has successfully reverse-engineered the algorithms that models use to manipulate these representations. A separate direction has identified one- dimensional representations & Tegmark, 2023; Marks & Tegmark, 2023; Heinzerling & Inui, 2024; Bricken et al., 2023). These findings have led to the linear representation hypothesis (LRH):: a hypothesis which has historically claimed both that 1. all representations in pretrained large language models lie along one- dimensioonl lines, and 2. model states are a simple sparse sum of these representations (Park et al., 2023; Bricken al., 2023). In this work, specifically call into question the first part of the LRH.!
high level concepts and quantities in large language models (Gurnee
An earlier version of this manuscript sparked discussion in the mechanistic interpretability community on the distinction between non-linear features and multi- dimensional features, and in fact this discussion directly led

---

Published as a conference paper at ICLR 2025
Days of the Week
Months of the Year
Years of the 20th Century
Menday mciresily
Hiss Surdby
20unryy March
Octiter Recember
1 1900
1950
1999
May
PCA axis 3
PCA axis
Suptemmer
PCA axis 4
PCA axis 2
PCA axis
PCA PCA axis 3
Figure Circular representations of days of the week, months of the year, and years of the 20th century in layer 7 of GPT-2-small colored by the token they fire on. These representations were discovered via clustering SAE dictionary elements, described in Section 4. Points are colored according the token which created the representation. See Fig. 14 for other axes and Fig. 15 for similar plots for Mistral 7B.
For the most part, these two directions have been disconnected: Yedidia (2023a) and Gould al. (2023) find intriguing hints of circular language model representations, and Bricken et al. (2023) speculate about the existence of feature manifolds, but these brief results only serve to further emphasize the lack of unifying and satisfying perspective on the nature of language model features. In this work, we seek to bridge this gap by formalizing, investigating, and systematically searching for niuti-dimennsooal language model features.
1.1 CONTRIBUTIONS
1. In Section 3, we generalize the one-dimensional definition of a language model feature to multi- dimensional features and provide an updated multi-dimensional superposition hypothesis to account for these new features.
In Section 4, we build on the definitions proposed in Section 3 to develop theoretically grounded and empirically practical test that uses sparse autoencoders find irreducible features. Using this test, we identify multi multi-dimensional representations automatically in GPT-2 and Mistral 7B, including circular representations for the day of the week and month of the year.
In Section 5, we show that Mistral 7B and Llama 8B use these circular representations when performing modular addition in days of the week and in months of the year. To the best of our knowledge, we are the first find causal cireular representations of concepts in language model. We additionally find that the model's 's circular representations respect continuous notion of time.
2 RELATED WORK
Farly word embedding methods such as GloVe and Word2vec, although only trained using co-occurrence data, contained directions in their vector spaces corresponding semantic concepts (Mikolov et al., 2013b; Pennington et al., 2014; Mikolov et al., 2013a). Recent research has found similar evidence of one-dimensional lincar representations in sequence models trained only on next token prediction, including Othello board positions (Nanda et al., 2023b; Li al., 2022), the truth value of assertions (Marks & Tegmark, 2023), and numeric quantities such as longitude, latitude, birth year, and death year (Gurnee & Tegmark, 2023; Heinzerling & Inui, 2024). These results have inspired the linear representation hypothesis (Park al., 2023; Elhage et al., 2022)
Linear Representations:
a consensus around the two part LRH we deseribe above (see (Olah, 2024; Csordás et al., 2024; Mendel, 2024; Kantamneni & Tegmark, 2025)). To clarify, we agree with these discussions, and believe the multi culti--immenionall features we find are "linear" in the sense that they are contained in low-dimensional linear subspace. but "non-linear" in the sense that this low-dimensional subspace is not one-dimensional (and this is the sense in the title).
2

---

Published as a conference paper at ICLR 2025
defined above. Jiang et al. (2024) provide theoretical evidence for this hypothesis, assuming latent (binary) variable-based model of language. Empirically, Bricken et al. (2023) and Cunningham et al. (2023) successfully use sparse autoencoders break down model's feature space into an over-complete basis of linear features. These works assume that the number of linear features stored in superposition exceeds the model dimensionality (Elhage al., 2022).
Multi-Dimensional Representations: There has been comparatively little research on multi- dimensional features in language models. Shai al. (2024) predict and verify that transformer trained on a hidden Markov model uses fractal structure to represent the probability of each next token, a clear example of a necessary lultii-dmennionall feature, but the analysis restricted to a toy setting. Yedidia (2023a;b) finds that GPT-2 learned position vectors form a helix, which implies a circle when "viewed" from below. Thus, we are not the first to find a circular feature in a language model. However, our work finds circular features that represent latent concepts from text, while the GPT-2 learned position vectors are specific to tokenization, separate from the rest of the model parameters, and causally implicated only due to positional attention masking. Another suggestive result, due to Hanna et al. (2024), is the presence of a U-shape in the representation of numbers between 0 and 100; however, Hanna et al. (2024) find that this representation is not causal, and they only show it exists within a specific prompt distribution. Recent work on dictionary learning (Bricken al., 2023) has speculated about multi-dimensional] feature manifolds; our work is similar this direction and develops the idea of feature manifolds theoretically and empirically. Finally, in a separate direction, Black al. (2022) argue for interpreting neural networks through the polytopes they split the input space into, and identifies regions of low polytope density as "valid" regions for potential linear representation.
Circuits: Circuits research seeks to identify and understand circuits, subsets of a model (usually represented as a directed acyclic graph) that explain specific behaviors (Olah et al., 2020). The base units that form circuit can be layers, neurons (Olah et al.. 2020), or sparse autoencoder features (Marks et al., 2024). In the first circuits-style work, Olah et al. (2020) found line features that were combined into curve detection features in the InceptionV1 image model. More recent work has examined language models, for example the indirect object identification circuit in GPT-2 (Wang al., 2022). Given the difficulty of designing bespoke experiments, there has been increased research in automated circuit discovery methods (Marks et al., 2024; Conmy et al., 2023; Syed et al., 2023).
Interpretability for Arithmetic Problems: Liu et al. (2022) study models trained modular arithmetic problems a + = с (mod m) and find that models that generalize well have circular representations for a and b. Further work by Nanda et al. (2023a) and Zhong et al. (2024) shows that models use these circular representations to compute e via a "clock" algorithm and a separate "pizza" algorithm. These papers are limited to the case of small model trained only on modular arithmetic. Another direction has studied how large language models perform basic arithmetic, including a circuits level description of the greater-than operation in GPT-2 (Hanna et al., 2024) and addition in GPT-J (Stolfo et al., 2023). These works find that to perform computation, models copy pertinent information to the token before the computed result and perform the computation in the subsequent MLP layers. Finally, recent work by Gould et al. (2023) investigates language models" ability to increment numbers and finds linear features that fire on tokens equivalent modulo 10.
DEFINITIONS
This section focuses on hypotheses for how hidden states of language models can be decomposed into sums of functions of the input (features). We focus on L layer transformer models M that take in token input for layers I, and output logit vectors .1..... Given set of inputs T, we let X1, be the set of
(ti.....) .t.) from input token distribution T. have hidden states X1,1. X,.,
all corresponding X,,- We write matrices in capital bold, vectors and vector valued functions in lowercase bold, and sets in capital non-bold.
3.1 -ULTI-DIMENSIONAL FEATURES
Definition (Feature). We define dy-dimensional feature as function that maps a subset of the input space into Rd4, We say that feature active the aforementioned subset.
3

---

Published as a conference paper at ICLR 2025
The input token distribution T induces d dimeenssiaa probability distribution over feature vectors f(t). As an example, let n = (so inputs are single tokens) and consider a feature that maps integer tokens to their integer values in R'. Then is a 1-dimensional feature that is active on integer tokens, and (t) is the marginal integer occurrence distribution from the token distribution.
differentiate "true" multi dinlti-dimeeioonll features from sums of lower dimensional features? We make this distinction by examining the reducibility of a potential multi-dimensionll feature. That is, f is a "true" multi-dimensional feature if it cannot be written the sum of statistically independent features and it cannot be written as the sum of two non-co-occurring features. Formally, we have the following definition:
How
Definition 2. A feature is reducible into features and b if there exists an affine transformation
+ Rf+
for some orthonormal d, x d matrix R and additive constant c, such that the transformed feature probability distribution p(a, b) satisfies at least one of these conditions:
1. p separable, i.e., factorizable product of its marginal distributions: p(a, b) p(a)p(b).
2. is a mixture, i.c., sum of disjoint distributions, one of which is lower dimensional: p(a, b) = (p)) + (1 w)p(a,
Here, is a probability density function that is conditional on the subset of T that f is active on. ò is the Dirac delta function, and w 1. By two probability distributions being disjoint, we mean that they have disjoint support (there is no set where both have positive probability measure, or equivalently the two features a and b cannot be active at the same time). Eq. (1), a is the first components of the vector Rf + and b is the remaining d, - k components. When is separable or a mixture, we also say that is separable or a mixture. We term a feature irreducible if is not reducible, i.e., if no rotation and translation makes it separable or mixture.
An example feature that is a mixture one hot encoding along a simplex; an example of feature that is separable is normal distribution In natural language, a mixture might be one hot encoding of "breed of dog", while a separable distribution might be the "latitude" and "longitude" of location tokens.
In practice, the mixture and separability definitions may not be precisely satisfied. Thus, we soften our definitions to permit degrees of reducibility:
Definition (Separability Index and e-Mixture Index). Consider a feature The separability index S(f) measures the minimal mutual information between all possible a and b defined in Eq. (1):
S(f) = min ((a; b)
where 7 denotes the mutual information. Smaller values of S(f) mean that f is more separable.
The e-mixture index (f) tests how often can be projected near zero while is active:
(f) max PEET (Iv f(t) + +cl < /Ellv f(t) + c))))
(3)
VER"S,CER CER
Larger values of Mf( mean that is more of mixture.
In Appendix B, we expand on the intuition behind why the separability and c-mixture indices as defined here correspond weakened versions of Definition 2.
We develop optimization procedures to empirically solve for the separability and e-mixture indices of two dimensional feature distributions. At high level, the separability procedure iterates over a sweep of rotations and estimates the mutual information between the axes for each angle, while the e-mixture index procedure performs gradient descent to find the band that contains the largest possible fraction of the feature distribution. For more details on the implementation of the tests, see Appendix B.2. In Section 4, we apply these empirical tests real language model feature distributions to find irreducible multi-dimensional features; we show the detailed test results on the "days of the week" cluster in Fig.
since any multidimensional Gaussian he rotated have diagonal covariance matrix
4

---

Published as a conference paper at ICLR 2025
(f(f)
0.4750
= 0.9506
600
Mutual Info (bits)
1.0
PCA 3
Count
400
0.5
200
11
0.0
PCA
-2
0 2
0
2
17
2
2n
Norm. f
Angle O
Figure 2: Empirical e-mixture index and separability index for the "days of the week" cluster along PCA components 2 and 3. Left: The e band parameterized by v and c that the optimization procedure found contained the highest fraction of points. Mid: Dot products of points in the feature distribution with the band; M, (f) is the percent of dot products within = 0.1 O. Right: Estimated mutual information for different rotations of the space: S(f) is the minimum over all rotations. This point cloud has a lower e-mixture index and higher separability index than PCA projections within typical clusters (see Fig. 3), indicating that is more likely to an irreducible multi-dimennional feature.
3.2 SUPERPOSITION
In this section, we propose an updated superposition hypothesis (Elhage et al., 2022) that takes into account multi-dimensional features. First, we restate the original superposition hypothesis:
Definition 4 (6-orthogonal matrices). Two matrices A₁ E Rdxd_ and A E Rdxd2 are 6-orthogonal |x| X2| S for all unit vectors X E colspace Ai) and X2 E colspace(AA2) Hypothesis 1 (One-Dimensional Superposition Hypothesis, paraphrased from (Elhage et al., 2022)).
Hidden states x are the sum many (> d) sparse one-dimensional features fi and pairwise 5-orthogonal vectors Vi such that (t) E. vtfi(t). We set fi(t) to zero when is outside the domain of ft.
In contrast, our new superposition hypothesis posits independence between irreducible multi- dimensional features instead of unknown levels independence between one-dimensional features:
Hypothesis 2 Multi-Dimensional Superposition Hypothesis, changes underlined). Hidden states x,, are the sum of many (*> d) sparse low-dimensional irreducible features f, and pairwise 6-orthogonal matrices V, E ddxdr, such that (t) = E. (tf. We set ft(t) to zero when t is outside the domain of f.
Note that since multi- multi-dimmnsional features can be written as the sums of projections of lower- dimensional features, next section, we will explore empirical evidence for our hypothesis, while in Appendix A, we prove upper and lower bounds the number of 5-almost orthogonal matrices V; that be packed into d dimensional space.
superposition hypothesis is stricter version of Hypothesis 1. In the
SPARSE AUTOENCODERS FIND MULTI-DIMENSIONAL FEATURES
In this section, we describe a method to identify multi-dimmnnionnl features in language model hidden states using sparse autoencoders (SAEs). Sparse autoencoders (SAEs) deconstruct model hidden states into sparse vector sums from an over-complete basis (Bricken al., 2023; Cunningham et al., 2023). For hidden states X1,., one-layer SAE of size with sparsity penalty À minimizes the following dictionary learning loss (Bricken et al., 2023; Cunningham al., 2023):
DL(,)) arg min
Σ
| D ReLU xx))l + A||ReLU(E
+|Il@+/|+BBDUUU (xeeLUEExx))lo xx))lo] (4)
B D
X
5

---

Published as a conference paper at ICLR 2025
In practice, the Ly loss on the last term is relaxed to Lp for 0 < p S to make the loss differentiable. We call the m columns of D (vectors in R3) dictionary elements.
We now argue that SAES can discover irreducible multi-dimensional features by clustering D. We will consider simple form of clustering: build complete graph D with edge weights equal to the cosine similarity between dictionary elements, prune all edges below a threshold T, and then set the clusters equal to the connected components of the graph. If we now consider the spaces spanned by each cluster, they will be approximately T Tortthogonal by construction, since their basis vectors are all T Toorthogonal. Now, consider some irreducible two-dimensional feature f; we claim that if the SAE is large enough and is active enough such that the SAE can reconstruct when is active, one of the clusters is likely be exactly equal f. If D includes just two dictionary elements spanning f, then these elements both must have nonzero activations Post- to reconstruct f (otherwise f is a mixture). Because of the sparsity penalty in Eq. (4), this two-vector solution to reconstruct f is disincentivized, so instead the dictionary is likely to learn many elements that span f. These dictionary elements will then have high cosine similarity, and so the edges between them will not be pruned away during the clustering process; hence, they will be in cluster.
Thus, we have a way to operationalize Hypothesis 2: clustering D finds T-orthogonal subspaces, and if irreducible nulti-dimensional features exist, they are likely to be equal to some of these subspaces. This suggests a natural approach to using sparse autoencoders to search for irreducible multi-dimensional features:
1. Cluster dictionary elements by their pairwise cosine similarity. We use both the simple similarity- based pruning technique described above, well spectral clustering; see Appendix F for details, including comments on scalability.
2. For each cluster, run the SAEs on all X X1, and ablate all dictionary elements not in the cluster. This will give the reconstruction of each x restricted to the cluster found in step (if no cluster dictionary elements are non-zero for given point, we ignore the point).
3. Examine the resulting reconstructed activation vectors for irreducible multi-dimensional features. This step can be done manually by visually inspecting the PCA projections for known irreducible multi-dimennional structures (e.g. circles, see Fig. 10) or automatically by passing the PCA projec- tions to the tests for Definition 3.
Pseudocode for this method is in the appendix in Alg. 1.
1.0
This method succeeds on toy datasets of synthetic irre- ducible multi-dimensionn apply this method to language models using GPT-2 (Rad-
Other cluster
features; see Appendix D. We
x
Cluster shown in Figure
S(f): more irreducible
0.8
ford et al., 2019) SAES trained by Bloom (2024) for every layer and Mistral 7B (Jiang et al., 2023) SAES that we train on layers 8, 16, and 24 (training details in Appendix F).
days
0.6
month ears
0.4
Strikingly, we reconstruct irreducible multi-dimennional features that are interpretable circles: in GPT-2, days, months, and years are arranged circularly in order (see Fig. 1); in Mistral 7B, days and months are arranged circu- larly in order (see Fig. 15). These plots contain the PCA dimensions that most clearly show circular structure; these best dimensions are usually the second and third because the first PCA dim is an "intensity" direction that manifests as the radius of the circle in Fig. 1 (thus the overall struc- ture for these multi-d features perhaps best thought of as a cone). See Fig. 14 for all PCA dimensions visualized).
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
M.(f:: more irreducible
Figure 3:
Mixture index and separability index of GPT-2 features. Features from Fig. 1, which we had manually identi- fied, score highly as candidate multidi- mensional features with these metries.
For each cluster of GPT-2 SAE features, we take the re- constructed activations and project them onto PCA components 1-2, 2-3, 3-4, and 4-5 (or fewer there are fewer features in the cluster) and measure the separability index and c-mixture index of each 2D point cloud as described in Appendix B.2. The mean scores across these planes are computationally tractable approximation of Definition 3. We plot these mean scores in Fig. 3, and find that the features which we had manually identified in Fig. I are among the top scoring features along
Code: https: http:///ittttt wwt.b/c.cmm//eaee com/:b////c/oos/eeeeeeeeee s /uul tiiiiiieesnss ona Keatures

---

Published as a conference paper at ICLR 2025
Table Aggregate model accuracy on days of the week and months of the year modular arithmetic tasks. Performance broken down by problem instance in Appendix I.
Tue
Jan
Feb
Mon
Mar
Wed
Dec
Apr
SStn
Nov
May
Oct
Sat
Thu
Jun
Model
Weekdays
Months
+
Sep
Juf
Aug
Llama 8B
29 /9
143/ / 144
Mistral 7B
/9
125 / 144
Figure 4: Top two PCA components on the a to-
GPT-2
/ 49 10 144
ken. Colors show Left: Layer 30 of Mistral on Weekdays. Right: Layer 5 of Llama on Months.
both measures of irreducibility. Thus, theoretical tests indeed be used to find interpretable irreducible features. We show the top 20 feature clusters, measured by the product of (1 e-mixture index ) and separability index, in Appendix G. Out of all 1000 clusters, the Fig. clusters rank 9, 28, and 15 by this metric, respectively."
5 CIRCULAR REPRESENTATIONS IN LARGE LANGUAGE MODELS
In this section, we examine tasks in which models use the multi-dimensional features we discovered in Section 4, thereby providing evidence that these representations are indeed the fundamental unit of computation for some problems. Inspired by prior work studying circular representations in modular arithmetic (Liu al., 2022), we define two prompts that represent "natural" modular arithmetic tasks:
Weekdays task: "Let's do some day of the week math. Two days from Monday is" Months task: "Let's ss do some calendar math. Four months from January is"
For Weekdays, we range over the 7 days of the week and durations between and days to get 49 prompts. For Months, we range over the 12 months of the year and durations between 1 and 12 months to get 144 prompts. Mistral 7B and Llama 8B (AI@ 2024) achieve reasonable performance the Weekdays s task and excellent performance on the Months task (measured by comparing the highest logit valid token against the ground truth answer), as summarized in Table 1. Interestingly, although these problems are equivalent to modular arithmetic problems + 3 = (mod m) for m = 7, 12, both models get trivial accuracy on plain modular addition prompts, e.g. "5 + (mod 7) =". Finally, although GPT-2 has circular representations, gets trivial accuracy on Weekdays and Months.
To simplify discussion, let be the day of the week or month of the year token (e.g. "Monday" or "April"), 3 be the duration token (e.g. "four" or "eleven"), and be the target ground truth token the model should predict, such that (abusing notation) we have a + 3 = r. Let the prompts of the task be parameterized by j. such that the jth prompt asks about a, 3y. and Yz-
We confirm that Llama 8B and Mistral 7B have circular representations of a on this task by examining the PCA projections of hidden states across prompts at various layers on the token. We plot two of these Fig. 4 and show all layers in Fig. 18. These plots show circular representations as the highest varying two components in the model's representation of many layers.
5.1 INTERVENING ON CIRCULAR DAY AND MONTH REPRESENTATIONS
We now experiment with intervening on these circular representations. We base our experiments on the common interpretability technique of activation patching, which replaces activations from a "dirty" run of the model with the corresponding activations from a "clean" run (Zhang & Nanda, 2023). Activation patching empirically tests whether specific model component, position, and/or representation has a causal influence on the model's 's output. We employ custom subspace patching
WW also tried an alternative ranking scheme: we sorted the clusters by separability and irreducibility and set the cluster score equal to the minimum sorted position between the two sorted lists. The Fig. I clusters rank 105, and 12 by this metric.
7

---

Published as a conference paper at ICLR 2025
Step 1: Lear Circular Probe P
Find
Mistral Weekdays
Mistral Months
Trainpo Ponts X21
Apply PCA
Learn Circuar
Marix Wei
Probe P
91
Average logit diff
2.5
Step
Intial point
Llama Weekdays
Liama Months
Apply PCA
Circular
Matrix
0
Apply interventin
Apply of FCA
-2
point
Apply Sinv!
Layer
No.op
-atch circle Averaage ablate
Patch layer Patch PCA
Figure 5: Visual representation of the circular intervention process. Top: We learn a circular probe on the PCA projec- tion of training set. Bot: To intervene, we change the circular representation to a' and average ablate other dimensions.
Figure 6: Mean and 96% error bars for intervening on the token across layers using different intervention methods. The circular intervention technique outperforms patching only the top 5 PCA components and leaving the rest un- changed, and almost reaches the upper bound performance of patching the entire layer.
method to allow testing for whether a specific circular subspace of a hidden state is sufficient to causally explain model output. Specifically, our patching technique relies on the following steps (visualized Fig. 5):
1. Find subspace with circle to intervene on: Using PCA reduced activation subspace to avoid overfitting, we train "circular probe" to identify representations which exhibit strong circular patterns. More formally, let x be the hidden state at layer token position for prompt
Let Wi, E pexdd be the matrix consisting of the top k principal component directions of x In our experiments, we set = We learn a linear probe P R from W.,- -X,1 unit circle in a. In other words, circle(a)) = [cos(2na//)) sin(2///7)) for Weekdays and circle(a) = [cos( (2Taa 11)), sin ain(2 12)] for Months, P is defined as follows:
P arg min
Σ
p' W., +. x circle(a)
(5)
P'ER2k
Intervene on the subspace: Say our initial prompt had a = aj and we are intervening with ay. In this step, replace the model's projection on the subspace P Wi,, which will be close to circle(as), with the "clean" point circle(ay). Note that we do not use the hidden state x"", from the "clean" run, only the "clean" label ay. In practice, other subspaces of x may be used concurrently by the model in "backup" circuits (see e.g. Wang et al. (2022)) to compute the answer, so if we just intervene on the circular subspace the remaining components of the activation may interfere in downstream computations. Thus, isolate the effect of our intervention, we the average ablate the portion of the activation not in the intervened subspace. Letting X,,7 be the average of x across all prompts indexed by and p be the pseudoinverse of P. we intervene via the formula
X = = W., (circceeaay)) x.,)
(6)
We run our patching on all 49 Weekday problems and 144 Month problems and use as "clean" runs the or other possible values for 3, resulting in a total of 6 patching experiments for Weekdays and 144 11 patching experiments for Months. We also run baselines where we (1) replace the entire subspace corresponding to the first PCA dimensions with the corresponding subspace from the clean run, (2) replace the entire layer with the corresponding layer from the clean run, and (3) replace the entire layer with the average across the task. The metric we use is average logit difference across all patching experiments between the original correct token (az) and the
8

---

Published as a conference paper at ICLR 2025
target token (a See Fig. 6 for these interventions on all layers of Mistral 7B and Llama 3 8B on Weekdays and Months.
The main takeaway from Fig. is that circular subspaces are causally implicated in computing 7, especially for Weekdays. Across all models and tasks, early layer interventions on the circular subspace have almost the same intervention effect as patching the entire layer, and are usually better than patching the top PCA dimensions from the clean problem.
Days
Task Duration Days
0
-1
-2
Patching experiments in Appendix J show a is copied to the final token on layers 15 to 17, which why interven- tions drop off there. Additionally, while in this section we train a probe on a dataset of prompts, Section 5.2, we show that intervening the circle discovered via SAE clustering in Section also works.
Task Duration 4 Days
Task Duration Days
To investigate exactly how models use the circular sub- space, we perform off distribution interventions. We mod- ify Eq. (6) so that instead of intervening on the circumfer- ence circle(a), we sweep over a grid of positions (r, (r,0) within the circle:
Monday
Wednesday
Friday
Sunday
Tuesday
Thursday
Figure 7: Off distribution interventions on Mistral layer 5 on the Weekdays task. The color corresponds to the high- est logit y after performing the circular subspace intervention that point.
x X + W., 2 P [rc((()))) cos((O,rssn(())))) - x.J) (7)
We intervene with
E [0.0.,..... 2].0 E
[0, 2T/ 2 1190,,..,997/100000 logit y after the forward pass. Fig. 7 displays these results Mistral layer 5 for 3 E [2.3, 3, 45). They imply that Mistral treats the circle as colti-dimensional representation with a encoded the angle.
and record the highest
5.2 INTERVENING WITH THE SAE PLANE
In the last section, we train probes manually on the PCA of the activations fit circle. A perhaps more natural approach is intervening with the precise circle we found in Section 4. To determine if this approach is feasible, we first project layer Mistral 7B Weekdays activations into the weekdays plane that was discovered by clustering (see Fig. 15; the plane is defined by PCA dimensions 2 and of the cluster). In the rest of this section, we call this plane the SAE plane. In Fig. 19, we find that indeed, the Weekdays representations projected into the SAE plane form a circle (see Fig. 19). We thus can fit a circular probe to this 2D plane as in Eq. (5). Similarly, we call this probe the SAE probe.
Because we only have layer 8 clustering results for Mistral, we train an SAE probe only on layer
Average logtt dogtteloggtiiiiin diffrrncc shtrr Intervention
Intervene w/ Leyer 8 Probe Intervene SAE
8. We then evaluate interventions using this SAE probe on layer of Mistral, but also at neighboring layers, since nearby layers should have similar representations (see e.g. Belrose al. (2023)). We compare two baselines: 1) training normal circular probe on the PCA projections of each layer as described in Sec- tion 5.1, and 2) training a circular probe only on layer 8 and then evaluating on adjacent layers (in the same way as for the layer 8 SAE probe). We show the results of these methods in Fig.
0.0
Intervene w/ Per Layer Probe
-1.0
-2.0
2.5
-3.0
10
Layer
Interventions on the Mistral 7B Weekdays task with different methods of deter- mining the probe.
Figure 8:
We find that on all layers, using the SAE probe only slightly decreases intervention performance as compared to training a circular probe (from -2.58 to -2.01 average logit difference on layer 8). Even more interestingly, the layer SAE probe is much more robust to layer shifts than the layer circular probe; for example, using the layer 8 circular probe on layer results in an average logit difference of 0.029, whereas using the layer 8 SAE probe results in average logit difference of

---

Published as a conference paper at ICLR 2025
-2.32. This is intriguing evidence that the SAE is perhaps finding more "true" (or at least more robust) features than our circular probing technique.
5.3 CONTINUITY OF CIRCULAR REPRESENTATIONS
In past sections, the representations of the interpretable numeric quantities we have discovered have been mostly discontinuous; that is, the days of the week and months of the year in Fig. I and Fig. 15 are clustered at the ver- tices of heptagon and dodecagon, and there is nothing "between" " adjacent weekdays or months along the cir- cle. In this section, we will examine the "continuity" of the circular features we have discovered. Although continuity of the representation is not requirement of Definition 3, it would further decrease the e-mixture index, and would also increase our subjective perception of the circular feature as an intrinsic model feature representing a continuous quantity (time). Thus, we create a synthetic dataset containing the text "[very early/very late] on [Mon- Tt.TTuaaay...SSndaaa
Brering
Morning Eveering Set
Pheming
Overing Wed
Merning Sun Evening Sun
Herning Wed
Evering Tue
Ese Weihhii Mon
Mon
Figure 9: Layer 30 Mistral 7B activa- tions for [morning/evening] plotted pro- jected into the PCA plane for [Mon- day/Tuesdays MayyTueedayy..//undaa//
[Mon- day/Tuesdayy../Sundaaa
and simply plot the projections of the layer 30 activations into the top two PCA compo- nents of the activations of The results, shown in Fig. 9, show that Mistral 7B indeed can map intermediate quantities to their expected place in the circle: the very early and very late version of each weekday are more to- wards the last and the next weekday along the circle, respectively. We show similar results for in Appendix Fig. 22.
"[morning/evening] on Monnday/Tussdlay../SSuddd
6 DISCUSSION
Our work proposes a significant refinement to the simple one-dimensional linear representation hy- pothesis. While previous work has convincingly shown the existence of one-dimensional features, we find evidence for irreducible multi-dimensional representations, requiring us to generalize the notion of feature to higher dimensions. Fortunately, we find that existing unsupervised feature extraction methodologies like sparse autoencoders can readily be applied to discover collti-dimensionall repre- sentations. However, we think our work raises interesting questions about whether individual SAE features are appropriate "mediators" (Mueller et al., 2024) for understanding model computation, if some features are in fact multi-dimeesional. Although taking a multi-dimensional representation per- spective may be more complicated, we believe that uncovering the true (perhaps multi )alti-ddmennsonna) nature model representations necessary for discovering the underlying algorithms that use these representations. Ultimately, our field aims to turn complex circuits in future more- more-capable models into formally verifiable programs (Tegmark & Omohundro, 2023; Dalrymple et al., 2024), which requires the ground truth "variables" of language models; believe this work takes important step towards discovering these variables.
Limitations: It unclear why we did not find more interpretable multi-dimennional features. We are unsure if we are failing to interpret some of the high-scoring multi-dimensional features, if most multi-dimeesional features lie dimensions higher than two, if our clustering technique is not powerful enough to find some features. or if there are truly not that many. Additionally, our definitions for irreducible features (Definition 2) are purely statistical and not intervention based, and also had to be relaxed to hold in practice, resulting in measures that return possibly subjective "degree" of reducibility (Definition 3). Thus, although this work provides preliminary evidence for the multi-dimension superposition hypothesis (Hypothesis 2), it is still unclear if this theory provides the best description for the representations models use. Future work might make progress on this question by investigating new techniques for decomposing model representations, exploring higher dimensional representations, or determining conclusively whether models use representations in ways that necessitate the representations are non-linear.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank (in alphabetical order) Dowon Baek, Kaivu Hariharan, Vedang Lad, Ziming Liu, and Tony Wang for helpful discussions and suggestions. This work is supported by Erik Otto, Jaan Tallinn, the Rothberg Family Fund for Cognitive Science, the NSF Graduate Research Fellowship (Grant No. 21 2141064), and IAIFI through NSF grant PHY-2019786.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altaa, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
AI@ Meta. Llama model card, 2024. URL UBLLtttp::/githbb..omm blob/maing htt/mmainn/ee MMDDEL CARD mdd
-astttallllmmnn 1lama/ /1ama/
Noga Alon. Problems and results in extremal combinatorics i. Discrete Mathematics, 273(1-3): 31-53, 2003.
Anthropic. The claude 3 model family: Opus, sonnet, haiku. Technical report, Anthropic, 2024.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.000 12, 2023.
Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, et Interpreting neural networks through the polytope lens. arXiv preprint arXiv:2211.12311 2022.
Joseph Bloom. Open source sparse autoencoders for all residual stream layers of gpt2 small.
2024.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, 7ac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity; Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023, https//ttansfomme-ecrcciss http://mrrrrrriituuuuup///////om pub/2023/mooosemaaitc features/index.hime
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 12712, 2023.
Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318-16352 2023.
Róbert Csordás, Christopher Potts, Christopher Manning, and Atticus Geiger. Recurrent neural networks leam to store and generate sequences using non-linear representations. arXiv preprint arXiv:2408. /0920. 2024.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.00 08600, 2023.
David Dalrymple, Joar Skalse, Yoshua Bengio, Stuart Russell, Max Tegmark, Sanjit Seshia, Steve Omohundro. Christian Szegedy, Ben Goldhaber, Nora Ammann, et al. Towards guaranteed safe ai: A framework for ensuring robust and reliable ai systems. arXiv preprint arXiv:2405.0666 2024.
11

---

Published as a conference paper at ICLR 2025
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Batfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of super- position. Transformer Circuits Thread, 2022. https: http::////aaasffomerrrrrree rcu ts t.. t..btt.pub pub/
2022 http://tymmmdeellleooe / html.
Simon Foucart and Holger Rauhut. Mathematical Introduction to Compressive Sensing. 2013.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2/01.000
2020.
Semyon Aranovich Gershgorin. üher die abgrenzung der eigenwerte einer matrix. Izvestiya Rossiīskoi akademii nauk. Seriya matematicheskaya, (6):749-754, 1931.
Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, interpretable attention heads in the wild. arXiv preprint arXiv:23/2.09230, 2023.
Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint arXiv:2310.02207, 2023.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Inter- preting mathematical abilities a pre-trained language model. Advances in Neural Information Processing Systems, 36, 2024.
Benjamin Heinzerling and Kentaro Inui. Monotonic representation of numeric properties in language models. arXiv preprint arXhv:2403113881 2024.
Nicholas J. Higham. Singular value inequalities. https: petps://nhigham... com00m//0ccmmm/0000///////
senggularvaalsee sateeeinnnuaaiiiees/eeoe May 2021.
Bill Johnson hutp::/maahboerffovv.ooe/s///// johnson).
Almost
orthogo-
nal vectors. MathOverflow. URL https://net
hwl.:ttpp//maahveeeefoooeee//g// (version: 2010-05-16).
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0663 2023.
Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. arXiv preprint arXiv:2403.03867 2024.
William B. Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into hilbert space. In Conference in modern analysis and probability (New Haven, Conn., 1982), volume 26 of Contemporary Mathematics, pp. 189-206. American Mathematical Society, Providence, RI, 1984. ISBN 0-8218-50304 doi: 10. (0.11900/comm/006677740000
Subhash Kantamneni and Max Tegmark. Language models use trigonometry to do addition. arXiv preprint arXiv:2502.00873 2025.
Silvio Lattanzi, Thomas Lavastida, Kefu Lu, and Benjamin Moseley. A framework for parallelizing hierarchical clustering methods. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Pro- ceedings, Part I, pp. 73-89 Springer, 2020.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring sequence model trained synthetic task. arXiv preprint arXiv:2210.1336. 2022.
Ziming Liu, Ouail Kitouni, Niklas S Nolte. Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651-34663 2022.
12

---

Published as a conference paper at ICLR 2025
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023.
Samuel Marks, Can Rager, Eric Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403. 19647, 2024.
Jake Mendel. Sae feature geometry is outside the superposition hyperpositionnbyootssiiis.. Al Aligmment Forum, 2024. URL http:////www..................rr sae saceffaeeee.etttttrrorooraatttea
Eric J Michaud, Isaac Liao, Vedang Lad, Ziming Liu, Anish Mudide, Chloe Loughridge. Zifan Carl Guo, Tara Rezaei Kheirkhah, Mateja Vukelić, and Max Tegmark. Opening the black box: program synthesis via mechanistic interpretability. arXiv preprint arXiv:2402.051 10, 2024.
Tomas Mikolov, llya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean, Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013a.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp. 746-751, 201 3b.
Depen Morwani. Benjamin L Edelman, Costin-Andrei Onceseu, Rosie Zhao, and Sham Kakade. Feature emergence via margin maximization: case studies in algebraic tasks. arXiv preprint arXiv:2311.07566 2023.
Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, al. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. arXiv preprint arXiv:2408.01411 014161 2024.
Neel Nanda and Joseph Bloom.
Transformerlens. 2022.
https://github.com
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023a.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2339.00941, 2023bb
Chris Olah. What is a linear representation? what is multidimennionnallatuureer Transformer Circuits Thread, 2024. URL httpss://transformmet http:r/c.....tttt.ppp/////////// http://o22///uyy--uaaaaa.eee index. # wewiinnrrrrrreeeee represeattrss
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
Zoom in: An introduction circuits. Distill, 2020. doi: 00it1///liill.00022..000
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03633 2023.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint TrXiv:2304.03277: 2023.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 2019.
13

---

Published as a conference paper at ICLR 2025
Adam Shai, Paul Riechers, Lucas Teixeira, Alexander Oldenziel, and Sarah
Marzen. Transformers represent belief state geometry in their residual https:: //wp:///wwwww/// al www..genntnonnummmmmmmmmmmm ottppppsts/gg222xeeiccc3... transformers represent-ber sttttsttteeeeeeeeee-eee-e----eee -etat---eemerrr-----tteeee 2024.
stream.
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning language models using causal mediation analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7035-7052, 2023.
Aaquib Syed, Can Rager, and Arthur Conmy, Attribution patching outperforms automated circuit discovery. arXiv preprint urXiv:2310. 10348, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Max Tegmark and Steve Omohundro. Provably safe systems: the only path to controllable agi. arXiv preprint wrXiv22309.09933 2023.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022.
Adam Yedidia. URL
Gpt-2's positional embedding matrix a helix, 2023a.
https: / /www lesswrong. cwm/posts posts sentss/quue /gwW33aRDaagKPPNss gpt 2ww---sppostiiinn-eeeeeeeee -emenndinebdddiinggg mattriaaaiii--ii is-a-helix helix. Accessed: 2024-09- 16.
Adam Yedidia. The positional embedding matrix and previous-token heads: how do they actually work?, 2023b. URL. https:: www. lesswrong. com ccmm/ppsts//AAABBFTTT..TR. e6/ cn-positiooaal wwwiiinnnneeeddineeeeeeeeeee wwwedddineeeeeeeeeeeedeedee maagg-maarrii marrixxxiiiiiiiii--iii--a and cessed: 2024-09-17.
how. Ac-
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arXiv:2309. (tV:::39..6642,, 2023.
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks, Advances Neural Information Processing Systems, 36, 2024.
14

---

Published as a conference paper at ICLR 2025
A MULTI-DIMENSIONAL FEATURE CAPACITY
The Johnson-Lindenstrauss (JL) Lemma (Johnson Lindenstrauss, 1984) implies that we can choose eads pairwise one- mee-imensional S-orthogonal vectors to satisfy Hypothesis 1 for some constant C, thus allowing us build the model's feature space with a number of one-dimensional d-orthogonal features exponential in d. We now prove similar result for low-dimensional projections (the main idea of the proof to combine 5-orthogonal vectors as guaranteed from the JI. lemma):
Theorem 1. For any d' and 6, it is possible to choose 1
Cild/d2)6 pairwise 5-orthogonal dmax
matrices A; E Rrxxdd for some constant C1. Furthermore, it is not possible to choose more than da(d-dmaro log( for some constant Cz.
We will first prove lemma that will help us prove Theorem 1.
Lemma 1. Pick pairwise 6-orthogonal unit vectors in V1,...; d Lety Rdd be unit norm vector that is a linear combination of unil norm vectors Vi,..., Va with coefficients 21 ..... 2n E IR. We can write A [Viy..., Vn. and [21,...y Zn so that have y Lk=1 2bVK = Az with lyll = 1. Then,
Σ
Zk =
2
n
1 - ôn
Proof. We will first bound the L2 norm of a is the minimum singular value of A, then we have via standard singular value inequalities (Higham, 2021)
|yll2 2
|2|2 S |
On
Thus now lower bound nn- The singular values are the square roots of the eigenvalues of the matrix A A, so we now examine A A. Since all elements A are unit vectors, the diagonal of A A is all ones. The off diagonal elements dot products of pairs of 3-orthogonal vectors, and so are within the range [-6, 8]. Then by the Gershgorin circle theorem (Gershgorin, 1931), all eigenvalues X of A A are in the range
(1 8(n 1), 1 5(n 1))
In particular, of A., 1 8(n 1), and thus on 2 v1 5(n 1). Plugging into upper bound for ||z||2, we have that | S 1/v1 8(n - 1). Finally, the largest In for a point on an n-hypersphere of radius is when all dimensions are equal and such point has magnitude /nir, so
||z|| V 8(n V ön
2rddd/d22 pairwise 5-orthogonal Theorem 1. For any d' and 5, it is possible to choose dmass matrices A, E R.n.dd for some constant C₁. Furthermore, it is not possible to choose more than dmssó d ()) for some constant Cz.
Proof. By the JL lemma (Johnson Lindenstrauss, 1984;
johnson), for any d and 6, we can choose CCd22 d-orthogonal unit vectors in Rd indexed as v., for some constant C. Let A; [Vdmax +i where each element in the brackets is column. Then by construction all A, are matrices composed of unique 5-orthogonal vectors and there are Jana Le Cd5² matrices A;.
Now, consider two of these matrices A, [V1...., vn, and A, [nj Un it will prove that they are oot)oortoaa for some function f. Let = Σn k= Ri,&VK be vector in the colspace of A; and y, Sk=1 Rg be vector in the colspace of A,, such that y, and y, unit vectors. To prove f(6) orthooontnyn we must bound the absolute dot product between y; and y
15

---

Published as a conference paper at ICLR 2025
y 8yl
Vi.VVV
Σ u
Rj
(2 Vara)
(Zyukc usa)
===1
Vis 182
Triangle Inequality
ky= k2=1
n2
|2 2jA
All va, My are orthogonal
=1==
Rig 22,02
1 1==1
6
Σ
Σ
2/
Factoring the product
km
74
n2
By Lemma 1
Eng,
8mg
5dmax
niy n2 S dmax by assumption
5dmax
Thus A; and A, are f(o)- for f(6) = 8dmax/ (1 - 5dmaax,, and so it is possible to choose e Cd6² pairwise f(6) projection matrices. Remapping the variable å with dmace
5 8 (8) 5/ (d + 8)), find that it is possible choose dmax cCa62 ((L+))((
pairwise 6-orthogonal projection matrices. Because + 8 is at most 2 with 8 (0, 1), we can further simplify the exponent and find that it is possible to choose duyaa ((d)) pairwise 5-orthogonal projection matrices. Absorbing the 4 into the constant C finishes the proof the lower bound.
For the upper bound, we can proceed much more simply. Consider l pairwise 5-orthogonal matrices A; E IRd'. Since these matrices are full rank, their column spaces each parameterize a subspace of dimension d', and so by result from (Alon, 2003) it is possible to choose CCd' 82 log(t) almost orthogonal vectors in this subspace. Furthermore, by our definition of 5-orthogonal matrices, all pairs of these vectors between subspaces will be d-orthogonal. Finally, again by (Alon, 2003) we cannot have more than e Cals log(t) 6-orthogonal vectors overall, so have that
ke Cdmar log(?)
Cdd2 log(!)
and simplfying.
k < e Old-dmax)) (0g())))))))))(()
These results imply that models can still represent exponential number of higher dimensional features. However, there is large exponential gap between the lower and upper bound we have shown. If the lower bound is reasonably tight, then this would mean that models would be highly incentivized to fit features within the smallest dimensional space possible, suggesting a reason for recent work showing interesting compressed encodings of multi-dimeesional features in toy problems (Morwani et al., 2023).
Note that the proof assumes the "worst case" scenario that all of the features are dimension dmax- while in practice many of the features may be 1 or low dimensional, so the effect on the capacity of a real model that represents multi-dimensional features is unlikely be this extreme.
16

---

Published as a conference paper at ICLR 2025
Finally, note that the dictionary learning literature may have discovered similar results in the past (which we were unaware of), see Foucart Rauhut (2013).
B
MORE ON REDUCIBILITY
B. ADDITIONAL INTUITION FOR DEFINITIONS
Here, we present some extra intuition and high level ideas for understanding our definitions and the motivation behind them. Roughly. we intend for our definitions the main text to identify representations in the model that describe an object or concept in a way that fundamentally takes multiple dimensions. We operationalize this finding a subspace of representations that has basis vectors that "always co-occur" no matter the orientation 2. is not made up of combinations of independent demes-dimeniional features.
1. The first condition is met by the mixture part of our definition. The feature in question should be part of an irreducible manifold, and so should "fill" a plane or hyperplane. There shouldn't be any part of the plane where the probability distribution of the feature is concentrated, because this region is then likely part of a lower dimensional feature. The idea of this part the definition is to capture multi-dimensional objects; if the entire multi-dimensional space is truly being used to represent a high-dimensional object, then the representations for the object should be "spread out" entirely through the space.
The second condition is met by the separability part of our definition. This part of the definition is intended rule out features that co-occur frequently but are fundamentally not describing the same object or concept. For example, latitude and longitude are not a mixture in that they frequently co- occur, but do not think is necessarily correct to say they are part of the same multi-dimensional feature because they are independent.
B.2 EMPIRICAL IRREDUCIBLE FEATURE TEST DETAILS
Our tests for reducibility require the computation of two quantities S(f) for the separability index and M. (f) for the e-mixture index. We describe how we compute each index in the following two subsections.
B.2.1 SEPARABILITY INDEX
We define the separability index Equation 2 as
S(f) = min (a;
where the min is over rotations used to split = Rf + into a and b. In two dimensions, the rotation is defined by single angle, so we can iterate over grid of 1000 angles and estimate the mutual information between a and b for each angle. We first normalize f by subtracting off the mean and then dividing by the root mean squared norm of f (and multiplying by v2 since the toy datasets in two dimensions). To estimate the mutual information, we first clip the data f to 6 by square centered on the origin. We then bin the points into a 40 by 40 grid, to produce discrete distribution p(a, b). After computing the marginals p(a) and p(b) by summing the distribution over each axis, obtain the mutual information via the formula
Σ pla, b) log p(a, b) a.b
[(a: b)
(8)
p(a)p(b)
B.2.2 e-MIXTURE INDEX
We define the e-mixture index in Equation 3 as
Mc(f)= = max P ((v. < <eel/ee//Vvlll f c)]
vER*1, CER
17

---

Published as a conference paper at ICLR 2025
representation dim
Mutual info (bits)
2000
count
1000
M.(f) = 0.6396
S(N)
0.3691
2m
representation dim
normalized v f
angle
Testing S(a) and M. (a) reducible feature
representation dim
Mutual info (bits)
[(()) 2.683
200
count
M.(f) 0.1784
100
1.5
-1.0
0.5 c
0.0
2m
representation dim 1
normalized
angle
(b) Testing S(b) and (b) irreducible feature b
Testing irreducibility of synthetic features. Left in each subfigure: Distributions of x. For feature a, 63.96% lies within the narrow dotted lines, indicating the feature is likely mixture. For feature b, 17.84% lies within the wide lines, indicating the feature is unlikely to be a mixture. The green cross indicates the angle that minimizes mutual information. Middle in each subfigure: Histograms of the distribution of v x with red lines indicating a 2e-wide region. Right each subfigure: Mutual information between a and b function of the rotation angle 0 of matrix R. Feature b has a large minimum mutual information so is unlikely to be separable; feature a has medium value of minimum mutual information of about 0.37 bits.
Figure 10:
The challenge with computing M, is to compute the maximum. We opted to maximize via gradient descent; and we guaranteed differentiability by softening the inequality < with sigmoid,
v v VE f+
M,r(f.v,c) =E
T
where T' temperature, which we linearly decay from
throughout training. We optimize for and using this loss Mf.(f, v, c) using full batch gradient descent over 10000 steps with learning rate 0.1. With the solution (v", c"), the final value of (o,r=o(f, v*, ..") )") then our estimate of (f).
We also run the irreducibility tests on additional synthetic feature distributions in Fig. 1la and Fig. 11b.
C ALTERNATIVE DEFINITIONS
In this section, we present an alternative definition of reducible feature that we considered during our work. This chiefly deals with mulli-diimensionll features from the angle of computational reducibility as opposed to statistical reducibility. In other words, this definition considers whether representations of features on a specific set of tasks can be split up without changing the accuracy of the task. This captures interesting (and important) aspect of feature reducibility, but because it requires specific set of prompts (as opposed allowing unsupervised discovery) we chose not to use as our main definition.
Our alternative definitions consider representation spaces that are possibly multi-diimessionll and defines these spaces through whether they can completely explain function h the output logits. We consider group theoretic approach to irreducible representations, via whether computation involving multiple group elements can be decomposed.
18

---

Published as a conference paper at ICLR 2025
representation dim 2
Mutual info (bits)
100
0.50
count
M.(f)
0.0
794
50
0.25
((r)
0.1156
0.00
2
2m
representation dim
normalized v
angle
(a) Testing S(c) and M. (c) reducible feature
representation dim
1000
Mutual info (bits)
1.0
M,(C)
0.259
count
500
0.5
((())
0.2541
0.0
0.0
2.5
2m
representation dim 1
normalized
angle
(b) Testing S(d) and (d) irreducible feature d
Figure 11: Testing irreducibility of synthetic features. Left in each subfigure: Distributions of x. For feature c, 7.94% lies within the narrow dotted lines, indicating the feature is unlikely to be mixture. For feature d, 25. 90% lies within the wide lines, indicating the feature is likely mixture. The green cross indicates the angle that minimizes mutual information. Middle in each subfigure: Histograms of the distribution subfigure: Mutual information between a and b function of the rotation angle 0 of matrix R. Both features have a small (< 0.5 bits) minimum mutual information and so are likely separable.
v x with red lines indicating a 2e-wide region. Right each
C.I ALTERNATIVE DEFINITION: INTERVENTIONS AND REPRESENTATION SPACES
Assume that restrict the input of prompts T (t/) to some subset of prompts and that have some evaluation function h that maps from the output logit distribution of M to real number. For example, for the Weekdays problems, 7' is the set of 49 prompts and h could be the arg max over the days of week logits. Abusing notation, we let M also be the function from the layer we are intervening on; this is always clear from context. Then we can define a representation space of x²,, as a subspace in which interventions always work:
Definition 5 (Representation Space). Given prompt T (t), rank-r dimensional repre- sentation space of intermediate value x(, is rank projection matrix P such that for all j.j'. h(M((1 - P)x, (00(11-))x_,p_ Px()) = h(M (x.)).
Note that it immediately follows that the rank d dimensional matrix Ta is trivially rank d representa- tion space for all prompt sets T.
Definition 6 (Minimality). A representation space P of rank is minimal if there does not exist lower rank representation space.
A minimal representation with rank > is multi-dimenniionn representation. Definition (Alternative Reducibility). A representation space of rank r is reducible there are orthonormal representation spaces P1 and Pz (such that P1 + P2 = P, P.P, 0) where
h ((M(())))) + M ((Px) (M x] Pax(,))
for all j.j'
Suppose T, and define the multiplication of two elements finite group of order n. Then we interpret the embedding vectors the group representations, definition of reducibility implies to the standard group- -hooretheoettcaal definition irreducibility - specifically, reducibility into a tensor product representation.
19

---

Published as a conference paper at ICLR 2025
D TOY CASE OF TRAINING SAES ON CIRCLES
To explore how SAEs behave when reconstructing irreducible features of dimension dj > 1, we perform experiments with the following toy setup. Inspired by the circular representations of integers that networks learn when trained on modular addition (Nanda et al., 2023a; Liu et al., 2022), we create synthetic datasets of activations containing multiple features which each 2d irreducible circles.
First however, consider activations for single circle - points uniformly distributed on the unit circle in IR?. We train SAEs on this data with encoder Enc(x) = (e.u(W.(X ba) + be) and decoder with the Adam optimizer and
Dec(f) Waf bd. We train SAEs with m and m learning rate of 10-3, sparsity penalty À = 0.1, for 20,000 steps, and a warmup of 1000 steps. In Fig. 12 we show the dictionary elements these SAEs. When m 2, the SAE must use both SAE features each input point, and uses dn to shift the reconstructed circle so it is centered the origin. When m = 10, the SAE learns dp ~ 0 and the features spread out across the circle, arranged close together, and only subset are active each input.
sparsity loss: 1.93
sparsity loss: 1.27, seed=2
sparsity loss: 1.28, seed=4
Recerobrution
Distienary
-1
Roconstructon
1
Reconstruction
-2
Dictionery alament
Diclionary elament
-2
11
1
Figure 12: SAEs trained to reconstruct single 2d circle with m 2 (left) and m = 10 (middle and right) dictionary elements. When there several SAE features, there no natural/canomical choice of feature directions, and the dictionary elements spread out across the circle.
We now consider synthetic activations with multiple circular features. Our data consists of points in R.00 where choose two orthogonal planes spanned by (e1. (c).ez) and (es, ea), respectively. With probability one half a points is sampled uniformly on the unit circle in the e -e plane, otherwise the point will be sampled uniformly the unit circle in the e3-e4 plane. We train SAEs with 64 on this data with the same hyperparameters as the single-circle case.
We now apply the procedure described in Section 4 circles. Encouragingly, we first find that the alive SAE features align almost exactly with either 2 to the
if we can automatically rediscover these
the e1-02 the C3-Ca plane. When we apply spectral clustering with n r_clusstrs features with the pairwise angular similarities between dictionary elements as the similarity matrix (Fig. 13, left), the clusters correspond exactly the features which span each plane. As described in Section 4, given cluster of dictionary elements S C (1,...,,) through the SAF, then filter out samples which don't activate any element in For samples which do activate element of S, reconstruct the activation while setting all SAE features not in S to have a hidden activation of zero. If some collection of SAE features together represent some irreducible feature, want to remove all other features from the activation vector, and we only allow SAE features in the collection to participate in reconstructing the input activation. We find that this procedure almost exactly recovers the original two circles, which encouraged to apply this method for discovering the features shown in Fig. and Fig. 15.
m). ), we run large set of activations
E TRAINING MISTRAL SAES
Our Mistral 7B (Jiang et al., 2023) sparse autoencoders (SAEs) are trained on over one billion tokens from subset of the Pile (Gao et al., 2020) and Alpaca (Peng al., 2023) datasets. We train SAFs
20

---

Published as a conference paper at ICLR 2025
Cosine sim between alive teatures
Cluster 1 reconstruction
Cluster 2 reconstruction
PCA axis
PCA axis
15
SAE
0
0
30
-1
-1
15
30
-1
-1
alive SAE feature
PCA 1
PCA
Figure 13: Automatic discovery of synthetic circular features by clustering SAE dictionary elements.
on layers 8, 16, and 24 out 32 total layers to maximize coverage of the model's representations.
We use 16x expansion factor, yielding a total of 65536 dictionary elements for each SAE.
To train our SAEs, we use an Lp sparsity penalty for p = 1/2 with sparsity coefficient = 0.012. Before an SAE forward pass, we normalize our activation vectors to have norm v /dmodel = 64 in the case of Mistral. We do apply a pre-encoder bias. We use Adam optimizer with weight decay 10-3 and learning rate 0.0002 with a linear warm up. We apply dead feature resampling (Bricken al., 2023) five times over the course of training converge on SAEs with around 1000 dead features.
F GPT-2 AND MISTRAL 7B DICTIONARY ELEMENT CLUSTERING
In this section, we first present pseudocode in Alg. for the overall high level technique that finds multi-dimensional features and that uses clustering as subroutine. We then provide the specific clustering algorithm implementations we use for GPT-2 and Mistral.
Algorithm 1: High Level Clustering Approach For Finding Multi-D Features
Input: Dictionary elements D, activation vectors X SAE
Output: Irreducible multi-dimensional features
Sig t Cwsinesiniii. D;, D,);
clusters + Cluster (S);
reconstructions ():
for cluster in clusters do
Reluster ids of dictionary elements in cluster; for Iıi in X;, do
encoding +- ReLU(E. æT,1);
if max max (encoddidng Retuster)) 0 then
r + D[:, Roluster] encoding; reconstructions recomstructions U {r):
end
end
end
features 0):
for R in reconstructions do
proj + PCA (R);
if TeestIrreuuciae
(proj) then
Add proj to features;
end end
return features
21

---

Published as a conference paper at ICLR 2025
F.1 GPT-2-SMALL METHODS AND RESULTS
For GPT-2-small, we perform spectral clustering on the roughly 25k layer 7 SAE features from (Bloom, 2024), using pairwise angular similarities between dictionary elements as the similarity matrix. We use n_cl p_cluste L.llssters = 1000 and manually looked at roughly 500 of these clusters. For each cluster, we looked at projections onto principal components 1-4 of the reconstructed activations for these clusters. In Fig. 14, we show projections for the most interesting clusters identified, which appear to be circular representations days the week, months of the year, and years of the 20th century.
Days the Week
Honday lussayy day
Sacurday CUher
PCA axis
PCA axis
PCA axis
PCA axis
PCA axis
PCA axis
Months of the Year
Febeuary
November December
Ray by August
PCA axis 2
PCA axis
PCA axis
PCA axis
PCA axis
PCA axis
Years of the 20th Century
1900
1950
1999
PCA axis 2
PCA axis 3
PCA axis 4
PCA axis
PCA axis 2
PCA axis
Figure 14: Projections of days of week, months of year, and years of the 20th century representations onto top four principal components, showing additional dimensions of the representations than Fig. 1.
F.2 MISTRAL 7B METHODS AND RESULTS
For Mistral 7B, our SAES have 65536 dictionary elements and we found it difficult to run spectral clustering all of these once. We therefore develop simple graph based clustering algorithm that we run on Mistral 7B SAEs:
1. Create graph out of the dictionary elements by adding directed edges from each dictionary element to its & closest dictionary elements by cosine similarity. We use A 2.
22

---

Published as a conference paper at ICLR 2025
2. Make the graph undirected by turning every directed edge into undirected edge.
3. Prune edges with cosine similarity less than threshold value We 0.5.
4. Return the connected components clusters.
We run this algorithm on the Mistral 7B layer 8 SAE (2 dictionary elements) and find roughly 2700 clusters containing between and 1000 elements. We manually inspected roughly 2000 of these. From these, re-discover circular representations of days of the week and months of the year, shown in Fig. 15. However, we did not find other obviously interesting and clearly irreducible features.
We also investigate the sensitivity of this method to T and k by varying 7 and k and showing the max Jaccard similarity between any of the resulting clusters and the days of the week cluster we show in Fig. 15. We show the results in Fig. 16, where we find that varying k has minimal effect, while varying 7 shows regimes: small causes all features to group in one cluster, the days of the week cluster is not found; medium causes the days of the week cluster to become identifiable; large causes all features be divided into their own clusters.
Days of the Week (Mistral 7B)
Months of the Year (Mistral 7B)
Monday
friday Saturday
January
Winters
Tuesday
February
Spring Sumtier
Wedr esday
Suntley
March
Thursday
weekend
May
July
August
September
PCA axis 3
PCA axis 3
Octobor
Novemberr
PCA axis 2
PCA axis 2
Figure 15: Circular representations of days of the week and months of the year which discover with our unsupervised SAE clustering method in Mistral 7B. Unlike similar features in GPT-2, we also find an additional "weekend" representation in between Saturday and Sunday representations (left) and additional representations of seasons among the months (right). For instance, "winter" tokens activate a region of the circle in between the representation of January and December.
Cluster Stability Acrass Parameters
Mean Cluster Size Across Parameters
0.2
Cluster part of large super cluster
10f
Mcan Cluterr Size [og ocalo
0.28
0.26
0.30
030
043
Cluster formed
035
10:
0.2
Cluster broken apart
0.59
1.00
U=2
kmd
bed
Top-k for Graph (k=2.3.4)
Top-k for Graph (k-2.3,4)
Figure 16: Hyperparameter regimes where the days of the week cluster exists. The cluster exists in the regime between all features clumping together and all features being in their own cluster; this regime seems reasonably stable.
23

---

Published as a conference paper at ICLR 2025
As future work, we think it would be exciting to develop better clustering techniques for SAE features. Our graph based clustering technique could likely be improved by more recent efficient and high-quality graph based clustering techniques, e.g. hierarchical agglomerate clustering with single-linkage (I (Lattanzi et al., 2020). Additionally, we believe we would see a large improvement by setting edge weights to be combination of both the cosine and Jaccard similarity of the dictionary elements, c.g. max(cosine, Jaccard).
G OTHER DISCOVERED CLUSTERS
In Fig. 17, we plot the top 11 ranked clusters by the product of a) the measured separability index and b) one minus the measured e-mixture index with 0.1 (this just one of many possible ways to get an ordered ranking from a two awo-parameter score). We color by both the current token (which results in clear patterns for all tokens) and the next token (to see if find belief states as found by Shai et al. (2024) in toy transformers). We note that weekdays are ranked 9 and so are shown in the plot. Additionally, the next token patterns of the 'such' cluster and the 'B' cluster do seem to display some clustering independently of the the current token pattern, which might lend the belief state hypothesis some support.
H FURTHER EXPERIMENT DETAILS
H.1 ASSETS INFORMATION
We use the following open source models for experiments: Llama 3 8B (AI@ Meta, 2024) (custom Llama license https: ////m/1laaaaa .etta. com/ commllama3 1ama3/1icense /). Mistral 7B (Jiang et al., 2023) (released under the Apache License), and GPT-2 (Radford al., 2019) (modified MIT license, see https: / /tle://gitthee /github..... com/ caena/opeaa/////eem /gptt/pttt//tt
H.2 MACHINE INFORMATION
Intervention experiments were run on two V100 GPUs using less than 64 GB CPU RAM; exper- iments can be reproduced from open source repository in less than day with this configuration. We use the TransformerLens library (Nanda & Bloom, 2022) for intervention experiments. e-mixture index measurements toy datasets took about minute each, on 8GB of CPU RAM. EVR experiments take seconds on 8GB CPU RAM and are dominated by time taken to human-interprr the RGB plots.
GPT-2 SAE clustering and plotting was run on a cluster of heterogeneous hardware. Spectral done CPUs only. We made reconstruction plots for 500 clusters, with each taking less than 10 minutes. Mistral 7B SAE reconstruction plots were made the same cluster. We made roughly 2000 reconstruction plots for Mistral 7B (and manually inspected each), with each taking less than 20 minutes to generate. Jobs were allocated 64GB of memory each.
clustering and computing reconstructions + plotting
Mistral SAE training was run on a single V100 GPU. Initially caching activations from Mistral 7B on one billion tokens took approximately 60 hours. Training the SAEs on the saved activations took another 36 hours.
H.3 ERROR BAR CALCULATION
In Fig. 6 we report 96% error bars for all intervention methods. To compute these error bars, we loop over all intervention methods and all layers and compute confidence interval for each (method, layer) pair across all prompts. Assuming normally distributed errors, we compute error bars with the following standard formula:
= =
where μ is the sample mean, 2 is the Z score (slightly larger than 2 for 96% error bars), and SE is the standard error (the standard deviation divided by the square root of the number of samples). We use standard Python functions to compute this value.
24

---

Published as a conference paper at ICLR 2025
Cluster 23 IRant Rough Desensbon: Latter It!
Clader 25 [Rann 2), to.ugh Descrppton: Letter 'N!
hed boinn top caland
Medt solen top bokaro colored
Cluster 30 (tark, 30, Rough Descnptian: Sutstring lee Caront sotercatinnal
Cluster iflnt 4), Rough Descnption Letter
Corent piinn titoa
top cales
Solan balane colond
Rough
Rough Descritionn
Connetsur tog coerectieeet
Cureet 28mn 32g zaam calred)
Cluster (Rank
Description
Lumen sken Iog ralord))
ND 20a
Cluster 96 (Rank LOV Rough Description: Tiken Suct!
Claster 65 HRakk 111. Rough Description: Lotter IV
next Eiten torecceeii
Mec Vale thi bilee rolend
Figure 17: Top 10 GPT-2 clusters by Mixture and Separability Index.
25

---

Published as a conference paper at ICLR 2025
Layer
Layer
Layer
Layer3
Layer
Layer5
Layer
Layer?
Layero
Layer'
Layer2
Layer
Layer
Layer
Layerd
Layer?
+
Layer
Layer
Layer 20 Layer 11 Layer 12 Layer 13 Layer 1e Layer 15
Layer 8
Layer
Laper 10 Layer
Layer 12 Layer 13 Layer 14 Layer 15
+
$35
Layer 16
Layer 17 Layer 18 Layer 19 Layer Layer Layer 22 Layer
Layer 16 Layer17
Layer 18
Layer 19 Layer 20 Layer 21 Layer 22 Layer
Layer 24 Layer
Layer
Layer 27 Layer
Layer 29 Layer Layer an
Layer 24 Layer
Layer
Layer
Layer 20 Layer Layer Layer
(a) Mistral 7B, Weekdays
(b) Llama 3 8B, Weekdays
Layer
LayerG
Layer?
Layero
Layer
Layer
Layer
Layer'
I' *
Layer
Layer
Layer11
13 Layer Lsyer1s
Layer
Layer
Layer
Layer
16 Layer
Loyer Enyer
Layer Layer
Layer
Layer
Layer
20
Layer
1144
Layer 24 Layar 25 Layer 26 Layer 27 Layer
29 Layer Layer 21
Layar 24 Layer
Layer 26 Layer
Layer 28 Layer
Layer
Layer 31
AV..
Golder
(c) Mistral 7B, Months
(d) Llama 3 8B, Months
Figure 18: Projections onto the top two PCA dimensions of model hidden states on the token show that circular representations of a are present in various layers.
The reason that the Months error bars are smaller than the Weekdays error bars is because there are more Months prompts: there are 12 12 * 11 1584 intervention effect values, rather than 7 7 6 294 intervention effect values.
1 MORE WEEKDAYS AND MONTES PLOTS AND DETAILS
1.0.1 BASIC PLOTS
Honcay Sursday
Wodnoaday Tharsday
May Satrday
Sunday
We show the results of Mistral 7B and Llama 8B on all individual instances of Weekdays that at least of the models get wrong in Table and present a similar table for Months in Table 3.
0.4
We show projections onto the top two PCA directions for both Mistral 7B and Llama 3 8B in Fig. 18 on the hidden layers on top of the a token, colored by a. These are similar plots to Fig. 4, except they are on all layers. The circular structure in is visible on many but not all layers. Much of the linear structure visible is due to B.
0.0
-0.4
-2.4
-0.2
0.0
0.2
5A
Projed on onto SQURI SEC PCA cotpproonn
Figure 19: Projections of Mistral 7B Weekdays task activations layer into the plane discovered by clustering layer SAE features.
1.0.2 INTERVENING WITH THE SAB PROBE
We show the results of projecting Mistral Weekdays representations into the plane discovered by clustering SAE features in Fig. 19. The result is clearly circular.
26

---

Published as a conference paper at ICLR 2025
Monddy
*Monday
0.4
is
is
0.3
0.2
two
two
0.2
days
0.1
days
0.1
from
from
0.0
10
20
30
10
20
30
Layer
Layer
(a) Mistral 7B MLP Patching
(b) Mistral 7B attention patching
*Monday
*Monday
0.4
is
is
0.2
two
two
0.2
days
days
from
from
0.0
0.0
10
20
30
10
20
30
Layer
Layer
(c) Llama MLP patching
(d) Llama attention patching
Figure 20: Attention and MLP patching results Weekdays. Results averaged over 20 different runs with fixed a and varying / and 20 different runs with fixed and varying a.
Two
0.3
*ww
0.4
months
months
0.3
0.2
from
from
0.2
*January
0.1
January
0.1
is
0.0
0
10
20
30
10
20
30
Layer
Layer
(a) Mistral 7B MLP Patching
(b) Mistral 7B attention patching
*wo
0.4
*Tw
0.3
months
0.3
months
0.2
from
from
0.2
January
JJanuary
0.1
0.1
0.0
10
20
30
0
10
20
30
Layer
Layer
(c) Llama 8B MLP patching
(d) Llama 3 8B attention patching
Attention and MLP patching results on Months. Results are averaged over 20 different runs with fixed and varying / and 20 different runs with fixed 3 and varying a.
Figure 21:
I.0.3 BASIC PATCHING
In Fig. 20 and Fig. 21. report MLP and attention head patching results for Weekdays and Months. We experiment on 20 pairs of problems with the same a and different 3 and 20 pairs of problems with the same 3 and different a, for total of 40 pairs of problems. For each pair
27

---

Published as a conference paper at ICLR 2025
Table 2: Weekdays finegrained results. Row ommited if both models get it correct.
Ground truth y
Mistral top y
Mistral correct?
Llama top 7
Llama correct?
Wednesday
Wednesday
Yes
Thursday
No
Friday
Friday
Yes
Tuesday
No
Saturday
Saturday
Yes
Thursday
No
Saturday
Saturday
Yes
Tuesday
No
4
Sunday
Sunday
Yes
Wednesday
No
Monday
Monday
Yes
Tuesday
No
2
Saturday
Friday
No
Saturday
Yes
Sunday
Sunday
Yes
Tuesday
No
4
Monday
Monday
Yes
Tuesday
No
Friday
Thursday
No
Friday
Yes
3
Monday
Monday
Yes
Tuesday
No
Saturday
Friday
No
Saturday
Yes
Sunday
Saturday
No
Wednesday
No
Monday
Sunday
No
Monday
Yes
4
Wednesday
Tuesday
No
Tuesday
No
Friday
Thursday
No
Thursday
No
Monday
Sunday
No
Thursday
No
Tuesday
Monday
No
Tuesday
Yes
Wednesday
Tuesday
No
Tuesday
No
4
Thursday
Thursday
Yes
Tuesday
No
Friday
Friday
Yes
Thursday
No
6
Saturday
Thursday
No
Thursday
No
Monday
Sunday
No
Tuesday
No
7
Tuesday
Sunday
No
Tuesday
Yes
Wednesday
Sunday
No
Wednesday
Yes
7
Thursday
Sunday
No
Thursday
Yes
Friday
Thursday
No
Tuesday
No
5
7
Saturday
Friday
No
Saturday Yes
Sunday
Friday
No
Thursday
No
of problems, patch the MLP/attention outputs from the "clean" the "dirty" problem for each layer and token, and then complete the forward pass. Defining the logit difference as the logit of the clean y minus the logit of the dirty 7, we record what percent of the difference between the original logit difference of the dirty problem and the logit difference of the clean problem is recovered upon intervening, and average across these 40 percentages for each layer and token. This gives us a score we call the Average Intervention Effect.
For simplicity of presentation, we clip all of the (few) negative intervention averages to 0 (prior work (Zhang & Nanda, 2023) has also found negative-effect attention heads during patching experi- ments).
1.0.4 CIRCLE CONTINUITY
Finally, in Fig. 22, show another example of the continuity of the circular days of the week representation in Mistral 7B.
J PATCHING
In this section, we present results to support a claim that MLPs (and not attention blocks) are responsible for computing 7. In Fig. 25, we deconstruct states on top of the final token (before predicting 7) Llama 3 8B Months (we show a similar plot for the states on the final token of Mistral 7B on Weekdays in the main text in Fig. 24. These plots show that the value of y is computed on the final token around layers 20 to 25. To show that this computation of occurs in the MLPs, we must show that no attention head is copying 7 from prior token or directly computing Y
28

---

Published as a conference paper at ICLR 2025
Table 3: Months finegrained results. Row ommited if both models get correct.
Ground truth y
Mistral top y
Mistral correct?
Llama top y
Llama correct?
May
April
No
May
Yes
6
4
November
October
No
November
Yes
July
June
No
July
Yes
0
7
August
July
No
August
Yes
September
October
No
September
Yes
November
October
No
November
Yes
7
January
December
No
January
Yes
February
January
No
February
Yes
7
7
March
February
No
March
Yes
May
April
No
May
Yes
4
9
February
February
Yes
January
No
10
January
December
No
January
Yes
8
10
July
June
No
July
Yes
11
January
December
No
January
Yes
2
11
February
December
No
February
Yes
11
March
February
No
March
Yes
11
July
June
No
July
Yes
11
August
July
No
August
Yes
11
September
August
No
September
Yes
0
12
January
December
No
January
Yes
We first perform patching experiment with the same setup Fig. 21 and Fig. 20 on individual attention heads on the final token. From the patching results we identify the top 10 attention heads by average intervention effect, For each attention head, we compute explanatory functions equal to one-hot functions of a and (resulting 14 functions g; for Weekdays and 24 for Months) and one with explanatory functions equal to one- hot functions of a, B, and 7 We find that for layers before 25, adding to the explanatory functions adds almost no explanatory power. Since we established above that the model has already computed y at this point, we know that attention heads do not participate in computing y.
Thu Earry
EVR run with
Wed
Thetw
La
Tue
Figure 22: Layer 30 Mistral 7B ac-
tivations for [very early/very late] MMonday/Tessday...SSummt projected into the PCA plane for [Mon- TayyTTesdayy..//Sndaay
plotted
To isolate the rough circuit for Weekdays and Months, perform layer-wise activation patching 40 random pairs of prompts. The results, displayed in Fig. show that the cireuit compute y consists of MLPs top of the and 3 tokens. copy to the token before 7, and further MIPs there (roughly similar to prior work studying arithmetic circuits (Stolfo et al., 2023)). Moreover, fine-grained patching in Appendix K shows that there just few responsible attention heads for the writes to the token before y. However, patching alone cannot tell use how or where 7 represented. For that, we need a new technique, which expand in the next section.
K EXPLANATION VIA REGRESSION (EVR)
So far, we have focused on examining and intervening on the representation for a, which we present cirele in the top PCA components top of the token. In this section, examine how the generated output, 7, is represented.
First, isolate the rough circuit for Weekdays and Months, perform layer-wise activation patching on 40 random pairs prompts. The results, displayed Fig. 21 and Fig. 20, show that the circuit compute consists of MI.ss top of the and tokens, copy to the token before
29

---

Published as a conference paper at ICLR 2025
layer 17
layer 18
layer 19
layer 20
layer
layer 22
layer 23
layer 24
layer 25
layer 20
layer 27
layer 28
layer 29
.2. 0.0%
0.015
22 0.0%
0.00
(2. 0.00
< 0.0%
0.0%
2 0.00
0.0%
(2. 0.0%%
. 0.0%
12 b.0%s
2 0.0%
12: 94.2%
22: 05.05
12 96.3%
22: 94.5%
22 91.8
12 90.0%
22: 37.3%
45.4%
22 41.0%
/2: 75.6%
12 14.5%
12: 72.5%
10.4%
Onde
Ande 14
98.0%
9%.0%
09.25
12. 96.0%
98.2%
12: 98.1%
07
km
Figure 24: EVR residual RGB plots on Mistral hidden states on the Weekdays final token, layers 17 to 29. From top to bottom, we show each residual RGB plot after adding the function(s) g; labelled just undemeath,, We also write "circle for x" meaning the inclusion of two functions gx(r) {cos, sin) (2Tx/7):
well as the resulting 7 value. We write "tmr" meaning "tomorrow" for 8 1.
7, and further MLPs there (roughly similar to what Stolfo et al. (2023) find in prior work studying arithmetic circuits). Thus, know where look for a representation of y: in the second half of the layers on the token before y. However, patching alone cannot tell use how is represented.
Unlike a, y has no obvious cireular (or linear) pattern in the top PCA components on these layers. To determine the representation for y. we introduce a more powerful technique we call Explanation via Regression (EVR): given set of token sequences with a corre- sponding set of hidden states Xi,,, we choose a set of interpretable explanation functions of the input tokens (t)). The r² value of a linear regression from {g, (t)} to X,,, tells us how much of the variance the activations the (1)} explain, and conversely the residuals show the exact components of the representation we have yct to explain.
Tue
Mon
Thu
2F
Sun
Sat
Figure 23: Top two PCA com-
ponents of residual errors after EVR with one-hot in a and B.. Mistral 7B Weekdays, layer 25, final token. Colored by 7,
K. USING
EVR TO UNCOVER A CIRCULAR REPRESENTATION FOR y
We first use EVR to determine the representation for y by plotting the top two PCA components of the layer 25 Mistral 7B activations after subtracting the components that can explained using a regression
with one hot functions in and B (i.e. gi (a 0), g: [ 1], g3 = (a .). The result, shown in Fig. 23, is an incredibly clear circle in y, which suggests that the model's generated representation of y lies along circle. A simple PCA projection because the representation for ? has interference from a and 33 which the EVR removes. This suggests that the models may be generating Y by using trigonometry based algorithm like the "clock" Nanda et al. (2023a) or "pizza" Zhong et al. (2024) algorithm in late MLP layers.
not enough to find this result
K.2 MORE EXPERIMENTS WITH EVR
We now apply EVR to Months and Weekdays to break down X completely into interpretable functions. We build a list g; iteratively and greedily. At each iteration, we perform a linear regression with the current list g1 ... gk. visualize and interpret the residual prediction errors, and build a new function g++1 representing these errors to add to the list. Once most variance is explained, can conclude that gı, ....... BA constitutes the entirety of what is represented in the hidden states.
30

---

Published as conference paper at ICLR 2025
This information tells us what and cannot be extracted via a linear probe, without having train any probes. Furthermore, we treat each g: as a feature (see Definition 1), then the linear regression coefficients tell us which directions in X;, these features represented in, connecting back to Hypothesis 2.
Since X consists of modular addition problems with two inputs and B3 as we iteratively construct gi....: & by making a heatmap with a and 3 on the two axes, where the made. More specifically,
visualize the errors
color shows what kind of error distribution and assign them to the colors red, green, and blue. We call the resulting heatmap a residual RGB plot. Errors that depend primarily on a, 3, or y show up as horizontal, vertical, or diagonal stripes on the residual RGB plot.
take the top PCA components of the
In Fig. 24, perform EVR on the layer 17-29 hidden states of Mistral 7B on the Weekdays task; additional deconstructions are in Appendix K. We find that circle in yy develops and grows in explanatory power, we plot the layer residuals after explaining with one hot functions in a and (i.e. gi lax O]. B2 [3 1]. g3 Ja ... in Fig. 23 to show this incredibly clear circle in 7. This suggests that the models may be generating by using a trigonometry based algorithm like the "clock" (Nanda et al., 2023a) or "pizza" (Zhong et al., 2024) algorithm in late MLP layers.
Table 4: Highest intervention effect attention heads from fine- fine-grained attention head patching, well as EVR results with one hot a, 3 and one hot a, 8,
(a) Mistral 7B, Weekdays.
(b) Llama 3 8B, weekdays.
L
H
Average
EVR R²
EVR R² One Hot
L
H
Average Inter- vention Effect
EVR R² One Hot a, B
EVR R²
Inter-
One Hot a,
One Hot
vention Effect
B, Y
a, 3,
28
18
0.22
0.39
0.73
17
0
0.18
0.98
0.99
18
30
0.17
0.95
0.96
0.08
0.98
0.98
15
13
0.17
0.94
0.95
19
10
0.08
0.95
0.96
22
15
0.11
0.77
0.82
0.07
0.85
0.90
16
21
0.09
0.92
0.93
17
3
0.07
0.93
0.95
28
16
0.08
0.42
0.69
27
0.06
1.00
1.00
15
14
0.06
0.98
0.99
31
22
0,05
0.37
0.78
30
24
0.05
0.43
0.79
21
9
0.04
0.73
0.78
21
26
0.04
0.53
0.63
20
28
0.04
1.00
1.00
14
2
0.04
0.93
0.95
30
16
0.04
0.73
0.85
(c) Mistral 7B, Months.
(d) Llama 3 8B, Months.
Average Inter- vention Effect
EVR Ri2 One Hot a, 3
EVR R² One Hot a, 3,
L
Average Inter- vention Effect
EVR R² One Hot a, B
EVR R²
One Hot
a, 3,
20
28
0.15
0.76
0.76
15
13
0.26
0.62
0.62
17
0
0.10
0.77
0.77
16
21
0.17
0.76
0.76
25
14
0.08
0.19
0.61
0.13
0.77
0.77
17
1
0.07
0.80
0.82
28
18
0.11
0.13
0.52
17
0.06
0.71
0.71
28
0.07
0.13
0.52
31
22
0.06
0.12
0.67
21
25
0.05
0.65
0.70
17
27
0.05
0.58
0.58
14
0.03
0.72
0.72
19
4
0.05
0.40
0.66
17
26
0.02
0.77
0.77
19
10
0.04
0.62
0.62
31
1
0.02
0.11
0.57
30
26
0.04
0.51
0.62
21
24
0.02
0.30
0.45
31

---

Published as a conference paper at ICLR 2025
lase
liver
laver 15
layer 16
laver
layer
18
Jayer
10
layer
20
layer
Iawer 22
layer
23
bayer
24
layer
layer
26
layer 27
layar
28
layer
orignall
Orignal
Onganaa origna
69.7%
94
ON
ON
74 M
465%
035
99
41.6
SOA A TOW
CIRE
CRU CIUE
CIUC
UICE
BICE
ORE
OICE TN
DEN DOR
DIL OF
15.1%
OPN TEE
16.2%
ON ISE
16.75
Figure 25: Iterative deconstruction of hidden state representations on the final token on Llama 8B, Months.
32


Reviews:
[{"review_id": "o08CAgRMHR", "reviewer": "Reviewer_x6ai", "summary": "The paper calls into question the Linear Representation Hypothesis (LRH), in particular the part that claims the representations of features in LLMs lie along one-dimensional lines. \nA clear visual example is provided in Figure 1, where a circular feature for days of the week is shown in a 2-dimensional PCA space.\nThe paper then formally defines irreducible, multi-dimensional features and proposes an updated version of Elhage et al.'s Linear Superposition Hypothesis to incorporate multi-dimensional features.\nThe specific definition of irreducibility is further used to derive a clustering-based method to automatically find irreducible features among the dictionary elements learned by sparse autoencoders (SAEs) trained on LLMs.\nFinally, the paper illustrates how steering the circular feature of day-of-week can alter the model's output in synthetic tasks.", "questions": "- Figure 1: is there any intuition for why specifically we find these features in the second and third PCA components? \n- Definition 1: What exactly is a $d_f$-dimensional point cloud in $\\mathbb{R}^{d_f}$? Is there a mathematical definition? (Is $d_f$ redundant?) If I have two points in $\\mathbb{R}^3$, what is the \"dimension\" of the point cloud they form?\n- Also, to be precise, should I understand the \"probability distribution over input tokens\" as that of the natural language? I also assume that this probability is restricted to $n$-length token sequences---is that correct? \n  - Later, in the definition of the two indices, the distribution comes up anyway as $\\mathbb{P}$. It may be helpful to formally define this notation/distribution up front.\n- Definition 2: in the mixture definition, what is the non-bold $b$? Also, does \"lower dimensional\" simply mean $k < d_f/2$ or does it mean low-dimensional, say, $k \\ll d_f$?\n- Is there intuition as to why it suffices to consider separable and disjoint features as the two types of reducible features? Could there be other types of reducible features?\n- Can you efficiently compute the separability index for features that are beyond two-dimensional?\n- Hypothesis 2: while this is a stylistic point, I think the use of a possessive in \"Our Superposition Hypothesis\" is unnecessary and un-scientific. It could have any other objective name, say, the \"Multi-Dimensional Superposition Hypothesis\". See, e.g., [Knuth et al.](https://jmlr.csail.mit.edu/reviewing-papers/knuth_mathematical_writing.pdf), page 2 item 6.\n- p. 5, lines 260--270: doesn't this claim depend heavily on (1) the two-dimensional feature being active in the LLM's representation space and (2) the SAE is large enough to capture this feature? I'm not even sure if this claim is necessary, as opposed to saying \"some of SAE clusters represent irreducible features\".\n- Figure 3: what are some of the other features that are shown  here to be more reducible than the features for Fig. 1?\n- Are there any other features that we may intuitively view as circular but not found by the SAE? Say, are color words (red-orange-yellow-green-blue-purple) represented in a circular fashion?\n- p. 8, \"In practice, ...\": this is somewhat confusing/concerning to me. If there are alternative pathways that may affect both the circular feature and the task output, then wouldn't they \"wash away\" the causal implication of the feature? In other words, in what sense exactly is this a \"causal\" implication rather than correlational?\n- Broad question: are there examples of multi-dimensional features that aren't circular?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper tackles a timely and important question in mechanistic interpretability by formalizing (inherently) multi-dimensional features in LLMs.\n- The visual examples of circular representations are clear and compelling.\n- I think the high-level idea of formalizing multi-dimensional, irreducible features is sensible and useful.\n- Overall, the experiments are comprehensive with a variety of interesting results.", "weaknesses": "- I think there are some conceptual questions about the definitions of features and irreducibility. \n    - Conceptually, I think the definition of irreducibility is somewhat incomplete. First, there can be concepts that are correlated but can still be disentangled for separate interventions, so equating separability with independence is can be limiting. Second, I can't tell for sure if the definition of reducibility is exhaustive. (Can there be a third category?)\n    - I have asked about some of the more specific unclear points in the questions below.\n- While the paper's main motivation is to call the LRH into question, the proposed alternative hypothesis appears to be a stricter version of the LRH (p. 5, lines 239--240), rather than a contradiction to it. This makes the overall message a bit confusing. (I'd be happy to be corrected if this is a misunderstanding.)\n- The proposed \"tests\" of irreducibility are not completely formal, in the sense that they are not statistical tests of significance but rather (possibly arbitrary) thresholding of a pair of metrics. This could mean that whether or not a feature is irreducible is ultimately a judgment call by the user. \n    - To be clear, I'm not saying this is something that can be readily addressed; it's a limitation that should be mentioned somewhere.\n- For the intervention tests, I'm not fully sure if the results demonstrate \"causal implication\" of the circular feature. The results are interesting, but I'm not sure if they are conclusive. See second-to-last question below.\n- Despite its interesting results, the paper's exposition is not very clear.\n    - I find that Section 3 is not easy to follow. The only visual aid is Figure 2, but it is not referenced or intuitive.\n        - E.g., why exactly are we looking at PCA 2/3, even for Fig. 1? What is the histogram in the middle, and what is the angle on the right? What are \"typical clusters\" and what are their mixture/separability indices?\n        - I believe some of these are answered in the referenced Appendix. While I am okay with most computational details being deferred to the Appendix, at least the plots should be self-contained. Also, some of the key computational details should be noted (that separability index is computed in 2-d and that mixture index is computed approximately via SGD). Otherwise, the exposition is quite hard to follow.\n    - While some of the plots are great, the experiments are quite dense and unorganized. The exposition would often refer to a figure in the Appendix, while some figures (2 and 4) in the main text are not referenced. This makes it difficult to read the paper linearly. \n        - I would recommend starting each subsection with a clear motivation, the key result(s), and then reference supplementary results in the Appendix.\n        - Perhaps a few more plots can be moved to the Appendix in favor of more explanation/motivation/examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "epIphIf9Es", "reviewer": "Reviewer_d8Uh", "summary": "This work's main focus is to set a theoretical framework for irreducible multidimensional features and to show their concrete existence in real-life LLMs. In particular, the authors: \n- update the superposition definition to account for irreducible multi-dimensional features.\n- propose an algorithm that---using an SAE-learned dictionary---identifies possible multi-dimensional features. \n- show evidence of features expressing circular behavior for the representation (and manipulation) of periodic data, such as months, weeks, and days. The models whose features are shown to exist are non-toy models, such as Llama 3 8B and Mistral 7B, further increasing their argument's validity.", "questions": "1) Do you have any qualitative insights on how big the threshold parameter T should be in order to find multidimensional features?\n\n2) Are there any extracted multi-dimensional features that show some interesting behavior, like the circular one shown in the paper? If not, do you think is it due to a structure difficult to interpret, or simply because probably most of the features are still one-dimensional?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The updated proposed definition for the superposition hypothesis is sound and may lead to the discovery of other interesting structures such as the ones presented in the paper. Even if this direction does not lead to further scientific discoveries, the discovered representations themselves are an interesting finding.\n\n2) In the paper, features are extracted from state-of-the-art LLMs, showcasing the existence of actual multi-dimensional features of circular nature **\"in the wild\"**.", "weaknesses": "1) If I understand correctly, the proposed algorithm can extract interpretable features, however, I imagine there is a good amount of them that is not easily--if at all--interpretable. Adding some of your insight on how many potentially interesting multi-dimensional features are among the ones extracted from the algorithm could improve the article.\n\n2) If I am not mistaken, there is not an ablation on how much the threshold parameter T affects the extracted clusters: Having an understanding of when circular features start to emerge could be an interesting experiment to add, and may potentially lead to some useful insights. \n\n3) Figures 2 and 3 can be improved in their overall presentation:\n- Figure 2: I would change the color for the PCAs scatter plot from black to something with less contrast (e.g. the green used in the angle subfigure of  Figure 2)\n- Figure 3:  This is a very minor complaint, but I would change the scatter icon for the \"other cluster\" points to something different than the one used to indicate the clusters shown in Figure 1, for example, something like an **x** would probably look a bit better.\n\n4) Another minor issue is that the captions for Figures 5 and 6 are a bit lacking, adding some more information to help understand their respective feature would probably be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ePBUpStn3E", "reviewer": "Reviewer_9dQp", "summary": "The authors define multi-dimensional features and identify these representations empirically using sparse autoencoders. They also show that some LLMs use circular representations for days of the week and months of the year to perform modular addition.", "questions": "1. As you mentioned, the title is confusing. How about 'Not All Language Model Features are One-dimensionally Linear'?\n2. Why do you use the term 'point cloud' in Definition 1? Does it mean that the range of the map $f$ is a discrete set?\n3. What is the source of randomness for $p(a,b)$ in Definition 2? Does it come from the distribution of input texts (in training set for LLMs)? Also, why do you state that statistical independence and separability are equivalent?\n4. In Figure 2, the histogram appears to show a mixture. Why did you say that the $\\epsilon$-mixture index of 0.4750 is low? What is the baseline? In Figure 10, you state that an index of 0.3621 indicates the feature is likely a mixture. This seems contradictory.\n5. In line 269, why do you state that 'These dictionary elements will then have a high cosine similarity'? I believe they could have low cosine similarity even if many elements span the feature.\n6. In Figure 1, how were the points labeled? What does the label 'Monday' mean? Does it indicate that 'Monday' would be the next token in the corresponding texts?\n7. In Section 5, you use specific texts and train new probes. Why didn't you use the circular representations previously identified by SAE clustering?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The extension of the linear representation hypothesis from one-dimensional to multi-dimensional one provides good insights for researchers in the mechanistic interpretability field. Concrete and extensive empirical results show that multi-dimensional features exist in LLMs. In particular, the circular representations of days of the week are interesting.", "weaknesses": "Some details in formalization and experiments are not clear. Please see the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2iTS4YD9nj", "reviewer": "Reviewer_hTfC", "summary": "The authors show convincingly that large language models have circular representations of periodic variables, and provide a means of discovering and probing these representations using sparse autoencoders and causal interventions.", "questions": "Is it mostly just circles? Given that they are still a one-dimensional variable, I'm wondering whether there are representations of variables with (irreducible) intrinsic dimension higher than one (like a spherical variable), which would seem to make better use of the scope of your Hypothesis 2. Otherwise, maybe the linear representation hypothesis has been violated in name but not in spirit (i.e. representation is still factorized into one-dimensional variables)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The results are straightforward, and backed up by causal experiments. Clearly connects to existing methods in the interpretability field, which could make it simple for others to replicate.", "weaknesses": "The clustering method was a little hard to follow for me at first, maybe pseudocode would be helpful (since it seems like it wouldn't be many lines).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)