From Data Statistics to Feature Geometry: How Correlations Shape Superposition
Authors: Lucas Prieto
Edward Stevinson
Melih Barsbey
Tolga Birdal
Pedro A. M. Mediano
Venue: ICLR 2026 Poster (2026)
Keywords: Mechanistic Interpretability
Superposition
Linear Representation Hypothesis
Feature Geometry
Feature Manifold
Primary area: interpretability and explainable AI

Abstract: A central idea in mechanistic interpretability is that neural networks represent more features than they have dimensions, arranging them in superposition to form an over-complete basis. This framing has been influential, motivating dictionary learning approaches such as sparse autoencoders. However, superposition has mostly been studied in idealized settings where features are sparse and uncorrelated. In these settings, superposition is typically understood as introducing interference that must be minimized geometrically and filtered out by non-linearities such as ReLUs, yielding local structures like regular polytopes. We show that this account is incomplete for realistic data by introducing Bag-of-Words Superposition (BOWS), a controlled setting to encode binary bag-of-words representations of internet text in superposition. Using BOWS, we find that when features are correlated, interference can be constructive rather than just noise to be filtered out. This is achieved by arranging features according to their co-activation patterns, making interference between active features constructive, while still using ReLUs to avoid false positives. We show that this kind of arrangement is more prevalent in models trained with weight decay and naturally gives rise to semantic clusters and cyclical structures which have been observed in real language models yet were not explained by the standard picture of superposition. Code for this paper can be found at: https://github.com/LucasPrietoAl/correlations-feature-geometry.

Published as a conference paper at ICLR 2026
FROM DATA STATISTICS TO FEATURE GEOMETRY: How CORRELATIONS SHAPE SUPERPOSITION
Lucas Prieto, Edward Stevinson, Melih Barsbey, Tolga Birdal ; Pedro A.M. Mediano" Imperial College London
ABSTRACT
A central idea in mechanistic interpretability is that neural networks represent more features than they have dimensions, arranging them in superposition to form an over-complete basis. This framing has been influential, motivating dictionary learn- ing approaches such sparse autoencoders. However, superposition has mostly been studied in idealized settings where features are sparse and uncorrelated. In these settings, superposition is typically understood introducing interference that must be minimized geometrically and filtered out by non-linearities such ReLUs, yielding local structures like regular polytopes. We show that this account is incomplete for realistic data by introducing Bag-of- Wordd Superposi- tion (BOWS), controlled setting to encode binary bag-of- bag-of-words representations of internet text in superposition. Using BOWS, we find that when features are correlated, interference can be constructive rather than just noise to be filtered out. This is achieved by arranging features according to their co-activation pat- terns, making interference between active features constructive, while still using ReLUs avoid false positives. We show that this kind of arrangement is more prevalent in models trained with weight decay and naturally gives rise to seman- tic clusters and eyclical structures which have been observed in real language models yet were not explained by the standard picture of superposition. Code for this paper can be found hhtpss / /ggthub.. com/LacasPrietoo. correlations feature- geometry
INTRODUCTION
The field of mechanistic interpretability (MI) aims to understand deep learning models by decom- posing them into interpretable components such as features circuits, and understanding how these interact (Olah et al. 2020). A central idea in this field is that models represent more features than they have dimensions, arranging them in superposition to form overcomplete basis (Elhage et al., (2022). at the cost of allowing interference between features. This framing has been influential, motivating dictionary learning approaches such as sparse autoencoders (SAEs), which can successfully recover
interpretable features from neural representations (Templeton et al. 2024 Gao et al. 2025).
While methods for finding features continue advance, our understanding of how feature representa- tions arrange themselves geometrically within high-dimensional activation spaces remains limited. The geometry features superposition determines which concepts interfere with each other, making it a key problem in MI (Sharkey et al. 2025) with implications for SAE training (Hindupur
2025), knowledge editing (Nishi et al. 2025), and adversarial robustness (Stevinson al.
2025 Gorton & Lewis, 2025). In existing toy models, superposition is typically understood as introducing interference that should be minimized geometrically and filtered out with non-linearities such as ReLUs (Elhage et al. (2022). This gives rise to arrangements such as regular polytopes, where pairwise dot products are small and negative interference can be suppressed.
However, this perspective does not account for the kinds of structure observed in real language models. Rather than local regular polytope structure, researchers have found ordered circles of features, such the months of the year (Engels et al. 2025). well anisotropic superposition, where related features cluster together rather than minimizing dot products (Bricken et al.) 2023) Templeton et al.) 2024). We argue that this discrepancy arises because realistic features are not sparse and uncorrelated.
When features are correlated, interference need not be purely harmful: it can also be constructive,
Joint Senior Authors

---

Published as a conference paper at ICLR 2026
BOWS
Non-linear
Linear Superposition
(This work)
Internet text
Superposition
Example:
Nov
[Elhage et. al. 2022)
Months of the year (cyclic structure)
Oct
Example: Antipodal pairs
Festure
Jan
Binary BoW (sice d)
Christmas
Feb
POSLI
ReLU filters out Interference
Months constructively Interfere to help recon-
Sep..
Halloween
Easter
W
Spring
Conditions:
Conditions
Aug
Apr
Latent Ipprrmtttioon
Feature
spersity
Conslnted feanures
m)
Vocab dim
Jul
features
Jun
May
Figure I: BOWS, our new framework to study superposition in realistic data (left) extends our current understanding of superposition (middle) by showing that interference can be constructive, allowing words like 'December' to contribute to the reconstruction of correlated words like 'Christmas' giving rise circular arrangement for the months of the year (right).
enabling efficient reconstruction in terms of weight norm and rank rather than always requiring non-linear filtering. ReLU-based filtering remains important for suppressing harmful interference, but correlated features can also be arranged that interference reinforces signal.
To study this a controlled setting, we introduce Bag-of- Words Superposition (BOWS), a framework in which autoencoder is trained to encode binary bag-of-wordh representations of internet text in superposition. BOWS provides realistic feature correlations together with known ground-truth features. Using this setup, we show that arranging features according their co-activation patterns naturally gives rise to semantic clusters and cyclical structures of the kind observed in real language models. We refer the regime in which low-rank structure the data supports constructive interference as linear superposition.
Our main contributions are follows:
We introduce BOWS as a controlled setting to study superposition with realistic features. We show that, when features are correlated, interference can be constructive rather than only acting as noise, and formalize regime of linear superposition in which this constructive interference is leveraged by non-linear AEs to enable efficient reconstruction in terms of weight norm and rank. We show that these solutions emerge prominently under tight bottlenecks or weight decay, and that they reproduce key geometric structures observed real language models, including semantic clusters and cyclical structure.
We introduce the distinction between presence-coding and value-coding features to explain the existence of structured representations in the absence of feature correlations.
2 BACKGROUND
We introduce definitions that distinguish whether superposed features can be recovered by linear decoder or require non-linear decoder, and present our setting for studying superposition under realistic data distributions. We define superposition for abstract features then discuss the relevance of superposition when these features are properties of data under the linear representation hypothesis.
2.1 DEFINITIONS
Consider d features indexed by [d] (1,..., d} with values f [fi,..., ... Sa] Rdd drawn from distribution Dr. We consider linear encoders W e mg with m < d, which represent each feature along a direction W; 1
Given a decoder x) R + Rd, we define the per-feature coefficient of determination as
R (,W, 1b) = 1 ED, [(/: v/ (W1).)
(1)
Varz, [ff]]
Definition (Superposition). A set of features F S [d] is represented in superposition if:
1. (Interference) For every E F, there exists j E F with + such that (w;, wy) * 0.
2

---

Published as a conference paper at ICLR 2026
2. (Recoverability) There exists decoder R + Rd such that (W, v)) 2 1 for all E F.
Definition 2 (Linear Superposition). Let F be set of features in superposition. A feature i E FF is in linear superposition there exists a linear decoder 4/iin II + IR such that I25 thin) 2 -
Definition 3 (Non-linear Superposition). A feature i E F is in non-linear superposition if:
1. There exists a decoder & with R, ()) 2 1 E, and
2. For all linear decoders Viin we have R (W, vlmn) < 1-
Superposition in deep learning models. The definitions above apply to any features f. However, superposition becomes central to mechanistic interpretability under the linear representation hy- pothesis (LRH). The LRH reflects the empirical finding that high-level concepts such as language (Gurnee al. 2023). entity attributes, or specific landmarks (Templeton al. 2024) are often linearly represented in model activations. Concretely, let Dx be a distribution over data samples x E X' (e.g.. an image or text) and let P Pa : X E. be interpretable properties (e.g.. "is written in French", "contains a dog"). These induce a vector of features f(x) = [pi(x).... Pa(x)] .((x))).
Definition 4 (L (Linear Representation Hypothesis). We say that a hidden representation h satisfies the linear representation hypothesis with respect to {e, } if there exist directions Wi,..., Wa E RM such that:
1(x) Eds(x) for ~ Dx. f=1
model that linearly represents more concepts than its hidden dimension must encode them in super- position for downstream Under the LRH, understanding how these concepts are geometrically arranged is thus a key challenge for mechanistic interpretability.
2.2 BOWS: REALISTIC DATA IN SUPERPOSITION
Studying superposition in the hidden representations of deep learning models requires postulating a set of features p(x) that are linearly represented. In most cases, however, we do not have access to ground-truth features or their representations. We therefore introduce Bag-of- Words Superposition (BOWS), setting for studying superposition with realistic feature correlations. Relative to the iid. and pairwise-correlttin settings considered in Elhage et al. (2022). BOWS induces richer covariance structure, including approximately low-rank structure, while retaining known ground-truth features.
Dataset. Let C be corpus of text segmented into records (lines or paragraphs). After
tokenisation, we construct a vocabulary of the V most frequent words, discarding common English stop-words and prepositions. This vocabulary includes words such as sun, code, and January which often correspond to linear features in sparse autoencoders trained on language data (Engels et 2025| Bricken et al. 2023). Each record is then encoded as a binary bag-of-words vector E {0,1)" 1J whose j-th component is 1 iff the j-th vocabulary word appears in the record. We choose a context size c E N. For every contiguous block of records we take the element-wise logical OR of their individual vectors, obtaining a single sample. The resulting dataset is
D {
x; E 10,
where N is the number of c-record chunks in the corpus.
Experiments in the main text use WikiText-103 (Merity et al., 2017) with V 10,000 and c 20. Data is split into N 1,621,198 samples for training, and 130,133 for validation. We include replication of the main results using WebTett in Appendix
Autoencoder. We use the autoencoder setup for superposition introduced in Elhage et al. (022) consisting of an encoder with weights W E R and bias b E IR. where the input f E R
reconstructed using a ReLU AE with loss: CRLLU-AE(X, W, b) ||f ReLU(W Wf
We also use a Linear AE as baseline with loss: LLinear-AE(E W, b) || (W7 WT b)ll
3 CONSTRUCTIVE INTERFERENCE IN NON-LINEAR AES
We now study how data covariance structure and optimization constraints shape the solutions learned by linear and non-linear autoencoders. We show that when features correlated, non-linear
3

---

Published as a conference paper at ICLR 2026
Data Cov.
m=2
m m=3
m=4
m=6
m=7
m=8
m=10
m=11
m=
Figure 2: Autoencoding synthetic correlated features shows two ways of handling interference. Weight inner products (WT W) convergence for AEs encoding covariance, varying latent size m. Top (Linear AE); captures the top-m principal components projecting all 12 features on the circular structure induced by the data covariance. Bottom (ReLU AE): Matches linear AE for small m, but forms antipodal pairs for E {6. 10) using the ReLU to filter interference.
12 features with cyclic
autoencoders can exploit interference constructively, arranging features so that shared variance supports reconstruction rather than treating interference noise to be filtered out. We characterize when each mechanism is favored, and how the two can coexist in the same model.
3.1 SUPERPOSITION AND INTERFERENCE
For tied-weight ABS with weight matrix W E mx columns [w:)i-r and activation function o, the reconstruction of feature fi decomposes into signal and an interference term:
f: a
[| f
(w., w,)ff
(4)
J
Signal
Interference I.
Let Σ = ] denote the feature covariance. In the standard picture of superposition, I₃ is treated as noise to be non-linearly filtered out (Elhage et al. (2022). We show that this interpretation is incomplete: when Σ has sufficiently strong low-rank structure, interference can align with the signal and become useful rather than purely harmful]
Interference as noise (weakly correlated, sparse features). When features are sparse and weakly correlated, Z; behaves as unstructured noise that is approximately uncorrelated with ft. Accurate reconstruction then requires filtering this interference via a non-linearity (e.g. a ReLU) together
with a negative bias such that E E/RcLU((I, + b:)] x 0, while maintaining ||w, 2 = 1 to preserve signal. This is the setting emphasized in prior work, and favors feature arrangements that minimize pairwise dot products.
Constructive interference (rank(>) Σ) < m)) For linear AEs (a = id the optimal map P W W
the orthogonal projector onto the top-m principal components Σ (Baldi & Hornik 1989 Jolliffe 2002). Defining the reconstruction residual E; = ft fú. we can rearrange Equation interference:
isolate
I, (1 ||
(5)
When (ank(E) < m, the data lies in the principal subspace, so = 0 and I, = (1- Pa)f In this case, interference is proportional
the signal rather than opposed it. This occurs because P. = (W,, w,) reflects the correlation between features i and within the principal subspace: each fy contributes to the reconstruction of fa proportion their shared variance. Through this lens, PCA can be viewed as form of superposition in which correlated features are arranged so that interference reinforces signal rather than requiring suppression!
The case of real data (E approximately low-rank). In real-world data, including the text data we consider, the covariance is often well approximated by a small number of principal components
Flhage al. (022) also consider pairwise correlated features, but their setup all principal components carry significant variance, so PCA collapses correlated pairs onto indistinguishable points.
IIn the linear case, the optimal hias satisfies b* (I W W)F so the AE effectively operates on centered features
Rlf] We therefore analyse the setting with = when a = id without loss of generality. In practice, P is learned from finite data and reflects the sample covariance: for test samples whose correlations deviate from the training distribution, interference will he imperfectly aligned with the signal.
4

---

Published as a conference paper at ICLR 2026
dn Natur civoorees
200 (a) wd
400 wd
800 (c) wd
800 (d) wd 1
Semantic categories
Figure 3: Linear superposition appears in ReLU AES which have small latent sizes (a) or are trained with weight decay (d), giving rise to semantic clusters. UMAP projections of word embeddings from AEs of different latent dimensions (m) and weight decay values (wd). Points are colored by semantic category (e).
(Deerwester et al. 1990, Blei et al. 2003 Udell & Townsend 2019). making E approximately
low-rank. In this regime, interference behaves as signal contaminated by residual 6; corresponding
to variance outside the top-m subspace, with ||=||2 Eksm A (E) 2 min
min rank(())) |Σ Σ
(Eckart & Young 1936). Thus, as the spectrum of becomes more concentrated, the residual becomes smaller and constructive interference becomes increasingly effective.
When do non-linear models exploit constructive interference? For models trained real data with weight decay, the interference-filtrrie and constructive-interfeenne solutions have different norm requirements. In the standard sparse- feature setting, recovering each feature while relying a ReLU to suppress interference requires feature directions with approximately unit norm, so the canonical interference-filerrng solution has | I scaling with the number of represented features, i.e. | [| z d. By contrast, when reconstruction achieved by projecting onto an m-dimensional low-rank subspace, the optimal linear solution is a rank-m projector P = W W, for which
WI = tr( E - m d.
(6)
Thus, weight decay combined with tight bottlenecks (m < d) biases even non-linear models to- ward solutions that exploit low-rank structure, since these can achieve accurate reconstruction with substantially smaller weight norm.
Complementarity of the two mechanisms. The constructive and filtering mechanisms are not mutually exclusive. When rank(E) S m, linear AE can arrange features so that interference on inactive features vanishes exactly. But when E is only approximately low-rank, the residual can still induce false positives on inactive features or negative reconstruction values. In this setting, both mechanisms operate together: the weight geometry exploits correlation structure that much of the interference aligns with the signal, while ReLU with negative bias suppresses the residual harmful interference introduced by Ei.
3.2 EVIDENCE OF LINEAR SUPERPOSITION IN NON-LINEAR AES
We start our empirical analysis with a simplified setting where two AEs of latent dimension m with and without a ReLU in the decoder are trained on 12 dimensional data with a cyclic covariance structure (Figure 2 left). Details for the data generation process are provided in Appendix [A] Figure (top row, Linear) shows the baseline learned by the linear AE, which learns the projection onto the principal subspace (Baldi Hornik] 1989). We then study the emergence linear superposition in non-linear autoencoders, specifically looking at autoencoders with ReLU in the decoder of the kind described in Section
Linear superposition. When the bottleneck is very tight (m < d). the ReLU AE recovers the circular structure dictated by the top principal components These are examples of non-linear AE leveraging linear superposition.
row, ReLU, m < 6).
Non-linear superposition. Figure 2 (bottom ReLU AE abandons the circular PCA structure and instead represents features as antipodal pairs. This specific geometry is one of the cases studied by Elhage al. (2022). whereby features placed
ReLU, 6) shows that increases, the
5

---

Published as a conference paper at ICLR 2026
Jen 232
Fdb.
223
Sep.
Hellowetry
nnow
Feb
Pro aedivatioon
Constraction
Sip.
Aug.
jol-
132
Jan
Peb
jul
Aug
Sumpee
saring
Apr
Interferance
Mar
Apr
May
Jul
Jun
May
Oytztra
condual content.
willuu: "Chrstmay"
(a) Data correlations
(b) PCA of the data
(c) PCA of W
(d) 'Christmas' example
Figure 4: Circular representation of months arises from data covariance via PCA. (a) Empirical correlation matrix of month words in the WikiText- 103 BOWS dataset, showing eyclic correlations. (b) PCA applied directly to the 12 month dimensions of the BOWS data vectors, projected onto the top PCs, reveals circle. (c) PCA applied to the 12 learned encoder features (W columns) for months from a ReL.U AE trained on WikiText-BOWS (V = 10k, m = 1000), projected onto their top 2 PCs, also recovers the circular structure. Seasonal words such as 'Christmas' and 'summer' align with the months with which they co-occur, allowing 'December' to contribute to the reconstruction of "Christmas' while interference on 'Christmas' cancels if all months are present (d).
in anti-correlated pairs such that activating one feature negatively activates its antipodal partner, with negative interference zeroed by the ReLU as an example of non-linear superposition.
Structure disappears features become orthogonal. As the latent size approaches the input size, the autoencoder weights converge to the identity, representing each feature orthogonally for a perfect reconstruction. As these models (m 12) approach perfect reconstruction of the data, the circular covariance structure is no longer reflected in the weights, and any notion of superposition is lost.
LINEAR SUPERPOSITION EXPLAINS FEATURE GEOMETRY IN REAL DATA
In Section we argued that when features are correlated, interference need not be purely harmful: it can also be constructive, allowing non-linear autoencoders to exploit low-rank structure in the data rather than relying only on ReLU-based interference filtering. In this section, we show that this mechanism accounts for the feature geometry observed in our main WikiText-BOWS setting.
Semantic clustering from constructive interference. A prominent observation in studies of LLM activations that learned features often form clusters based on semantic relatedness, leading to anisotropic superposition, where features are not arranged to minimize pairwise dot products (Bricken al. 2023 Templeton et al. 2024). Such structure is difficult to reconcile with the standard picture of superposition as purely harmful interference. By contrast, it arises naturally if model exploits constructive interference to capture low-rank structure in realistic data.
In Figure 3 we show UMAP (McInnes al., 2018) projections of the word embeddings (columns of W) learned by ReLU AE trained on WikiTex--BOWS (V = 10,000) with varying latent dimensions In panel (a), where compression is strong (m 200), observe distinct clusters corresponding to semantic categories (c.g. verbs, proper names, sports). This clustering weakens as the latent size increases to 800 (Figure but reappears when weight decay is introduced (Figure These results suggest that semantic clustering can arise as a consequence of constructive interference under compression, and help explain why similar structure is observed in language models trained with weight decay.
Cyclical structures inherited from data statistics. Another notable form of feature geometry observed in real models is circular structure in the principal components of feature embeddings, for concepts such as months of the year or days of the week (Engels et al. 2025).
Consider the features corresponding to the twelve months of the year. In Figure 4a we show that these features exhibit cyclic correlations WikiText: for example, January co-occurs more often with February and December than with August. This covariance structure drives the leading principal components of the month activations, producing a circle the 2D PCA plot in Figure 4b)) Notably, the learned latent representations exhibit the same geometry (Figure 4c). For fully linear AE, this would

---

Published as a conference paper at ICLR 2026
hot (-0.442) threshold
665
one-hot R
Abwsss 0 (81%)
PeLU threshoid
.... bias -0.271
2.35
Starr
6.4
Lotenge
Interference is beneficial
Harrison
2.50
Pre-activation
Starr
De
Harrison
fish"
McCartney
2.02
McCartney
m -0.003
Lennon.
Lennon
6.12-
-81 Beatles
Beatles
McCartney
Beatle tles
6:0
-6.33 -025 ois
Li0
Beathes correiated
Per-word
'Beaties' decoder pre-activation
uncomelated contect
(without Beatkss
contest
Figure 5: Constructive interference and interference filtering coexist in realistic data. (Left) Terms related 'Beatles' achieve high validation Ht² despite poor one-hot reconstruction, indicating that they benefit from contextual interference. (Middle) For 81% of validation samples containing 'Beatles', interference improves reconstruction relative the one-hot case. (Right) In supportive contexts, correlated words contribute positive pre-activation to 'Beatles'; when these contexts occur without the target word, the ReLU and negative bias suppress false positives.
be expected, since its weights span the PCA subspace (Baldi & Hornik 1989) (see Appendix D). The fact that our non-linear AE displays the same structure supports the presence of linear superposition within a non-linear model.
Furthermore, seasonal words such as 'Christmas' and 'summer' align with the months with which they co-occur. This provides a direct example of correlated features being arranged so that interference reflects shared structure in the data rather than being eliminated. We show as example, that interference from 'December' contributes to the reconstruction of 'Christmas' when they are both present in the context, while interference from all the months almost cancels out and does not lead to false positives when 'Christmas' not present (Section 4). More detailed results for 'Christmas' and other examples where interference is constructive are provided in Appendix C
To verify that the month features are in linear superposition, we train linear decoder (without a ReLU) reconstruct their inputs. We obtain R,(W menths: thin)) 0.98 + 0.00015, with none of the months orthogonal to one another. By Definition the month features are therefore represented in linear superposition.
Constructive interference and interference filtering coexist in realistic data. In realistic data, constructive interference and ReLU-based filtering can coexist within the same model, and even for the same feature. This can be seen by comparing reconstruction in isolation and in context. If interference were purely harmful, a feature would be reconstructed best in the one-hot setting, where no other active features contribute interference. However, when model exploits correlations between features, the one-hot setting need not be optimal: correlated context can improve reconstruction by contributing constructively to the target feature.
We observe exactly this phenomenon for words associated with The Beatles. These words are not frequent enough to occupy nearly independent dimensions, yet 'Beatles' and the surnames of the band members achieve validation R22 values above 0.5 while having near-zero one-hot R²2 Thus, interference from correlated words (middle), for 81% % of validation samples containing 'Beatles', the contextual reconstruction is better than the one-hot reconstruction. Figure [S] (right) shows the mechanism more directly: correlated words provide positive evidence for the target word, while the ReL.U and negative bias suppress false positives when similar contexts occur without the target. This is precisely the realistic setting in another.
merely tolerated; it is beneficial. As shown in Figure
which constructive interference and interference filtering complement
Word frequencies in natural text are highly heterogeneous ZZipi 1949 Clauset et al. 2009 so different features need not be represented by the same mechanism. Some can exploit correlation structure and remain in linear superposition, while others are represented through stronger interference filtering become nearly orthogonal. Performing linear superposition test the 800 model used in Figure 9 and using = 0.5 as a threshold, we find that 4073 words are represented in non-linear superposition, while the most frequent 522 words are represented in linear superposition.
7

---

Published as a conference paper at ICLR 2026
Months
Jan
Aug
Roman
Myg
Wix
numerals
Apr
Snapshot 1
Apr
Snapshot 2
More Dec
m
|G diag
0.8
0.4
0.2
0.0
10a
10a
10
Latent dimension
Figure 6: Different feature structures disappear at different latent sizes. As the latent size of the AE increases, the structure in the weights appears and disappears at different times for different groups of features. We look 2D PCA plots of months and roman numeral embeddings at two key latent sizes. At Snapshot (green line), both groups of features appear in order with some structure, while at Snapshot (red line), the representations the months have already become orthogonal while the roman numerals still represented in ordered structure.
We illustrate this heterogeneity in Figure by comparing two groups of words: the months of the year and the first 10 Roman numerals. We isolate the weights W moentss and W roman and measure the off-diagonal Frobenius norms of WT monts W months and WT W roman- which quantify the extent roman to which features in each group interfere with one another. An off-diagonal Frobenius norm of zero indicates that all features in the group mutually orthogonal. Figure Tigure@ssows shows that small latent sizes both groups exhibit ordered structure consistent with their covariance. As the latent size increases, however, the month structure disappears earlier, as W months becomes closer orthogonal while the Roman numerals continue to display ordered structure. This illustrates that in realistic data, where features differ in frequency and correlation structure, different groups of features can occupy different parts of the spectrum between linear and non-linear superposition.
5 VALUE-CODING FEATURES: GEOMETRY WITHOUT INPUT CORRELATIONS
While the BOWS framework can replicate the kind of semantic structure observed in the hidden representations of language models, there are some examples, like the circles that appear in models performing modular addition (Power et al. 2022; Nanda et al. 2023), which appear in the absence of correlations in the data. To explain this kind of structure, we introduce the distinction between value-coding and presence-coding features and explain how value-coding features can give rise to apparent structures in features that not actually represented in superposition.
Presence-coding features. We say that representation h(r) E Rd contains a presence-coding feature if some binary variable y(r) (e.g. "this token the word car") is recoverable by linear classifier. Formally, there exist weights {wk, b&) such that j(x) = arg max h(x)) by. predicts y(x) with low error. Presence-coding features thus behave as detectors for discrete properties, and different values of are treated as separate classes without requiring any particular geometric relation between them in representation space a priori. For presence-coding features structured representations contingent correlations in the data and capacity constraints that lead the features to be represented linear superposition.
Value-coding features. contrast, we say that a representation h(x) contains a value-coding feature real-valued variable v(x) E IR (e.g. angle, a coordinate, continuous latent factor) is linearly decodable. That is, there exist w E gd and v(z:) with low error. A collection of such value-coding features (x), ... VA (Ir) defines a low- dimensional value space R plotting examples this space can reveal semantically meaningful structures (for instance, 2D map from (z. y) coordinates, addition). Crucially, these structures are fully accounted for by the existence of linear value codes for the underlying variables and therefore exist even in the absence of superposition.
R such that 0(x) = h(x)) + b approximates
circle from (sin 6, cos 0) in modular
8

---

Published as a conference paper at ICLR 2026
Empirical evidence for value-coding features. To exemplify this, look at simplified modular addition setup similar to that in Nanda et al. (2023), as well as a relative map position setup inspired by Gurnee& & Tegmark| (024). In the latter, take the top 1,000 most populated cities in the US and calculate their relative positions terms of quadrants (e.g. Seattle is north-west of Denver). The cities are embedded separately, fed through a 1-hidden layer ReL.U MLP (details in Appendix [A]. This latter dataset is designed to incentivize the model to learn 2 value-coding features encoding the coordinates of each city since relative positions of the cities can easily be calculated by subtracting coordinates. For both of these tasks, each integer or city pair only appears once across the train and validation sets so no pair of cities or integers are correlated and we do not expect any kind of low rank structure coming from linear superposition.
We validate that the cardinal direction model is learning value coding features for the coordinates by training a linear probe to predict the coordinates of a subset of the cities from their embeddings. We then use this probe to predict the coordinates of some held-out cities. The result an average Re validation score of 0.98 and correct arrangement of the held out cities on the US map when projected onto the directions identified by the linear probes (Figure 7 Similarly, we validate that the relevant sine and cosine values are linearly represented by projecting the representations onto the corresponding Fourier components in the case of modular addition (details Appendix A).
We have shown two examples where neural networks need linearly represent values like coordinates or trigonometric functions, in order to perform computations with them. We have then shown in Figure 7 that in doing this, models implicitly create structures like circles maps when we project the data onto these value-coding features.
Ablating the subspace orthogonal to the value-coding features. Having iso- lated the value coding features both in the modular addition and map datasets, we ablate the subspace of the embed- ding space that is orthogonal to these value-coding features. In both cases, the ablation results show that most of the test accuracy preserved if we replace over 90% of the dimensions with their mean (Table []. Conversely performance breaks down if remove only the value coding features. This serves further evidence that these value coding features are the units of computation that the model is using to perform these tasks (replicating the results from Nanda al. (2023) in the modular case).
Table 1: Value-coding (VC) ablations on two datasets. VC keeps the VC subspace and zeros its orthogonal complement; VC ablates VC coordinates.
MAP
Key-freq
Condition
Loss |
Acc. (%) f
Loss
Acc. (%) f
Baseline
0.0536
97.94
0.0001
100.00
VC+ VC
0.2879
93.16
0.1649
93.99
6.3943
22.43
11.7464
3.11
Distinguishing feature geometry and feature manifolds. At first glance, feature manifolds such as those in Figure 7 may appear deceptively similar to geometric arrangements like Figure Yet, our findings offer principled way to distinguish the two. When features exhibit recognizable structure, we can ask whether that structure reflects genuine co-activation patterns. In Figure [4] for instance, the latent arrangement clearly aligns with co-activations among month-related features, pointing to a superposition- based representation. In contrast, the patterns in Figure eigure despite inputs being uncorrelated, suggesting the model has instead learned value-coding features through task-driven projections. Disentangling these phenomena in more complex, real-world scenarios remains a compelling direction for future research.
RELATED WORK
Superposition. Initial works in MI studied interpretable monosematic neurons in DL models (Olah et al., 2020): Cammarata et al.] 2020) but faced challenges in interpreting polysemantic neurons which activate for seemingly unrelated concepts. [lhhae et al. (2022) introduced superposition as an explanation for neuron polysemanticity. This view of DL models inspired further studies (Scherlis)
et al. 2025) and sparse dictionary learning approaches like sparse autoencoders to decompose model activations into an overcomplete basis of linear features (Gurnee et al.) 2023 Huben et al. 2024 Bricken et al., 2023). This approach has successfully been scaled to frontier language models and multimodal models by Gao al. 2025) and Templeton et al. (024))

---

Published as a conference paper at ICLR 2026
Seattle
integer indox
Dentver
Chi
City
Waslpington
ton
Athiaaee San Bilego"
Auskin
train validation
12 200
CO0 20 2
Miaaml
Figure 7: In the embeddings of a model performing modular addition, the circular structure is isolated by projecting onto directions corresponding sine and cosine values (left). In the case of the map directions dataset, two linear probes reconstruct the city coordinates, reconstructing their positions on the map for both train and validation samples. (right).
Feature geometry. Park et al. (2024) proposed a formalization of the LRH and proposed an inner product that preserves language structure. Park et al. (2025) studied how features with hierarchical relations are encoded in language models while they show that categorical features which form polytopes, we note that these are different from regular polytopes posited by |Elhage et al. (2022). Lee et al. (2025) studied geometric similarities in token embeddings of different language models, while Zhao et al. 2024) studied the kind of structure that emerges in the representations of models trained on next token prediction. Li et al. (2025) showed that language models represent integers in helix structure to perform modular addition echoing the results from Nanda et al. (2023) and
Liu et al. )022) on transformers trained for modular addition. Gurnee & Tegmark (2024) showed that longitude and latitude well notion of time, are encoded as linear features in language models. The main results highlighted in this paper are circular structures formed by features and semantic feature clusters, described Engels et al. (2025) and Bricken et al. (2023) respectively. These structures have also been studied as feature manifolds in Modell et al. (2025) and Hindupur al. (025) highlighted the importance of understanding feature geometry when designing SAES. These findings sparked a discussion around the potential limitations of SDL approaches and the LRH suggested by this non- -ineeily encoded semantic information (Sharkey al. 2025). Understanding feature geometry has implications beyond interpretability and generalization, such as for robustness
and compression (Stevinson et al. 2025 Barsbey et al. (026).
Structure in word representations. Classic work on distributional semantics and word embeddings (e.g., Word2Vec (Mikolov et al. 2013). Glo (Pennington et al. 2014)) demonstrated that training simple models on large text corpora leads vector spaces where geometric relationships capture surprisingly sophisticated semantic and syntactic relationships. |evv & Goldberg (2014) showed that methods like Word2Vec with negative sampling implicitly factorize the Pointwise Mutual Information (PMI) matrix shifted by a constant, while others show connections to PCA or SVD on co-occurrence counts or PMI (Allen & Hospedales) 2019).
7 DISCUSSION & CONCLUSION
We introduced BOWS as controlled setting for studying superposition with realistic feature correla- tions. Our main result is that, when features are correlated, superposition does not only introduce harmful interference to be filtered out: it can also exploit constructive interference, arranging features according to their oo-activation patterns. This mechanism yields norm- and rank-efficient reconstruc- tions and accounts for semantic clusters and cyclical structures of the kind observed in real language models (Bricken al., 2023 Engels et al.) 2025). We show this both in synthetic settings and in realistic internet text, and find that such solutions are especially prevalent under tight bottlenecks and weight decay. Finally, we distinguish these correlation-driven structures from value-coding feature manifolds, which arise for functional reasons.
Limitations and future work. BOWS is intentionally simple, and does not capture the full richness of language model representations. Our results show that constructive interference and ReLU-based interference filtering can coexist, but we do not yet provide a complete mathematical characterization of when each mechanism dominates. Important directions for future work include extending the analysis to untied autoencoders and more realistic representation settings, and using BOWS as a benchmark for SAE evaluation with known ground-truth feature geometry.
10

---

Published as a conference paper at ICLR 2026
Acknowledgmennts. This work was supported by the UKRI Centre for Doctoral Training in Safe and Trusted AI (EP/S0233356/1). TBB acknowledges support from the Engineering and Physical Sciences Research Council Igrant EP/X01 1364/1]. TE was supported by a UKRI Future Leaders Fellowship Igrant number MR/Y018818/1). MB was supported by the EPSRC Project GNOMON [EP/X01 1364/1].
LLM USAGE STATEMENT
This work used LLM assistance for literature search, language editing, coding, and initial semantic grouping of 4,000 words that was subsequently inspected and refined by hand. All M.M-assisted code and writing were validated by the authors, who take full responsibility the final manuscript and released code.
REPRODUCIBILITY STATEMENT
We include code to reproduce the main results of this paper (including Figure Figure and Figure 4 in the supplementary material. Details about the BOWS setup are given in Section 2.2 and further details about our experiments are provided in appendix A
REFERENCES
Carl Allen and Timothy Hospedales. Analogies explained: Towards understanding word embeddings. In Kamalika Chaudhuri Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 223-22222 2311. PMLR, 09-15
Jun 2019. URL httptppp///ppcocedings... mir. pwwpp htt///9////ala aal.aanennnl1lg html
Pierre Baldi and Kurt Hormik. Neural networks and principal component analysis: Learning from ex-
amples without local minima. Neural Networks, 2(1))53-58, 1989. ISSN 0893-6080. doi: https:
//doi.or//10.0016600
URL C http://wwwwww...neecccccce rectteeddieee....eoo rect com/ scinnce/
Melih Barsbey, Amonno H Ribeiro, Umut Simşekli, and Tolga Birdal. On the interaction of compressibility and adversarial robustness. In International Conference Learning Representations, 2026.
David M. Blei, Andrew Y. Ng. and Michael I, Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:923-1022, 2003.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison. Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter. Tom Henighan, Christopher Olah. Towards monose- manticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. http::/ttansffmmrrrciicittttttt http:///ssfooooorrrccoo///////// wwwo:////0.../mmmmmmmmiiiooooo hettttiiffffatuesenne..tttma
Tim Maxwell, Nicholas Joseph, Zac
Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schubert, Michael Petrov, and Chris Olah. Curve detectors. Distill, 2020. doi:
Aaron Clauset. Cosma Rohilla Shalizi, and M. E. Newman. Power-law distributions in empirical data. SIAM Review, 51 6703 2009, doi: 10.11377070101111
Michael A. A. Cox and Trevor Cox. Mulridimmensioal Scaling, pp. 315-347. Springer Berlin Heidelberg. Berlin, Heidelberg, 2008.
Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):991-407, 1990.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3): 211-218, 1936.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, 7ac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://taansoommeeccccuussppppp pubp:l22spppo222o0a /ttp:///iiiiiiii.o/oooo/o//o//ie wwwameewwn/indexe.iiiii html.
Nelson Elhage. Tristan Hume, Catherine Olsson. Nicholas Schiefer. Tom Henighan, Shauna Kravec, Zac Hatfield- Dodds, Robert Lasenby, Dawn Drain. Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. http::/ttaasfommeecccccuussppppp httph//0022hhoomo//dddeeeee html.
11

---

Published as a conference paper at ICLR 2026
Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features one-dimensionally linear. In The Thirteenth International Conference on Learning Representations, 2025.
URU/httts:://peereeele
forum? Loo
Leo Gao, Tom Dupre Tour, Henk Tillman, Gabriel Goh, Rajan Trall, Alec Radford, llya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representationss 2025, /ttp://ts////pprr.....eeeeeeeees
GeoNames. GeoNames cities1000 dataset. 2025, Accessed: 2025-05-19
Liv Gorton and Owen Lewis. Adversarial examples are not bugs, they are superposition, 2025. URL nt
//arxiv.org abs/ 2508. 17456
Wes Gurnec and Max Tegmark. Language models represent space and time. In The Twelfth Interna- tional Conference Learning Representations, 2024. URL htttpa://openrevierr net/ net//orum? forum?
id= ju 188xbmv in
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitkkii, and Dimitris Bertsimas, Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023.
ISSN 2835-8856. [RL[/tttss///peerreee wwwpeerrwwwwwo.eeeorruunuaaaaaaa www.eeeeoeoddddiooo......
Sai Sumedh R. Hindupur, Ekdeep Singh Lubana, Thomas Fel, and Demba Ba. Projecting assumptions: The duality between sparse autoencoders and concept geometry, 2025. URI
2503. 01822]
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey, Sparse autoencoders find highly interpretable features language models. In The Twelfth International Conference Learning Representations, 2024. htttp:
wwwm/oort////eeeeeeeeeeeeeeeeeet 2 6bwRSLcK
Ian T. Jolliffe. Principal Component Analysis. Springer, edition, 2002.
J. Dennis Lawrence, Catalog of Special Plane Curves. Dover, 1972.
Andrew Lec, Fernanda Viégas, and Martin Wattenherg. Shared global and local geometry of language model embeddings. In Second Workshop on Representational Alignment ar ICLR 2025, 2025. URL. https pepeeneeieww.nt/Eorrre
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.. 2014. URL https: httppps:/pprccceednggge wnw.seeewd..nnnnnnn...........e. neurips. cc/ paper fir_ffileee /a 966 httee///c0eccccceeeeeeeeaarrrraa
Yuxiao Li, Eric J. Michaud, David D. Back, Joshua Engels, Xiaoging Sun, and Max Tegmark. The geometry of concepts: Sparse autoencoder feature structure. Entrpyy, 27(4), 2025. ISSN 1099-4300. doi: 10.3390/ c27040344, URL httssht:s:// /www, mdpi. com/ 40000m11100944447773333333
Ziming Liu, Ouail Kitouni, Niklas Nolte, Fric J Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave. and Kyunghyun Cho (eds.). Advances in Neural Information Processing Systems, 2022.
did.ummmiieeaa net. worumm 1idiidd6 bat 6rm312m
R. Marques, C. Bouville, M. Ribardière, L.-P. Santos, and K. Bouatouch. Spherical fibonacei point sets for illumination integrals. Computer Graphics Forum, 32(4):134 143, 2013.
Leland McInnes, John Healy. Nathaniel Saul. and Lukas Großberger. Umap: Uniform manifold approximation and projection. Jourmal of Open Source Software, 3(29):861, 2018
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL [ttpps://openrevieet net/ forum? 3y377ddy7
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In Ist International Conference on Learning Representations, ICLR 2013, Scotisdule, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013. URL http::/xx//xx org /ab//bb/aa 11011. 3781
Alexander Modell, Patrick Rubin-Delanchye and Nick Whiteley. The origins of representation manifolds in large language models, 2025. URL htt.:/ttpss//aaww..rrr//aaass/
12

---

Published as a conference paper at ICLR 2026
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. The Eleventh International Conference on Learning Representations, 2023. not URL/htttps://ppeneeiiic forum forum/odummmd--oo 9XE 9tt.dd--xXeDDmmmnn
Kento Nishi, Rahul Ramesh, Maya Okawa, Mikail Khona, Hidenori Tanaka, and Ekdeep Singh Lubana.
Representation shattering in transformers: A synthetic study with knowledge editing. In Forty-second International Conference on Machine Learning, 2025, URL htl.htttps///peereeee//
net/forum?
BK00eyZa00
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction circuits. Distill, 2020. doi: 10.239 2/2911////slll..0002.00000 https://dcaillllu//////00drrrrir in.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference Machine Learning, 2024. URL https
Kiho Park, Yo loong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. In The Thirteenth International Conference on Learning Representations, 2025. net forum? LeerrummLd-bVV...CR.0
URL
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation.
In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on
Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-15433 Doha. Qatar, October 2014.
Association for Computational Linguistics. doi: 10.1115vlL/D44 1162. [ /tt.tts://aaaaaett
orq/D14-111
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization be- yond overfitting on small algorithmic datasets, 2022. Uttptt //ap:////aiv//// www.arwwwww...aaaaaaa/r org/ wtggaba//g//.....///////g// 02177
Adam Scherlis. Kshitij Sachan, Adam S. Jermyn. Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks, 2025. [tttts:::://aaae org. abs 2210. 01892
Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq. Nicholas Goldowsky- Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonse Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J. Michaud, Stephen Casper. Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, and Tom MeGrath. Open problems in mechanistic interpretability, 2025. URL https://arxiv: wrw./bs/////ooo.oooo.o.../... abs 2501. 16496
Edward Stevinson, Lucas Prieto, Melih Barsbey, and Tolga Birdal. Adversarial attacks leverage interference between features in superposition, 2025. URL ptctps://arxiv.orrt www.orgggaba//.ooo 11709
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindscy, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL pub/2024/ scaling www.semant.ttoommmmmoommmmm www.emaannntttttttteeeoeetm wnw.x..ddeee..eee html
şi circiits
Ozan Tuncer, Vitus J Leung, and Ayse K Coskun. Pacmap:
Topology mapping of unstructured communication patterns onto non-contiguous allocations. In Proceedings the 29th ACM on International Conference on Supercomputing, pp. 37-46, 2015.
Madeleine Udell and Alex Townsend, Why big data matrices approximately low rank? SIAM Journal on Mathematics of Data Science, 1(1):144-166 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Re- html
search, 2779-257,-,, 2008. /ttp///ttpp//...oocoooo/////e//r
Yize Zhao, Tina Behnia, Vala Vakilian, and Christos Thrampoulidis. Implicit geometry of next-token prediction: From language sparsity patterns to model representations. In First Conference on Language Modeling, 2024. net forum? f wwt.idddiiyyyiiie.ne.
George Kingsley Zipf. Human Behavior and Principle of Least Effort. Addison- Wesley, Cambridge, MA, 1949.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A IMPLEMENTATION DETAILS
WikdText-BOWS. All the models trained in the WikiTexi BOWS setup use a cosine annealing scheduler with a starting learning rate of le - 3 and are trained for 20 epochs with a batch size of 1024.
Synthetic "Months" dataset. Each document is a 12-bit vector E {0, 1 1 whose entries stand for the calendar months. One sample generated follows.
1. Latent month angle. Pick a discrete month m E {O,..., 113 (uniformly or by cycling) and add Gaussian blur:
2Tm/ 12 E, 2. Embed on the unit cirele. == [cos 0, sin 0]
N(0,08..
E IR?
3. Project onto month directions. Let
(coss
2nk 12
sin 2
E 12X2
=
A-o
whose k--th row corresponds to month & Compute log-odds la = W&2 + b. where b < 0 fixes the global sparsity and 3 > controls sharpness.
4. Binary activations. Draw the bits independently:
Bernoull((((42)),
aru))
1 146-01
k k=
12.
With 00=0 and large 8 the code is nearly one-hot; decreasing 3 or increasing de mixes neighboring months, producing a rank-2 correlation structure that is analytically tractable yet retains the extreme sparsity of real bag-of-words data.
Let a, E O...., , 112 make an input pair of integers with the task being addition modulo 113. We learn a shared embedding matrix E e For a given pair, we look up E[a] and [b... concatenate them into 200-dimensional feature vector, and feed this through a sarre-hidden-lee logits over the 113 possible sums. We train the entire model and-to-end via cross-entropp loss using the Adam W optimizer with weight decay set to 4.
Modular addition.
that maps each integer a 100-dimensional vector.
MLP. Each hidden layer has 200 ReLU neurons. The output layer produces
Relative map positions. From the Cities1000 Dataset (GeoNames) 2025 take the top 1,000 most populated cities in the US and sample two subsets city pairs out the 1,000, 000 possible pairs. For each pair of cities a, b, the task is to predict their relative position on the US map out of eight possible classes (North, South, East, West, North East, North West, South East, South West). We learn 200 dimenssooa embedding matrix E E I 100000200 x200 to map each city to 50- d--dienniional vector; for each pair (a, 6). we concatenate their embeddings [la] and ET]] into dinddimensional feature vector, which is then fed through single hidden-layer MLP with 200 ReL.U units. The MLP's output layer produces logits over the eight classes, and the entire model is trained end-to-end using cross-entropy loss and an Adam optimizer.
UMAP plots and semantic clusters. For the UMAP plots in Figure and Figure 3, the categories created by using Gemini 2.5 Pro to split top 4000 words into categories, with each category inspected and refined by hand. The exact word to category mappings can be found in the code provided in the supplementary material. The UMAP plots are made with 15 neighbors, min distance of 0.01. and a cosine metric.
Linear probes. In this work, we argue that non-linear AEs sometimes linearly encade low-rank structure of the data. To quantify how linear the representations learned by the ReLUUAE are, we deploy a simple linear probe. After fully training the ReI.U-AF, we freeze its encoder and collect the latent activations h = Wx E TR. A probe is single linear layer P E R xm that is trained from scratch to reconstruct the input without any non- linearity::
Xymbe
Eerobe (x. P) = ||x Sprobe ||
To measure how much of the ReLU-AB's predictive power be captured by linear mapping use the Fraction of Explained Variance (FEV) of the probe relative the ReLUUAE:
FEV = 1 Ex| Sx poobe 1
(
Σ. Xx - XReLU
where i indexes data points and RReLU is the mean ReLU-AE reconstruction over the evaluation set. An FEV of 1 indicates that purely linear map can reproduce the ReI.U-AE's outputs perfectly; FEV of 0 means the prohe does no better than predicting the mean. We report the FEV on a held-out validation split.
14

---

Published as a conference paper at ICLR 2026
B ToY TRANSFORMER SETTING
Nov
Oct
Dec
Sep
UMAP-2
PC2
Aug
Jan
Jul
Feb
Jun
May
Mar
Apr
UMAP-1
PC1
(a) UMAP of BOWS word embeddings
(b) PCA of unembodding weights
Figure 8: Tow-layer transformer trained on Wikitext-103 for the multi-task token-recovery task exhibit semantic clusters and ordered circular representations for the months of the year. This replicates the main results in Figure 4 and Figure 3 on different dataset. Colors correspond to different semantic categories (Figure 311)
We train two-layer transformer on WikiText- 103 to predict, at cach position, the set vocabulary items that has appeared so far in the causal context.
Architecture. One encoder block with pre-norm attention and MLP:
Token embedding dimension daodel 768, tied to the output projection.
Multi-head self-attention with 8 hcads, dropout O, causal masking, and learned positional embeddings.
Feed-forward network of width Admola (GELU activation). followed by layer normalization.
Data and tokenization. WikiText- 103 is tokenized at the word level with a fixed vocabulary of 16,000 tokens. The tokenizer and reserves <pad> and <unk>. Sequences are constructed with window length 512 and stride 512, padding full length.
Targets. For sequence position, target is multi-hot vector over the vocabulary indicating whether the token has appeared anywhere in the prefix (inclusive). This is computed from the sequence with padding tokens masked out.
Loss and optimization. Training binary cross-entropy with logits over the multi-hot targets. Optimization uses Adam with learning rate 3 10 10-4, weight decay x 10 batch size 8, and cosine annealing schedule. Gradients are clipped to norm 1.0.
In Figure S we see that, similarly to the BOWS setup. semantic clusters and circular structures appear in the residual stream of the transformer byproduct of compression.
C COMBINING CONSTRUCTIVE INTERFERENCE AND INTERFERENCE FILTERING
The main text argues that realistic models need not rely single pure mechanism. In practice, correlated features contribute constructively reconstruction, while the ReLU and negative bias still suppress false positives. The examples below illustrate this coexistence for features that are only partially reconstructable isolation.
To complement the 'Beatles" example in Figure 5
we now show the same analysis for 'Christmas'. While "Christmas' is frequent enough to be allocated a large weight norm and achieve positive reconstruction whenever it present the input, even in unrelated context, only achieves reconstruction of 0.2 in the one-hot case. In
15

---

Published as a conference paper at ICLR 2026
200
400
600
1000
Figure 9: UMAP embeddings of features from ABS trained with context size of 20 records with wd=0 (top) and wd=4 (bottom) across different latent sizes. The plot shows that semantic structure remains for larger fraction of context sizes when using weight decay. Colors correspond to different semantic categories (Figure 3d).
Rovorea O (3%)
one-hot (-0.281) threshold
threshold blus
Brangee PRITI
Interference
0.6
Pre- Pre-activation
January
Density
044
+0.023
Christmap
Christmas
gift December
0.3 030
'Christmas' correlated
'Chiistmae uncorrelated
comelated contaxt
Per-word
CCriismaas decoder pre-activation
(without "Christmass'
Figure 10: Words like 'December' and 'Day' contribute to the reconstruction of 'Christmas'. (left) "Christmas' is better reconstructed with than without interference. (middle) For most samples (81%) where 'Christmas' appears in the validation set, interference is beneficial, allowing for reconstructions closer to 1. (right) When 'Christmas' appears in the right context it has a positive reconstruction thanks the interference from related terms. However, if the related context present but 'Christmas' is not, the model uses the ReLU and a negative bias to avoid false positives at the cost of having smaller reconstructions for 'Christmas' when appears in unusual context.
order to get closer to the target of 1, the reconstruction of 'Christmas" relies on words like 'December' or day Figure II
D PCA & GEOMETRY PRESERVATION
compute correlations data by the Pearson correlation coefficient. Pearson correlation matrix Gram (inncr-product) matrix of the standardized vectors. Due to the normalization, the Euclidean distance (chordal) distance monotonically links to the angular distances:
d, = x x, 11 - cos x xxxx) 2
where | = 1, and Ris is the tb and 7th entry of R. So correlation induces Euclidean geometry (up to rotation/reflection)) on the embedded points. Hence, performing PCA on R is equivalent to performing classical multidimensional scaling (MDS) (Cox Cox 2008) chordal distances, which explicitly aims embed
16

---

Published as a conference paper at ICLR 2026
Knder Arcer
threshold bis -0.301
559
cre-hot 8?
0
25
hPa
24
inHg
Interference is beneficial
Pre- Pre-activation
0.4
orange
Density
milk
barometric
02-
+000007
inifg
mbar
mbar
barometric
0.2
644
Per-word
'mbar' decoder pre-activation
(without
context.
context.
Figure 11: Atmospheric terms like 'barometric' and 'inHg' contribute to the reconstruction of 'mbar'. (left) 'mbar' better reconstructed with than without interference. (middle) For every sample where 'mbar' appears in the validation set, interference is beneficial, allowing for reconstructions closer to (right) When 'mbar' appears in the right context has a positive reconstruction thanks to the interference from related terms. However, if the related context is present but 'mbar' is not, the model uses the ReLUU and negative bias minimize false positives the cost of not being able to reconstruct 'mbar' when it appears in an unusual context.
vectors such that between-voetor distances are preserved as well as possible. Hence PCA yields embeddings which reflect the geometry induced by the correlations in data.
E OEENWE REPLICATIONS
Sep.
Oct.
Jul.
May. Apr.
Jul.
Nov.
Aug.
g
Jun.
Dec.
2
Feb.
Sep.
May.
an.
Jan.
Apr.
Feb.
Oct.
NovDec.
Mar.
(a) UMAP of BOWS word em- beddings
(b) PCA of the data
(c) PCA of the weights
Figure 12: Autoencoders trained on OpenWebTexx exhibit semantic clusters and ordered circular representations for the days of the week. This replicates the main results in Figure 4 and Figure on a different dataset. Colors correspond to different semantic categories (Figure 3l).
replicate the main results of the paper showing that the appearance of circular structure for the months of the year and semantic clusters is phenomenon limited WikiText. These results from an OpenWeb BOWS setup in the main text, we observe semantic clustering of word embeddings and that circular structures appear both when taking the PCA of the data and the trained encoder weights.
In Figure 12
10, 000. e
a stride 10. Similarly to the WikiText case studied
WEIGHT DECAY AND SUPERPOSITION
The main text argues that weight decay favors solutions that exploit shared low-rank structure because these achieve good reconstruction with smaller weight norm than feature-by beatuee interference filtering. The extended UMAP comparison in Figure supports supports this interpretation: semantic structure remains visible across broader range of latent sizes when models trained with weight decay.
17

---

Published as a conference paper at ICLR 2026
Dct
Per He
Pa Dec
Wk
200
400
600
800
1000
2000
4000
Figure 13: Reconstructions different latent-vector sizes. Top: "Months" dataset; middle: "Roman
numerals"; bottom: corresponding latent size m.
G MORE DETAILED EXAMPLE OF GROUPS OF FEATURE GEOMETRY IN BOWS
In the main paper only show the feature structures for representative latent sizes due space constraints. Figure 13 we show the structures studied in Figure for extended range latent sizes.
also show zoomed version one of the UMAP plots Figure [6 This figure highlights the rich structure of the features beyond simple clustering of high-level classes. We see that words corresponding to sciences are clustered together, but within this high level cluster, sub-groups like words about medicine (top left), astronomy (lower left), chemistry (lower center) and biology (center) are also grouped in smaller clusters.
G.1 SOME EXAMPLES BEYOND 2D
Beyond the 2D examples presented in the main paper. include 2 examples showing that the days of the week and months of the year have structure beyond 2D cirele (Figure [T4). This is clear in the case of the months where ondulation third principal component is present heyond 2D circular structure.
H OTHER CORRELATION STRUCTURES IN SYNTHETIC DATA
In Figure 155 we show the superposition patterns for the values of m missing in Figure 1, as well as examples for autoencoders trained on data with eegure-f-eight correlation structure spherical structure. In all cases we see that the Gramm matrix structure is similar in the linear but they diverge at larger latent
ReLU cases sizes as the ReL.U-ES start leveraging non- non-linear superposition which indicated by sparse interference patterns (for example for d 8).
2
We reuse the "synthetic months" pipeline described in Appendix but change only latent curve z(-) and the feature directions W: Steps (phase selection), 4 (Bernoulli sampling), and the log-odds & 3 W 2+ b are identical, with defaults 3 5.0, b -2.0, noise 0.1, seed 42.
(a) Months
the Figure-8 (Lissajous)
Lawrence 1972), we replace Steps 2-3 by:
sin 0 sin (20)
F-1
z(0)
(sin you, sin(2px))
k=o" ipk
For the Sphere ($²), we replace Steps 2-3 by 3D unit-sphere embedding:
2 ~ )*
W- (cos 0% sin Φx, sin 0g sin Φk, cos Φxx KmD
where a Fibonacci lattice (Maqques et al. 2013) gives approximately uniform feature directions:
(b) Days
dv arccus 218+0.5)
Figure 14: 3-D PCA of the beddings for the words and the days in WikiText BOWS setup.
6, = <(1 + v5) (1: + 0.5).
Full implementation details are provided in the supplementary material.
18

---

Published as a conference paper at ICLR 2026
8010
2231
BaLh Useer Eigar BAL Ploeed Liew Ball Oreeer Uemee
Figure 15: Extension of Figure I include all values of between 2 and 12, well comparison with the weight patterns for AEs trained on data drawn from data with different correlation strucutres: -gure-of-eight and spherical.
I A TAIL OF PARTIALLY RECONSTRUCTED FEATURES
An important implication our results is that meaningful geometry does not necessarily imply that feature accurately represented. In fact, the opposite can occur: when capacity is limited, a model may place a feature in the correct semantic region because is projecting it onto shared directions used by correlated features, while still failing capture most of its individual variance. this sense, semantically meaningful geometry reflect a best guess induced by surrounding features rather than a precise representation the concept itself.
This what we observe Figure 17 Even when we restrict to features with poor reconstruction scores (H²² < 0.3), they still form semantic clusters. A natural explanation is that the autoencoder has learned low-rank representation of dominant co-activation structure and projects many features, including rela- tively uncommon ones, onto this shared subspace. Such features inherit meaningful geometry from the data statisties, even when the model does allocate enough capacity to reconstruct them accurately.
Higt
merrally
Figure provides concrete example of this effect. The figure shows zoomed-in portion of the UMAP for the model with latent size 200, regime in which many words in these clusters are still only partially reconstructed. Nevertheless, the local geometry highly meaningful: words related medicine, astronomy. chemistry, and biology form distinct sub-clusters within the broader science region. This suggests that the model has already learned where these features belong relative to one another before has sufficient capacity represent each them accurately on own. In other words, shared structure the data can organize the representation geometry well before it supports accurate feature-level recovery.
tohting
Figure 16: Zooming into the cluster for science features in the UMAP plot with a latent size of 200, we observe sub- clusters within it. Medical features in the top left while astronomy features are in the lower left and chemistry features are in the lower center.
IMPLICATIONS FOR THE LINEAR REPRESENTATION HYPOTHESIS
While the linear representation hypothesis (LRH) is of the pillars of current mechanistic interpretability (MI) approaches. There
no consensus on correct formulation of this hypothesis. The LRH can be
taken to mean that internal features of model correspond to activations along one- -ie-dimensional directions in activation space (Engels et al. 2025). However, the LRH can also be formalized around the mathematical notion of linearity meaning the representation of two features is the addition of their representations and scaling feature corresponds to scaling its representation (Elhage
2021
19

---

Published as a conference paper at ICLR 2026
100
200
400
600
800
1000
Figure 17: UMAP embeddings at different latent-vector sizes including only features with R < 0.3 (top) and 2 > 0.3 (bottom). Semantic clusters at different latent sizes are still observed in both, although this effect is combined with an increase in the number of features above the threshold the lower one.
While works have suggested that observed feature geometry like the ordered circles formed the months undermine first definition (Engels et al. 2025 Sharkey et al. (2025). our results show that these structures can emerge from the compression and reconstruction structures do not necessarily undermine either formulation of the LRH. On the other hand, our results in Section 5 do suggest that some features used by DL models can be value-coding meaning they encode concrete trigonometric values or coordinates along lincar directions which do fulfill the constraints for mathematical linearity. For example scaling the value a cosine-coding feature leads to diffcrent (and potentially invalid) cosine value, rather than a stronger activation of the same cosine value.
one-dimensionaly linear features. This means that these
An interesting line of research would be to explore if presence-coding features can have value-coding components. Findings like the fact that city representations language models be projected lincarly onto coordinates subspace (Gurnee& Tegmark (2024), that integers can projected onto helix subspace (Li al. 2025) could be understood through this lens. In this view, city representations could have a coordinate-couee coding component and integers could have a size-coding component as well as sine and cosine coding components which combine to make a helix structure.
Overall, findings show that rich feature geometry be explained away lincar superposition recovering the structure inherent in the data, without appealing to non- non-linearly encoded information with a functional role calculation. However, we believe the existence value-coding features could be in conffict or an exception to features being mathematically linear.
K OTHER DIMENSIONALITY REDUCTION METHODS
To verify that the semantic clusters not dependent on the choice of dimensionality reduction method, include t-SNE (van der Maaten & Hinton 2008 and PaCMAP Tuncer et al. (2015) two alternatives in fig.
Wusic Audoo
Media
Pobticx Govemment
Geography Piccs
Torrsooratinn Travel
Time, Numbers 6 Neasrremen
Seletce, famure Envionmeer
People, Society & ftslss
Propor numes
Verbs
Figure 18: We show the latent representations of the top 4000 most frequent words using UMAP (left) t-SNE (middle) and PaCMAP (right) to highlight that these semantic clustering results are not dependent on the choice of dimensionality reduction technique.
20

---

Published as a conference paper at ICLR 2026
Linear Probe
PC dimension (B5 var)
Other words
Months
Word Frequeec
Ce2 C-10
Labent Size
sample
(b)
(c)
Figure 19: Larger context windows strengthen shared structure and make constructive interfer- ence more effective. (Left) Increasing the amount of text encoded by each sample (C) increases the fraction of ReLU-AE reconstruction variance that can be recovered by a linear probe from the latent space. (Middle) Increasing C' decreases the number of principal components needed explain 95% of the variance while increasing the average number of active features per sample. (Right) Word frequencies follow a power-law distribution (Zipf) 1949), helping explain why some feature groups. such months, receive more capacity and retain structured geometry longer than rarer groups such as Roman numerals.
L CONTEXT SIZE, EFFECTIVE RANK, AND FEATURE FREQUENCIES
The main text argues that constructive interference becomes more useful when the data contains stronger shared structure. One simple way to increase such structure in BOWS is to enlarge the context window used to construct cach sample. This makes co-occurrence patterns denser and more informative, increasing the extent which reconstruction can be supported by shared low-rank directions rather than only by ceature-specift. interference filtering. Figure quqantiiis this trend from three complementary perspectives: the lincarity of the learned reconstructions, the effective rank and sparsity of the data, and the frequency heterogeneity words that helps explain why some groups of features retain structured geometry longer than others.
21


Reviews:
[{"review_id": "BxV8AdKxLA", "reviewer": "Reviewer_WkYu", "summary": "The paper investigates the geometric structure of internal features in mech interp, aiming to bridge the gap between \"toy model\" theories of superposition and observations in real-world language models. Prior work suggested superposition creates interference-managing structures (like polytopes), but empirical studies of LLMs instead find semantically-rich structures (like semantic clusters or ordered circles for months).\n\nThe authors introduce a framework called Bag-of-Words Superposition (BOWS), where a simple ReLU autoencoder is trained to compress high-dimensional, sparse, binary bag-of-words vectors derived from real text. The use of the BOW space in such analysis is interesting as it provides a proxy for the platonic, objective, or \"ideal\" embedding space for features that we cannot access.\n\nThe paper makes two key claims:\n\nIt distinguishes between non-linear superposition (as seen in toy models), which uses non-linearities like ReLU to create local structures (e.g., antipodal pairs) to manage interference between uncorrelated features, and linear superposition, which emerges when features are correlated.\n\nIn the linear superposition regime—which is induced by tight bottlenecks or, significantly, by weight decay—the non-linear AE simply learns to linearly encode the low-rank structure (i.e., the principal components) of the data's covariance matrix.\n\nThe authors demonstrate that this linear superposition mechanism is sufficient to reproduce the exact semantic clusters and circular representations seen in LLMs. This suggests these structures are a \"parsimonious\" explanation, arising merely as a byproduct of efficient compression, and may not have a specific functional, computational role.", "questions": "My question would be whether \"byproduct\" structures also be \"functional\"? Couldn't the model leverage the fact that the optimal compression scheme (PCA) naturally arranges features in a computationally convenient way (e.g., a circle for months)? Or perhaps a little bit more challenging (in future works), what happens when the low-rank structure does not facilitate (or even adverserial for) a geometry that makes underlying computation (like modular arithmetic) easier?\n\nMy question, however, does not impact my perception of this paper as a novel and solid contribution to mech interp community.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1, The paper's primary strength is its clarity. The distinction between \"linear superposition\" (PCA on correlated data) and \"non-linear superposition\" (local, ReLU-dependent structures for uncorrelated data) is a very useful and clear conceptual framework. And the flow of the paper is natural, too.\n\n2, The BOWS framework offers a good trade-off, it's more realistic than toy models but far more controllable than a full LLM. Using it to show the emergence of semantic clusters and circles from data statistics alone is highly effective. The experiment showing that a ReLU AE transitions from a linear (PCA) regime to a non-linear (antipodal) one as the latent dimension $m$ increases is clean and convincing.\n\n3, This work provides an interesting for feature geometry. It challenges the assumption that structures like semantic clusters or \"month circles\" are necessarily functional or computationally constructed by the model. The idea that they are simply a byproduct of compression (driven by data correlations and weight decay) is a fundamental claim that the mech interp community must consider.", "weaknesses": "1, The main limitation is the simplicity of the BOWS setup. An AE trained on static BoW vectors does not necessarily capture all the SAE variants that exist in literature today, since many of them explicitly make architectural changes to cater to the space where features live.\n\n2, Also, the paper seems to strongly implies that these structures are just byproducts and not functional. This dichotomy might be false. A model could (and likely would) exploit this emergent, PCA-driven structure for computation. Framing it as byproduct OR functional is perhaps too strong and I think the two are not mutually exclusive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ER6RQvQTYL", "reviewer": "Reviewer_2aH1", "summary": "This work studies correlations between (input) features in a “toy models of superposition” (Elhage et al. 2022) type setting, and introduces a distinction between linear and non-linear superposition.  While Elhage et al. focused on non-linear superposition, this work argues that linear superposition is more common in real data.\n\nThe submission also introduces a Bag-of-Words Superposition (BOWS) setup, which uses co-occurrence in real documents to generate correlated binary feature vectors.  (BOWS is a bit of a misnomer since bag-of-words representations capture multiplicity, but the representation in this work does not).  The work also postulates that these co-occurrence statistics may be largely responsible for the particular geometric patterns observed in LLM representations (e.g. months being arranged in a circle), although I don’t believe this hypothesis was actually tested.", "questions": "“However, polytope-like structures have not been observed in standard LLM activations” seems to refer specifically to Elhage et al.’s 2022 work; however, Park et al. (2025) claim to have found polytopes in the activations of Gemma.  Does this invalidate the quoted claim?  How are the polytopes discussed in Park et al. related to those from Elhage et al. (2022)?\n\nHow do the experiments in Section 5 differ from what was already done by Gurnee & Tegmark (2024)?\n\n“In contrast, the patterns in Figure 7 emerge despite inputs being uncorrelated,” Can you please make the claim about inputs being uncorrelated precise and support it?\n\nI’m not convinced that “circles and clusters resemble the global semantic organization first reported in distributional word embeddings”.  Can you elaborate on and substantiate this claim?\n\nThe abstract states: “Our findings suggest that the semantically meaningful structures observed in language models could arise driven by compression alone, without necessarily having a functional role beyond efficiently arranging feature representations.”  What would it mean for this to be the case?  Are there experiments that could falsify this as a hypothesis?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The observation that linear superposition is more common (if correct), seems important.\n\nThe BOWS setup seems like a nice, sensible approach to introducing realistic correlation structure, and has the bonus of enabling researchers to bring their knowledge of words’ semantics to the analysis of experiments.", "weaknesses": "(major): I believe this work is in need of more rigor in establishing its central definitions and concepts (e.g. “superposition”, “linear superposition”).  For instance, I’m not sure what is the content of the statement “This explicitly shows that linear dimensionality reduction enables a form of superposition (d = 12 > m = 2) by exploiting feature correlations, without requiring any non-linearity.”  This just sounds like it’s saying “you can reconstruct an input well using a few principle components when input dimensions are highly correlated”, but that’s nothing new.  Similarly, “features in linear superposition inherit their structure from their covariance matrix” seems vacuous if linear superposition basically just means PCA.  While superposition is a more established concept, it relies (in my experience) on assuming some ground-truth set of underlying features.  In this case, I guess these are the words in the BOWS representation, but this should still be made more explicit.  I would like the discussion of superposition and linear superposition to be grounded in a rigorous mathematical exposition of the concepts.\n\n(major): The submission claims that Elhage et al. (2022) don't study linear superposition or correlated features. However, Elhage et al. do consider correlated features, and state “when there isn't enough space to represent all the correlated features, it will collapse them and represent their principal component instead”.  This is a significant mischaracterization of key related work, and makes me question the novelty of this submission.\n\n(moderate): I think some of the statements in this submission overstate the successes of mechanistic interpretability, eg:\n- “These approaches have successfully uncovered interpretable units corresponding to semantic concepts, syntactic roles, or specific input patterns.”\n- “Initial works in MI studied interpretable monosematic neurons”\nI think referring to neurons as 'interpretable' is generally accurate.  They may seem more or less interpretable, but so far I’ve not seen sufficient evidence to justify claims that we really understand what neurons are doing, except perhaps in the context of particular tasks / distributions of inputs.  This is an important distinction, as overstating the success of interpretability can provide false assurances.\n\n(minor): The related work section should do more to connect the referenced works to the submission.  \n\n\n(nit): The acronyms SDL and LRH should be introduced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DRGRTJ7pVu", "reviewer": "Reviewer_vBLf", "summary": "This paper introduces BOWS, which induces the formation of complex structures of features in an autoencoder network that tries to compress and reconstruct binary bag-of-words vectors. With both a linear and nonlinear autoencoder, BOWS shows that networks do not need any goals besides compression alone in order to form these structures. The feature structures form as a result of feature correlations in the data, which are exploited by the autoencoders for compressive purposes. Experiments show the successful recovery of feature structures known to exist in LLMs (such as circular month-of-the-year features and latitude-longitude features), from the learned autoencoder representations. The authors offer a distinction between value-coding and presence-coding features, and argue that value-coding features are used to perform computation. BOWS sheds some light on the reason that LLMs form complex feature structures through superposition, arguing that feature compression, without any additional goal, is the cause.", "questions": "I am presuming below that as indicated in the abstract, the main goal of the paper is to show that compression alone can be responsible for complex feature structures in LLMs, when features statistics are correlated.\n- It seems like this goal is already achieved with just PCA, so why ever bother with the autoencoder?. In what case does the autoencoder reveal any structures to you that PCA/probing does not? As you mention in Section 3.1, the linear autoencoder will be recovering the top two principal components anyways.\n- It seems like you designed the autoencoder as the simplest possible example where compression can be demonstrated as the root cause for the emergence of the feature structures. Why complicate the matter with a nonlinear autoencoder? Is it to take into account the possibility that the presence of nonlinearity can disrupt the usage of the complex structures as the most efficient compressed representation?\n\nMuch of the rest of the analysis describes how the autoencoders behave, but this is not necessary in order to see that the structures are inherited from data statistics, since it is already evident from PCA and probing.\n\nFor line 397, is there converse evidence that the presence-coding features are _not_ used? Also, I find this experiment unconvincing of the importance of the difference between value-coding and presence-coding. It looks like you are defining the features that are known to be used in a particular way as the value-coding ones, and the rest as presence-coding, and then using that to argue that the value-coding ones are the useful ones, which I do not see the point of.\n\nRandom:\n- Figure 7 right: You are trying to probe one city's coordinates from an 8 dimensional vector which predicts the relative positioning between two cities. Which of the two city's coordinates are you trying to probe, and which city is plotted?\n- Figure 3 typo: (a-c) instead of (a)\n- Please try to keep the figures placed close to where they are referenced.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper provides convincing evidence that complex structures of features can emerge purely from the task of compressing features for storage, when there are correlations in the feature distribution.\n- Figure 6 provides convincing evidence that complex structures of features form and then disappear as the latent size changes, when training a network to compress information.", "weaknesses": "The paper is unnecessarily complex and convoluted for it's goal in helping interpretability efforts.\n- BOWS is much less a \"framework\" than an application of a pre-existing technique for demonstrating superposition, in Elhage (2022) [1], but on a custom bag-of-words dataset. Most of the \"framework\" parts of the work (techniques for analysis of results, PCA, visualization, training methodology) are inherited from Elhage (2022).\n- The framework's goal is to show that compression alone can be responsible for complex feature structures learned in LLMs, but this can already be shown by taking a PCA of the data (implicit compression), making the autoencoder analysis an unnecessarily complicated solution.\n- The distinction made between presence-coding and value-coding neurons is weak/useless. The difference as defined seems to boil down to an artifact of the way the English language treats some things as quantifiable/measurable and others as not, rather than intrinsic to an LLM's internal functioning, so it is trivially true that there exist neurons for both presence-coding and value-coding things. The \"value of the north-south coordinate\" is just \"the presence of northerliness\". While this may seem contrived, there are plenty of fuzzy edge cases to be made (\"lots of hair\" (presence) vs \"many hairs\" (value)).\n  - Altogether, I do not see why making this distinction of quantifiableness helps us understand the internal functioning of LLMs any better than before. In general, I don't see how the mathematical behavior of the autoencoder is supposed to depend on things outside of the correlation structure of the features, such as our external understanding of the feature's meaning as the presence vs value of something.\n  - If the authors want to instead define presence-coding versus value-coding based on how the features are being used, they should clarify precisely what conditions are sufficient to qualify the feature as one or the other. An example does not suffice as a definition, as in lines 343-347.\n  - The authors give no examples of how making this distinction helps us to reason about other aspects of the features. Being able to identify the features as \"value-coding\" does not help tell us how it's used. Rather, identifying how it's used allows us to label it as \"value-coding\", and there doesn't seem to be any purpose served after that.\n\n[1] Elhage, et al., \"Toy Models of Superposition\", Transformer Circuits Thread, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)