Superposition as Lossy Compression — Measure with Sparse Autoencoders and Connect to Adversarial Vulnerability
Authors: Leonard Bereska
Zoe Tzifa-Kratira
Reza Samavi
Stratis Gavves
Venue: Accepted by TMLR ()

Abstract: Neural networks achieve remarkable performance through superposition: encoding multiple features as overlapping directions in activation space rather than dedicating individual neurons to each feature. This phenomenon challenges interpretability; when neurons respond to multiple unrelated concepts, understanding network behavior becomes difficult. Yet despite its importance, we lack principled methods to measure superposition.
We present an information-theoretic framework measuring a neural representation's effective degrees of freedom. We apply Shannon entropy to sparse autoencoder activations to compute the number of effective features as the minimum number of neurons needed for interference-free encoding. Equivalently, this measures how many "virtual neurons" the network simulates through superposition. When networks encode more effective features than they have actual neurons, they must accept interference as the price of compression.
Our metric strongly correlates with ground truth in toy models, detects minimal superposition in algorithmic tasks (effective features approximately equal neurons), and reveals systematic reduction under dropout. Layer-wise patterns of effective features mirror studies of intrinsic dimensionality on Pythia-70M. The metric also captures developmental dynamics, detecting sharp feature consolidation during the grokking phase transition.
Surprisingly, adversarial training can increase effective features while improving robustness, contradicting the hypothesis that superposition causes vulnerability. Instead, the effect of adversarial training on superposition depends on task complexity and network capacity; simple tasks with ample capacity allow feature expansion (abundance regime), while complex tasks or limited capacity force feature reduction (scarcity regime).
By defining superposition as lossy compression, this work enables principled, practical measurement of how neural networks organize information under computational constraints, in particular, connecting superposition to adversarial robustness.

Published in Transactions on Machine Learning Research (12/2025)
Superposition as Lossy Compression Meaaure with Sparse Autoencoders and Connect to Adversarial Vulnerability
Leonard Bereska'; Zoe Tzifa- Frata-Kratira, Reza Samavi?:3| Efstratios Gavves' 1
University of Amsterdam
Toronto Metropoliton: University
Vector Institute for Artificial Intelligence
Reviewed OpenReview: /ttps://openreeee net/ pormn id= qaNP665quu
Abstract
Neural networks achieve remarkable performance through superposition: encoding multiple features as overlapping directions in activation space rather than dedicating individual neu- rons to each feature. This phenomenon challenges interpretability: when neurons respond to multiple unrelated concepts, understanding network behavior becomes difficult. Yet de- spite its importance, we lack principled methods to measure superposition. We present an information-theorciicc framework measuring neural representation's effective degrees of freedom. We apply the Shannon entropy to sparse autoencoder activations to compute the number of effective features as the minimum mumber neurons needed for interference-free encoding. Equivalently, this measures how many "virtual neurons" the network simulates through superposition. When networks encode more effective features than they have actual neurons, they must. accept interference the price of compression. Our metric strongly cor- relates with ground truth in toy models, detects minimal superposition in algorithmic tasks (effective fcatures approximately equal neurons), and reveals systematic reduction under dropout. Layer-wise patterns of effective features mirror studies of intrinsic dimensionality on Pythia-70M. The metric also captures developmental dynamics, detecting sharp feature consolidation during the grokking phase transition. Surprisingly, adversarial training can increase effective features while improving robustness, contradicting the hypothesis that su- perposition causes vulnerability. Instead, the effect. of adversarial training on superposition depends task complexity and network capacity: simple tasks with ample capacity allow feature expansion (abundance regime), while complex tasks limited capacity force feature reduction (scarcity regime). By defining superposition as lossy compression, this work en- ables principled, practical measurement of how neural networks organize information under computational constraints, in particular, connecting superposition to adversarial robustness.
1 Introduction
Interpretability and adversarial robustness could be two sides of the same coin (Räuker et al., 2023). Adver- sarially trained models learn more interpretable features (Engstrom et al., 2019; llyas et al., 2019), develop representations that transfer better (Salman et al., 2020), and align more elosely with human perception (Santurkar et al., 2019). Conversely, intepretabilty--elaaningg techniques improve robustness: input gradi- ent regularization (Ross Doshi- Velez, 2017: Boopathy al., 2020), attribution smoothing (Etmann al., 2019), and feature disentanglement (Augustin et al., 2020) all defend against adversarial attacks. Even architectural choices that promote interpretability, such as lateral inhibition (Eigen & Sadovník, 2021) and second-order optimization (Tsiligkaridis & Roberts, 2020), yield more robust models. This pervasive duality demands mechanistic explanation.
"Corresponding author. Email: leonard, bereaka@gmail.com EEuua supervision

---

Published in Transactions on Machine Learning Research (12/2025)
neurons
features
superposition measure
y
(a) Observed network
(b) Hypothetical disentangled model
(c) Superposition as features per neuron
Figure 1:
Defining superposition for neural network layer. (a) Observed net with compressed repre-
sentation where multiple features share neuronal dimensions. (b) Hypothetical disentangled model where each effective feature occupies its own neuron without interference (Elhage et al., 2022b). (c) Superposition measure & quantifies effective features per neuron. Here, the network simulates twice as many effective features as it has neurons. Figure adapted from (Bereska & Gavves, 2024).
The superposition hypothesis offers a potential mechanism. Elhage et al. (2022b) showed that neural net- works compress information through superposition: encoding multiple features overlapping activation patterns. When features share dimensions, their interference creates attack surfaces that adversaries might exploit. If, by this mechanism, superposition cansed adversarial vulnerability, this would explain i.) adver- sarial transferability as shared feature correlations (Liu et al., 2017), vi.) the robustness-accuracy trade-off as models sacrificing representational capacity for orthogonality (Tsipras et al., 2019), and ii.) robust models becoming more interpretable by reducing fcature entanglement (Engstrom et al., 2019). Also, this nuperposiiion-vuheraaa
hypothesis predicts that adversorial training should reduce superposition.
Testing this prediction requires measuring superposition in real networks. While Elhage et al. (2022b) used weight matrix Frobenius norms, this approach requires ground truth features; available only in toy models. We need principled methods to quantify superposition without knowing the true features.
We solve this through information theory applied to sparse autoencoders (SAFs). SAEs extract interpretable features from neural activations (Cunningham et al., 2024; Bricken et al., 2023), decomposing them into sparse dictionary elements. We measure each feature's 's share of the network's representational budget through its activation magnitude across samples.
The exponential of the Shannon entropy quantifies how many interference-free channels would transmit this feature distribution, the network's effective degrees of freedom. We call this count effective features F' (Figure 1b): the minimum neurons needed to encode the observed features without interference. We interpret this as "virtual neurons": the network simulates this many independent channels through its N physical neurons (Figure 1b). The feature distribution compresses losslessly down to exactly F neurons; compress further and interference becomes unavoidable.
We measure superposition +e F/N (Figure 1c), counting virtual neurons per physical neuron. At tt = the network operates at its freee limit (no superposition). At 1e = 2, it simulates twice as many channels as it has neurons, achieving 2x lossy compression. Thus, we define superposition as compression beyond the lossless limit.
Our findings contradict the simple ppperposition-vulerabbii hypothesis. Adversarial traïning does not uni- versally reduce superposition; its effect depends on task complexity relative to network capacity (Section 7). Simple tasks with ample capacity permit abundance: networks expand features for robustness. Complex tasks under constraints force scorcity: networks compress further, reducing features. This bifurcation holds across architectures (MLPs, CNNs, ResNet-18) and datascts (MNIST, Fashion-MNIST,, CIFAR-10).
We validate the framework where superposition is observable. Toy models achieve r = 0.94 correlation through the SAE extraction pipeline (Section 5.1), and under SAE dictionary scaling the measure con- verges with appropriate regularization (Section 5.2). Beyond adversarial training, systematic measurement across contexts generates hypotheses about neural organization: dropout seems to act as capacity constraint, reducing superposition (Section 6.1), compressing networks trained on algorithmic tasks seems to not cre-
2

---

Published in Transactions on Machine Learning Research (12/2025)
ate superposition (eb Σ 1) likely due to lack of input sparsity (Section 6.2), during grokking, capture the moment of algorithmic discovery through sharp drop in superposition at the generalization transition (Section 6.3), and Pythia-70M's layer-wise compression peaks in early MLPs before declining (Section 6.4); mirroring intrinsic dimensionality studics (Ansuini et al., 2019).
This work makes superposition measurable. By grounding neural compression information theory, we en- able quantitative study of how networks encode information under capacity constraints, potentially enabling systematic engineering of interpretable architectures.
2 Related Work
Superposition and polysemanticity. Neural nelworks employ distributed representations, encoding information across multiple units rather than in isolated neurons (Hinton, 1984; Olah, 2023). The discovery that semantic relationships manifest as directions in embedding space, exemplified by vector arithmetic like "king man + woman queen" (Mikolov al., 2013), established the lineor representation hypothesis (Park et al., 2023). Building this geometric insight, Elhage et al. (2022b) formulated the superposition hypothesis: networks encode more features than dimensions by representing features nearly orthogonal directions. Their toy models revealed phase transitions between monosemantic neurons (one feature per neuron) and polysemantic neurons (multiple features per neuron), governed by feature sparsity. Recent theoretical work proves networks can compute accurately despite the interference inherent in superposition (Vaintrob et al., 2024; Hänni et al., 2024).
While superposition (more effective features than neurons) inevitably creates polysemantic neurons through feature interference, polyscmanticity (multiple features sharing neuron) also emerges by other means: rotation of features relative to the neuron basis, incidentally (Lecomte et al., 2023) (e.g. via regularization), or forced by noise (such as dropout) as redundant encoding (Marshall & Kirchner, 2024) (as we show in Section 6,1, dropout shows the opposite effect on superposition). Scherlis et al. (2023) analyzed how features compete for limited neuronal capacity, showing that importance-weighten feature allocation can explain which features become polysemantic under resource constraints.
Sparse autoencoders for feature extraction. Sparse autoencoders (SAEs) tackle the challenge of ex- tracting interpretable features from polyscmantic representations by recasting it as sparse dictionary lcarning (Sharkey el al., 2022; Cunningham et al., 2024). SAEs decompose neural activations into sparse combina- tions of learned dictionary elements, effectively reversing the superposition process. Recent architectural innovations such as gated SAEs (Rajamanoharan et al., 2024), TopK variants (Gao et al., 2024; Bussmann et al., 2024), and Matryoshka SAEs (Bussmann et al., 2025) improve feature recovery. While our experiments employ vanilla SAFs for conceptual clarity, our entropy- based framework remains architecture-agnosiic: im- proved feature extraction yields more accurate measurements without invalidating the theoretical foundation.
SAEs scale to ofate-of-the-art models: Anthropic extracted millions of interpretable features from Claude 3 Sonnet (Templeton et al., 2024), while OpenAT achieved similar results with GPT-4 (Gao et al., 2024). Crucially, these features are causally relevant: activation steering produces predictable behavioral changes (Marks et al., 2024). Applications span attention mechanism analysis (Kissane el al., 2024), reward model interpretation (Marks et al., 2023), and automated feature labeling (Paulo et al., 2024), establishing SAEs as foundational for mechanistic interpretability (Bereska & Gavves, 2024).
Information theory and neural measurement. Information-ttherrll, principles provide rigorous foun- dations for understanding neural representations. The information bottleneck principle (Tishby et al,, 2000), when applied to deep learning (Shwartz-Ziv & Tishby, 2017), reveals how networks balance compression with prediction. Each neural layer acts as a handwidth-limite channel, forcing networks to develop efficient codes (i.e. superposition) to transmit information forward (Goldfeld et al., 2019). This perspective recasts super- position as an optimal solution to rate-distortion constraints.
Most pertinent to our work, Ayonrinde et al. (2024) connected SAEs minimum description length (MDL). By viewing SAF features compression codes for neural activations, they showed that optimal SAFs
3

---

Published in Transactions on Machine Learning Research (12/2025)
sparsity $
importance 0
W
toy
W
toy
increasing spansity
S
S 0.9
0.99
X
W
sac
Z
WT
sae
Z
P
F
increasing
2.
3.
Wio W
ReLU
-10
ReLU
1
(a) Toy model architecture
(b) Effect of sparsity
(c) SAE
(d) Measurement
Figure 2: From loy model to practical superposition measurement. (a) Toy model bottlenecks features f through fewer neurons æ, with importance gradient determining allocation. (b) Sparsity enables interference-
based compression: matrices W Wtay show off-diagonal terms growing v increases from to 2.5. (c) Sparse autocncoders learn sparse codes reconstructing activations æ. (d) Measurement: extract activations Z, derive probabilities compute = eH(p), measure v = F/N.
balance reconstruction fidelity against description complexity. Our entropy-based framework extends this perspective, measuring the effective "alphabet size" networks use for internal communication.
Quantifying feature entanglement. Despite its thcoretical importance, measuring superposition re- mains unexplored. Elhage et al. (2022b) proposed dimensions per feature metric for analyzing uniform importance settings in toy models, which when inverted could measure features per dimension. But this approach requires knowing the ground truth feature-to-neuron mapping matrix, limiting its applicability to controlled settings. Traditional disentanglement metrics from representation learning (Carbonneau et al., 2022; Eastwood & Williams, 2018) assess statistical independence rather than the representations ional compres- sion characterizing superposition. Other dimensionality measures like effective rank (Roy Vetterli, 2007) and participation ratio (Gao al., 2017) quantify the number of significant dimensions in a represcntation but do not directly measure fealure-to-neuron compression ralios.
Entropy-based measures have proven effective across disciplines facing similar measurement challenges. Neu- roscience employs participation ratios (form of entropy, see Appendix A.3 for connection to Hill numbers) to quantify how many neurons contribute to population dynamics (Gao et al., 2017). Economics uses entropy to quantify portfolio concentration (Fontanari et al., 2021). Quantum physics applies von Neumann entropy to count effective pure states in entangled systems (Nielsen Chuang, 2011). Recent work applies entropy measures to neural network analysis (Lee al., 2023; Shin al., 2024). Across fields, entropy naturally captures how information distributes across components: exactly what need for measuring superposition.
3 Background on Superposition and Sparse Autoencoders
Neural networks must transmit information through layers with fixed dimensions. When neurons must encode information about many more features than available dimensions, networks employ superposition Saapaiiiiiipppiiinnggggg multiple features into shared dimonsions through interference. This compression mechanism enables repre- senting more features than available neurons at the cost, of introducing crosstalk between them. Superposition is compression beyond the lossless limit.
packing
We examine toy models where superposition emerges under controlled bandwidth constraints, making inter- ference patterns directly observable (Section 3.1). For real networks where ground truth remains unknown, we extract fcatures through sparse autoencoders before measurement becomes possible (Section 3.2).
3.1 Observing Superposition in Toy Models
To understand how neural networks represent more features than they have dimensions, Elhage et al. (2022b) introduced minimal models demonstrating superposition under controlled conditions. The toy model com-
4

---

Published in Transactions on Machine Learning Research (12/2025)
presses feature vector f E RM through bottleneck RN where M > N (Figure 2a):
a W toy f' = ReLUUWWi toy b)
(1)
Here M counts input features, N counts bottleneck neurons, and Wtoy RNN maps between them. The model must represent M features using only N dimensions; impossible unless features share neuronal resources.
Each input feature ft samples uniformly from [0, 1] with sparsity (probability of being zero) and importance weight wi. Training minimizes ayportance-weighhee reconstruction error L(f) = E willfi f!ll?, revealing how networks optimally allocate limited bandwidth.
As sparsity increases, the model packs features into shared dimensions through nearly-orthogonal arrange- ments (Figure 2b). The interference matrix W Wtoy revcals this gcometric solution: low compression, strong diagonal with minimal ofl-diagonal terms; high compression, substantial off-diagonal interference features share space. These interference terms quantify the distortion networks accept for increased capacity, The ReLU nonlincarity proves essential, suppressing small interference to maintain reconstruction despite feature overlap.
Elhage et al. (2022b) proposed measuring "dimensions per feature" as D N/| toy ll for analyzing uniform importance scttings, where the Frobenius norm || | Ilab = Ei W aggregates weight magni- tudes. While this metric was not intended for general superposition measurement, we nevertheless adopt its inverse baseline, as it provides the only existing weight-based comparison point for toy model validation:
W
l toy Frob N
Prob
This weight-based approach requires knowing the true feature-to-neuron mapping (unavailable real net- works) and lacks scale invariance (multiplying weights by any constant arbitrarily changes the measure). We need principled framework quantifying compression without ground truth features.
3.2 Extracting Features Through Sparse Autoencoders
Real networks do not reveal their features directly. Instead, must untangle them from distributed neural activations. Sparse autoencoders (SAEs) decompose activations into sparse combinations of learned die- tionary elements, effectively reverso-engineering the toy model's feature representation (Cunningham et al., 2024; Bricken et al., 2023).
Given layer activations æ RN, an SAE learns digherrdi sparse code z RD where D (Figure 2c):
ReLU en b)
(3)
The reconstruction combines these sparse features:
D z' Wdec
(4)
where columns d, of Waee form the learned dictionary.
Training balances faithful reconstruction against sparse activation:
C(x, 2) = x 113 + A||
(5)
The & penalty creates explicit competition: the bound on total activation Σ; | forces features to justify their magnitude by contributing to roconstruction. This implements resource allocation where larger |ay indicates greater consumption of the network's limited representational budget (see Appendix A.2 for rate- distortion derivation).
5

---

Published in Transactions on Machine Learning Research (12/2025)
SAE design choices. We tie encoder and decoder weights (Weee W eue) to enforce features as directions in activation space, maintaining conceptual clarity at potential cost. to reconstruction (Bricken et al., 2023). Weight tying can also prevent feature absorption artifacts (Chanin et al., 2024a). We omit decoder bias following Cunningham et al. (2024) for a transparent baseline, accepting slight performance degradation. The l regularization provides clean budget. semantics, though alternatives like TopK (Gao el. al., 2024) could work within our framework.
If networks truly employ superposition, SAEs should recover the underlying features enabling measurement. Recent work shows SAE features causally affect network behavior (Marks et al., 2024), suggesting they cap- ture genuine computational structure. Our measurement framework remains architecture-agnostic improved SAE variants enhance accuracy without invalidating the theoretical foundation.
4 Measuring Superposition Through Information Theory
We quantify superposition by determining how many neurons would be required to transmit the ob- served feature distribution without interference. Information theory provides a precise answer: Shannon's source coding theorem establishes that any distribution with entropy H(p) can be losslessly compressed to e"(r)) uniformly-allocatte channels. This represents the minimum bandwidth for interference- transmis- sion-the network's effective degrees of freedom.
We formalize superposition as the compression ratio sb F/N, where N counts physical neurons and F (²()) measures effective degrees of freedom extracted from SAE activation statistics (Figure 2d)'. When f 1, the network operates at the lossless boundary. When &é 1, features necessarily share dimensions through interference. For instance, in Figure 2b, 5 features represented in 2 neurons yields ab 2.5.
Feature probabilities from resource allocation. Consider a layer with N neurons whose activations have been processed by an SAE with dictionary size D. Across S samples, the SAE produces sparse codes Z ReT.UU(ssae E RDx where X E R contains the original activations. Fach feature's probability reflects its share of total activation magnitude":
budget allocated to feature i total representational budget
Pi
1=1
(6)
Σ
[
The SAE's & regularization ensures these allocations reflect computational importance. Features activating more frequently or strongly consume more capacity, with optimal |2:| proportional to marginal contribution to reconstruction quality (derivation in Appendix A.2).
Effective features as lossless compression limit.
Shannon entropy quantifics the information content
of this distribution: H(p) = - log Its exponential:
F F=c H())
measures effective degrees of freedom, the minimum neurons needed to encode p without interference. This is the network's lossless compression limit: the feature distríbution could be transmitted through F neurons with no information loss. Using fewer than F neurons guarantees interference as features must share dimen- sions; using exactly F achieves the interference-Ff free boundary; the actual layer width N determines whether compression remains lossless (N 2 F) or becomes lossy (N < F). The ratio
F
(8)
'In toy models where the input dimension M is known, F. ranges from N to M depending spansity; in real networks M is undefined and estimate directly.
2Why not mcasure SAE weights instead of activations? Weight magnitude [[s indicates potential representation but misses actual usage: "dead features" may exist the dictionary without ever activating. Empirically, weight- measure succeeds only in toy models (Figure 3a); and small toy transformer models already require our activation approach (Section 6.2).

---

Published in Transactions on Machine Learning Research (12/2025)
non-linear
e regularization
scallin model
scaling dictionary
3.0
corrolation
Wou (WW
dasa
perfect conilct
2.0
0.99
0.88
0.95
0.94
1.0
wrong scale
1.0
3.0
0.1
0.4
0.7
L.D
2.0
4.0
2.0
3.0 4.0
Wina Way
Veak
Wous
sac)
Voar (2)
coellicient
input dim
Imput dam
(a) Correlation with observable patterns
(b) Robustness across hyperparameters
Figure 3: Validation of superposition metrics. (a) Our measure maintains high correlation whether applied to toy weights (r = 0.99) or SAE activations (r = 0.94), while the Frobenius norm fails on SAE weights. (b) Performance remains stable across l regularization, model scale, and dictionary size variations. Shaded regions show 95% confidence intervals across 100 modcl-SAE pairs.
then measures superposition as lossy compression.
While the SAE extracts interpretable features, semantic concepts humans might recognize, our measure quantifics effective features, the interference-free channel capacity required for their activation distribution. A network might use D 1000 interpretable features but need only F 50 effective features most activate rarely.
Our measure inherits desirable properties from entropy. i.) For any D-component distribution, the output stays bounded 1 S F(p) < D, bounded by single-feature dominance and uniform distribution. ii.) Unlike threshold-based counting, features contribute according to their information content: rare features matter less than common ones, weak features less than strong ones. This enables the interpretation as effective degrees of freedom, beyond "counting fcatures".
sufficient samples until convergence (see convergence analysis in Section 6.4). For convo- lutional layers, we treat spatial positions as independent samples, measuring superposition across the channel dimension (Appendix A.7). While, in general, the data distribution for extracting SAE activations should reflect the training distribution, for studying adversarial training's effect, we evaluate on both clean inputs and adversarially perturbed inputs for contrast.
In practice,
This framework enables quantifying superposition without ground truth by measuring each layer's compres- sion ratio; how many virtual neurons it simulates relative to its physical dimension.
5
Validation of the Measurement Framework
5.1 Toy Model of Superposition
We validate measure using the toy model of superposition (Elhage et al., 2022b), where interference pallerns are directly observable. This controlled setting tests whether sparse autoencoders can recover accurate feature counts from superposed representations.
Following Elhage et al. (2022b), we generate 100 toy models with sparsity S E [0.001, 0.999]. Each model compresses features through 5-neuron bottleneck, with importance weights decaying as w; = 0.7*. After training convergence, we extract 10,000 activation samples and train SAEs with 40-dimensional dictionaries (8x expansion) and & coefficient 0.1. This two-stage process mimics real-world measurement where ground truth remains unknown.
Validation strategy. Our validation procceds in two steps. First, we establish reference values by mea- suring superposition directly from Wrny, where the interference matrix WT els: diagonal dominance indicates orthogonal features; off-diagonal terms show interference (Figure 2b). Both our entropy-based measure and the Frobenius norm bascline (Eq. 2) achieve near-perfect correlation (r 0.99 0.01) when applied toy model weights, confirming both track these observable patterns (Figure 3a).
toy Way reveals compression lev-

---

Published in Transactions on Machine Learning Research (12/2025)
60
1.0
no dropout
feature count
feature count
accuracy
0.9
neurons features
27
40
23
0.8
semi-stable region
b-16
1r32
dropout
22
0.7
keep most important features
bx122
0.0
0.4 ratio
0.8
0.0
0.4
0.8
dictionary size
dropout ratio
dropout ratio
(a) Dictionary scaling
(b) Dropout reduces features and performance
(c) Dropout effect on features
Figure 4: Measurements multi-lask sparse parily dataset. (a) Dictionary scaling plateaus with proper regularization (e, 2 0.1), validating intrinsic structure measurement. Weak regularization (6 0.01) shows unbounded growth through arbitrary subdivision. (b) Dropout monotonically reduces effective features and accuracy. (c) Capecity-dependent response: larger networks show reduced scnsitivity while narrow networks exhibit sharp feature reduction, distinguishing polysemanticity (neurons encoding multiple features) from superposition (compression beyond lossless limit).
Second, we test whether each metric recovers these reference values when given only SAE outputs, the realistic scenario for measuring real networks. Here the Frobenius norm fails catastrophically on SAE weights, producing nonlinear relationships and incorrect scales (0.1-0.7 versus expected 1-4); the & regularization fundamentally alters weight statisties. Our activation-based approach maintains strong correlation (r 0.94 + 0.02) with the reference values even through the SAE bottleneck.
Hyperparameter stability. We test sensitivity across three axes: & strength (10 to 10"), model scale (8 32 input dimensions), and dictionary expansion (2x to 32x). Figure 3b shows stable performance across most configurations. Correlation degrades when extreme regularization (f1 10) suppresses features, when dictionaries lack capacity to represent the feature set, when loy models are too small or too large to train reliably, or when very large dictionaries enable feature splitting (see Section 5.2). These failure modes reflect limitations of the toy model SAE training rather than the measure itsclf.
5.2 Dictionary Scaling Convergence
Mcasuring natural coastline with finer ruler yields longer measurement; potentially without bound (Mandelbrot, 1967). As SAE dictionaries grow, might we discover arbitrarily many features at finer scales?
We test convergence using multi-task sparse parity (Michaud et al., 2023) (3 tasks, 4 bits cach) where ground truth bounds meaningful features. Networks with 64 hidden neurons trained across dictionary scales (0.5x to 16x hidden dimension) and & strengths (0.01 to 10.0).
Figure 4a reveals two regimes. With appropriate regularization (4 2 0.1), feature counts plateau despite dictionary expansion, indicating we measure the network's representational structure and not arbitrary decomposition (i.e. feature splitting (Chanin et al., 2024b)). Weak regularization (e, = 0.01) permits continued growth across all tested scales- this reflects fcature splitting rather than gemuine superposition, where the SAE decomposes single computational features into spurious fine-grained components. Excessive regularization (&1 10.0) suppressos features entirely.
The dependence on dictionary size means absolute counts vary with SAE architecture, but comparative measurements remain valid: networks analyzed under identical configurations yield meaningful relative dif- ferences, even changing those configurations shifts all measurements systematically.
6 Applications and Findings
We measure superposition across four neural compression phenomena: capacity constraint under dropout (Section 6.1), algorithmic tasks that resist superposition despite compression (Section 6.2), developmental
8

---

Published in Transactions on Machine Learning Research (12/2025)
transformer
trained hrom-scratch
compressed
trained-from scrathh
compressed
compression
feature count F
feature count
F=N
sorting
20
ReLl
10
acourocy
10
reversing
Dla part
Jeyss 9
stream
10
20
residual dimension
30
40
10
20
30
40
10
residual
20
30
40
50
dimension
10
20
30
40
50
(a) Architecture
(b) Task: sequence reversal
(c) Task: sequence sorting
Figure 5: Algorithmic tasks under compression. (a) Compression architecture projects activations through ReLU(W Wx) to force features into fewer dimensions. (b) Sequence reversal: progressive compression from native 45D increases superposition from & s 0.3 toward = (leftmost to rightmost points approaching F=N line). Once reaching the F=N boundary, further compression causes performance degradation and then collapse (x markers) rather than superposition beyond & = 1. (c) Sorting exhibits identical dynamics with F N throughout compression. Both tasks resist genuine superposition (+/ > 1), operating at the Jossless limit where each neuron encodes one effective feature, likely due to lack of input sparsity for sequence operations.
dynamics during learning transitions (Section 6.3), and layer-wise representations ional organization in language models (Section 6.4).
Each finding here is preliminary, exploratory analysis specific architectures and tasks. Our primary con- tribution remains the measurement tool itself. These findings illustrate its potential utility while generating testable hypotheses for future systematic investigation across broader experimental conditions.
6.1 Dropout Reduces Features Through Redundant Encoding
We investigate how dropout affects feature organization using multi-task sparse parity (3 tasks, 4 bits cach) with simple MLPs across hidden dimensions h E {16, 32, 64, 128} and dropout rates [0.0, 0.1, .... 0.9].
Marshall Kirchner (2024) showed dropout induces polysemanticity through redundancy: features must distribute across neurons to survive random deactivation. One might expect this redundancy to increase measured superposition. Instead, dropout monotonically reduces effective features by up to 50% (Figure 4b).
We propose this reflects the distinction between polysemanticity and superposition (Figure 4c). If dropout forces each feature to occupy multiple neurons for robustness, this redundant encoding would consume capacity, leaving room for fewer total features within the same dimensional budget. Under this interpretation, networks respond by pruning less essential features, consistent with Scherlis et al. (2023)'s competitive resource allocation framework.
The capacity dependence supports this account: larger networks show reduced dropout sensitivity while narrow networks exhibit sharp feature reduction, suggesting capacity constraints mediate the effect.
6.2 Algorithmic Tasks Resist Superposition Despite Compression
Tracr compiles human-rcadable programs into transformer weights with known computational structure (Lindner al., 2023). We examine sequence reversal (*123" - "321") and sorting ("213" "123"), com- paring compiled models at their original dimensionality (compression factor 1x) against compressed variants and transformers trained from scratch with matching architectures.
Following Lindner et al. (2023), we compress models by projecting residual stream activations through learned compression matrices. Our compression scheme (Figure 5a) applies ReLU(WT Wx) where W e NNX compressing from originally M dimensions N. The ReLU activation, absent in the original Tracr compression, allows small interference terms to cancel out following the toy model rationale (Elhage et al., 2022b).
9

---

Published in Transactions on Machine Learning Research (12/2025)
fecture count (x 1000
mpp resdual embedding
mlp layer
1.0
phose transition
20
170
8
accurocy
40
lecture count
LLC
6
0.5
sast
Joes
10
30
160
uscuritty
Scamure COURT
150
2
0.0
20
LEC
4 layer
10
10
10'
10
100
100
number of samples
epoch
epoch
(a) Pythia-70M layer analysis
(b) Grokking dynamics
Figure 6: (a) Non-monotonic feature organization across Pythia-70ke 70M. MLP layer 1 peaks at 10,000 features (20x neurons). Convergence analysis shows saturation ion after x 10a samples. (b) Feature dynamics during grokking on modular arithmetic. Sharp consolidation generalization transition (epoch 60) follows smoother LLC decay. Strong correlation (r 0.908, p < 0.001) with LLC suggests feature count functions as a measure of model complexity.
The compression dynamics reveal limits on superposition in these algorithmic tasks (Figure 5b, 5c). Both compiled Tracr models and transformers trained from scratch converge to 12 features for reversal and 10 for sorting" far below their original compiled dimensions (45D for reversal), revealing substantial dimensional redundancy Tracr's compilation.
As compression reduces dimensions from 45D toward the task-intrinsic boundary, superposition increases from vb s 0.3 toward & 1. However, compression stops increasing superposition once models reach the F = N diagonal: further dimensional reduction causes linear drop in elfective features and eventually performance collapse (x markers) rather than superposition beyond &े = 1, resisting genuine superposition (ib 1) entiroly.
This resistance likely stems from algorithmic tasks violating the sparsity assumption required for lossy com- pression (Elhage al., 2022b). The toy model of superposition requires features to activate sparsely across inputs: most features remain inactive most samples, kceping interference managable. Algorithmic tasks break this assumption; sequence operations require consistent activation patterns across inputs. Without sparsity, interference becomes destructive rather than enabling compression. While we originally anticipated this setting would enable controlled validation across superposition levels, the systematic F x N tracking, coupled with performance collapse when dimensions drop below this boundary, instead provides indirect evidence that our measure captures genuine capacity constraints, detecting minimal superposition as the sparsity prerequisite fails.
6.3 Capturing Grokking Phase Transition
Grokking (sudden perfect generalization after extended training on algorithmic tasks) provides an ideal testbed for developmental measurement (Power et al., 2022). We investigate whether feature count dynamics can detect this phase transition and how they relate to the Local Learning Coefficient (LLC) from singular learning theory (Hoogland et al., 2024).
We train two-path MLP on modular arithmetic (a (b) mod 53. Figure 6b reveals distinct dynamies: while LLC shows initial proliferation followed by smooth decay throughout training, our feature count exhibits sharp consolidation precisely at the generalization transition.
This pattern suggests the measures capture different aspects of complexity evolution. During memorization, the model employs numerous superposed features to store input-output mappings. The sharp consolida- tion coincides with algorithmic discovery, where the model reorganizes from distributed lookup tables into compact representations that capture the modular arithmetic rule (Nanda al., 2023). Strong correlation
While generally recommend comparative interpretation due measurement limitations (Section the systematic F N boundary tracking performance decline when violated suggest our measure may provide meaningful absolute effective feature counts in sufficiently constrained computational settings.
10

---

Published in Transactions on Machine Learning Research (12/2025)
MINIST
MNIT
feature count ratio
feature count ratio ratio
1
baselnw
bxselie
0.5
0.5.
2- clclass
3-class
10-class
FoshionMMNiST
2
2-class
3-class
10-class
FashionMNIST
2
0oev1
0.5
liwwu
cocY 2
ofversakd
dean
0.5.
Inecr
Inecr
deen adveraccua
0.0
0.1
0.2
0.3
0.1
0.2
0.3
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.1
0.2
0.3
0.1
0.2
0.3
adversarial training strength (e)
adversarial training strength (e)
(a) 3-layer CNN on (Fashion)JMNIST
(b) 3-layer MLP on (Fashion-MMNTST
Figure 7: Higher task complexity shifts adversarial training's effect from feature expansion toward reduction.
Each panel shows results varying dataset (MNIST top, Fashion-MNIST bottom) and number of classes (2, 3, 10). (a) CNNs show clear complexixy-dependeen transitions: simple tasks enable feature expansion while complex tasks (10 classes) force reduction below baseline. (b) MLPs exhibit similar patterns with
more pronounced layer-wise variation. Fashion-MNIST consistently amplifies the reduction effect compared to MNIST, suggesting that representational demands drive defensive strategies beyond mere class count. Dashed lines: clean data; solid lines: adversarial examples. Feature count ratios normalized to e 0 baseline. Error bars show standard error across seeds.
(r 0 0.001) between feature count and LLC positions superposition measurement develop- mental tool for detecting emergent capabilities through their informationn-heorrdii signatures.
6.4 Layer-wise Organization in Language Models
We analyze Pythia-70M using pretrained SAFs from Marks et al. (2024), measuring feature counts across all layers and components. Convergence analysis (Figure 6a) shows saturation after 2 x 10 samples. Feature importance follows power-law distributions: while 21,000 SAE features activate for MLP 1, our entropy-based measure yields 5,600 effective features, automatically downweighting rare activations.
MLPs store the most features, followed by residual streams, with attention maintaining minimal counts, consistent with MLPs knowledge stores and attention as routing (Geva al., 2021). Features grow early layers (MLP achieves 20x compression), compress through middle layers, then re-expand before final consolidation.
This non-monotonic trajectory parallels intrinsic dimensionality studies (Ansuini et al., 2019): both reveal "hunchback" patterns peaking in early-middle layers. Intrinsic dimensionality measures geometric mani- fold complexity (minimal dimensions describing activation structure), while we count effective information channels (minimal dimensions for lossless encoding), both measuring aspects of representational complexity.
7 Connection between Superposition and Adversarial Robustness
Testing the superposttion--wheerabiil hypothesis. The superposittion-vuueeabbbiil hypothesis pro- posed by Elhage et al. (2022b) predicts that adversarial training should universally reduce superposition, as networks trade representational efficiency for orthogonal, robust features. We test this prediction systemat- ically across diverse architectures and conditions, finding that the direction of the effect expansion versus redeetion-depends on task complexity and network capacity.
We employ PGD adversarial training (Madry et al., 2018) across architectures ranging from single-layer to deep networks (MLPs, CNNs, ResNet-18) on multiple datasets (MNIST, Eashion-MNISS CIFAR-10). Task complexity varies through both classification granularity (2, 3, 5. classes) and dataset difficulty. Network capacity varies through hidden dimensions (8-512 for MLPs), filter counts (8-64 for CNNs), and width scaling (1/4x-2: for ResNet-18). For convolutional networks, measure superposition across channels
11

---

Published in Transactions on Machine Learning Research (12/2025)
fecture count ratto
N
N
N
fecture count ratio
koyw
2
4
Joys
0.3
Joya:
bedia 2-class
3-class
10-class
1
CNN Maager
0.5
cdveraniia
0.00
0.01
0.02
0.03
0.01
0.02
0.03
0.01
0.02
0.03
adversarial training strength
adversarial training strength (e)
(a) Widening 1-layer NNs MNIST
(b) Narrowing ResNet-18 CIFAR-10
Figure 8: Higher network capacity shifts adversarial training's effect from feature reduction toward expan- sion. (a) Single-layer networks on MNIST demonstrate aspacit-ddepnndent transitions: MLPs with hidden dimensions h E {8, (8, 128, 512) (top) and CNNs with filter counts {8, 16, 32, 64) (bottom) show that narrow networks reduce features while wide networks expand them across task complexities. (b) ResNet-18 on CIFAR-10 with width scaling (1x, 1/2x, 1/4x) reveals layer-wise spocialization: carly layers reduce fea- tures while deeper layers (layer 4) expand dramatically, with this paltern dampening as width decreases. Dashed lines: clean data; solid lines: adversarial examples. Feature count ratios normalized to bascline.
by reshaping activation tensors to treat spatial positions as independent samples (see Appendix A.7 for details). All SAEs use 4x dictionary expansion with 4 = 0.1. Measurements on adversarial examples match the training distribution; models trained with e = 0.2 are evaluated on e 0.2 attacks.
Statistical methodology. To quantify adversarial training effects, we extract normalized slopes repre- senting how feature counts change per unit increase in adversarial training strength (e E (0.0, 0.1, 0.2, 0.3)). Positive slopes indicate adversarial training increases features; negative slopes indicate reduction. For each experimental condition, we fit linear regressions to feature counts across epsilon values, pooling clean and adversarial observations increase statistical power. These slopes are normalized by baseline (e == feature counts, making effects comparable across layers with different absolute scales.
Since networks contain multiple layers, aggregate layer-wise wise measurements using parameter- weighttd aver- aging, where layers with more parameters receive proportionally higher weight. This reflects the assumption that computationally intensive layers better represent overall network behavior. For simple architectures, pa- rameter counts include all weights and biases; for ResNett-18, we implement detailed counting that accounts for convolutions, batch normalization, and skip connections.
Testing Adversarial Training Effects Superposition
We test three formal hypotheses:
H1 (Universal Reduction): Adversarial training uniformly reduces superposition across all conditions, directly testing Elhage et al. (2022b)'s original prediction.
H2 (Complexity 4): Higher task complexity shifts adversarial training's effect from fea-
ture expansion toward reduction. We encode complexity ordinally (2-class=1, 2, 5-class=3, 10-class-4) and test for negative linear trends in the adversarial training slope.
H3 (Capacity t): Higher network capacity shifts adversarial training's effect from feature reduction toward expansion. We test for positive log-linear relationships between capacity measures and adversarial training slopes.
All statistical tests use inverso-variance weighting account for measurement uncertainty, with random- effects meta-analysis when significant heterogeneity exists across conditions.
12

---

Published in Transactions on Machine Learning Research (12/2025)
MLP
CNN
1-layer MLP
1-layer CNN
ResNet-18
NhiST FishionMNST
2
Jic assss
1.0
slope normeiino pert beazllne
S-cass
0.5
10dliss
0-
40
0
-0.5
0
3
10
8
32
128
512
32 channel dimension
16
64
174
1/2
number of classes
number of classes
hidden dimensionn
layer width multiplier
(a) Complexity & (number classes + dataset)
(b) Complexity + Capacity 1
(c) Capacity
Figure 9: Statistical analysis of adversarial training effects on superposition. Normalized slopes quantify fea- ture count changes per unit adversarial strength E; positive slopes indicate adversarial training increases fea- tures, negative slopes indicate reduction. (a) Task complexity (number of classes + dataset difficulty) shows consistent negative relationship with slopes: higher complexity yields more negative slopes. Fashion-MNIST (green) produces systematically lower slopes than MNIST (yellow), consistent with its greater difficulty. (b) Single-layer werksrks on MNIST show capacity-dependent transitions: narrow networks (8 32 units) have negative slopes regardless of task complexity, while wide networks (128-51) 512 units) have positive slopes. (c) ResNet-18 on CIFAR-10 demonstrates log-lincar scaling: wider networks show dranatically more positive slopes. Error bars show standard errors.
Complexity shifts adversarial training toward feature reduction (H2 supported). Contrary to H1's prediction of universal reduction, adversarial training produces bidirectional effects whose direction depends systematically on task complexity (Figures 7 and 9a). Our meta-analysis reveals significant het- erogeneity across conditions (Q = 8.047, df = p 0.045), necessitating random-effects modeling. The combined effect confirms H2: negative relationship between task complexity and the adversarial training slope (slope = -0.745 + 0.122, = -6114, p < 0.001), meaning higher complexity shifts the effect from expansion toward reduction.
Binary classification consistently yields positive slopes, with feature counts expanding up to 2x baseline. Networks appear to develop additional defensive features when task demands are simple. Ten-class problems show negative slopes, with feature counts decreasing by up to 60%, particularly in early layers. Three-class tasks exhibit, intermediate behavior with inverted-U curves: moderate adversarial training (e 0.1) initially expands features before stronger training (e 0.3) triggers reduction.
Dataset difficulty amplifies these effects. Mashion-MNIST produces systematically more negative slopes than MNIST (mean difference = -1.467 + 0.16,, ((7) = -..405, p = 0.047, Cohen's d = -0 -0.85,, consistent with its design as a more challenging benchmark (Xiao et al., 2017). This suggests that representational demands, beyond mere class count, drive defensive strategies.
Layer-wise patterns differ between architectures: MLP first layers reduce most while CNN second layers reduce most. We lack mechanistic explanation for this divergence.
Capacity shifts adversarial training toward feature expansion (H3 supported). Network capacity exhibits a positive relationship with the adversarial training slope, strongly supporting H3 (Figures 8, 9b, and 9c). Single-layer networks demonstrate clear capacity thresholds (meta-analytic slope = 0.220 + 0.037, = 5.90, p < 0.001). Networks with minimal capacity (8 hidden units for MLPs, 8 filters for CNNs) show negative slopes slopes-reddee reee-iirduinng features across all task complexities while high-capacity networks (512 units/64 filters) show positive slopes, expanding fcatures even for 10-class problems.
This capacity dependence scales dramatically in deep architectures. ResNet-18 on CIFAR-10 exhibits a strong log-linear relationship between width multiplier and adversarial training slopes (slope = 31.0 + 2.0 per log(width), t(2) 15.7, p = 0.004, R² 0.929). An 8-fold width increase (0.25x to 2x) produces a 65-unit change in normalized slope. At minimal width (0.25x), adversarial training barely affects feature counts; at double width, networks show massive feature expansion with slopes approaching 80.
13

---

Published in Transactions on Machine Learning Research (12/2025)
The layer-wise progression in ResNet-18 reveals hierarchical specialization: early layers (conv1, layerl) reduce features by up to 50%, middle layers remain stable, while deep layers (layer3, layer4) expand up to 4x. Systematically narrowing the network dampens this pattern: at 1/4 width, late-layer expansion vanishes while carly-layer reduction persists but weakens. This could reflect vulnerability hicrarchics, where early layers processing low-level statistics are easily exploited by imperceptible perturbations, necessitating feature reduction, while late layers encoding semantic information can safely expand their representational repertoire.
Two regimes of adversarial response. Our findings reveal a. more nuanced relationship between su- perposition and adversarial vulnerability than originally thoorized. Rather than universal feature reduction, adversarial training operates in two distinct regimes determined by the ratio lask demands to network capacity.
Bifurcation Driven by Task Complexity to Network Capacity Ratio
Adversarial training's effect on superposition depends on the ratio of task demands to network ca- pacity:
Abundance regime (low complexity / high capacity): Adversarial training increases effective features. Networks add defensive features, achieving robust through elaboration.
Scarcity regime (high complexity / low capacity): Adversarial training decreases effective features. Networks prune to fewer, potentially more orthogonal features, as predicted by the superppsitionnvvueerbbllll hypothesis.
Unexplained patterns. Several patterns in our data remain unexplained. We observe non-monotonic inverted-U curves where moderate adversarial training (e 0.1) expands features while stronger training (e 0.3) reduces them below baseline. The gap between clean and adversarial feature counts varies umpredictably; sometimes negligible, sometimes substantial. Some results contradict our complexity hypothesis, with 2-class MLPs occasionally showing lower feature counts than 3-class. CNN experiments consistently yield stronger statistical significance (p < 0.02) than equivalent MLP experiments (p R 0.09) for unknown reasons.
Implications for interpretability. Our findings complicate simple accounts of why robust models often appear more interpretable (Engstrom et al., 2019). Tf interpretability benefits arose purely from reduced representational complexity, we would expect universal feature reduction under adversarial training. The existence of an abundance regime where feature counts increase suggests alternative mechanisms: perhaps non-interpretable shortcut features are replaced by richer, more human-aligned representations, or perhaps interpretability benefits are confined to the scarcity regime. Resolving this requires interpretability metrics beyond the scope of our current framework.
The bidirectional relationship between robustness and superposition suggests that achieving robustness with- out capability loss may require ensuring sufficient capacity for defensive claboration. Whilc our experiments demonstrate that increased robustness can coincide with either increased or decreased superposition depend- ing on the regime, establishing the exact causal connection between superposition and robustness remains an important direction for future work.
8 Limitations
Our superposition measurement framework is limited by its dependence on sparse autoencoder quality, theoretical assumptions about neural feature representation, and should be interpreted as proxy for repre- sentational complexity rather than literal feature count:
Sparse autoencoder quality. Our approach inherently depends on sparse autoencoder feature extrac- tion quality. While recent architectural advances (gated SAFs (Rajamanoharan et al., 2024), TopK variants
14

---

Published in Transactions on Machine Learning Research (12/2025)
(Gao et al., 2024), and end-to-end training (Braun et al., 2024)) have substantially improved feature recov- ery; fundamental challenges remain. SAE training exhibits sensitivity to hyperparameters, particularly & regularization strength and dictionary size, with different initialization or training procedures potentially yiclding different feature counts for identical networks. Ghost features, i.c. SAE artifacts without computa- tional relevance (Gao al., 2024), can artificially inflate measurements, while poor reconstruction quality may deflate them.
Assumptions on feature representation. Our framework rests on several assumptions that real net- works systematically violate. The linear representation assumption (that features correspond to directions in activation space) has been challenged by recent discoveries of circular feature organization for tempo- ral concepts (Engels et al., 2024) and complex geometric structures beyond simple directions (Black et al., 2022). Our entropy calculation assumes features contribute independently to representation, but neural networks exhibit extensive feature correlations, synergistic information where feature combinations provide more information than individual contributions, and gating mechanisms where some features control others' activation. The approximation that sparse linear encoding captures true computational structure breaks down in hicrarchical representations where low-level and high-level features are not substitutable, and in networks with substantial nonlinear feature interactions that cannol be decomposed additively.
Comparative rather than absolute count. Our measure quantifies effective representational diversity under specific assumptions rather than providing literal feature counts. This creates several interpretational limitations. The measure exhibits sensitivity to the activation distribution used for measurement. SAE training distributions must match the network's operational regime to avoid systematic bias. Feature granu- larity remains fundamentally ambiguous: broader features may decompose into specific ones in wider SAEs, creating uncertainty about whether we're discovering or creating features. Our single-layer analysis poten- tially misses features distributed across layers through residual connections or attention mechanisms. Most critically, we measure the effective alphabet size of the network's internal communication channel rather than counting distinct computational primitives, making comparative rather than absolute interpretation most appropriate.
The limitations largely reflect active research areas in sparse dictionary learning and mechanistic inter- pretability. Each advance in SAE architectures, training procedures, or theoretical understanding directly benefits measurement quality. Within its scope comparative analysis representational complexity un- der sparse linear encoding assumptions the measure enables systematic investigation of neural information structure previously impossible.
9 Future Work
Cross-model feature alignment. Following Authropic's crosscoder approach (Templeton et al., 2024), training joint SAEs across clean and adversarially-trainnn models would enable direct feature comparison. This could reveal whether the abundance regime involves feature elaboration (creating defensive variants) versus feature replacement (substituting vulnerable features with robust ones).
Multi-scalc and cross-layer mcasurement. Current layer-wise analysis may miss features distributod across layers through residual connections. Matryoshka SAEs (Bussmann et al., 2025) already capture fea- ture hierarchies at different granularities within single layers; extending this to cross-layer analysis could reveal how abstract features decompose into concrete features through network depth. Applying our en- tropy measure at each scale and depth would quantify information organization across both dimensions. Implementation requires developing new SAE architectures that span multiple layers.
Feature co-occurrence and splitting. Our independence assumption breaks when features consistently co-activate, yet this structure may be crucial for resolving feature splitting across dictionary scales. As we expand SAE dictionarics, single computational features can dccompose into multiple SAE features artificially inflating our count. Features that always co-oceur likely represent such spurious decomposi- tions rather than genuinely independent. components. We initially attempted eigenvalue decomposition of
15

---

Published in Transactions on Machine Learning Research (12/2025)
feature co-occurrence matrices to identify such dependencies, but this approach faces fundamental rank constraint: covariance matrices have rank at most. N (the neuron count), making it impossible to detect superposition beyond the physical dimension. Alternative approaches include mutual information networks between features or hierarchical clustering of co-occurvence patterns. Combining these with Matryoshka SAEs' multi-scale dictionaries could reveal which features remain coupled across granularities (likely repre- senting single computational primitives) versus those that split independently (likely representing distinct features). This would provide principled solution to the dictionary scaling problem: count only features that disentangle across scales.
Causal intervention experiments. While demonstrate correlation between adversarial training and superposition changes, establishing causality requires targeted interventions: i.) artificially constraining superposition via architectural modifications (e.g., softmax linear units (Elhage et al., 2022a)) then measuring robustness changes; ti.) directly manipulating feature sparsity in synthetic tasks; Wi.) using mechanistic interpretability tools trace how specific features contribute to adversarial vulnerability.
Validation at scale. Testing our framework on contemporary architectures Vision Transformers, diffusion models) would reveal whether findings generalize. Scale might expose new phenomena in adversarial training: very large models may escape capacity constraints entirely, scaling laws might reveal limits on compression efficiency while maintaining robustness. If validated, our metric could guide architecture search for interpretable models by incorporating superposition measurement inlo training objectives or architecture design.
LLMs,
Connection to model compression. Our lossy compression perspective parallels findings model com- pression research (Pavlitska et al., 2023). Both superposition (internal compression) and model compression (parameter reduction) force networks to optimize information encoding under constraints. Formalizing this connection through rate-distortion theory could yield theoretical bounds the cobustaees-compprssiio tradeoff, explaining when compression helps versus hurts.
10
Conclusion
This work provides precise, measurable definition of superposition. Previous accounts characterized su- perposition qualitatively, as networks encoding "more fcatures than neurons", we formalize it as lossy com- pression: encoding beyond the interference-free limit. Applying Shannon entropy to sparse autoencoder activations yields the effective degrees of freedom: the minimum neurons required for lossless transmission of the observed feature distribution. Superposition occurs when this count exceeds the layer's actual dimension.
The framework enables testing previously untestable hypotheses. The superppsttion-ruunerbbbllly hypoth- esis (Elhage et al., 2022b) predicts that adversarial training should universally reduce superposition networks trade representational efficiency for orthogonality. We find instead that the effect depends the ratio of task demands to network capacity: an abundance regime where simple tasks permit feature expan- sion, and a scarcity regime where complexity forces reduction. By grounding superposition in information theory, this work makes quantitative what was previously only demonstrable toy settings.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Author Contributions
L.B. conccived the project, developed the theorctical framework, designed and conducted all experiments except grokking, performed statistical analyses, and wrote the manuscript. Z.T.-K. conducted the grokking experiment and LLC comparison. R.S. and E.G. supervised the research.
Acknowledgments
We thank Daniel Sadig for insightful discussions on adversarial training mechanisms and Hamed Karimi for detailed feedback on the manuscript that improved clarity and presentation. We are grateful to Jacqueline Bereska for valuable suggestions on manuscript. organization and prioritization. We thank the anonymous TMLR reviewers for their rigorous feedback, particularly on statistical methodology and theoretical founda- tions, which substantially strengthened this work.
Part of this research was conducted during L.B.s visit to the Trustworthy AT Lab (TAILab) at Toronto Metropolitan University. We are grateful for the stimulating research environment that facilitated the development of the core conceptual framework.
17

---

Published in Transactions on Machine Learning Research (12/2025)
References
Kartik Anand, Ginestra Bianconi, and Simone Severini. The shannon and the von neumann entropy of random networks with helerogeneous expected degree. Phys. Rev. E, March 2011. 23
Alessio Ansuini, Alessandro Laio, Jakob Macke, and Davide Zoccolan. Intrinsic dimension of data represen- tations in deep neural networks. NeurIPS, May 2019. 3, 11
Maximilian Augustin, Alexander Meinke, and Matthias Hein. Adversarial robustness on in- and out- distribution improves explainability. ECCV, July 2020.
Kola Ayonrinde, Michael T. Pearce, and Lee Sharkey. Interpretability compression: Reconsidering sae explanations of neural activations with mdl-saes. CoRR, October 2024. 3
Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety a review. TMLR, August 2024. 2,
Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guovara, Beren Millidge, Gabriel Alfour, and Connor Lealy. Interpreting noural networks through the polytope lens. CoRR, November 2022. 15,
Akhilan Boopathy, Sijia Liu, Gaoyuan Zhang, Cynthia Liu, Pin-Yu Chen, Shiyu Chang, and Luca Daniel. Proper network interpretability helps adversarial robustness in classification. JCML, October 2020. 1
Dan Braun, Jordan Taylor, Nicholas Goldowsky-Din and Lee Sharkey. Tdentifying functionally important features with end-to-end sparse dictionary learning. ICML MI Workshop, May 2024. 15
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuils Thread, October 2023. 2, 5,
Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. CoRR, 2024. 3
Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning multi-level features with matryoshka sparse autoencoders. CoRR, 2025. 3, 15
Marc-André Carbonneau, Julian Zaidi, Jonathan Boilard, and Ghyslain Gagnon. Measuring disentanglement A review of metrics. IEFE Trans. Neural Netw. Learn. Syst., May 2022. 4
David Chanin, Tomás Dulka, Hardik Bhatnagar, and Joseph Bloom. Toy models of feature absorption in saes. Less Wrong, October 2024a.
David Chanin, James Wilken-Smith, Tomás Dulka, Hardik Bhatnagar, Satvik Golechha, and Joseph Bloom. A for absorption: Studying feature splitting and absorption in sparse autoencoders. CoRB, September 2024b.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interprotable features in language models. ICLR, Jamuary 2024. 2, 3, 5, 6
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. ICLR, February 2018. 4
Henry Eigen and Amir Sadovnik. Topkconv: Increased adversarial robustness through deeper interpretability. TCMLA, December 2021.
18

---

Published in Transactions on Machine Learning Research (12/2025)
Nelson Elhage, Tristan Hume, Olsson Catherine, Nanda Neel, Tom Henighan, Scott Johnston, Sheer FlShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-oodss, Jack- son Kernion, Tom Concrly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacobson, Eli Tran- Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuits Thread, 2022a. 16
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac HatfieldDoodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. Transformer Circuits Thread, 2022b. 2, 3, 4, 5, 7, 9, 10, 11, 12, 16, 24
Joshua Engels, Isaac Liao, Eric J. Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear. CoRR, May 2024. 15, 24
Logan rug,trom, Andrew Nyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness as a prior for learned representations. CoRR, September 2019. 1, 2, 14
Christian Etmann, Sebastian Lunz, Peter Maass, and Carola-Bibiane Schönlieb. On the connection between adversarial robustness and saliency map interpretability. ICML, May 2019.
Andrea Fontanari, Iddo Eliazar, Pasquale Cirillo, and Cornelis W. Oosterlee. Portfolio risk and the quantum majorization of correlation matrices. IMA Journal of Management Mothematica, 2021.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, December 2020. 26
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, llya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. CoRR, June 2024. 3, 6, 15, 24
Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. A theory of multineuronal dimensionality, dynamics and mcasurement. bioRwie, November 2017. 4
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. CoRR, September 2021. 11
Ziv Goldfeld, Ewout van den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. Estimating information flow in deep neural networks. 1CML, May 2019. 3
Kaarel Hänni, Jake Mendel, Dmitry Vaintrob, and Lawrence Chan, Mathematical models of computation in superposition. ICML MI Workshop, August 2024. 3
Mark O. Hill Diversity and evenness: A unifying notation and its consequences. Ecology, March 1973. 22
Geoffrey E Hinton. Distributed representations. Carnegie Mellon University, 1984. 3
Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet.
The developmental landscape of in-context learning. CoRR, February 2024. 10
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. NeurIPS, August 2019. 1
Edwin T. Jaynes. Information theory and statistical mochanics. Phys. Rev., May 1957. 23
Frederick Jelinck, Robert L. Mercer, Lalit R. Bahl, and Janet. K. Baker. Perplexity-a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, December 1977. 22
Lou Jost. Entropy and diversity. Oikos, May 2006. 22, 23
19

---

Published in Transactions on Machine Learning Research (12/2025)
Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. Interpreting attention layer outputs with spanse autoencoders. CoRR, June 2024. 3
Victor Lecomte, Kushal Thaman, Trevor Chow, Rylan Schaeffer, and Sanmi Koyejo. Incidental polyseman- ticity. CoRR, 2023. 3
Sangyun Lee, Hyukjoon Kwon, and Jae Sung Lee. Estimating entanglement entropy via variational quantum circuits with classical neural networks. CoRR, December 2023.
David Lindner, János Kramár, Sebastian Farquhar, Matthew Rahtz, Thomas McGrath, and Vladimir Miku- lik. Tracr: Compiled transformers as a laboratory for interpretability. CoRR, 2023. 9, 25
Yanpei Liu, Xinyum Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. CoRR, February 2017. 2
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. CoRR, 2018. 11
Benoit Mandelbrot. How long is the coast of britain? statistical self-similarity and fractional dimension. Science, May 1967. 8
Luke Marks, Amir Abdullah, Luna Mendez, Rauno Arike, Philip Torr, and Fazl Barez. Interpreting reward models in rlhf-tuned language models using sparse autoencoders. CoRR, October 2023.
Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. CoRR, March 2024. 3, 6, 11, 26
Simon C. Marshall and Jan H. Kirchner. Understanding polysemanticity in neural networks through coding thoory. CoRR, Jamuary 2024. 3, 9
Eric J. Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling, NewrIPS, March 2023. 8, 25
Tomas Mikolov, 1lya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. NeurIPS, October 2013.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. ICLR, January 2023. 10
Michael A. Nielsen and Isaac L. Chuang, Quantum Compulation and Quantum Information. Cambridge University Press, January 2011. 4, 23
Chris Olah. Distributed representations: Composition & superposition. Transformer Circuils Thread, 2023.
3
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. NeurIPS Workshop on Causal Representation Learning, November 2023.
Gonecalo Paulo, Alex Troy Mallen, Caden Juang, and Nora Belrose. Automatically interpreting millions of features in large language models. CoRR, October 2024. 3
Svetlana Pavlitska, Hannes Grolig, and J. Marius Zöller. Relationship between model compression and adversarial robustness: A review of current evidence. CoRR, November 2023. 16
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting small algorithmic datascts. CoRR, Jamuary 2022. 10
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Licberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders. CoRR, April 2024. 3, 14
20

---

Published in Transactions on Machine Learning Research (12/2025)
Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. TMLRR August 2023.
Andrew Slavin Ross and Finale Doshi- Doshi-Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. AAAT, November 2017.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. 2007 15th European Signal Processing Conference, September 2007.
Hadi Salman, Andrew Nyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? NewrIPS, December 2020. 1
Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew llyas, Logan Engstrom, and Aleksander Madry. Image synthesis with single (robust) classifier. NeurIPS, August 2019.
Adam Scherlis, Kshitij Sachan, Adam S. Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks. CoRR, July 2023. 3, 9
Erwin Schrödinger. Discussion of probability relations between separated systems. Mathematical Proceedings of the Cambridge Philosophical Sociely, 1935. 23
Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoencoders.
Al Alignment Forum, 2022. 3
Myeongjin Shin, Seungwoo Lee, Junseo Lee, Mingyu Lee, Donghwa Ji, Hyeonjun Yeo, and Kabgyun Jeong. Disentangling quantum neural networks for unified estimation of quantum entropies and distance measures. Phys. Rev. A, December 2024. 4
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. CoRR, April 2017.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, and Brian Chen. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. 3, 15
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. CoRR, April 2000. 3
Theodoros Tsiligkaridis and Jay Roberts. Second order optimization for adversarial robustness and inter- pretability. CoRR, September 2020. 1
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. ICLR, September 2019.
Dmitry Vaintrob, Jake Mendel, and Kaarel Hänni. Toward a mathematical framework for computation in superposition. Al Alignment Forum, 2024. 3
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for beuchmarking machine learning algorithms. CoRR, September 2017. 13
21

---

Published in Transactions on Machine Learning Research (12/2025)
A Theoretical Foundations
A.1 Networks as Conurre-Constraieedd Communication Channels
Neural networks must transmit information through layers with limited dimensions. Each layer acts as a communication bottleneck where multiple leatures compete for neuronal bandwidth. When a network needs to represent F features using only N < F dimensions, it uses lossy compression (== superposition).
This resource scarcity creates a natural analogy to communication theory. Just as telecommunications sys- tems multiplex multiple signals through shared channels, neural networks multiplex multiple features through shared dimensions. Our measurement framework formalizes this intuition by quantifying how efficiently net- works allocate their limited representational budget across competing features.
A.2 L1 Norm as Optimal Budget Allocation
The sparse autoencoder's & regularization creates an explicit budget constraint on feature activations:
LSAE = l W
sae 2112
A||2|11
(9)
The penalty term A||/|11 = A, |ai| enforces that the total activation budget Σ; |as| remains bounded. This creates competition where features must justify their budget allocation by contributing to reconstruction quality.
From the first-order optimality conditions of SAE training, the magnitude |2y] for any active feature satisfies:
|=:| = 1 10 (h - W x--z
(10)
where W-, excludes feature i. This reveals that |2s] mcasures the marginal contribution of feature re- construction quality-exacth) the budget allocation that optimally balances reconstruction accuracy against sparsity. Our probability distribution therefore has meaning as "relative feature strength":
E 2
expected budget allocation to fcature total representational budget
P:
Σ,
(11)
This fraction represents how much of the network's limited representational resources are optimally allocated to feature i under the SAE's constraints. Alternative norms fail to preserve this budget interpretation. The l norm E[2]] overweights outliers and breaks the linear connection to reconstruction contributions through squaring. The lo norm captures only peak activation while ignoring frequency of use. The lo norm provides binary active/inactivy finactive information but loses the magnitude data essential for measuring resource allocation intensity.
A.3 Shannon Entropy as Information Capacity Measure
Given the budget allocation distribution p, the exponential of Shannon entropy provides the theoretically optimal feature count. The exponential of Shannon entropy, exp(If), formally known as perplexity in information theory and the Hill number (order-1 diversity index) in ecology (Hill, 1973; Jost, 2006):
(-E Pi log =
PP(p) =exp
II
p P
(12)
This quantifies the effective number of outcomes a probability distribution: how many equally likely outcomes would yield identical uncertainty. information theory, represents the effective alphabet size of communication system (Jelinek et al., 1977). In ecology, it quantifies the effective number of species in
22

---

Published in Transactions on Machine Learning Research (12/2025)
ecosystem (Jost, 2006). In statistical physics, it relates to the number of accessible states in system (Jaynes, 1957). In quantum mechanics, it corresponds to the effective number of pure quantum states in a mixed state (Schrödinger, 1935).
Shannon entropy uniquely satisfies the mathematical properties required for principled feature counting (Anand et al., 2011). The measure exhibits coding optimality, equaling the minimum expected code length for optimal compression. It satisfies additivity for independent feature sets through H(p@q) H()) + (g). Small changes in feature importance yield small changes in measured count through continuity. Uniform distributions where all features equally important maximize the count. Adding features with positive probability monotonically increases the count. These axioms uniquely characterize Shannon entropy up to a multiplicative constant, making (x)( the theoretically principled choice for aggregating feature importance into an effective count.
In quantum systems, von Neumann entropy S(p) To(p mcasures entanglement, with eS(o) rep- resenting effective pure states participating in mixed quantum state (Nielsen & Chuang, 2011). Neural superposition exhibits parallel structure: just as quantum entanglement creates non-separable correlations that cannot be decomposed into independent subsystem states, neural superposition creates feature repre- sentations that cannot be eleanly separated into individual neuronal components. Both phenomena involve compressed encoding of information: quantum entanglement distributes correlations across subsystems re- sisting local description, while neural superposition distributes features across neurons resisting individual interpretation. Our measure (()) captures this compression by quantifying the effective number of fea- tures participating in the neural representation, analogous to how ((p) quantifies effective pure states in an entangled quantum mixture.
Higher-order Hill numbers provide different sensitivities to rare versus commin features:
1/(1-4)
DD
P
(13)
where g gives exponential entropy measure (via L' rule), q 0 counts non-zero components, and q = 2 gives the inverse Simpson concentration index (participation ratio in statistical mechanics).
A.4 Rate-Distortion Theoretical Foundation
Our measurement framework emerges from two nested rale-distortion problems that formalize the intuitive resource allocation perspective. The neural network layer itself solves:
min RNN(D)= ((Nl)))
T(X; H)
(14)
where the layer width constrains the mutual information 1(X; H) that can transmitted, while represents acceptable task performance degradation. When the optimal solution requires representing >> N features, superposition emerges naturally as the rale-optimal encoding strategy.
The sparse autoencoder solves complementary problem:
RSAE(D) p( b( A1
min
=||
(15)
where sparsity ||=||| acts as the rate constraint and reconstruction error as distortion. This dual struc- ture justifies SAF-based measurement: we quantify the effective rate required to represent the network's compressed internal information under sparsity constraints.
The SAE optimization can be viewed as an information bottleneck problem balancing information preser- vation E[ ( against information cost [B[[[=]]]- Under this interpretation, [[[[=]]] represents the
23

---

Published in Transactions on Machine Learning Research (12/2025)
information cost of including feature i in the compressed representation, making our probability distribution natural measure of information allocation across features.
A.5 Critical Assumptions and Failure Modes
Our method measures effective representational diversity under sparse linear encoding, which approximates but does not exactly equal the number of distinct computational features. We must carefully assess the conditions under which this approximation holds.
Feature Correspondence Assumption. We assume SAE dictionary elements correspond one-to-one with genuine computational features. This assumption fails through feature splitting where one computa- tional feature decomposes into multiple SAE features, artificially inflating counts. Feature merging combines multiple compulational features into one SAE feature, deflating counts. Ghost features represent SAE ar- tifacts without computational relevance (Gao et al., 2024). Incomplete coverage occurs when SAEs miss computationally relevant features entirely.
Linear Representation Assumption. We assume features combine primarily through linear superpo- sition in activation space. Real networks violate this through hicrarchical structure where low-level and high-level features aren't interchangeable. Gating mechanisms allow some features control whether oth- ers activate (Elhage et al., 2022b). Combinatorial interactions emerge when meaning comes from feature combinations rather than individual contributions (Black et al., 2022).
IapnotaddeImpootanne Correspondence. We assume |ai reflects feature i's computational impor- lance. This breaks when SAE reconstruction preserves irrelevant details while missing computational es- sentials, when features interact nonlinearly in downstream processing (Engels al., 2024), when feature importance depends heavily on context rather than magnitude.
Independent Information Assumption. We assume Shannon entropy correctly aggregates information across features. This fails when correlated features don't contribute independent information, when synergis- tic information means feature pairs provide more information together than separately, or when redundant encoding has multiple features encoding identical computational factors.
The approximation captures genuine signal about representational complexity under specific conditions. The measure works best when features combine primarily through linear superposition, activation patterns are sparse with balanced importance, SAFs achieve high reconstruction quality computationally relevant information, and representational structure is relatively flat rather than hicrarchical. The approximation degrades with highly hierarchical represenlations, dense activation patterns with complex feature interac- tions, poor SAE reconstruction quality, or extreme feature importance skew. Despite these limitations, the measure provides principled approximation rather than exact counting, with primary value in comparative analysis across networks and training regimes.
A.6 Why Eigenvalue Decomposition Fails for SAE Analysis
Following the quantum entanglement analogy, one might consider eigenvalue decomposition of the covariance matrix:
Σ AA n
(16)
where A represents the activation matrix. Eigenvalues (A1, 22.22..... , And represent explained variance along principal components, normalized to form a probability distribution:
X
pi Σ"
(17)
A
24

---

Published in Transactions on Machine Learning Research (12/2025)
This approach faces fundamental rank deficiency when applied to SAEs. Expanding from lower dimension (N neurons) to higher dimension (D > N dictionary elements) yields covariance matrices with rank at most N, making detection of more than N features impossible regardless of SAE capacity.
Our activation-basee approach circumvents this limitation by directly measuring feature utilization through activation magnitude distributions rather than intrinsic dimensionality. This enables superposition quantifi- cation with overcomplete SAE dictionaries.
A.7 Adaptation to Convolutional Networks
Convolutional neural networks organize features across channels rather than spatial locations. For CNN layers with activations X E RBxCxH*W, we measure superposition across the channel dimension while accounting for spatial structure.
We extract features from each spatial location's channel vector independently, then aggregate when comput- ing feature probabilities:
Ps E/=1 Ebtt.w (bbj,h..
(18)
where Sb,i,h, represents feature i's activation at spatial position (h, w) in sample b.
This aggregation treats the same semantic feature activating at differont spatial locations (c.g., edge detectors firing everywhere) as evidence for a single feature's importance rather than separate features.
B Experimental Details
B.1 Tracr Compression
We compile RASP programs using Tracr's standard pipeline with vocabulary (1,2,3,4,5 and maximum soquence length The sequence reversal program uses position- basod indexing, while sorting employs Tracr's built-in sorting primitive with these parameters.
Following Lindner et al. (2023), we train compression matrices using dual objective that ensures compressed models maintain both computational equivalence and representational fidelity:
C = Xone Lont + Atayer Llayer Alayer
(19)
Lout = (L(sftmmax (y.). softmax(y.))
(20)
Llayer
1 L
Σ |b (0)
h 1
(21)
where y. and yo denote compressed and original logits, and h 11) h represent original and compressed activations at layer i.
Hyperparameters: Aout 0.01, Alayer = 1.0, learning rate 10-3, temperature - = 1.0, maximum 500 epochs with carly stopping at 100% accuracy. We use Adam optimization and train separate compression matrices for each trial. For each compressed model achieving perfect accuracy, we extract activations from all residual stream positions across trials. SAFs use fixed dictionary size 100, L1 coefficient 0.1, learning rate 10-3 training for 300 epochs with batch size 128, We analyze the final layer activations (post-MLP) for consistency across compression factors.
B.2 Multi- Task Task Sparse Parity Experiments
Dataset Construction. We use the multi-task sparse parity dataset from Michaud al. (2023) with 3 tasks and 4 bits per task. Each input consists of 3-dimensional one-hot control vector concatenated with
25

---

Published in Transactions on Machine Learning Research (12/2025)
12 data bits (total dimension 15). For each sample, the control vector specifies which task is active, and the label is computed the parity (sum modulo 2) of the bits corresponding to that task. This creates a dataset where ground truth bounds the number of meaningful features while maintaining task complexity.
Model Architecture. Simple MLPs with architecture Imput(15) + Linear(h) + ReLU + Linear(1)) where E {16,35 32, 64, 128, 256] for capacity experiments. We apply interventions (dropout) to hidden activations before the ReLU nonlinearity. Training uses Adam optimizer (Ir=0.001), batch size 64, for 300 epochs with Loss Dataset split: 80% train, 20% test with stratification by task and label.
Intervention Protocols. Dropout experiments: Applied hidden activations with rates [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9). Dictionary scaling: Expansion factors [0.5, 1.0, 2.0, 4.0, 8.0, 16.0) relative to hidden dimension, with L1 coefficients [0.01, 0.1, 1.0, 10.0), maximum dictionary size capped
1024. Each configuration lested across 5 random seeds with 3 SAE instances per configuration for stability measurement.
SAE Architecture and Training. Standard autoencoder with tied weights: z = ReLU W enc + b), x' WeecZ where Wdec = WT enc- Dictionary size typically 4x layer width unless specified otherwise.
Loss function: L = ||x x + A||z|| with L1 coefficient À 0.1 (unless testing sensitivity). Adam optimizer (Ir=0.001), batch size 128, 300 epochs. For stability analysis, we train 3-5 SAE instances per configuration with different random seeds and report mean + standard deviation.
B.3 Grokking
Task and Architecture. Modular arithmetic task: (a+ b) mod 53 using sparse training data (40% of all possible pairs, 60% held out for testing). Model architecture: two-path MLP with shared embeddings.
ea Fabbedding(a, dim = 12)
(22)
e, = Embedding((,, dim 12)
(23)
GELU(WIe, + Wges)
(24)
logits Wyh
(25)
where WT, W2 e 12 and W3 553448
Training Configuration. 25,000 training steps, learning rate 0.005, batch size 128, weight decay 0.0002.
Model checkpoints saved every 250 steps (100 total checkpoints). Random seed for reprodueibility.
LLC Estimation Protocol. Local Learning Coefficient estimated using Stochastic Gradient Langevin Dynamics (SGLD) with hyperparameters: learning rate 3 x 10-3, localization parameter y 5.0, effective inverse temperature n2 = 2.0, 500 MCMC samples across 2 independent chains. Hyperparameters seleeted via 5 x 5 grid search over epsilon range [3 x 10-5,3 5,3 2,3x x 10-1] ensuring é > 0.001 for stability and < 100 for (-independence.
B.4 Pythia-70M Analysis
Data Sampling and Preprocessing. 20,000 samples from Pile dataset (Gao et al., 2020), shuffled with seeds [42, 123, 456] for reproducibility. Text preprocessing: truncate to 512 characters before tok- enization to prevent memory issues. Tokenization using model's native tokenizer with max max__ength=512, truncation-True, no padding. Samples with empty text or tokenization failures excluded.
Model and SAE Configuration. Pythia-70M model with layer specifications: embedding layer, and fattn _utt, out, resid ] for layers 0-5. Protrained SAEs from Marks et al. (2024) with dictionary size 64 x 512 32, 768 features per layer. SAE weights loaded from subdirectories following pattern: 2010003278//aa/ pp.
26

---

Published in Transactions on Machine Learning Research (12/2025)
Activation Processing. Activations extracted using nnsight tracing with error handling for failed for- ward passes. Feature activations accumulated across all token positions and samples: feature_ssu = postiosss /z/. Feature count computed from accumulated sums using entropy-based measure. Mem- ory management: explicit cleanup of activation tensors and CUDA cache clearing between sceds.
B.5
Adversarial Robustness
B.5.1
Model Architectures
Simple Models (Single Hidden Layer)
SimpleMLP: Input(784) + Linear(h) - ReLU - Linear(output)
Hidden dimensions E 8, 32, 128, 512
SimpleCNN: Input - Conv2d(h, 5x5) - ReLU - MaxPool(2) - Linear(output)
Filter counts h E8, 16, 32, 64
Standard Models
StandardMLP: Imput(784) - Lincar(4h) ReLU - Linear(2h) + ReLU Linear(h) - ReLU
+ Linear(outputt)
Base dimension h 32, yielding layer widths [128, 64, 32]
StandardCNN; LeNet-style architecture
Conv2d(1, h, 3x3) + ReLU - MaxPool(2)
Conv2d(h, 2h, 3x3) ReLU MaxPool(2)
Linear(4h) ReLU + Linear(output)
Base dimension
CIFAR-10 Models
CIFARIOCNN: Three-block CNN with batch normalization
Conv2d h, 3x3) + BN - ReLU + MaxPool(2)
Conv2d(h, 2h, 3x3) BN ReLU MaxPool(2)
Conv2d(2h, 4h, 3x3) BN ReLU - MaxPool(2)
Dropout - Linear(output)
Base dimension
ResNet-18: Modified for CIFAR-10
Initial: Conv2d(3, 64, 3x3, stride=1, padding-1)
MaxPool replaced with Identity
Standard ResNet-18 blocks [2, 2, 2,
WedeResNet: ResNet-18 with variable width
Width factors: 1/16, 1/8, 144,1/2,1,2,4
Initial channels: 16 x width factor
Block channels: 16, 32, 64, 128 x width factor
27

---

Published in Transactions on Machine Learning Research (12/2025)
B.5.2 Training Protocols
MINIST/Fashioo-MMINNSS
Optimizer: SGD with momentum 0.9
Learning rate: 0.01, MultiStep docay at epochs [50, 75]
Weight decay: 10-4
Epochs: 100
Batch size: 128
PGD: 40 steps, step size 0.01
FGSM: Single step, a =
CIFAR-10:
Optimizer: SGD with momentum 0.9
Learning rate: 0.1, MultiStep decay at epochs [100, 150]
Weight decay: x 10-4
Epochs: 200
Batch size: 128
PGD: 10 steps, step size a = 2/255
FGSM: Single step, a =
B.6 SAE Configuration
Dictionary size: 4N (4x layer width)
Ll cocfficient: 0.1
Optimizer: Adam, learning rate 10-3
Training: 800 epochs with early stopping (patience 50)
Activation collection: 10,000 samples from test
Separate SAEs trained per layer
28


Reviews:
[{"review_id": "EzuqMAlcPD", "reviewer": "Reviewer_nNad", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper defines superposition as lossy compression and proposes a practical metric based on SAE activations: compute a feature‑usage distribution ($p$), set ($F=e^{H(p)}$) as the effective number of features, and measure compression/superposition as ($\\psi=F/N$). The metric aligns with toy‑model interference, indicates minimal superposition for compiled algorithmic tasks, shows dropout reduces ($F$) (adds redundancy), captures grokking consolidation, reveals a non‑monotonic layer profile in a small LM, and—most notably—finds that adversarial training can either increase or decrease ($F$) depending on task complexity × model capacity.\n\nKey strengths: principled information‑theoretic metric (effective counts), practicality once SAEs exist, broad demonstrations, improved clarity/statistics in the robustness section.\n\nKey weaknesses: dependence on SAE quality (feature splitting/ghost features), independence/linearity assumptions, small‑scale experiments, and remaining descriptive elements in the adversarial section.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The measurement claims are well supported: toy‑model checks track observable interference; hyperparameter/dictionary studies show stability/plateau; applications (dropout, Tracr, grokking, layer‑wise structure) demonstrate comparative usefulness. The adversarial‑training result—now with formalized hypotheses and meta‑analysis—supports a two‑regime story rather than a universal reduction, while acknowledging non‑monotonic layer behavior. Overall, evidence is sufficient for a measurement contribution.", "audience_yn": "Yes", "audience_explanation": "Readers focused on mechanistic interpretability, representation analysis, and robustness will value a portable metric that ties superposition to effective degrees of freedom and connects interpretability phenomena (dropout, grokking, ID) with a nuanced view of adversarial training (capacity×complexity).", "requested_changes": "1. Foreground the “lossy compression” framing and assumptions: make near‑optimality and lossless‑before‑lossy assumptions explicit; clarify that ($F$) counts virtual neurons (effective degrees), not human‑interpretable features; disambiguate superposition vs. polysemanticity.\n2. SAE ablation for measurement stability: on a representative subset (e.g., dropout + adversarial settings), add a comparison with a Gated or Top‑K SAE to show that qualitative ordering of ($F,\\psi$) is stable to known ($\\ell_1$) issues.\n3. Tighten adversarial‑training analysis: integrate the pre‑registered hypotheses and compact stats tables into the main text; report effect sizes/CIs and an ablation over capacity proxies (depth/width) to support the “capacity enables expansion” story. Keep anomalous layer trends visible.\n 4. Add a small comparison to effective rank / participation ratio, emphasizing conceptual differences (subspace usage vs. feature compression) and where they agree/diverge.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NUG21GkwuH", "reviewer": "Reviewer_CPqo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:** The authors introduce a new metric to quantify \"superposition\" in neural networks, i.e., the phenomenon in which some neurons encode multiple features in overlapping directions in activation space rather than dedicating each neuron to individual features. Existing methods to quantify this phenomenon have relied on knowledge of the true underlying features in the data and were thus scoped only to toy models. The authors instead propose to train a sparse autoencoder (SAE) on the activations of a model to extract a vocabulary of distinct features and use this to quantify the effective \"number\" of features using the entropy of the distribution of features extracted by the SAE (the probabilities are calculated over a fixed number of samples). Overall, this is aimed at quantifying the effective information capacity in the activations of a layer and yields a comparative metric that can be used to measure superposition in a relative sense between different conditions. The authors apply this metric to yield several insights, chiefly: (1) showing that dropout increases redundancy when it induces polysemanticity, thus reducing the effective number of features represented given fixed capacity; (2) analysing grokking and detecting it using feature counts; (3) tracking the effective representational capacity across layers in a transformer; and (4) showing that the relationship between adversarial training and the effective number of features depends on task complexity rather than universally causing feature counts to decrease. Overall, the paper aims at providing a better and more widely applicable metric for feature counting and measuring superposition for mechanistic interpretability in neural networks.\n\n**Strengths:**\n* The authors have done a good job of motivating their contribution, and the writing and figures are mostly clear (although the paper is quite information-dense).\n* The proposed metric is intuitive and principled in its design.\n* The paper lists several practical applications and experiments using the metric to analyse the effect of training and design choices on neural network features (in terms of counts and organisation). As such, it moves beyond previous work that was restricted in scope to toy models.\n\n**Weaknesses & Suggestions:**\n* The metric relies on the assumption that SAEs can recover the \"actual\"/\"ground truth\" features involved in the neural network's computations. This means that the metric is sensitive to how well the SAEs are trained, and I imagine in most practical cases there are not identifiability guarantees for these features. Thus, in practice, the metric would potentially always be computed over several random seeds and hyperparameter configurations for the SAEs and an average value over these configurations would probably be a better estimate to compare. To the authors' credit, they do show in some of their toy experiments that the metric is fairly robust in reasonable hyperparameter ranges.\n* However, related to my last point above, it is unclear how expensive computing this metric would be for larger (billion+) parameter models, and how well SAEs would scale to these models. I imagine there would be several more training challenges at that scale and assumptions underlying the metric violated as well.\n* Again on a related note, the authors have only tested fairly small models in their experiments, e.g., simple MLPs and CNNs, and Pythia-70M which is a much older and smaller model compared to modern LLMs. I understand that this may be compute-intensive, but truly evaluating the metric's utility for interpretability at scale would require analysing larger models (more modern CNN architectures such as ConvNeXt and ResNeXt, and modern open LLMs that are not too expensive to deploy, such as Llama 2 or Mistral).\n* The assumptions underlying the metric, such as feature independence, linear representation assumption, etc. are fairly restrictive and may not hold for most models. However, this is not unique to this paper alone, and I think the authors have done a good job of listing their limitations and have managed to get interesting insights with smaller models despite these limitations.\n* One of my key concerns, in general with several works in mechanistic interpretability, is the focus on static layer-wise representations and the lack of modelling dynamics (either through several layers or over several input tokens). Modelling activation dynamics can constrain the features you identify (hence neuroscience's focus on population dynamics [1,2,3]), so it is possible that you might get closer to identifying the actual computation being performed and better feature counts [4]. There are also identifiability guarantees you can get with modelling dynamics [5,6].\n* Using the metric in its current form ignores features that largely co-occur or represent the same underlying computation. Instead of calculating the entropy from the raw extracted features directly, the authors could attempt to improve their feature counts by adjusting for \"motifs\" of features that always co-occur by counting feature pairs across samples. This could help deal with the feature splitting issue.\n* Some results such as those in section 6 are quite dense, and could be summarised (maybe at the end) so readers can understand the main insights clearly.\n\n**References:**\n1. Vyas, Saurabh et al. \"Computation Through Neural Population Dynamics.\" Annual review of neuroscience vol. 43 (2020): 249-275. doi:10.1146/annurev-neuro-092619-094115\n2. Williamson, Ryan C et al. \"Bridging large-scale neuronal recordings and large-scale network models using dimensionality reduction.\" Current opinion in neurobiology vol. 55 (2019): 40-47. doi:10.1016/j.conb.2018.12.009\n3. Pandarinath, C et al. \"Inferring single-trial neural population dynamics using sequential auto-encoders.\" Nat Methods 15, 805–815 (2018). doi:10.1038/s41592-018-0109-9\n4. Fernando, Jesseba, and Grigori Guitchounts. \"Transformer Dynamics: A neuroscientific approach to interpretability of large language models.\" arXiv preprint arXiv:2502.12131 (2025).\n5. Lippe, Phillip, et al. \"Citris: Causal identifiability from temporal intervened sequences.\" International Conference on Machine Learning. PMLR, 2022.\n6. Balsells-Rodas, Carles, Yixin Wang, and Yingzhen Li. \"On the identifiability of switching dynamical systems.\" arXiv preprint arXiv:2305.15925 (2023).", "claims_and_evidence_yn": "Yes", "claims_explanation": "I think the main claim that the proposed metric effectively quantifies superposition has been justified through clear experiments with toy models (Section 5), and also applied to understand feature counts and organisation in several other setups (Section 6). Furthermore, the other main claim that superposition or feature counts may not simply decrease as a result of adversarial training, but change depending on the task the network is performing, has been empirically justified (Section 6.4). Finally, the method's robustness to sensible hyperparameter values/ranges has been demonstrated (Figures 3b & 5a), although I do have one question related to this (see Additional Comments).", "audience_yn": "Yes", "audience_explanation": "This submission focuses on mechanistic interpretability of large models, which is an active and important area of research in the machine learning community. This research has a clear audience among TMLR's readership, i.e., researchers interested in mechanistically understanding the inner workings of deep learning models; it is also clearly aligned with one of the core themes in TMLR's CfP, i.e., \"new approaches for analysis, visualization, and understanding of artificial or biological learning systems\".", "requested_changes": "I am not an expert in mechanistic interpretability (my background is in computational neuroscience), so the following points should be viewed more as suggestions to improve the work rather than critical adjustments:\n\n1. Testing the method on models of a larger scale, i.e., more modern and larger CNNs and LLMs.\n2. Adjusting feature counts based on features that regularly/always co-occur, thus reducing the effect of feature splitting in SAEs.\n3. Incorporating analysis of features across several layers or input tokens, i.e., bringing the method closer to identifying the \"dynamics\" of computation.\n4. Exploring other ways to quantify feature contribution, e.g., the change in outputs or performance by lesioning each feature?\n\nI view the first three points as more important, and especially 1 & 2 as relatively easier to implement in the current framework.", "broader_impact_concerns": "I do not see any ethical concerns that could arise uniquely from this contribution. There are several ethical concerns associated with LLM/foundation model research that such methods could help alleviate. However, it is also possible that existing models (or potential future models) violate the foundational assumptions used in this work to extract features from model activations, and I believe these limitations have already been discussed adequately in the submission.", "additional_comments": "As stated earlier, I work on computational neuroscience and am not an expert in mechanistic interpretability – so my comments might be more high-level and inspired by similar research directions in my field.\n\nFor now, I had one additional question: There is a downwards trend in correlation values when scaling the dictionary size beyond 8x the true number of features. Could the authors expand on why this is happening and maybe try to see what happens at, e.g., 64x or beyond?", "extra_scores": {}, "extra_text": {}}, {"review_id": "YvvvekTZ04", "reviewer": "Reviewer_5fcf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThe authors propose a novel metric for measuring superposition in neural network layers and use it to analyze several deep learning phenomena. The authors quantify superposition as the ratio of the effective number of features to the layer's dimensionality, where they calculate the effective number of features using a two-step procedure. During the first stage, the features encoded in a layer are disentangled with a sparse auto-encoder (SAE) trained over the layer's output. Then, the total number of features is quantified using an entropy-based metric calculated over the SAE's feature probabilities. Section 5 validates this metric on several problems (toy problem of superposition, Tractr, and sparse parity). Section 6 uses this metric to analyze several phenomena in deep learning: dropout, grokking, intrinsic dimensionality, and adversarial training. The authors specifically focus on adversarial training and show that this technique either increases the number of features in the abundance regime or decreases the number of features in the scarcity regime.\n\n\n**Strengths**\n\n1. The considered problem, the quantification of superposition, seems relevant for the interpretability of neural networks.\n2. SAE seems to be an appropriate technique for feature disentanglement.\n3. The choice of the entropy-based metric seems motivated by the related works.\n4. Section 6 sheds light on the mechanism of some standard learning techniques. For instance, I think the conclusions of Sections 6.1 and 6.4 highlight some interesting underlying mechanisms of the considered phenomena.\n\n\n**Weaknesses**\n\n1. The metric is not well-motivated overall, even though I understand the logic behind some parts of its construction. Specifically, I do not understand what the effective number of features really measures (i.e., the semantics of this metric). For instance, I do not understand in which situations the entropy-based measure performs better than threshold-based measures (e.g., the dimensionality of SAE minus the number of \"dead\" neurons). Additionally, I do not think that the ratio of the number of features to the layer's dimensionality is well-motivated.\n2. Additionally, I do not understand some choices made in Section 3.2. For instance, the authors omit the bias term in feature decoding and use weight tying for encoding and decoding matrices. These choices are not standard in the literature. Typically, bias terms are present during both encoding and decoding stages, and encoding and decoding matrices are optimized independently, even though they might share the initialization (Bricken et al., 2023, Appendix \"Advice for Training Sparse Autoencoders: Autoencoder Architecture\"). Moreover, the l1-regularization is often avoided due to the feature suppression problem (Wright and Sharkey, 2024; Rajamanoharan et al., 2024).\n3. Section 5 does not sufficiently validate the metric. I will expand on this point in the following box.\n4. The findings in Sections 6.1, 6.2, and 6.3 are not thoroughly tested. Each of these sections only considers one learning task and one architecture.\n5. I think that the mechanisms proposed in Section 6.4 are not sufficiently validated. I will expand on this point in the following box.\n\n\n**References**\n\nBricken, T., et al. (2023). Towards monosemanticity: Decomposing language models with dictionary learning.\n\nWright, B., and Sharkey, L. (2024). Addressing feature suppression in SAEs.\n\nRajamanoharan, S., et al. (2024). Improving sparse decomposition of language model activations with gated sparse autoencoders.", "claims_and_evidence_yn": "No", "claims_explanation": "Given the outlined weaknesses, I think that the article does not provide sufficient evidence for its claims.\n\n1. I do not understand the scope of validation in Section 5. Specifically, I do not understand which properties of the metric are validated. Moreover, I think that the validity of the proposed tests is insufficient. Now, I will cover all subsections in more detail.\n2. Section 5.1 compares the metric with the \"weight mass\" metric, allegedly proposed by Elhage et al. (2022). First, I want to note that Elhage et al. (2022) do not propose the \"weight mass\" metric. Elhage et al. (2022) only proposed a similar \"dimensions per feature\" metric, which they only used as a secondary metric in the analysis of uniform superposition (when all features have the same importance). However, the current manuscript uses this metric in the case of decaying feature importance, in which Elhage et al.'s (2022) metric does not semantically correspond to the ground truth. Moreover, the authors implicitly assume that $\\psi_{ours}(W_{toy})$ constitutes a ground truth number of features in the toy setting, but they do not justify this claim.\n3. Section 5.2 applies the new metric to Tractr-generated transformers. First, while the authors claim that they analyzed original transformers, I do not see these results in the submitted manuscript. Second, I do not understand the scope of validation in this setting. The authors only claim that feature counts stay below the $F=N$ line; this finding seems more fit for Section 6. (Also, note that Figure 4a is incorrect; Lindner et al. (2023) use transposed data and weight matrices, hence, all matrices in Figure 4a should also be transposed.)\n4. Section 5.3 applies the new metric to sparse parity tasks and shows that feature count does not explode under an appropriate regularization. While I think that this finding strengthens the applicability of the authors' metric, I again do not understand what constitutes the ground truth for comparison. Specifically, I do not see how the authors conclude that they \"we measure intrinsic structure, not artifacts\" using their metric.\n5. As I outlined above, I think that the scope of evaluation in Sections 6.1, 6.2, and 6.3 is insufficient.\n6. As for Section 6.4, I think the current evaluation lacks rigor and mathematical precision. For example, in Figure 7 in the MNIST row, we can notice that the relative number of features of the first layer (linear 1 or conv 1) increases when we switch from 2-class classification to 3-class classification. Since 3-class classification is harder than 2-class classification, this observation contradicts the statement that \"task complexity determines defensive strategy\". Similarly, the behavior of conv, layer 1, and layer 3 is inconsistent in Figure 8b. I suggest that the authors use statistical tools to test these hypotheses to avoid confusion.", "audience_yn": "Yes", "audience_explanation": "I think that the considered question is important for the interpretability of neural networks, and the mechanisms outlined in Sections 6.1 and 6.4 are interesting.\n\n1. Superposition seems to be one of the main roadblocks to the interpretability of neural networks (e.g., in terms of neurons). Thus, metrics that quantify this phenomenon might be useful for research in this direction.\n2. Section 6.1 suggests that dropout consumes the capacity of neural work, which refines the applicability of this technique.\n3. Section 6.4 suggests that mechanisms of adversarial training might differ between small and big networks, which could potentially help with the analysis and application of this technique.\n4. Overall, Section 6 demonstrated how the superposition metric might complement the analysis of several deep learning phenomena, which suggests that this metric is a valuable methodological tool.", "requested_changes": "1. Please motivate the effective feature count metric in more detail.\n2. Please explain the scope and the goal of experiments in Section 5.\n3. For Section 5.1, please explain the motivation behind the \"ground truth\" measure or change the scope/positioning of evaluation.\n4. For Sections 5.2, 6.1, 6.2, and 6.3, please either make the conclusions sound more speculative or extend the scope of experiments.\n5. For Section 5.3, please explain the motivation in more detail and refine the scope of the conclusion.\n6. For Section 6.4, please formalize the tested hypothesis and present the results of statistical testing.\n7. Please motivate your design choices in Section 3.2.\n8. (minor) Please fix Figure 4a and the corresponding description.\n9. (minor) Please present findings for the original transformers in Section 5.2.\n10. (minor) I think the paper would benefit from the discussion of effective rank (Roy and Vetterli, 2007) and participation ratio (Recanatesi et al., 2022) metrics, given that they approximate the rank of the matrix, which is close to the effective number of features.\n\n\n**References**\n\nRoy, O., and Vetterli, M. (2007). The effective rank: A measure of effective dimensionality.\n\nRecanatesi, S., Bradde, S., Balasubramanian, V., Steinmetz, N. A., and Shea-Brown, E. (2022). A scale-dependent measure of system dimensionality.", "broader_impact_concerns": "I do not identify direct ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SmfeQOPP2z", "reviewer": "Reviewer_kgBT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new method to measure superposition—how neural networks encode multiple features into the same neurons. The authors use sparse autoencoders (SAEs) and an information theory formula (the exponential of Shannon entropy) to count the \"effective number of features\" in a network layer. The key finding contradicts the popular theory that superposition causes adversarial vulnerability. Instead of reducing superposition, the research shows that adversarial training often increases the number of features a model uses. The effect seems to be depending on the situation", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the claims made in this paper are supported by accurate, convincing, and clear evidence. It is a combination of theoretical grounding, validation in controlled settings, and large-scale empirical experiments—to build a strong case for both the authors proposed measurement technique and their subsequent scientific findings. In reviewing the paper, The submission's evidence is particularly strong because it is systematic and thorough. The authors don't just present a new metric; they rigorously validate it against known ground truths, test its stability, and then apply it to answer a significant open question in the field.", "audience_yn": "Yes", "audience_explanation": "In reviewing the paper, The submission's evidence is particularly strong because it is systematic and thorough. The authors don't just present a new metric; they rigorously validate it against known ground truths, test its stability, and then apply it to answer a significant open question in the field. I believe the audience will particularly find the main claim - relationship between superposition and adversarial robustness is more nuanced than previously thought—is especially compelling due to the breadth of experiments across different models, datasets, and conditions.", "requested_changes": "(1) Add Qualitative Examples of Feature Granularity - The paper notes that feature granularity is fundamentally ambiguous. I would recommend supplementing the quantitative dictionary scaling analysis (Figure 5a) with a qualitative example. Visualize a single, broad feature discovered by an SAE with a small dictionary and then show how it splits into multiple, more fine-grained features when a larger dictionary is used. This would make the abstract concept of granularity much more concrete.\n(2) Propose a Method for Multi-Layer Analysis - The current analysis is performed on a per-layer basis. Including a \"future work\" section to outline a potential methodology for extending the analysis across layers or entire network blocks. \n(3) Provide a experiment to Probe Causality - The paper demonstrates a strong correlation between adversarial training and changes in superposition but explicitly notes that establishing a direct causal link is an open question. It would benefit to strengthen the conclusion by proposing a concrete future experiment to test for causality.", "broader_impact_concerns": "the work presented  is foundational and methodological, focusing on creating a tool to measure an internal property of neural networks. As such, it does not have direct ethical implications that would necessitate a detailed Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)