Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learning
Authors: Xinting Huang
Michael Hahn
Venue: ICLR 2026 Poster (2026)
Keywords: Mechanistic Interpretability
Unsupervised Learning
Representation Space Geometry
Primary area: interpretability and explainable AI
TL;DR: We introduce an unsupervised method to decompose representation space into interpretable subspaces

Abstract: Understanding internal representations of neural models is a core interest of mechanistic interpretability. Due to its large dimensionality, the representation space can encode various aspects about inputs. To what extent are different aspects organized and encoded in separate subspaces? Is it possible to find these "natural" subspaces in a purely unsupervised way? Somewhat surprisingly, we can indeed achieve this and find interpretable subspaces by a seemingly unrelated training objective. Our method, neighbor distance minimization (NDM), learns non-basis-aligned subspaces in an unsupervised manner. Qualitative analysis shows subspaces are interpretable in many cases, and encoded information in obtained subspaces tends to share the same abstract concept across different inputs, making such subspaces similar to "variables" used by the model. We also conduct quantitative experiments using known circuits in GPT-2; results show a strong connection between subspaces and circuit variables. We also provide evidence showing scalability to 2B models by finding separate subspaces mediating context and parametric knowledge routing. Viewed more broadly, our findings offer a new perspective on understanding model internals and building circuits.

Published as a conference paper at ICLR 2026
DECOMPOSING REPRESENTATION SPACE INTO INTERPRETABLE SUBSPACES WITH UNSUPERVISED LEARNING
Xinting Huang
Michael Hahn
Saarland University (xhuang, mhahn) @lst uni-s- -asssaaraaadd.. de
ABSTRACT
Understanding internal representations of neural models is a core interest of mech- anistic interpretability. Due to its large dimensionality, the representation space can encode various aspects about inputs. To what extent are different aspects organized and encoded in separate subspaces? Is it possible to find these "natural" subspaces in a purely unsupervised way? Somewhat surprisingly, we can indeed achieve this and find interpretable subspaces by a seemingly unrelated training objective. Our method, neighbor distance minimization (NDM), learns -on-basis-aligned subspaces in an unsupervised manner. Qualitative analysis shows subspaces are interpretable in many cases, and encoded information obtained subspaces tends to share the same abstract concept across different inputs, making such subspaces similar to "variables" used by the model. We also conduct quantitative experiments using known circuits in GPT-2; results show strong connection between subspaces and circuit variables. We also provide evidence showing scalability to 2B models by finding separate subspaces mediating context and parametric knowledge routing [1] Viewed more broadly, our findings offer a new perspective on understanding model internals and building circuits.
INTRODUCTION
Mechanistic Interpretability research aims to understand the internal mechanisms of neural networks.
Unlike behavioral research, the community seeks deeper algorithmic descriptions or circuits of models. Insights from this kind of research can serve our purpose of building safer, more controllable,
and more reliable AI systems (Bereska & Gavves 2024), and making progress in other scientific
fields by learning how data is modeled by these models (e.g. Futrell & Mahowald 2025 2023 Celeghin et al. 2023).
Oota et al.)
Researchers have made significant progress in this direction in recent years, uncovering circuits consisting of various types of basic units (or mediators (Mueller et al. 2024)). including components (attention heads and MLPs) of transformers (Wang et al. 2022 Hanna al. (202)). sparse features obtained from Sparse Auto-Encoders (SAEs) or its variants (Cunningham et al. 2023 Bricken et al.
Dunefsky al.) 2024: Ameisen al. 2025), and subspaces (Geiger al. 2024 Wu et al. 2023 Geiger et al. 2025). While providing valuable insights, they are limited different ways. The information transferred between components is hard understand, and researchers need to design specialized counterfactual data targeting certain aspect of the input to perform causal intervention, or use easy but much less reliable methods such observing attention weights or projecting to vocabulary space. Sparse features provide a direct way to interpret (by reading from the inputs they activate on), but resulting circuits are input-dependent, in other words, specific for each input sequence, as different features are activated in different inputs. This approach also usually modifies the model's computation, replacing activations or MLPs with reconstruction from pparsity-encourage new modules, in order to better analyze feature-to-feature interactions. Lastly, current subspace-basee approaches require supervision from an abstract causal model specified by human. The abstract
2023
Link for code and trained matrices Stcceessttaee httpt//cccctt//////////////o///n

---

Published as a conference paper at ICLR 2026
model represents clear hypothesis of how the real model works, thus requiring anticipating possible mechanisms. We refer to Mueller et al. (2024) for a good summary of different mediators.
In this paper, explore new direction: unsupervised representation space decomposition. We show we can learn a partitioning of representation space by doing neighbor distance minimization (NDM) among subspace activations. We explain the intuition behind this training objective in terms of feature groups and mutual information and show that it looks for partition such that subspaces are independent as possible. Similar Geiger et al. (2024), an orthogonal matrix is learned, which rotates and reflects the space before partitioning on the transformed basis. It captures the distributed nature of neural representations. While they aim find the subspace for certain target variables by supervised learning, our method is unsupervised and makes use of "natural" structure inside the model. We first demonstrate the effectiveness of NDM in toy setting, then move on to real language models. Somewhat surprisingly, we find that there is indeed meaningful inner structure of representation space in real- models. We show that resulting subspaces are usually interpretable with Inversion View (Huang et al. 2024b). We quantitatively test alignment between subspaces and variables in known circuits in GPT-2 Small using causal intervention. Moreover, we demonstrate its applicabiliit on larger 2B models by showing that NDM finds separate subspaces mediating context knowledge and parametric knowledge.
Our method also points out potential future direction: subspace circuits. Subspaces given by our method are suitable as new basic units for circuit analysis, as we can directly analyze model weights understand, for example, which subspaces certain attention head reads from and writes to. By building connections between subspaces across layers with model weights, we could construct input-independent circuits, as the space decomposition and model weights are input-independent. In these circuits, subspaces are like variables, each of which encapsulates a group of features it can take as values. Therefore, we believe our method and the possibilities it enables will benefit the interpretability community.
2 MOTIVATION AND BACKGROUND
One of the key concepts in interpretability literature is superposition, which means a model can represent more features than its dimensionality by representing them as non-orthogonal directions. The phenomenon is well-illustrated in (Elhage et al. (022). where they use toy model as follows:
h -
x' = ReLU(W" h +
where x, x' E R h E W E Rdx= d < 2. The high- nigh-dimensional x is the ground truth feature vector, each entry F; represents a "feature" and its value ranges in [0, 1]. it is encoded in low-dimensional h. The model output x' is trained to reconstruct æ. So each column in W corresponds to a direction in the lower-dimensional space that represents feature 24.
In this toy setting, (Elhage et al. 2022) found that sparsity is necessary to allow superposition to occur. In this paper, we argue that mutual exclusiveness could be more fundamental condition. More specifically, for a group of features, if only one (or none) occurs at a time, they are said to be mutually exclusive, and can be represented with superposition without much loss of information. Although sparsity encourages mutual exclusiveness, they are different, since the former concept treats features uniformly while the latter emphasizes non-uniform view under which some sets of features can be more dependent on each other.
If mutual exclusiveness holds, superposition works well; the only discrepancy between x and x' is because of small interference between almost orthogonal features. On the contrary, if multiple features are activated, they might cancel each other certain directions, causing loss of information, so that reconstructed x' can be totally different from x (Fig. (6a and 6b in App.). In other words, given two or more specific features, if they often co-occur, superposing these features would not work well incentivizing the model to represent them more orthogonally. On the other hand, if they mutually exclusive (because the concepts they represent are mutually exclusive in the real world), the model would tend to make use of this fact and superpose their features.
2

---

Published as a conference paper at ICLR 2026
Feature Groups and Subspaces It would be more interesting if we consider groups of mutually exclusive features. Features are mutually exclusive with features in the same group but independent of features in different groups. So each model activation h can result from multiple activating features, but each of them comes from a different group. As we discussed earlier, features in the same group tend to be superposed, while co-occurrence would encourage features from different groups to be more orthogonal. Therefore, in such an ideal case, each feature group can form a dedicated subspace, superposition happens inside the subspace, but subspaces are orthogonal to one another.
One might ask whether the mutual exclusiveness property is likely to happen in the real world. We think it is actually pervasive. For example. if the previous token is "the". it cannot be any other tokens in the vocabulary; if the subject of sentence is "Alice", it cannot be "Bob"; if the sum of two digits 5, it cannot be simultaneously other numbers. model's 's processing, like an algorithm, might have "variables" (like those searched by DAS (Geiger et al.) 2024)) to encode certain aspects of information, and each variable can take only one value at time, and each corresponds to a feature group, or subspace. So we think such mutual-excluiiveness property and group structure can be very common in the model, and there are "natural" partitions of orthogonal and interpretable subspaces. In the following section show evidence in toy setting, and in Sec. []] we show evidence in real neural language models. In addition, we find our notion of feature groups quite relevant to multi-Dimensional Superposition Hypothesis (Engels et al. )024). each feature group can be thought of as one multi-dimennional irreducible feature in this hypothesis, because of intra-group dependence and inter-group independence.
Toy Model of Intra-group Superposition In the toy setting (Elhage et al. 2022) where we have access to real features, we can straightforwardly verify superposition and orthogonality. We use the aforementioned toy model (Eq. [. As a start, we set = 40, d = 12, and features are split into two groups, 20 for each. When sampling æ, for each group, 0.25 of the time no feature is activated, 0.75 of the time one feature is randomly chosen and its value is sampled uniformly from [0, 1]. Features are equally important (not weighted when computing reconstruction loss).
After training, fraction of variance unexplained (FVU) is 0.059, and W is shown in Fig. la We see that products of features coming from the same group are non-zero, while features in different groups are always orthogonal each other. There are two orthogonal subspaces containing the two feature groups, respectively. Closer inspection of singular values shows that each feature group is encoded in 5-dimensional space. In App. [.I we show different configurations of feature groups and the resulting W W. In sum, we can expect similar orthogonal subspaces when features follow the aforemennioned group structure.
METHODOLOGY
If orthogonal structure exists, i.e., orthogonal partition of representation space such that each subspace contains one feature group, how can we find these subspaces? In toy setting, we can take all feature vectors from the same group, and then the subspace spanned by these what we are looking for. But we aim to create method that works for real-world models, i.e., without access to ground truth features & or projection W. We need a method that only requires model activations h. this section, we describe method, NDM, which finds the correct subspace given only model activations.
The intuition behind our method is illustrated in Fig. Tb In a 3D space, assuming xy plane contains group of 3 features (blue arrows), and dimension contains another group of features (orange arrows), so there are subspaces orthogonal to each other and each contains a feature group. If intra-group mutual exclusiveness holds, data points will only lie in the pink planes. as each point is a sum of only one blue and one orange. Given the feature groups. the correct partition is thus (xy, 2). If the 3D space is partitioned in this way, the data points' projection on the xy subspace only lies on blue arrows, and similarly for the subspace. Thus, given some data points, inside each subspace (consider their projection) points are concentrated on few lines, and it is easy for each point to find another point in a close distance. However, if it is partitioned into xz and y, then the data points, after being projected to xz plane, cover the entire plane, causing larger distance between each point and their neighbor. In other words, the correct partition will make the distance to the nearest neighbor inside subspaces smaller, whereas incorrect partitions entail large distances because they arbitrarily combine features from different groups
Intuition
3

---

Published as a conference paper at ICLR 2026
(d)
(a)
(b)
Figure 1: (a): W W. W E 12 (Sec. 2). Each column of W is feature direction, its first 20 columns belong to one feature group, the last 20 to the other. We can see the bottom left and top right parts zero, which means feature vectors of different groups orthogonal. (b): Illustration for the intuition of our method (Sec. 3 correct partition can make data points casier to model. (c) (d): RW before and after NDM training (Sec. 4 The final dimension configuration is dj 6, d2 6. The corresponding toy model shown in (a). After training, the first subspace (first 6 coordinates)
only depends the first feature group (first 20 coordinates), and likewise for the other subspace.
Optimization We define partition tuple consisting of an orthogonal matrix R and dimension configuration e (number of subspaces and their dimensionality). R. rotates and reflects the space as needed (similar to (Geiger al. (2024), then the transformed space is partitioned on its standard basis (simply grouping coordinates) according to c. We optimize R. as follows: given N model activations [h1 hN] where hn E IRd, and an orthogonal matrix to be trained R 1 F dimension configuration e = [d,, ds]. where the number of subspaces, Σ dg = d, we do
(1)
hn
henl = R(h, hnl. [h .h...hpp
hn
h
Rdd.
(2)
(8)
n" = arg min dist (hi (2())
(3)
N..., N
m
min
1 N #=1==1
(e) dist (hn hn-)
s.t. S.t. R
(4)
where the function dist(-) (-) the distance metric we can experiment with. we optimize the orthogonal matrix to minimize the distance with the nearest neighbor in subspaces.
Besides the aforementioned perspective in terms of feature superposition, there also heuristic interpretation of our method minimizing total correlation between subspaces, which generalizes mutual information (MI)'s definition from two variables to multiple variables and measures the redundancy and dependence among them. Loosely speaking, neighbor distance reflects the entropy; our method reduces entropy subspaces. Entropy of the whole space is invariant under orthogonal transformation: thus, total correlation is reduced (details in App. B.II.. In other words, look for a partition of representation space such that the resulting subspaces are as independent as possible, thus they suitable perspective justifies our method even when the previous feature group hypothesis does not strongly or strictly hold in real neural models.
as the elementary units of interpretation. This MI/total correlation
Determining the Subspace Dimension Configuration However, there is still a question: how to find the correct dimension configuration (number of subspaces and their dimensions)? is not
2 Libraries like Pytorch Paszke
2019 contain parametrization ensure orthogonality of matrix
4

---

Published as a conference paper at ICLR 2026
differentiable and is unclear what is the appropriate objective optimize for. In our method, use MI to address this problem. The procedure is as follows: we start with a configuration of small and equal-sized subspaces and train the orthogonal matrix. We measure the MI between subspaces regularly during training using KSG estimator (Kraskov et al., 2004). If the M is higher than a threshold, merge the two subspaces, the nearest neighbor should be found in the joint subspace afterwards and keep training the orthogonal matrix. We stop training when we cannot find a pair of subspaces whose MI is above the threshold. This procedure is in line with our goal of reducing the dependency between subspaces, as we merge them if their MI is unavoidably high. We describe We also experimented with other methods for determining the configuration, see App. G.4 In sum, NDM finds a partition containing both R and c.
our method in detail in Algorithm and App. [
4 EXPERIMENTS IN TOY MODELS
We first test our method in the toy model setting described See 2 using a slightly different and less scalable version of Algorithm| (see details in App. C.2 We use Euclidean distance as the dist(.) Eq. and because we find cosine similarity to perform worse in preliminary experiments. The toy model setting is an ideal testbed we have access to the ground truth feature vectors and subspaces. As described, we rotate and reflect the hidden activations h from [q[]] which results in h = RWæ. Subspace activations, which are defined to be a continuous span of coordinates of h. should ideally result from only one feature group. Therefore, a good R means rows in RW corresponding to one subspace (subvectors of h) have nonzero values only at columns corresponding one feature group (subvectors of æ). So, the important question is, can we find subspaces encoding each feature group by minimizing distances? Despite our argument before, this still sounds uncertain a priori, as we do not optimize R to associate subspaces with underlying groups but rather optimize the indirect objective of neighbor distance, as we want to avoid using W and æ. Somewhat beyond our expectation, the answer is yes, we find good subspaces by using NDM.
The result is shown in Fig. 1d We initialize R as an identity matrix, and the RW after training is almost perfect. We present more experiments in Fig. 10 in App. C.2 where numbers of features and groups vary. The results overall show strong performance of NDM, we are able to find the subspace partition close to the ground truth in all cases. In sum, in the toy setting where there exist orthogonal subspaces dedicated to each group, we indeed can find these subspaces by NDM.
5 EXPERIMENTS IN LANGUAGE MODELS
The key question is NDM's s applicability to real-world neural models. Unlike the toy setting, we do not know ground-trute truth features, we design new ways to do evaluation. In this section, we denote residual stream activations as h (mid.pppo where is the layer index (starting from 0), mid means after adding the attention sublayer's output, and post means after adding the MLP sublayer's output.
5.1 QUANTITATIVE EVALUATION BASED ON GPT-2 CIRCUITS
The evaluation is based on subspace activation patching, or subspace interchange intervention (Geiger ett al.] (2024). The main intuition is that, when processing inputs, key intermediate results should ideally lie in a single subspace. For example, an induction head circuit where the residual stream contains the previous token, the previous token should be placed in a single subspace, independent of which token it is from. Therefore, if we patch subspace activations with the value they take in sequences with different previous tokens, the effect should be concentrated on that correct subspace. We refer to the previous token as the target information, which is changed in the counterfactual input. Different target information should ideally be concentrated in different subspaces. We describe subspace patching in detail in App. D.I and address potential concerns from (Makelov et al. (002)) in App. G.2 In sum, we aim to explore whether the high-level "variable" lies in the same subspace no matter what value it takes by measuring inequality or concentration of subspace patching effect.
Test Suite and Metric We construct test suite, making use of known circuits in GPT2 Small. It consists 5 tests. The first 4 tests are based on the IOI circuit (Wang et al. 2022). where we intervene
In practice, we divide the mutual information value by the sum of the two subspaces' dimensions.
5

---

Published as a conference paper at ICLR 2026
Method
test
test 2
test 3
test 4
test 5
Avg
d,
Var,
d,
Var,
d,
Var,
d,
Var,
d,
Var,
Identity
0.33
0.05
0.23
0.32
0.11
0.25
0.40
0.12
0.19
0.31
0.04
0.15
0.32
0.12
0.19
0.21
Random
0.36
0.10
0.11
0.36
0.16
0.16
0.32
0.11
0.12
0.33
0.10
0.12
0.39
0.16
0.18
0.21
PCA
0.43
0.42
0.37
0.46
0.22
0.52
0.50
0.30
0.51
0.38
0.22
0.36
0.35
0.24
0.36
0.38
PCA 2
0.66
0.56
0.39
0.26
0.19
0.24
0.28
0.19
0.29
0.40
0.20
0.19
0.40
0.26
0.27
0.32
Feature
0.60
0.44
0.60
0.61
0.41
0.67
0.73
0.39
0.54
0.74
0.31
0.54
0.71
0.56
0.61
0.56
NDM
0.89
0.89
0.90
0.71
0.72
0.83
0.63
0.70
0.38
0.50
0.55
0.79
0.72
0.67
0.78
0.71
Table 1: Results on GPT-2 Small test suite, higher is better. Test 1-5 measure the concentration of various target information in various places in GPT-2 Small (details in App. D.2) based on ground truth circuits (Wang et al.] 2022 Hanna et al. 2023). Numbers are Gini coefficient over subspace patching effects ("-"), over effects normalized by subspace dimension ("d,"), and over effects normalized by subspace variance ("Var,"), Last column is the average values over the whole row. We compare NDM with 5 baselines. An identity matrix ("Identity"); random orthogonal matrix ("Random")) "); PCA 1 and are matrices composed of eigenvectors after applying PCA on activations. We use dimension configuration [dl..... ds] of NDM result as the configuration for all baselines. [d1.... dg] is sorted in descending order, eigenvectors can be sorted in either way according to eigenvalues, resulting in PCA 1 and 2 meaning either that the smallest subspace receives the biggest ("PCA I"") or smallest ("PCA 2")) directions of variance; and subspaces obtained by clustering SAE features ("Feature"), using a method adapted from (Engels et al. 2024 (details are in App. F2 Importantly, as shown in App. D.4 the degree of concentration of target information becomes satisfactory when Gini coefficient > 0.6.
on the following pieces of information respectively: (1) previous token information contained in
layer 4 residual stream (2) subject position in layer 6 (3) subject position and (4) subject name in layer 8 residual stream. The last test is based on Greater-than circuit (Hanna et al., 2023): (5)
information about the last two digits of the starting year in layer 9. We measure patching effect as the change in logit difference for IOI-based tests and change in valid answer probability for the Greater-than based test. See more details in in App. D.2 We then quantify the inequality of patching effect using Gini coefficient. Though very unconventional in machine learning, we find Gini coefficient a suitable metric. Because we want to measure how strongly patching effect is focused on just one or a few subspaces, this notion captured by Gini coefficient as it measures inequality or degree of concentration. If the target information is mainly encoded in one subspace, the patching effect for this one would be much higher than others, thus resulting in Gini coefficient close to 1. On the other hand, evenly distributed patching effect results in O. In App. D.4 we show examples of patching effect distributions and their corresponding Gini coefficients, from which we conclude that the coefficient should be 0.6 to be considered satisfactory (i.e., the effect is mainly concentrated in one subspace). We also calculate the Gini coefficient over patching effect normalized by dimension d, and variance Var, of subspaces.
Results We apply NDM to find subspace partitions in GPT-2 Small (Radford et al. 2019). details are described in App. [33] The results of NDM using the best hyperparameters are shown in Table together with 5 baselines, which, like NDM, are unsupervised and purely computed from activations. As we can see, NDM clearly outperform baselines. Though there are a few cases where Gini coefficient is low for NDM, the overall average value shows a strong concentration of target information (0.71, significantly higher than 0.6). Figure 22 29 and 30 show the qualitative examples of the subspaces obtained by NDM with the highest effect Test 1, 2, 5 respectively (See later section for how to read them). In App. [.3 we show ablation studies, providing experimental data on how different factors affect final results. Meanwhile, the estimated MI between subspaces decreases significantly after training (Fig. 15). matching the explanation of NDM's training objective.
5.2 QUALITATIVE ANALYSIS OF OBTAINED SUBSPACES IN GPT-2
Preliminaries (Hang et al., 2024b) show that we can interpret activations in terms of their preimage, which is the set of inputs that are mapped to the same activation by the model. Simply speaking, to interpret an activation (a vector, called query activation) at a certain place in the model (e.g., layer 3 residual stream, called activation site), we find those inputs that, when fed into the model, would give rise to roughly the same activation at that activation site. Therefore, when query activation is observed by downstream components, any input in preimage could be the current input,

---

Published as a conference paper at ICLR 2026
loai Somee
Vermscc
Spamasiion
?
Layer representation
Layer i representation
prey
100-117
100-119
poc-119
poor
Rolate and Reffect
Iolera
Input lext:
exprighted work under
a
different Eceens
Input text:
copyrighted
under
different
license
Figure 2: Illustration of our subspace decomposition. On the right, the interpretation of the column of token "a" is obtained by inspecting Figure Likewise, each row from bottom to top corresponds to Figure 21 23
22
24 respectively.
Query QueryActivation Activation
Query Activation
see 500 tttrrghtt fomee ue Cffioree teeree
LUR rightt capyl grad Muudee
Sulopacc 2 Preinage
Sobapace Preimmgge
wannas dryy abracia unee condlsss
spencd
reture
samm soytranderre ae aod by fordde the StV seleral
pur
see 10--0. the dally appin meyen oeher dotpot te
claddes un giolp piocod wgg.gacccoiinieoooooooooo
indic merteclude andeeliea
mescriennt pootive
use sas 2010 costectivs a oll musties spor
6.00 country thee indiper de from Spai 2508 20 Spart Sulrve
(a)
(b)
Astivation
decsape war wile af iineeii
Sabopace 7 Preimage
Vete "cmmdde ofoertennt
are par 2o delgura the tuto Meter nou
wind
sen a thin dge acali droip
caneoo (t oow veli He (ieeeee
(d)
Figure 3: Preimages of the subspace activation corresponding to the same token "a" in 4 different subspaces of the residual stream h 4.ppt in GPT-2 Small. Tokens highlighted with red background are those where the activations are taken from. Tokens highlighted with dashed boxes reflect the encoded information. These activations are projected into the subspaces and the first column "sim" is the similarity between this projection in the preimage and in the query activation. The second column "pos" is the position indices of the highlighted tokens. We showing only the most recent part of the context and as well as future tokens, the actual input sequences are much longer. Interpretation of encoded information: (a) current token "a". (b) the prior token "under". (c) current position, around 1 18.. (d) the topic, the overall context is about licence/permission.. In sum, we can see in different subspaces of the same layer, different aspects of the current context are encoded.
Input:
Planci wos Powi Literature Makker at Notel Piuz Eomuricc
way Naload
targes entity
Faran Corrupt: Noah Vesk
ves the Maca Poss Litoratura A Mirwe whethe Natorl Prize Fearoiar. Noah Vask Milee
noterop eratty
Context Cocrupt:
Noeh Vesk
Literature. Nicre
Planck
Figure 4: Clean and counterfactual inputs used in knowledge conflict experiments. Patching with activation from Param Corrupt can cause the model to increase the probability of context answer, while patching with Context Corrupt increase the probability of parametric answer.
7

---

Published as a conference paper at ICLR 2026
thus the commonality in preimage is the available information. In other words, query activation (any activation we want to interpret) "represents" the whole preimage.
Setting Instead of the whole space, we apply it to subspaces. (See discussion in App. for why it works better for subspaces). Instead of training decoders as in (Huang et al. 2024b), by, we collect around 1M model activations and conduct vector search among them find preimages (see details in App. E. Though without a decoder the content of preimage is restricted by fixed set of inputs (2000 sequences from MiniPile + 2000 IOI and greater-than sequences), it is easier to implement. We use cosine similarity to define the preimage, so it includes inputs whose corresponding activations are in a similar direction to the query activation. We set 0.85 as the threshold for preimages in most cases. For all qualitative results in the paper (including the web application), the subspace partition is given by the best configuration of NDM according to the test results from Sec. 5.1 (last row of Table D
Results Fig. 2 illlustrates our findings. In Fig. 3 we show 4 preimages corresponding to the same token at the same layer's 's residual stream. We see clearly that in this contextualized representation, different aspects of the context are encoded in different subspaces, i.e., the current token, position, prior token, the topic. NDM indeed decomposes the representation space in a meaningful way. Moreover, we also find that the function/interpretation of subspaces consistent. For example, Fig. Bla) shows subspace 2 encodes the current token, and find it always encodes the current token. This is also true for the other 3 subspaces shown in the other 3 subfigures (See Fig. [2] to 24 in App.). Though being consistent, we also find that the meaning of subspaces can vary, depending factors such as whether a certain pattern occurs or a certain mechanism is triggered. We provide an extensive discussion for the qualitative results in App. E.3 In sum, qualitative results show that most of the time, the preimages are interpretable, and we can find a high-level concept to summarize the encoded information in subspaces across inputs because of good consistency. Due to space limits we only present a very limited number of examples, we strongly recommend checking more in our web application
5.3 APPLICABILITY TO LARGER MODELS
To test the applicability on larger models, we apply NDM on residual stream activations of Qwen2.5- 1.5B (Qwen et al. 2025) and BBmma-2-2B (Team et al. 2024 As very detailed ground-truth circuits are available for them, we design prompts that are likely to elicit separate mechanisms inside the model, and see if they are mediated by separate subspaces. This would constitute strong evidence that our partition is meaningful.
Setting We construct inputs that elicit knowledge conflicts: "Max Planck won the Nobel Prize in Literature. ... Max Planck won the Nobel Prize in". The model can either predict "Physics" (parametric answer) by using its parametric knowledge, or predict "Literature" " (context answer) by copying the context knowledge. We refer "Max Planck" as target entity. We construct examples following this form with different target entities, using RAVEL dataset (Huang et al. (2024)). We do not use the dataset the way it is designed, see App. D.6 for discussion and experimental details. To test whether a subspace plays a role in either of these two circuits quantitatively, we again use subspace patching and construct two types of counterfactual inputs (Fig. 4) 1) Param Corrupt: It aims to knock out the role of the parametric knowledge. We replace both occurrences of the target entity with made-up entity, and other parts remain unchanged. Since the model does not have parametric knowledge for this made-up entity, the circuit for parametric knowledge is broken, while the circuit for context knowledge should not be affected much. 2) Context Corrupt: We replace only the first occurrence of the target entity with made-up up entity, other parts remain unchanged. This would presumably only break the context knowledge retrieval, as in the last sentence the target entity does not match any previous entity. We filter entities to ensure the model has parametric knowledge, and construct more than 500 prompts for both models, and measure the average patching effect over them. When patching, patch all activations across token positions for single subspace, since do not know the underlying circuit, and our subspace partition is uniform across the entire sequence.
httpps: / htt////sssppppppaeeeeeee inarrttiion tion. www.iionssteeeeeeeeeeeo stammml at.
Both without post training
8

---

Published as a conference paper at ICLR 2026
Param Comupt
Param Comupt
10
negiiibbe offect
sided effectt
context
modkating only one type of knowledge
14
porametric
Shoon offect
Contost ComusE
Context Comupt
Figure 5: Subspace patching effect. Left: NDM partition on h 1.mid of Qwen2.5-1.5B. The model originally predicts the context answer around 60% of the time ("Clean Run"). If the entire space is patched by activations from Param Corrupt, the model always chooses the context answer ("d=1536"), and likewise for Context Corrupt. NDM finds subspaces that only mediate parametric knowledge routing ("d2 256", context answer is 100% under Param Corrupt while parametric answer rate remains the same as clean run under Context Corrupt) or only mediate context knowledge routing ("d10 64"). See Fig. 19 in App. for baseline results. Right: NDM on h. miid of Gemma-2-2B. Similarly, some subspaces given by NDM have strong effect under one kind of counterfactual inputs but negligible effect under the other. See Fig. 20 in App. for baseline results.
Results We show results in Fig. 5. We can see that NDM finds some subspaces that have non-trivial patching effect under one kind counterfactual inputs, indicating they play role in for only one mechanism. For example, in h 11. of Qwen2.5-1.55 subspace seems to do the heavy lifting for routing parametric knowledge, while subspace 10 is crucial in routing context knowledge. Both subspaces are part of the same representation space. In contrast, baselines (Fig. 19 20 in App.) typically show correlated patching effects under two types of counterfactual inputs. Importantly, we need to be careful when interpreting these results since the detailed circuits are unknown, see caveats in App. D.6 Moreover, also try qualitatively interpret some important subspaces, subspace 2 and 6 for Qwen2.5-1 1.5BB. We found that the two subspaces indeed encode parametric knowledge while being different stages of knowledge routing, Subspace encodes local knowledge (consistently focused on the current entity/n-gram), while subspace 6 encodes knowledge propagated from elsewhere by attention layers (could be about a previous entity). See details in App. E.4
6 DISCUSSIONS
Related Work
The internal mechanisms of model can be analyzed and described using various
units at different levels, such as components (Wang et al. 2022 Hanna et al. 2023 Stolfo et al. 2023
Prakash 2024. Yu & Ananiadou| 2024). sparse features (Cunningham et al.) [023 Bricken et al.
2023 Dunefsky et al. 2024 Ameisen et al., (023). token positions (Haklay et al. 2025 Bakalova ct al. 2025), parameters (Braun et al., 2025] Bushnaq et al. 2025), and most relevantly - subspaces. Interpretable and causally important non-neuron-basi-aalleee
multi-dimensional subspaces have been identified in many studies, though typically only those relevant to a specific sub-task or supporting certain arguments. For example, subspaces for gender bias (Bolukbasi et al.) [016]] Belrose et al. 2023). linguistic concepts (Belrose et al.) [2023] Guerner et al.][ greater- than and equality relationships (Wu et al. 2023) Geiger et al.] (022),, estimation of current state (Shai et al. 2024), refusal behavior (Wollschläger et 20.35 variables in abstract reasoning (Yang et al. 2025). days of the week, months, years (Engels al. 2024 Clarke et al. 2024), and separate subspaces for digits of different significance in addition dHu et al. (202)). Like the toy setting described in this paper, discussion of feature dependency leads findings about interpretable subspaces (Engels et 2024
Clarke et al. 2024).
As have shown, the resulting subspaces often align with meaningful "variables", representing groups of infinitely many features sharing a higher- concept. We believe they better fundamental units to build circuits. Unlike feature eircuits where connec-
Future Directions and Limitations

---

Published as a conference paper at ICLR 2026
tions specific to concrete features, circuits defined over such "variables" capture relationships at more abstract, high-level, and inpu--indeeendent level, offering greater potential for algorithm- like description (see more discussion in App. G.3).
Our current approach shows a promising direction, but we expect that even stronger results could be achieved using substantial computational resources and engineering efforts from industry. With more compute, applications to larger models or learning better subspace partition on 2B models can produce more interesting subspaces (Compute usage of this paper is in App. H. More experiments can be done to explore more fine-grained partitions, instead of start merging from subspaces of at least 32 dimension. a small set (e.g., sentiment), thus encoded in tiny subspaces. Moreover, we interpret directions in subspaces by using cosine similarity to define preimages, we rely on the Linear Representation Hypothesis (Elhage et al.) 2022 Park et al. 2023). which could be untrue in some cases (Csordás) al. 2024). Furthermore, as mentioned in App. H..3 we do not find clear interpretations for some small subspaces. Besides reasons mentioned before, it could be because the orthogonal matrices require more training, or because we assume strong orthogonality between meaningful subspaces, or some abstract variables used internally by the model may be hard interpret on the basis of inputs.
very likely that the model uses intermediate variables that take value from
7 CONCLUSION
In this paper, we present novel way to decompose representation space into non-basis-aligned multi-dimensional subspaces by optimizing under an unsupervised objective, neighbor distance minimization. Our method uncovers interpretable and less dependent subspaces from only data distribution. Different from previous work, we aim to investigate and describe the whole space by studying each subspace. We present both quantitative and qualitative evidence supporting that obtained subspaces are interpretable and consistent most of the time. On the other hand, we also highlight some limitations of the current version of the method, e-g-, partition is not fine-grained enough. We believe many of them can be improved in the future. In sum, we believe our method can serve as useful tool for the interpretability and, more broadly, machine learning community.
claim
ACKNOWLEDGMENTS
Funded by the Deutsche Forschungsgemeiischaatt (DFG, German Research Foundation) - Project-IID 232722074 SFB I 102. We thank Aleksandra Bakalova, Yuckun Yao, and Kate McCurdy for discussions and feedback.
REFERENCES
Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro. David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL https //transformer- circuits pwwptt25.pp////2//// wttp//////ttttibuiioo p
graphs /eatsssssmehhhs...mmmm html
Aleksandra Bakalova, Yana Veitsman, Xinting Huang, and Michael Hahn. Contextualize-
aggregate: Circuits for in-context learning gemma-2 2b. arXiv preprint arXiv:2504.00132 2025.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference machine learning, pp. 531-540. PMLR, 2018.
Nora Belrose, David Schneider-Joseph. Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. Advances in Neural Information Processing Systems, 36:66044-66063, 2023.
10

---

Published as a conference paper at ICLR 2026
Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety review. arXiv preprint arXiv:2404. 14082, 2024.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai, Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29, 2016.
Dan Braun, Lucius Bushnaq, Stefan Heimersheim, Jake Mendel, and Lee Sharkey. Interpretability in parameter space: Minimizing mechanistic description length with attribution-based parameter decomposition. arXiv preprint wrXiv:2501.14922 2025.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly. Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity; Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https:/ttanssfrmer-cicciii http:///oooettttiiiiooo//////onn pub/2202/mmooooemmmiicc features/index.htm..
Lucius Bushnaq, Dan Braun, and Lee Sharkey. Stochastic parameter decomposition. arXiv preprint arXiv:2506.20790, 2025.
Alessia Celeghin, Alessio Borriero, Davide Orsenigo, Matteo Diano, Carlos Andrés Méndez Guerrero, Alan Perotti, Giovanni Petri, and Marco Tamietto. Convolutional neural networks for vision neuroscience: significance, developments, and outstanding issues. Frontiers in Computational Neuroscience, 17:1153572, 2023.
Matthew Clarke, hrdkbhatnagar, and Joseph Bloom. Compositionality and am-
biguity: Latent co-occurrence and interpretable subspaces. LESSWRONG, URL
compos i tiona https:// wwww http:://wwwwwwwssogw....oe http:///o/o....///coc/////tta///
2024.
amb wtwp:aaddddamiiittlllaa-leeeeeee latent-teetttte co -ow.-oooooonooo--aaa......
Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. Recurrent neural networks leam to store and generate sequences using non-linear representations. arXiv preprint arXiv:2408. 10920, 2024.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.0880 086000 2023.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng. Jeff Johnson, Gergely Szilvasy, Baere-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable I1m feature circuits. Advances in Neural Information Processing Systems, 37:24375-24410,, 2024.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac HatfieldDDodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/ttransformer circuitss wwtpuuisssppp2221111 httb://22//rammeooo//nndoo/
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac HatfieldDDodds Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. https:/ttrassfommrr bircuits.pub/2022/o wtwp:/2002oooyyoo/////ee....mmm model/index.html
Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are one-dimensionally linear. arXiv preprint 24Xiv:2405.1448 2024.
Richard Futrell and Kyle Mahowald. How linguistics learned to stop worrying and love the language models. arXiv preprint 25X1:2501.7704 2025.

---

Published as a conference paper at ICLR 2026
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pp. 160-187. PMLR, 2024.
Atticus Geiger, Duligur lbeling. Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, et al. Causal abstraction: A theoretical foundation for mechanistic interpretability. Journal of Machine Learning Research, 26 (83): 2025.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 2020.
Clément Guerner, Anej Svete, Tianyu Liu, Alexander Warstadt, and Ryan Cotterell. A geometric notion of causal probing. arXiv preprint arXiv:2307.15054 2023.
Tal Haklay, Hadas Orgad, David Bau, Aaron Mueller, and Yonatan Belinkov. Position-aware automatic circuit discovery. arXiv preprint arXiv:2502.04577, 2025.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Inter- preting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36:76033-6006, 2023.
Xinyan Hu, Kayo Yin, Michael Jordan, Jacob Steinhardt, and Lijie Chen. Understanding in-context learning of addition via activation subspaces. arXiv preprint arXiv:2505.05145, 2025.
Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. Ravel: Evaluat- ing interpretability methods on disentangling language model representations. arXiv preprint 2770000 2024a.
Xinting Huang, Madhur Panwar, Navin Goyal, and Michael Hahn. Inversionview: a general- purpose method for reading information from neural activations. Advances in Neural Information Processing Systems, 37:137903-137964, 2024b.
Jean Kaddour. The minipile challenge for data-efficient language models. AXiv:2304.08442, 2023.
arXiv preprint
L. F. Kozachenko and N. N. Leonenko. Sample estimate of the entropy of random vector. Probl. Pered. Inform., 23:9, 1987.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E-Statisticall, Nonlinear, and Soft Matter Physics, 69(6):066138 2004.
Aleksandar Makelov, Georg Lange, and Neel Nanda. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching. arXiv preprint 1rXiv:2311.17033 2023.
Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, et al. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. arXiv preprint arXiv:2408.01441 01416, 2024.
SubbaReddy Oota, Manish Gupta, and Mariya Toneva. Joint processing of linguistic properties in brains and language models. Advances in Neural Information Prucessing Systems, 36: 18001- 18014, 2023.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint 2rXiv:2311.03653 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, gegeorerformance deep learning library. Advances in neural information processing systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2026
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. arXiv preprint arXiv:2402.14811 2024.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL httpp:://axxisi o alss / 15115
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Adam Shai, Lucas Teixeira, Alexander Oldenziel, Sarah Marzen, and Paul Riechers. Transformers represent belief state geometry in their residual stream. Advances in Neural Information Processing Systems, 77:75012-750344 2024.
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. arXiv preprint arXiv:2305.15054, 2023.
Milan Studený and Jirina Vejnarová. The multiinformation function as tool for measuring stochastic dependence. Learning in graphical models, pp. 261-297, 1998.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanezyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic. Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choor Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Llmczak-PPluiinsee Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen. Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia I Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, I Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mchran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth. Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev, Gemma 2: Improving open language models at a practical size, 2024. URL https://eat wwwpxxvvvorgggaoooo 2408 408 00118
Harleen Batra,
13

---

Published as a conference paper at ICLR 2026
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022.
Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of research and development, 4(1 (())666-222 82, 1960.
Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. In International Conference on Machine Learning, pp- 11000-10900. PMLR, 2021.
Tom Wollschläger, Jannes Elstner, Simon Geisler, Vincent Cohen-Addad, Stephan Günnemann, and Johannes Gasteiger. The geometry of refusal in large language models: Concept cones and representational independence. arXiv preprint arXiv:2502.1 17420, 2025.
Zhengxuan Wu, Atticus Geiger, Thomas Ieard, Christopher Potts, and Noah Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. Advances in neural information processing systems, 36:78205-78226 2023.
Yukang Yang, Declan Campbell, Kaixuan Huang, Mengdi Wang, Jonathan Cohen, and Taylor Webb. Emergent symbolic mechanisms support abstract reasoning in large language models. arXiv preprint arXiv:2502.2033: 2025.
Zeping Yu and Sophia Ananiadou. Interpreting arithmetic mechanism in large language models through comparative neuron analysis. arXiv preprint arXiv:2409. 14144, 2024.
A ADDITIONAL FIGURES FOR MOTIVATION AND BACKGROUND
(a) Left: input fcatures æ, blue arrow represents the only activating feature; Right: yellow arrows represent reconstructed features x
(b) Left: input features æ, two fcatures are activated, their sum h is aligned with the y-axis; Right: recon- structed features x'
Figure 6: Toy model representing 5 features in 2D space, each black arrow corresponds to column in its weight matrix W
B METHODOLOGY DETAILS
B. TOTAL CORRELATION PERSPECTIVE
The entropy of continuous random vector be estimated based the distance kth-nearest neighbor (Kozachenko & Leonenko 1987). Formally, given random samples (.))) of the random vector X, the entropy estimate ((X) is as follows
fI())
log e(n) + const.
N
<(n) is twice the distance from H, to its kth neighbor, const. is constant if N and are fixed. Meanwhile, the total correlation (Watanabe) 19600) or multi-information (Studený & Vejnarová 1998) defined as
Σ H(X.) - -..,....., Xs) a=1
(6)
14

---

Published as a conference paper at ICLR 2026
Algorithm NDM
Require: Activations Buffer B, !',,... hvr,, threshold т, initial configuration (di,.... dsl where Es=1 d, = d. initial orthogonal matrix R + 1, batch size b, search steps n, block size m, merging start delay p. merging interval maximum update steps K. for k = to K do
Hg < B.pop())
b batch of activations, deleted after pop
Hg -- RHg
11 nearest [[] For S]
D Dnearest f [| For to b] For to S
Stop gradient tracking:
for I to do
H& B.next()
> batch of size m, without being deleted from B
H₂ +- RH₂
for to S do
D pair__ pair_wise__s wwwpwwissseede_eeeeeeeeee
(is)( ()())))
D Rbxm
for = 1 to do
if mii()
nearest [s] then
Dncarest
[s]
t min (D())
H Hecarest 8
Hg aarg (Dn((())
endi if end for
end for end for D Deearest []
for 1o S do
b> Start gradient tracking
H t R.Hnearest
d= dist( (is((Fr(a) (1)0)) ())
d E R
D Dacarest [s] E. d end for
3 Σ. D Duearest (s)
D Final loss
update_ (R, VRf)
b Gradient descent, orthogonality ensured by special parameterization
if k > p and & mod t 0 then
mutual info(R, B)
b I R
if any de, +d., fonileal Sort by d., +d,, and select top 5/8 non-intersecting pairs > 7 Update configuration (and R if necessary, keep the correspondence between and
4p111a > r then
sub-matrices of R)
else
break end if end if
end for return R.
15

---

Published as a conference paper at ICLR 2026
which generalizes mutual information's binary case to multiple random variables and measures the redundancy and dependence among them. Therefore, when we minimize the distances in Eq the entropy of individual subspaces decrease and because the last term of Eq is constant as the entropy of the joint subspace is invariant under orthogonal transformation, we actually minimize the total correlation.
B.2 ALGORITHM DETAILS
Here we supplement Sec. and Algorithm I with some important details and explanations. The MI is measured by KSG estimator (Kraskov et al. 2004). We use the first algorithm presented in the paper (Estimator ( (X, Y)) and normalize the variance of each subspace before estimation, by dividing subspace activations by the square root of the of the variance of each dimension in the subspace. Note that MI is invariant under this transformation in theory but it helps reduce errors in practice (Kraskov al. 2004). We measure pair-wise MI for subspaces, and then divide them by the sum of dimensions of subspace pairs. We use subspace dimension as a very rough estimate of entropy. which forms the upper-bound of MI. So the intuition behind normalizing MI with dimensionality is to roughly estimate the ratio of entropy removed after the other variable is observed. We set a threshold for normalized MI, we select and sort all pairs whose normalized MI above the threshold. We then iterate over these pairs, selecting pairs that do not intersect with selected pairs. For example, for subspace a, b, c, if (a, b) is selected, (b, c) will not be selected. We merge at most S/8 pairs, where is the number of subspaces. Note that merging does not change the weights of R (or just reordering), merging affects the training as we now search the nearest neighbor in the joint space and optimize the distance in the joint space. When merge subspaces, the optimizer states (for the whole R. are also re-initialized.
Note that although have experimented with some alternatives, these specific details are not necessarily optimal, each part of the method can be studied and optimized more extensively in the future.
Lastly, to clarify which hyperparameters we refer to throughout the rest of the paper. We assign them the following names: the initial dimensionality for subspaces referred as unit size (i.e. di. dg in Algorithm| [ which are equal at the beginning). We search the nearest neighbor among N activations (including the point itself, so only N candidates), we refer this as search number (i.e. equal to n x m in Algorithm []. The orthogonal matrix is first trained for some steps, during this stage no merging happens, we call it merging start delay. After this stage, we measure MI and merge subspaces every / step, we call / merging interval. Lastly, the normalized MI threshold is called merging threshold.
C MORE EXPERIMENTS IN ToY SETTING
C.I TOY MODEL FEATURE GEOMETRY
In this section, we extend the experiment in See with more different configurations. We keep model the same, while changing the feature group structure and model dimensions to bring more comprehensive view. We use group=(20, 20) to refer to groups, each group has 20 features, other cases follow this pattern. All experiments in toy setting is done with batch size of 128, training steps of 10000, Adam optimizer, and linear learning rate scheduler which decrease learning rate from 0.003 0.0003. The trained model is tested on 12800 samples. Overall, we strong orthogonal structure in all cases.
Group Sparsity We refer the probability for a whole feature group being not activating as group sparsity S. In Sec we have experimented with S 0.25. We keep other parameters the same (2= 40, d = 12, group=(20, 20)), but set S = 0.0 (Fig. 7a S = 0.5 (Fig. 7b and S = 0.75 (Fig. [c]. We train the model until convergence and final FVU is 0.028. 0.028 and 0.035 respectively. From the figures we can see as the S increases, features in different groups are less likely to co-occur, thus more similar to mutual exclusive condition, which encourages superposition. So we see weaker
In our preliminary experiments, found including the log(-) in Eq 5 as part of the training objective does help.
16

---

Published as a conference paper at ICLR 2026
(a) S
(b)
(c) S = 0.75
Figure 7: WTW, 40, 12, group=(20, 20)
(a) d= 10
(b) d 12
(c) d = 16
Figure W 2 40, group=(5, 15,5, 15), S = 0.25
orthogonality (bottom left and top right submatrices are nonzero, though the absolute values smaller) when = 0.75.
More Groups and Different Hidden Dimension We also experiment with group=(5. 15, 15),, 2 = 40, S = 0.25, and d = 10, 12, 16, corresponding to subfigures in Fig. g Final FVU are 0.166, 0.066, 0.016 for 10, 12, 16 respectively. We still see similar orthogonal structure. When = 10, the dimensionality is not enough to allocate orthogonal subspaces to each group, and the features from 2nd and 4th groups are superposed (as their frequency lower than those from the other two groups), and the model cannot reconstruct very well. When d = 16, as the dimensionality constraint becomes more relieved, orthogonality also occurs within groups.
More Features We increase the total number of features and experiment with configurations. (a) = 32, = 160, 0.25, group=(80, 80), Fig. Da final FVU is 0.025, (b) d = 64, = = 400, = 0.25, group=(40, 80, 120, 160), Fig. 9b final FVU IS 0.033. Again, the orthogonal structure is very obvious.
C.2 ToY MODEL SUBSPACE PARTITIONING
In this section, describe more experiments of NDM in toy setting. We apply for toy models trained in previous section (Sec. C.I whose orthogonal structure is clear.
Experimental Details For toy setting, we use slightly different version from Algorithm Specifically, we find it is not necessary to search over huge amount of subspace activations, so we search for the nearest neighbor inside the batch (Eq. 3 thus simplify the algorithm. In addition, we find R usually get stuck at sub-optimal point. When observing RW, usually find one subspace is computed from two feature group, and this is usually accompanied with higher overall training loss, and more training steps and changing hyperparameters in Adam optimizer do not apply the following heuristics: we take the subspace with the largest loss and measure MI between this and all other subspaces, and multiply the MI values with loss of all other subspaces, then divide
to help. We
17

---

Published as a conference paper at ICLR 2026
www
www
10
-10
(a) 32, 160, 0.25, group=(80. d 64, 400. S 0.25, group=(40, 80)
80, 120, 160)
Figure WTW
the resulting values (always > 0) by their sum as the probability by which we randomly choose the other subspace. This process selects two large loss subspaces which encodes similar information. We multiply their corresponding rows in R with a random orthogonal matrix (E [d (d()+d))x((4 +d2)), thus re-initialize the weights for these two subspaces by mixing these two subspaces, while the whole matrix R remains orthogonal. We find that this technique helps R achieve lower training loss and produce a more ideal RW. In other words, we have a better chance find global minimum. But we also find this becomes less effective when number of features and dimension become larger. This also calls for more study on this atypical optimization problem, i.e., with orthogonality constraint and ignorance of distance change to all but the nearest neighbor when computing gradients.
For reproducibility we report the hyperparameters used as follows: We use euclidean distance as the optimization objective. We find 1 cosine similarity leads to worse results, we also find taking the square or logarithm of the distance does not help. We use 2 as the unit size. We use search number N 4. We find small search number is crucial, it can be much smaller than the number of features, but we are unclear about why small search number helps. For re-initialization loss, we evaluate the loss (distance to nearest neighbor) which we find more accurate for selecting the correct subspace (subspace that contains more than 1 feature group). We use batch size of 128, learning rate of 0.001, and Adam optimizer. As we described previously, the training step is not fixed. Merge start delay is 60k steps, during this stage two subspaces are randomly re-initialized every 3k step. The optimizer states for the whole matrix are also re-initialized. After the 60k steps, we measure MI and merge subspaces with merge interval of 3k steps, and merge threshold of 0.04. Importantly, we use the same set of hyperparameters for all experiments in toy setting, regardless of the varying number of features and activation dimension.
largest much larger search number, N 1024,
Experimental Results We train R for toy models whose orthogonal structure is clear. Concretely, we present experiments here, corresponding to toy models visualized in Fig. Ta Fig. 8b Fig. 9a and Fig. 9b The results shown in Fig. TO 12 13 We
the proposed method is
quite effective. Before merging, each subspace gradually learns to encode only one feature group. For example, in Fig. [0] each 2D subspace in h is computed either from the first or the second feature group (e.g., the first rows only have nonzero entries in first 20 columns, corresponding to the first group). In Fig. [ at the beginning the third subspace encodes all 4 groups (a), then after 3k steps it only encodes the 1t and 3rd group (b), then later encodes only the 1st group. We can clearly see how NDM training objective leads R to evolve towards the desirable direction. Loss values also confirm the alignment between neighbor distances and "purity" of subspace (encoding one group instead of a mixture of groups). We denote the loss when search number is 4 as (4=4, and similarly IN=1024- We take the mean of loss across subspaces instead of sum when reporting them. For example, Fig. 10p's loss values are: IN=1= 0.212 and IN=1024 0.011. After more training when subspaces are "pure" (subfigure c), loss values are: IN=4 = 0.194 and IN=10 = 0.004. Another example, Fig. 11 loss values are: N 0.340 and 10=4 0.016 for subfigure b, and
18

---

Published as a conference paper at ICLR 2026
(a) W or RW before training. R is initialized
RW after first 3k training step. Corresponding
identity matrix
configuration dj 2, 2, dg
(c) RW 60k training step before merging. Corre-(d) RW after entire training procedure completes. sponding configuration di = 2, dg 2, .ds = 2 Corresponding configuration d1 = 6, d2 = 6
Figure 10: RW at different stages of training. The toy model is the from Fig. la where d = 12 40, group=(20, 20), = 0.25
(a) W or RW before training. R is initialized as an (b) RW after first 3k training step. Corresponding
identity matrix
configuration d1 = 2, 2, .. do = 2
(d) RW after entire training procedure completes. (c) RW after 60k training step before merging. Corre- Corresponding configuration dj = 4, d2 = 4, dg sponding configuration
2, d2 ..... dg 2 2, d4 2
Figure 11: RW at different stages of training. The toy model is the one from Fig. 8b where d = 12
40, group=(5, 15. 5, 15), $ 0.25
19

---

Published as a conference paper at ICLR 2026
W RW before training. R. is initialized as an identity matrix
(b) RW after first 3k training step.
Corresponding configuration 2, d2
2. .... die =
(c) RW after 60k training step before merging. Corresponding configuration dj
2, d2 ..... dia 2
(d) RW after entire training procedure completes. Corresponding configuration d1
18, d2 14
Figure 12: RW at different stages training. The toy model is the one from Fig. 9a where d 32
= 160, g80up=(80, 80), S = 0.25
20

---

Published as a conference paper at ICLR 2026
(a) W or RW before training. R is initialized as an identity matrix
(b) RW after first 3k training step. Corresponding configuration
2, d2
2. .... die =
(c) RW after 60k training step before merging. Corresponding configuration d1
2, d2 = 2, ... dis 2
(d) RW after entire training procedure completes. Corresponding configuration dj
18, du = 16, d3 16, d, 12, dg = 2
Figure 13: RW at different stages of training. The toy model is the one from Fig. 9b where d 32 400, group=(40, 80, 120, 160), S = 0.25
21

---

Published as a conference paper at ICLR 2026
(N=4 0.322 and Ix=1024 0.004 for subfigure Importantly, we also that R. sometimes does not find global minimum. In Fig. 12 and [3] we see some sub-optimal subspaces, such as the 3rd subspace from the last in Fig. [2c which is the reason for later imperfection of final R. Morcover, also see that the proposed MI-based merging works well. It correctly merges the subspaces encoding the same group. For example, in Fig. IT. normalized MI between 1st and 5th subspace is 0.55, and between 2nd and 6th is 0.49, normalized MI between all other pairs is below 0,03. In Fig. [00 and II] we get the correct number of subspace and their dimension. In Fig, T2 and 13 the final configuration is imperfect but close to the ground truth. As mentioned before, we generally find the imperfection results from R being unable find global minimum or disentangle feature groups for few subspaces, the MI estimate usually works well.
In summary, we find the training objective of NDM indeed encourages subspaces encode only one feature group. Together with MI based merging, the final subspace partition is close to the groundtruth, with each subspace encoding one feature group. While training objective aligns well with our goal, the orthogonal matrix sometimes gets stuck at local minimum of this objective, resulting in sub-optimal partition. Nevertheless, it might not be necessary to achieve perfect partition to make itself useful in practice. In subfigure b of Fig. [0] observe that these sub-optimal points already have good deal of disentanglement, which could potentially be very useful when using these subspaces as analyzing unit instead of random subspaces.
lastly, more experiments can done in toy setting, we have not done extensive experiments exploring the effect of varying unit size and search number, much more feature groups, and many other aspects. More rigorous study can be done in this ideal testbed, but we would like to focus more on real world language models, in order address more urgent concerns about whether the orthogonal structure even exists in real models.
D QUANTITATIVE EXPERIMENTS ON LANGUAGE MODELS
D.I SUBSPACE PATCHING
Here we describe our method detail. Formally, suppose hetn and hem are the activations from clean and corrupt inputs respectively, at the same layer and token position. We patch subspace as follows:
hap
Rhor
Rhaetn
(8)
(1)
foorp
hein
hee
(x)
(a)
(9)
(S)
(8)
h
(1)
h
h
hap
(a)
(10)
(8)
h h= R
(11)
after patching, the model uses h to finish the rest of the forward pass the clean input. To measure the patching effect, we first run the model on clean input, recording the original logits or probability, which refer clean run. Then activations hap which we refer to as corrupted run. Finally we run the model on clean input, with certain subspace activation being patched with the value saved in corrupted run, and record the
the model on corrupted input, to save the corrupted
22

---

Published as a conference paper at ICLR 2026
Legend
When
10
Mars
Key/Volue Value
and
S1 John
Query
Class Heads Layse
Cutput
S1+1
Ivent
Previous Token Heads
store
Negative Name Mover Heads
52 Dohn
Duplicete Towen leeds 0J JUR)
Induction Heads Sa 63 BB 5311
11:0
Narre Mover Heeds oo ou
drink
END K3
S-inhibition Heads 73 19 86 B10
Deckup Name Mover Heeass s0 27 10) 102 D.D 12
Figure 14: IOI circuit, Fig. 2 from (Wang et al. 2022). Attention heads in this figure is denoted as fayer_index head_ ndex. E.g., The second previous token head, 4.11, refers to layer 4 and attention head 11 in that layer. Note that indices start from O.
patched logits or probability, which we refer to as patched run. We compare the model output from clean run and patched run to measure the subspace patching effect.
D.2 GPT-2 SMALL TEST SUITE
The Indirect Object Identification (IOI) task consists of sentences like: "When Mary and John went to the store, John gave drink to _", which should be completed with "Mary". (Wang et al. 2022) found a circuit in GPT-2 Small responsible for solving this task (Fig. [4]. Besides computational graph, they also reveal the information moved between the nodes of the graph.
Tests based on 101 circuit
To understand our intervention, we briefly describe the information flow inside the circuit: At token S1+1 (see Fig. [4] for the notation), the model attends the previous token and moves the name from which will be used as key in later induction heads. Right after layer 4, the model finishes moving previous token information, and this information is not yet used by downstream components. At token S2, the induction heads attend to S1+1, moving the relative position of S1 (i.e., it occurs before or after IO, or equivalently, SI+1 is "and" or "went"). Meanwhile, Duplicate token heads attend to S1 from S2, moving S name and relative position of S1. Therefore, at S2's residual stream, right after layer 6 the model finishes moving position information of S1, and it is not yet moved by any downstream component. Then END token, S-Iniibition heads attend to $2 and move the position information of $1 and as well as the S name. Right after layer 8, the model finishes moving position information of SI and information of S name, and they are not yet used by any name mover heads. Lastly, name mover heads avoid attending SI by using those information and attend to I0 to move IO name. Note that the claims about the moved information is supported by strong causal evidence from Wang et al. 2022). and is confirmed again with a different method in (Huang et al. 20245).
Importantly, we emphasize key places in the circuit in our description (see the italics). Specifically, his+1 G.post pegt They are ideal for causal intervention because in those places we know the existing information, and all heads that gather the information are before the place and all heads that use the information are after the place. In contrast, if we intervene on 3.post to change previous token information, the changed information might be overwritten by head 4. resulting in negligible effect if we intervene the right subspace. This is also the reason why we do not pick the layer after duplicate token heads, because similar information will be added later by induction heads. Moreover, we intervene on 5.post to change previous token information, it will only affect induction head 6.9, while the original information was already moved by heads in layer 5, We do not select h 11,post because we suspect there is a big "output subspace" after the last layer and our method would not be very interesting there.
Based on the information used in the circuit, we design two kinds of counterfactual inputs. (1) Position Swap. We swap the position of the 10 and SI in the sub-clause at the beginning. For example, clean input: "When Mary and John went to the store, John gave a drink to": corrupted input: "When John and Mary went to the store, John gave drink to".. (2) Name Swap. We swap the names
23

---

Published as a conference paper at ICLR 2026
of IO and S (both S1 and S2). For example, clean input: "When Mary and John went to the store, John gave a drink to"; corrupted input: "When John and Mary went to the store, Mary gave a drink to".
Finally, we design following tests:
1. Do subspace patching in residual stream 4.pust and Mo0+! (IO+1 means the token following I0), using Position Swap. If the information about previous token is successfully changed, then in the patched run, induction heads would attend 10+1 instead of S1+1. Therefore, the induction heads would move the position of IO instead of SI, and pass the changed information to later components and cause name mover heads to avoid attending IO.
2. Do subspace patching in residual stream hs2 6.pss using Position Swap. If patching in the right subspace, corrupted activations should contain a different position of $1, thus cause the model to change towards an opposite prediction, similar to the explanation above.
3. Do subspace patching in residual stream REND using Position Swap. This is to change the position of S1 in the residual stream of END.
4. Do subspace patching in residual stream HEND using Name Swap. This is to change the name in the residual stream of END.
For all 4 tests in IOI circuit, we measure the drop in logit difference, we denote it as ALD. So ALD = (logit logit (logit - logit where logit the logit value for IO name on the clean run, and peb is the logit value for IO name on the patched run, and similarly for S. We use 1000 examples, which are clean inputs, and we construct counterfactual inputs on-the-fly. In these examples, the order of IO and SI in the sub-clause is uniformly random. The names are picked randomly from a list of common one-token names. The template ("When ... went to the store, gave drink to", "Then, ... had a long argument, and afterwards ... said to",, etc.) is also picked randomly from a pre-defined list. The final reported effect ALD is averaged over the 1000 samples.
10
Test based Greater-than circuit The Greater-than task (Hanna et al. 2023) consists of sentences in the form "The <noun> lasted from the year XXYY to the year XX". such as "The war lasted from the year 1732 the year 17", the model should assign higher probability years greater than 32, thus performing the simple greater-than math operation. (Hanna et al.) 2023) revealed a circuit in GPT-2 Small responsible for solving this task. Briefly speaking, some attention heads between layer 5 to layer 9 (both inclusive) move YY to the last token's 's residual stream, and MLP layer 8 to 11 (both inclusive) specify which years are greater than YY. Therefore, right after layer 9, the model finishes moving YY to the last token, so we intervene at EN to change the YY, where END denotes the last token.
Regarding counterfactual inputs, we use the same as the original paper: the YY in clean input is replaced by 01. For example, clean input: "The war lasted from the year 1732 to the year 17"; corrupted input: "The war lasted from the year 1701 to the year 17".
We do one test based on Greater- than circuit (continue the indexing from 10I tests):
5. Do subspace patching in residual stream REND using 01 counterfactual input. If the information YY is successfully replaced with 01, the sum of the probability of years greater than YY will decrease.
9.post
Unlike IOI, we do not compare logits of two tokens as there are many possible answers. Instead, we compute the drop in sum of the probability valid answers, we denote as AP, our running example where YY=32, it means AP Pan (next token E (33, (3,,34. 34, ... .99)), where Pan and peh denotes probability in clean and patched run respectively. ,99,)))))))))
; .99)) Ppean (next token
Similar to 101, the inputs for this task is also constructed from random nouns, years and random templates, while restricting years such that they are always 2-token long. The final reported effect AP is averaged over the 1000 samples.
Measuring target information inequality Therefore, we have tests in total for GPT-2 Small, and 4 layers are involved. To test training configuration of NDM, we apply the same procedure
24

---

Published as a conference paper at ICLR 2026
and hyperparameters to train 4 orthogonal matrices, for post MLP residual stream at layer 4, 6, 8, 9 respectively. Then apply the 5 aforementioned tests to each of the subspaces found in corresponding layer.
Each test changes one rype of information. Analogously it means a fixed "variable", while the "variable" can take many different values. To measure if the target information (the "variable" are interested in) only lives in one subspace, we quantify the inequality of patching effect across subspaces by calculating Gini coefficient. Same as before, is subspace index, for the patching effect A, (can be either ALD, or PP),, Gini coefficient is
G EA
S Laz D., 25 2 == A.
A
(12)
A Gini coefficient of O means the effect is evenly distributed across all subspaces. If the effect is nonzero for only one subspace and zero elsewhere, the Gini coefficient will be close 1.0 (the more subspaces there are, the closer it gets). If the effect is concentrated in about half of the subspaces, the Gini coefficient will fall somewhere in between. So by using Gini coefficient, the incquality of effects across subspaces or the inequality of the amount of target information across subspaces is summarized as one value, which makes our evaluation easier.
Importantly, there is one trivial way to get high value in this metric while not matching our expectation: there can be one big subspace and many tiny subspaces (e.g., one dimensional), the big one would presumably encode the target information, but as well as many other types of information. Therefore, we also calculate Gini coefficient after dividing the patching effect by the dimension of the subspace, which computes the "density" of target information in each subspace, or "amount of target information per dimension". ". Moreover, there can be another trivial way to get high value in the second metric: one can use PCA to concentrate variance compress information into few subspaces, thus these subspaces would have high "density" for all kinds of information. This again goes against our expectation, we want subspaces to have high "density" for one or few kinds of information and low "density" for all others. Therefore, the third metric is to calculate Gini coefficient after dividing the patching effect by the sum of the variance of each axis in the subspace. We refer to this sum of variance as Var,. We report all 3 variants (original effect, normalized by dimension, normalized by variance) in the results. Ideally, a good subspace partition should achieve high values in all 3 metrics.
D.3 NDM TRAINING DETAILS ON GPT-2 TEST SUITE
We use the NDM algorithm described in Sec. (with more details provided in App. B and Algorithm
D
For reproducibility, we report the hyperparameters used as follows: We use euclidean distance as the optimization objective. We use 32 as the unit size. We try different search numbers N {I x 214, 5 x 214, 25 x 224]. We do not do random re-initialization during the pre-merging stage like we did for the toy setting: preliminary results show this is not helpful for language models. We use batch size 128, learning rate 0,0003, and Adam optimizer. We try different merging intervals, 4k and 8k, and set the merging start delay 2.5 times the merging interval. Longer merging intervals mean more training steps in general. We set 100k as the maximum training steps, which is usually not reached. We also try different merging threshold (0.02, 0.03, 0.04, 0.05). We use MiniPile (Kaddour 2023) as the data to run GPT-2 Small when collecting activations, instead of the Open Web Text dataset which is similar to GPT-2 pretraining data. Therefore, we do hyperparameter search over all the combinations of different search numbers, merging interval, and merging threshold, while other hyperparameters are always the same. For each hyperparameter configuration, we train 4 orthogonal matrices for pLst-MLP residual stream at layer 4, 6, 8, 9 respectively, they are needed in the test suite.
As shown in later section, the best configuration is search number 25 214), merging interval 8k, merging threshold = 0.04. In Fig. [5 we show the MI after training, compared with the identity matrix, which is used to initialize R.
In later section, we show ablation studies, and some of the hyperparameter choices will be justified by those results.
25

---

Published as a conference paper at ICLR 2026
NDM
NDM
NOM
1-1
53
1623
Identity
Iambiity
Idantity
Icencity
0-
06
36-30
R1
81.00 9:042
60 23 C9:1
74:
8-
00
10-10
10
5.
(a) h 4pport
(b) h Sppost
(c) h 8 3posst
(d) h 9,post
Figure 15: MI between each pair of subspaces. Top row: subspace partition given by NDM. Bottom row: subspace partition using the identity matrix and the dimensionality configuration given by NDM. Columns correspond to 4 layers in GPT-2 Small. We use the same MI estimation method as the one used during training. The MI values are not normalized. Bigger values in the top left corner because subspaces are sorted in descending order according to dimensionality. We can see NDM produces more independent subspaces.
index
0
2
3
4
5
6
7
8
9
10
11
12
dg
128
96
96
64
64
64
64
32
32
32
32
32
32
Var, (x103)
2.1
2.7
1.3
0.6
0.8
1.4
1.7
1.0
1.0
0.4
4.0
4.8
0.3
AP. (x
5.4
0.7
1.8
1.9
29.7
0.6
0.6
0.3
0.6
0.8
1.2
0.7
0.7
AP. (x10-4)
4.2
0.8
1.9
3.0
46.4
1.0
0.9
0.9
2.0
2.4
3.9
2.0
2.3
AP. Var,
(x10 5)
2.6
0.3
1.4
2.9
39.6
0.5
0.3
0.3
0.6
2.2
0.3
0.1
2.2
Table 2: Test 5 raw data for NDM results shown in main paper (Table AP, is 0.72, for row AP, is 0.67, and for row APs Var, is 0.78.
Gini coefficient for row
de
D.4 REPRESENTING PATCHING EFFECT DISTRIBUTION WITH GINI COEFFICIENT
We do not show the patching effect for each individual subspace, as there would be too many numbers, but rather summarize the degree of concentration using Gini coefficient. To provide sense of how high the Gini coefficient should be for the partition to be considered good, we show some raw data in Table 2 4 We can see that, when the Gini coefficient is above 0.6 (Table 2}, the distribution of the effect is quite satisfactory, as there is a single value much greater than all others. When Gini coefficient around 0.5 (third and fifth
in Table 3) we do some degree of concentration, but effect is concentrated in quite a few subspaces. Lastly. if Gini coefficient is lower than 0.3 (Table J. we do not see any significant degree of concentration. Therefore, we can say that Gini coefficient should be greater than 0.6 to be considered "good", , and the higher the better. In other sections of the paper, we only report Gini coefficients for the GPT-2 Small test suite.
D.5 ABLATION STUDY ON GPT-2 TEST SUITE
We also show hyperparameter searching results in Table in order to provide information on how each factor affects the training result. First of all, we can see is better to have larger merging interval, in other words, more training steps. Due to compute limits, we do not further experiment with more training steps, which would probably produce better results. Moreover, 16 using search number as the variable on x-axis. We can see that larger search number generally produces better results, though depending on merging threshold. However, considering both search
show the same data in Fig.
26

---

Published as a conference paper at ICLR 2026
index 8
5
6
dg
192
128
64
64
64
64
64
32
32
32
32
Var, (x (x10")
0.3
0.3
0.2
0.2
0.3
0.3
0.4
0.3
0.3
0.4
10.7
ALD, (x10
14.8
11.6
7.6
4.2
3.1
2.5
2.9
1.4
1.6
0.9
2.2
ALDs (x10
7.7
9.1
11.9
6.6
4.8
3.9
4.6
4.4
5.0
2.9
6.8
de ALD, (x10-4)
49.5
37.6
38.5
18.1
11.0
7.5
7.1
5.5
5.2
2.2
0.2
Var,
Table 3: Test 2 raw data for PCA baseline results shown in main paper (Table 1 Gini coefficient
for row ALD, is 0.46, for row ALDs
is 0.22, and for row ALDs is 0.52.
d,
index
0
1
2
3
4
5
7
8
9
10
de Var, (x (x10*)
128
128
96
96
64
64
64
32
32
32
32
0.7
1.0
0.5
7.9
0.4
0.2
0.2
0.1
0.1
0.1
0.1
ALD, à (x 10-1)
3.9
3.7
2.3
2.5
1.6
1.7
1.7
0.7
0.8
0.8
0.7
ALD. (x10-3)
3.0
2.9
2.4
2.6
2.4
2.6
2.7
2.2
2.4
2.4
2.2
ALDs (x10-1)
5.5
3.6
4.7
0.3
4.0
8.1
9.1
5.2
8.0
7.7
7.0
Var
Table 4: Test raw data for identity baseline results shown in main paper (Table 1). Gini coefficient for row ALD, is 0.33, for row ALD, is 0.05, and for row Var, 0.23.
ALD,
d,
number and merging interval, when given fixed compute budget, it is a trade-off between searching among more activations per step and performing more training steps. Determining which side we should spend more compute on is important question and should be explored more extensively in future work.
We also consider other alternatives for other hyperparameters. In Table ( we compare different distance metric, under different condition. We use good hyperparameters for merging threshold (0.04 or 0.05) and merging interval (8000) based our previous finding. When using cosine similarity as distance metric, losses (average distances) from different subspaces are weighted by the subspace dimensions. We found it important, because in some preliminary experiments where there is no such weights, the orthogonal matrix prioritizes maximizing cosine similarity for small subspaces while the big subspaces somewhat ignored. From the results we can see that cosine similarity produces similar or worse results, which is consistent with our choice in toy setting.
In Table 7 we compare different data sources, i.c., the text data on which we run GPT-2 Small to collect activations. Again, we use merging threshold of 0.04 or 0.05, and merging interval of 8000, We can see that Open WebText produces worse results in all cases, some of them are much worse. After checking Gini coefficient for individual tests, we found that using Open WebText only hurts performance on test 5. We speculate that it probably due greater diversity MiniPile.
A key hyperparameter is unit size, in the paper we normally use attention head dimension divided by Though we think this is very important aspect, we haven't done extensive testing on different
Uree 02, H400
0.700
Brcd 02, 2000
011-0.03. (020)
Ciri coeficeent
0.675
mr.000 /nbress_22000
0.050
04,
nntervi40000
0.825
0h1-0.0 n100
dh, inder00-00000
0.600
0.575
0.530
0.525
X214
3
25
search
Figure 16: Average Gini coefficient on GPT2 test suite, as a function of search number, under different conditions. Same data as Table X-axis is in log-scale.
27

---

Published as a conference paper at ICLR 2026
search number (x214)
merging threshold
merging interval
number of subspaces
avg coefficient
0.02
4000
6,8,6.5
0.55
0.02
8000
9,9,9,9
0.64
0.03
4000
9,11,11,10
0.66
0.03
8000
9,11,11,10
0.61
0.04
4000
12.13,12,14
0.57
0.04
8000
12.12,13.14
0.63
0.05
4000
14,16,17.19
0.52
0.05
8000
13, 17,18,18
0.58
0.02
4000
5,8,7,6
0.58
0.02
8000
9,9.9.9
0.63
0.03
4000
8,8,9,10
0.64
5
0.03
8000
9,9.9.9
0.64
5
0.04
4000
11, 12. 14, 14 10, 12, 13, 13 12, 15, 16.
0.65
0.04
8000
0.65
0.05
4000
0.61
5
0.05
8000
12, 15, 16, 16
0.63
25
0.02
4000
6.
0.57
25
0.02
8000
9,9.9.9
0.66
0.03
4000
7,9,8, 10
0.63
25
0.03
8000
9,9, 9.
0.65
25
0.04
4000
10, 10, 12, 13
0.68
25
0.04
8000
11. 11, 12,
0.71
0.05
4000
10,
13,
15,
0.63
25
0.05
8000
12,
15,
14, 15
0.71
Table Combinations different search number, merging threshold, and merging interval, and their resulting number of subspaces in the 4 layers used in the test (from low to high) and resulting average Gini coefficient on GPT-2 test suite. The one shown in Table the third row from the last, which is slightly better than the last row if we show more precision. Other hyperparameters fixed, and described in D.3
avg coefficient
search number (x214)
merging threshold
dist=1-cosine
dist=euclidean
0.04
0.62
0.63
0.05
0.58
0.58
0.04
0.64
0.65
5
0.05
0.64
0.63
0.04
0.65
0.71
25
0.05
0.68
0.71
Table 6: Average Gini coefficient on GPT2 test suite. We set merging interval 8000. The last column comes from Ta Compared to euclidean distance, using cosine similarity produces in similar slightly worse results
search number (x214)
merge threshold
avg coefficient
Oeta=Open
data=MiniPile
0.04
0.54
0.63
0.05
0.54
0.58
5
0.04
0.59
0.65
5
0.05
0.58
0.63
25
0.04
0.61
0.71
25
0.05
0.60
0.71
Table 7: Average Gini coefficient on GPT2 test suite. We set merging interval 8000. The last column comes from Table Compared to MiniPile, using Open WebText produces worse results. Closer look shows that, instead of degrading uniformly across all tests, using WexTeet only hurts performance on test 2 5.
28

---

Published as a conference paper at ICLR 2026
unit sizes as the search space is big and our compute budget is small. Moreover, the interaction of unit size and search number is also worth exploring, big search number might be less helpful when subspace dimension is small, thus could spend compute more training steps. More importantly there could be many other changes, one could set search number depending on the dimension of the subspace instead of using the same search number for all subspaces throughout the whole training process, one could also do merging until a target number of subspaces is reached, instead of based on fixed threshold. In sum, we think there are many possibilities, we have only explored a small part of them, there is lot of potential improvements for NDM.
D.6 KNOWLEDGE CONFLICT EXPERIMENT
Unlike the original purpose of RAVEL dataset, we do not aim to find separate subspaces for different types of attributes. Because we find many attributes given in RAVEL have high MI, e.g.. the continent and country of a city, we think these attributes would not "naturally" been placed in separate subspaces, even if so, our method might merge them because of high MI. In contrast, we think there might be separate circuits routing context and parametric knowledge, and subspaces involved in these two circuits might be more separable.
We filter entities to make sure language models have parametric knowledge about them. Specifically, for each entity in RAVEL prize winner split, we prompt the model to complete "A. Michael Spence won the Nobel Prize in Economics. [target entity] won the Nobel Prize in", and only keep entities where model successfully predicts the ground truth. Morcover, we remove entities whose ground truth answer has more than one token. This includes "Linus Pauling" and "Marie Curie". This process produces 552 entities for Qwen2.5-1.58 and 719 for Gemma-2-22 after filtering, among total 927 entities in RAVEL'S prize winner data split.
We only use Nobel prize winner data in RAVEL, because it satisfies several conditions: 1) Language models do not completely lean to one side. When there is no intervention, Original model choice should be ideally equally distributed between the context answer and parametric answer. Nobel prize winner data has a good distribution for both Qwen2.5-1.3 1.5B and Gemma-2-2B with this respect, make possible to observe effect on both sides. 2) should not be too hard for 2B models, such that we have enough instances to give a reliable result. 3) Answers are single tokens for the ease of implementation (not a real problem).
We patch subspace activations at all token positions together. In order to keep simple one-to-one relationship between patching source and destination tokens, the made-up entity always has the same number of tokens as the target entity. We do so by prompting GPT-4o to generate random names that resemble real names, but are not associated with any well- person. The length of generated names varies, ensuring that each target entity has at least one made-up name of the same number of tokens.
Hyperparameters for NDM are the same as those for GPT-2 (App. D.3), except for the following ones: We use 64 as the unit size for Qwen2.5-1.5B and 72 for Gemma-2-2B. We use N = 25 x 2 as the search number. We set 150k maximum training steps. We try different merging threshold (0.015, 0.02, 0.03, 0.04] for Qwen and add one additional value 0.01 for Gemma. We try all these combinations and choose the best one. For Owen2.55-1.5B it is threshold 0.02, merging interval 8000; FOr Gemma-2-2B, is threshold 0.01, merging interval 4000; We apply NDM to h 11.mdd of Qwen2.5-1.5B (totally 28 layers) and h 9,mid of Gemma-2-2B (totally 26 layers). We choose the layer according to preliminary experiments where we patch the whole activation at the last token position of target entity.
After training the estimated MI decreases significantly, see Fig. 17 and [8] Interestingly, as we can see, the original basis of the representation space of Qwen2.5-1.5B is quite special, simply partition on standard basis can produce many independent subspaces.
We show baseline results in Fig. 19 and 20
Some caveats for interpreting results We say the two mechanisms is mediated by different subspaces, instead of saying these subspaces encode contextual/paramettri knowledge, as we do not know the underlying cireuit, encoded information could also be others that support information routing, e.g., the value of target entity (as key for retrieval), the fact that the entity is repeated in the
29

---

Published as a conference paper at ICLR 2026
NDM
Identily
Random
30.
88
550300
922
02
65
20
43
6 55 944
33
55 93
4
6.7
62 0.
00
21
ED
81
0.
46
45
16
64
22
85
84
35
Figure 17: MI between each pair of subspaces. Activation site: h 11, NDM: subspaces given by NDM. Identity: subspaces given by the identity matrix and the dimensionality configuration from NDM. Random: subspaces given by the identity matrix and the dimensionality configuration from NDM. We use the same MI estimation method as the one used during training. The MI values are not normalized, We can see NDM produces more independent subspaces.
NUM
Identity
Handom
0-
10
2- 47
13
2 2-52
2-00..1 3.
0
4-631
50
6
-4
36
24
23
2x
23
19
L5
60 23
82
84
84
84
6.5 811 04 9/1 64 93
8T
Figure 18: MI between each pair of subspaces. Activation site: h 9.idd See the previous Fig. 17 for explanation.
context (may trigger certain operation). They are different types of information and may be encoded in separate subspace, so effect does not have to be concentrated in one subspace. Moreover, also understandable if certain subspace is used in both mechanisms, thus having effect on both sides. So even if subspace partition is perfect, does not necessarily mean all subspaces only have effect on one side, but finding such subspaces is nevertheless good sign. In addition. it is not necessarily the case that the two mechanism are using different subspaces in all layers. In early layers the current affect both mechanism, while in later layers, the conflict may have been resolved
token subspace and the final result is encoded in one subspace. The middle layer is likely the place where retrieval is already happened but one type of knowledge is not yet overwritten by the other.
E QUALITATIVE ANALYSIS DETAILS
E BUILDING ACTIVATION DATABASE
As we mentioned in main paper, we apply Inversion View to subspaces and simply do vector search to find preimages. Here are the details: (1) The text data includes 2000 random sequences in MiniPile, 1000 1O1 prompts and 1000 Greater-than prompts. We made 1OI and Greater-than prompts using the same procedure as we did for clean inputs patching experiments. We run GPT-2 Small on these text, producing 1178970 activations for each activation site (i.c., post MLP residual stream at layer 4, 6, 8, 9). Each activation has a corresponding token position from which it is collected. We maintain their correspondence such that we can always retrieve the input tokens that result in certain activation (i.e., all tokens in its previous context). (2) For a subspace partition trained for an activation site, i... orthogonal matrix R and configuration
make S (number of subspaces) databases by using FAISS (Douze et al. 2024) index. Activations in whole space are converted into subspace activations (multiplied by R and split according to c) and are added to each database respectively. Therefore, make separate databases for each layer (activation site) and each subspace. (3) Then for any interested subspace activation (query activation), we search for other activations in corresponding subspace
30

---

Published as a conference paper at ICLR 2026
10-
02
0.
Contect Carupt
Cunimal Comuupt
(a) Identity
(b) Random
Panam Compat
Param Cormupt
1.0
1.0
6.5
08
Pogo tee
0.
1.0
1.0
0.8
02
50
0.0
Context Comupt
Context Comupt
(c) PCA
(d) PCA2
Figure 19: Subspace patching effect. Partition given by identity matrix. random orthogonal matrix, cigenvectors whose corresponding cigenvalues are ascending order, eigenvectors whose corresponding eigenvalues are in descending order, on h mlmid Qwen2.5-1.55 This figure supple- ments the main paper Fig. [ (the right one) with baselines. We can see the effect under two types of counterfactual inputs tend to correlate.
31

---

Published as a conference paper at ICLR 2026
Corrppt
02
0.
Gontect Carupt
Cuniual Comuupt
(a) Identity
(b) Random
Pwnam Comppt
Param Cormupt
1.0
1.0
6.5
08-
6.6-
0.6
4-14
2-17
1.0
1.0
0.8
02
50
0.0
Context Comupt
Context Comupt
(c) PCA
(d) PCA2
Figure 20: Subspace patching effect. Partition given by identity matrix, random matrix, eigen- vectors whose corresponding eigenvalues are in ascending order, cigenvectors whose corresponding eigenvalues are descending order, on 9. Gemma-2-2B. This figure supplements the main paper Fig. (the right one) with baselines. We can see the effect under two types of counterfactual inputs tend to correlate.
32

---

Published as a conference paper at ICLR 2026
database whose cosine similarity is above the threshold. We then show the inputs corresponding to query activation and searching results.
E.2 INPUT ATTRIBUTION DETAILS
As language model's representations are contextualized, sometimes activations encode information far from the token position corresponds to, i.e., long-range dependency. This makes interpretation difficult, need to search commonality in a large context. We thus make use of input attribution method to highlight important tokens. We use integrated gradients with 10 interpolations. Specifically, given query subspace activation and a sequence of tokens, we first obtain the initial residual stream activations from these tokens, and multiply them with a = 0.1, 0.2,... .............................. 1.0 to create 10 interpolations. We continue the forward pass to get subspace activations. Importantly, we freeze all attention weights to the values they take when running on clean inputs (a 1.0) during the forward pass. We finally compute the cosine similarity between resulting subspace activation and the given query subspace activation. The gradient (not via attention weights) of the cosine similarity with respect to each interpolations are averaged and multiplied with residual stream activations obtain the final token- level attribution score. The scores are normalized by the sum of scores of all tokens in the same input sequence before showing in the web application. So the scores sum 1. But they usually spread somewhat evenly across many tokens, resulting in invisible highlights, so we also provide options in the web application to scale these scores together. Because requires time to compute attribution scores, we pre-compute them for 50 random query activations and show them in a separate page in the application.
The reason for fixing attention weights is that, we empirically find it produces more plausible attribution. The idea behind it to emphasize what is moved by the attention circuit, while ignoring how this circuit is formed. For example, for induction head on input "A B ... A", it would only highlight "B" though all 3 tokens play a role. The same intuition is also used in other works (Ameisen) al., 2025). Importantly, we find the effectiveness of our attribution method is limited, the scores are usually too concentrated at the most recent token, readers should be cautious when using them. This also highlights importance of better input attribution, as in larger and more capable models long-range dependency might be more prevalent.
E.3 MORE EXAMPLES AND DETAILED FINDINGS
24 correspond to subfigures in the main paper Fig. in each of them we fix the subspace and vary inputs to observe the consistency.
Examples supplementing Fig. Fig. 21
Subspace interpretation is consistent while being dependent on the input When say the subspace's 's interpretation or function is consistent, it does not mean must align exactly with any human concept. For example, as shown in Fig. 25 subspace 0 encodes only the current token in first two subfigures, but it encodes 2 more previous token in the last subfigure. This does not contradict with our claim of consistency, as the 3 tokens form frequent phrase, and can be regarded one.
More importantly, as shown in Fig. 26 we found subspace interpretation can depend on input in an interesting and different way: the subspace encodes the occurrence of repetition if the current token is relatively less common and repeated, otherwise it "falls back" to encode the current token. This behavior is understandable, certain meaningful pattern does not always occur, the subspace might have an "default/empty state". When it occurs, it overwrites the information in the subspace. We also think the phenomenon for subspace Fig. might be because the subspace partition is not done very well, so that it mixes "repetition subspace" with "current token subspace".
Several subspaces encode local context, while being slightly different We found that subspace 0, 1, 2, 3, 5 encode short-range local context, most of the time they only encode information about most recent 1-2 tokens. But they are slightly different. Fig. 27 shows an example, where subspace encodes
coarse-rrained information for current token but slightly longer context, and in contrast subspace I encodes a more fine- grained information for current token but shorter context. Among these subspaces, found subspace 2 encodes mostly only the exact current token.
33

---

Published as a conference paper at ICLR 2026
Query Activation
Query
per serype tif right to korne cecopprighe whghgghh offercet
sopei
Subspace Preimagge
Subssaace 2 Preinage
perc ppas 560 poourrr able to practso you maaiiing tands dlncet the riss panti of Secuitt
wen prn frues to attry stet ade evoiding robeccu traondos
serv
to writin
wen OM 245, ceeeteee list i5 yoars extrting vovr spensdly
pennt pync s--jendekkere henkeennioeee Acse onders
wen sor bye Shigg butti rest o frrn eecteecct ingincly
der salt 2605 slang in South piove EEEES pe tith
veriggio leingyes other we non who yelr wen C183 Ansugl in movo rech atudy cinsg retest All sectiona atodyy
meen pae 207: ner citcchsl romentic moths, turth Ladt the cle
M22 MSS esse stat prskive high andrney
soe 3000 countoos andee derton mustoro Avtyrto Thnntur
perne pas sur notractioon plo ning. The
Stotenter 2018
lerroughyy somet e peopgg wufl prrject makes
(a>
(b)
Querry Activation
Query Activattio
art 2.80 pou fioecc the capylghttt fightd yor uundrr dffecr
L000 piu LAk te fonnce the copyrightd work under Sifaad lonotivaa
2ubspaoe 2 Preimage
Subapace 2 Preimage
sutt par ent, firily jeu 1o horl Cileect das Ningflis
R.M4 eummvet right and prople tohell -lasons
miitthoug
and 700 pourss code appoarred by 350 stys locust cabblin)
sair pas Alk par of light relect at irarfaa ot conracc Lrvien
aim & nit 71 ailorre rets it passit to gegeeee Doarat provisetesstouue
ane saiit par dl splutic wis stucled by yurt surickigtaaae contanng Solutine conorrattonn sedeccad tharnall mplicaalll
open
PR 21, N0P rodooeeeaa 'o eeceettk
Seotie
pastrper 205 rai dime putoticig sumy thee Shout tie Hko
ane pos 67 for
4ltent pausiia
ane
of wnlyy with apeci
(d)
Figure 21: Subspace 2 at h A.ppsst Preimages of the residual stream activation corresponding to continuous span of tokens ("under", "a", "different", "licence") the same subspace after the 5th layers in GPT-2 Small. Interpretation of encoded information: (a) the current token "under". (b) the current token "a". (c) the current token "different". (d) the current token "license". But there are not enough samples above cosine similarity of 0.85, we lower the threshold to 0.75 to check more samples. It is probably because encodes more than just the current token.
Query Activation
Query Activation
cappii
Proinsage
1T resing the here hew weel Deguntyle
dinnes
glotling
maity
the cer saroenss, ove sproaee cure glest drii bheee
the can the Apecie Soltvae 'ho sirt
"eltor foods. "hhe EEE firs ON--.
IoC aammauer rroocre commer floeees
une cart she minghe autorratie satirtul
ahe cortpe 21P11 The mecedd revi Copyrigh bousee ATYTO
LiA coertyy inleppectene from
(a)
(b)
Query Activation
Query Activattoon
Aude nos Luk /o rolceca it coppriah graid buuder Soae Comoorith
1itvv tee raper thec
Sabep sow Preimmgge
Sutspace Preinago
sAn mn), de sere waku poli
freewie by se betiiiieeeeee
ardiili (P31) becuueett the nasse others winlod relll under seranno The
SAA (80: starrssti proa thet offiree Cf the
are mos W0. time tie remarr off show talhoo
own,, aus parnall wrlles of soeg supaccs diffee
till tett eeflyeu parter
ardddl 225/ the ceriday t soage wnwt
The do the aneet expoementttteee
east the Alse s wnterrant the Al bere wi
sas pou aoa of sacering ofliree [20 mlndio
(c)
(d)
Figure 22: Subspace 6 at h Appot, Preimages of the residual stream activation corresponding to a continuous span of tokens ("under"", "a", "different", "licence") in the same subspace after the 5th layers in GPT-2 Small. Interpretation of encoded information: (a) the previous two tokens "copyrighted" and "work". We have to lower the threshold to 0.7 to have enough samples inside preimage. In those samples, we can see some contain "work" and others contain "COPYRIGHT" in previous tokens. We think that both tokens are encoded, that is why having only one of them cannot produce high similarity. (b) the previous token "under". (c) the previous token "a". (d) the previous token "different". This subspace is the one with the largest patching effect in test 1. Its un-normalized effect 3.60, much greater than the second largest value 0.03.
34

---

Published as a conference paper at ICLR 2026
Query Activation
Query Activatton
det par sul, light forow cop iight offfrent
reltone the soprighttd work Eraraiiinn
Sobepace 7 Preimage
Sabquaae Preimage
deniie par 1ol skla the coclucioon any coffil car poopery moan
10,, live piofit coneete ia ger onlly made pusk rith alanal powerki fun form
Foa nordedtw fall trer Tanla
denidai, pas protected fancli andiocatiin
bymmet, bl
Aesgiiii vor Br
venttrt you :doccppdenta entampin
1000 sccrstiyfuutt | samored
serttitl Mendehhett--tteee benthy Jami wer E
1.00 J4l ee, wnle ph bo Extil eneeeepde
sertiitt..m ont iiluuattion whit sict rerhereeeeee
you com
sern you "relladdioeee aes't Minoiee "Lngled
0MP aey 200.1 mesoueph
appiiits importaere
(a)
(b)
Queryy
prodiam
Selopase Preinege
m00 RD- 11 Simittrr Aerorr cn coloeting efject chundd yu tet sruuto proeied the
ene por 537 Vestraatinn ewcy
400 del gie supprr usee + araant 1013,
fo aide
per to new puotiion to Melendes uncerraed the baing
dist 41 Oet bule Io pf bovg du afwec 4590
ye yow thihee 1t
(c)
(d)
Figure 23: Subspace 7 at h 4.sst, Preimages of the residual stream activation corresponding to two tokens adjacent to the in Fig. (])) ("under", "different"). ") and two tokens in very different positions
(at position 565 and 384) in the same subspace after the 5th layers in GPT-2 Small. Interpretation of encoded information: (a) current position is around 1 117. (b) current position is around 119. (c) current position is around 565. (d) current position is around 384. For this subspace, one can increase
the threshold and obtain samples of very similar positions as shown in Figure 31
(c)
(d)
Figure 24: Subspace 8 at h 4.postt Preimages of the residual stream activation corresponding to two tokens adjacent to the one in Fig. ((c) ("under", "different"), and two tokens in very different context in the same subspace after the 5th layers in GPT-2 Small. Interpretation of encoded information: (a) context is about software license/permission.. (b) discussing software development. (c) describing detailed medical or physical processes in technical contexts. (d) online advertisements and announce- ments, more concretely, suggesting readers do something. We can see as (a) and (b) are close, the topic information is also similar, while it is very different from (c) and (d)
35

---

Published as a conference paper at ICLR 2026
Ovey delaaioo
Query Activaiion
Query Actimstoon
uolceee all ths heur offert, eerr rnd
Preinage
Subeppood Prereoge
ahe siuizen 511 manci foe ouE cvee thes levp A
hast 30 soegg on civeree
ain aatteen 4NC aeoudder
199 feet
you
sory purtt patir you rece
calicane an low piople repilerl coseed liek corserts O
yeu aottiinitinn vomer ring 4E300
ane nastran de maaape lic cant chatl ataersdigg
cattyyer S0g sod fel
tas mo demeliiss oit livel salig boikding
mertedd oft e2003
(a)
(b)
(c)
Figure 25: Subspace O at h p.ppo Preimages of the residual stream activation corresponding to three tokens in a continuous span. While encoded information for (a) and (b) are the current token ("for", "a"), the encoded information for (c) is "for while" instead of just "while" (we confirm this by checking more samples and varying threshold). So the high-level interpretation of the subspace is not current token, but more like current semantic unit. This also shows that sometimes several recent tokens are merged in representation, but sometimes they are not, indicating some kind of "internal tokenization".
An Cef bodil
ogn prrseeaati
(b)
unde Rone
(d)
Figure 26: Subspace 4 at h post, Preimages of the residual stream activation. Different from other preimage figures, we show input attribution scores here, because the encoded information depends on long-range context, we need these hints to help find commonality. The blue color represent the attribution score (see Sec. [2.2 for details), they are amplified by a factor of 20 for visibility. The token where the activation is taken from is marked by bold and underscored font. Encoded information is: (a) the current token is repeated multiple times. (b) The current token is repeated multiple times. Note that we check the entire previous context of the query activation, and there are indeed more "water"" tokens there. We don't 't show it due to space limit. (c) The current token "above" (if we only consider similarity > 0.85) (d) The current token "ac".. We checked the entire context and found the token "ac" is not repeated. Overall, we found the following pattern: when the current token is relatively less frequent and is repeated, this subspace clusters together activations corresponding to different tokens, otherwise it clusters activations according to current token. The signal might be important to activate induction heads in later layers.
36

---

Published as a conference paper at ICLR 2026
Activation
Query
ahe pre GED Raddko, NiShime Corn
Subspace Preimage
pace 1 Praimage
aftn silert pas sait Lag, darmo and foos Eaona
ser net is, dewduttaai---eee the wonci tht Bereaoe Singhen
pee 755 d du-loste hesly
afe 0.00 pes 200, Plenodd Lfffeet la rice Carnet
ater 222 good dieatig tine stoud and
sle ect un, hospital walle ded bog conqu pupare
ane 960 20, Aitaere, Sa
aftn (.07/ PAl andee Farrice lininn,
ster 6.307 200 come sour Who bee
and p.000 22tt ooyden feacherd, Sam lickey
ater 401 wane vitties
(a)
(b)
Figure 27: Subspace 0 and subspace at 4.ppstt Preimages of the residual stream activation corresponding to the same input. Both subspace O and in this case encode local context, but they slightly different: subspace O encodes more coarse-grained information for current token (i.e., a first name) and encodes a slightly longer context (i.e., there are more than one name in context); subspace I encode shorter context (only the current token) but more fine- anneggraiedd information (i.e., exactly "John").
20 Voiae all hettke un tou
(a)
(b)
(c)
Figure 28: Subspace 9 at h 4.post (a) we show the histogram of cosine similarity between a activation and all other activations in the subspace. We can see all activations in the subspace are in similar direction. (b) (c) we show some examples inputs and the L2 norm of their corresponding activations in this subspace, can that while direction is similar, the norm can vary between 10-50.
Some subspaces are hard to explain We also find the remaining two subspaces and 10 hard to explain. For subspace 9, we find that, somewhat surprisingly, almost all activations are in a similar direction, as shown Fig. 28] We also find the norm of activations in this subspace varies, the main variance in this subspace is from this direction. If this is single feature, what is the pattern that affect the magnitude in this direction? However, we do not have very confident conclusion for this question. We find that the norm is almost always small if the current token is in the middle of a multi-token word (e.g., Fig. 28(b)ss 's bottom row) and very big z 2000 for "end of sequence" (EOS) token. One point we believe is that, the information encoded in this direction is quite independent with the information in other subspaces. Interestingly, subspace 9 in h &post also shows this pattern, i.e., all activations point to similar direction and big norm for EOS. But subspace in that layer is the one with the highest effect in test described in Sec. [5.] it has an un-normalized effect of 2.31 (dg = 32), the second and third largest is 0.78 (dz = 96) and 0.30. This means such an subspace indeed encodes certain meaningful information. In IOI domain it is the position of the subject (outputs from S-Inhibition heads), but it might be part of more general mechanism, like a signal to not attend to certain part of context. That explains why it so hard to interpret.
Coming back to layer 4, for subspace 10. we also see many activations in the same direction, but not as often as subspace 9. For both 9 and 10, we do not have a clear conclusion for the information encoded by them. But we believe they are worth more extensive study, as it might reveal some unexpected features used by the model.
So far we have described every subspace we Summary of subspace interpretation for h 4post found for one layer. Here is brief overview:
Subspace (dg = 128), (di = 128), (d2 = 96), 3 (d3 96), 5 (d5 = 64): short- range local context, usually only most recent few tokens. They encode slightly different information.
Subspace 4 (d4 = 64): Repetition in the context.
37

---

Published as a conference paper at ICLR 2026
(a)
Figure 29: Subspace 4 h 8.post, The subspace is the one with the highest patching effect in test described in Sec. [5.1] so it should encode output of induction heads. (a) "ay" is moved by induction mechanism. We show the entire context for each input. (b) ""17 is moved by induction mechanism. We check the entire context and confirm that the first, second sample's context indeed contain ("One". ".") combination, and the third from the last indeed contains a ("er", ",") combination. We scale attribution scores by 10 so. Importantly, we pick these on purpose, because in many the induction mechanism not "activated". those cases the information more about the current token. In many other cases there simply not enough samples in preimage due to limited size of our activation database.
Subspace 6 (dg = 64): Previous token, almost excluding the current token.
Subspace 7 (d7 32): Position information.
Subspace 8 (dg 32): Longer-range context, topic or the main described object.
Subspace 9 (dg 32), 10 (d10 32): No clear conclusion.
Subspaces in later layers After exhaustively exploring subspaces in the residual stream after layer (the 5th layer), we move on to take a brief look at subspaces in later layers. We show some preimages in Fig. 29 and Fig. 30, which correspond to two subspaces with the highest patching effect in quantitative evaluation. The web application includes results for all 4 layers (4, 6, 8, 9) used in quantitative evaluation, we encourage readers to check our web application.
To check interpretability of subspaces in later layers, we also try to interpret every subspace of layer and summarize our findings follows:
Subspace 0 (dg 128), (d; 96). 2 (d2 96), 3 (dg 64), 8 (dg 32): local context. Compared to layer 4, they tend to cover a longer span of context, such as sub-clause. They encode slightly different information.
Subspace 4 (dg 64): Induction output.
Subspace 5 (ds = 64): Longer-range context, topic or the main described object.
Subspace 6 (ds 32): Certain kind of syntactic role of the current token, not necessar- ily aligned with human definition, such as whether the current token is the subject, or starting/ending of sub-clause/sentence.
Subspace 7 (d7 32): Position information.
Subspace 9 (dg = 32), 10 (d1o = 32), 11 (d11 = 32), 12 (d12 = 32): No clear conclusion.
Subspace 10 shows similar pattern as subspace 9 of layer 4 residual stream h 4.post
Again, we see some subspaces that are hard to interpret. they tend to be quite independent from other subspaces (Fig. d))..
E.4
SUBSPACES FOR PARAMETRIC KNOWLEDGE ROUTING IN QWEN2.5-1.5B
In Fig. 5(a), we see that there are two subspaces, 2 and 6, playing important roles for parametric knowledge routing. We apply the same qualitative analysis what is the information encoded
38

---

Published as a conference paper at ICLR 2026
cudh euc
(d)
Figure 30: Subspace 4 at h P:ppst The subspace is the one with the highest patching effect in test 5 described in Sec. [..]] the effect of all subspaces is shown in Table So should encode the starting year information in Greater-than prompts. (a) Encoded information is (approximately) "79". We can see how the same mechanism in Greater-than prompts being used in more natural text. Note that we lower the threshold to 0.7 in order to include more samples. We can see though having the same starting year "79", the four Great-than prompts are not close enough to query activation, so some other context is also slightly encoded in this subspace. (b) Encoded information is "10", Here we can see how this circuit is useful in a broader domain. (c) Encoded information is "ant". We can see this subspace is essentially induction output subspace. (d) It seems that the encoded information is a mixture of "activities" and "performance", having only one of them (like the last sample) is not enough to get close to the query activation. So the induction circuit is triggered quite often, can be used to predict the noun after "the".
Query Acthation
Query Activation
ater. 300 217, ue geette nel carce cheopp grted Eomeeet
comu you 119, tore Soonee the copyrigtna vnd
Exper (enecadd
pace Preimage
Subspece Preimage
sent mes jaw API cree wrs and full awdd i terond the sare hoo wit eg
the 8:90 you 1SIt iu ligg oo theprng Cuncee Gill enterrondiingg
orgari Eullll tolaros sillows tee sume pou ios, wenta the aaitaa (i0l sas
1.04 186 and
sten 1ot 175, thin wenge then pa- ee has dynam
<ofl wile
sten ao poe 113, Ai and Missort, The sooe towe tell of now some cond) improvee tobi..
ahe 5.54 you 120, abeutt Hhe rrweted esteraheb thin throughot the eve goitig
marry
dle may paac aur Cille fine ingllsh ang gage desioo of rected ve maci audf rour titosay
1.90 you 130, sa sew pleadd 399 puotraaggry woldit
1:00 you 121, aque hooping Copoorand nande vos tommur geto
(a)
(b)
Quary Activation
Query Activation
1,00 pau 565, la because this the probl 252et
a 905 publy Mapo
Subspace Preimage
Subopace 7 Preimagge
sindd pas sik, baw bare vily orlic for tullingt prosecute por A of the to eegarsone
a the tuil bdidkar Intall beddewuue
paz S56, open
port oges
slev wio pou 2ie, to have any iniuinni sigglss Soaii know deea thane
3055 you 576, that govarment
undoree era caprictrs
massage
time pat vis, doteeittee snleinng cleent, de you whucae (Tat dievery single
Ver 5t6 di sttrnd in the sare twertyy thousee
pic
compacc
alen wos pos all, cirector of poo biibyl vo gooup
3u54 you SP4, Kings Caryen haom aatt- ree slovaiion the imbo
see 1050 557, wetheer wereese rilp, one (2 wi been teting syve!
ales 9et mer, department soeer meeer odd and of Michae atrn sol por 280 four gewrgeeendooeee fomby
per
ruiling
(c)
(d)
Figure 31: Subspace 7 at h 4.post, Same examples Figure 23 but the threshold for cosine similarity is increased to 0.99, we obtain samples of very similar positions. Remaining error could be because the optimization is not optimal, or the model has forgotten the information bit at this layer.
39

---

Published as a conference paper at ICLR 2026
Query Actiation
Activatce
1000 awe momen Yoldl U
femee eer't inbbed tn
sosh wis thethaaale Passii
Sabepace 3 Prelmagge
Sulepaoc Proinage
comterr undedd Oeverey
we
ane wiecc mara lro what cndmaaaa becomes So Pogers reg Carch
une cas ceniiiee fecien OOT mocee locieai Jaln dardes aeriici tie
tocdal
are coc.ooooe eiy 200 ousiped by Cain Chapraa olE tary tw powered Fors to sou aiggen
ane jaur
oactighe anterter proezzane duntaa
shrugges Mechiliwgghll
uns uos woithersle sale misiicheenay cnct Seetttr
the rreddhhue Johe teatter wo the Prenr ii dreg lilggna use vode Plylica
Query Actioation
Acthvation
Arion
A000, war diocude riasi conaaa ner
aoe Preinage
So Sabaaace Preeiwage
cun pen defest di Agiie Litalaan Ougins chiell catt you Matucleo
cam dary Recier bitect Cotemmit Minal
com awn7: Airadi-du Mhd
(c)
(d)
Figure 32: All statements shown are true, there is no made-up entity. (a) & (b): Subspace 2 at h. mi.d in Qwen2.5-1.55 1.55B. The subspace is the one with the highest patching effect under Param Corrupt in shown in Fig. 5 a). So it should play an important role for routing parametric knowledge. Encoded information shown in (a) simply "first name" or "partial person name", while information shown in (b) is the knowledge retrieved from model's parametric memory, "the person is economist". Therefore, we hypothesize that this subspace receives knowledge retrieved from MLP layer. When only part of the name is observed, the only "knowledge" is that "this is a first name", when enough tokens are observed (not necessarily the full name) to identify a memorized person, the subspace is filled with retrieved knowledge. like the field of work. Note that for this subspace cosine similarity to 0.7 in order to include enough samples. So it seems to also encode other kinds of information. Better training might help disentangle them. (c) & (d): Subspace 6 at h 11, in Qwen2.5-1.5B. This is the with the second highest effect under Param Corrupt in shown in Fig. 5 a). That means it also plays a non-trivial role for parametric knowledge. Encoded information shown in (c) is "Literature in the context". in (d) is "Economics in the context".. From these two preiamges, we hypothesize that this subspace receives knowledge broadcasted from the most recent recognized entiry. At first name token position, the current name "Kenneth" is not enough identified any memorized person, so this subspace contains the previous entity's 's knowledge "Literature", other normal tokens such as "Prize", "Nobel", ..* also receive this knowledge, and similarly for (d). Note that for this subspace we sometimes do not find clear interpretation for individual preimage. Possible reason could be that sometimes current entity is "half recognized"' and its knowledge is mixed with previous entity.
lower the threshold of
40

---

Published as a conference paper at ICLR 2026
in them, and if they differ in some way. Like in quantitative tests, test and remove entities that model does not have knowledge for. We randomly sample 250 entity pairs from them, each is used to construct 2-shot example. We then run Qwen2.5-1.5B on these examples, together with 2000 text sequences from MiniPile, and collect activations to build the database. Fig. 32 shows some examples. As we can see, they indeed differ. Though they both contain retrieved parametric knowledge, subspace 2 encodes the knowledge about the current n-gram, while subspace 6 encodes knowledge that does not depend on current token. We therefore have the following hypothesis: when enough tokens are observed and the name is identifiable, MLP retrieves knowledge and places it into subspace 2, this knowledge is then emitted by attention heads to many other tokens' residual stream (or in other words, attended by other tokens), this information arrives at certain subspace similar to subspace 6, but at later layers. This behavior is probably distributed and repeated across multiple middle layers, so subspace 6 in this layer receives information from certain subspaces in earlier layer that resemble subspace 2 of this layer. So the information transfer is not done all at once. To verify this hypothesis, more rigorous experiments should be done. We encourage readers to check more examples. Due to its large size and various limits we did not manage to deploy the database for Qwen2.5-1 our web application. But we released all trained orthogonal matrices together with the code. Readers can follow the instructions in our repository and easily reproduce the database.
F COMPARISON TO SPARSE AUTOENCODERS (SAEs)
Due to the popularity of SAEs, in this section, explain the difference and connections between them and our method. Readers who are familiar with SAFs might find this section particularly helpful.
F.1 FAQ
(1) Why not compare with SAEs?
Quantitative evaluation In previous quantitative evaluations, we patch each subspace and see its average effect on the output. SAES do not provide such subspaces directly. If we have to make comparison, we can change the evaluation method to apply an analogous evaluation SAEs. One way is to patch the basis elements that SAES provide, the features. This means that we replace feature's activation value with its value on the corrupted input. For example, in the clean input on the token after "John", feature representing *"previous token is John" "* is highly activated while, in the same position in the corrupted input, feature representing ** "prrevious token is Mary " highly activated. If we patch the "prev-Mary" feature, the new activation would say ** "previiuus token is both John and Mary"*, "* # If we patch the "prev-John" feature, the new activation has no information about previous token. In other words, we need to add a feature while removing another one (and it's hard to select which one since there are multiple active features), which requires understanding of feature dependency that SAEs do not provide ("prev-Mary" and "prev-John" More importantly, patching the "prev-John" feature would not have any real effect on inputs where the names are not John. In other words, conceptually, one would expect such patching to only have effect on a few specific inputs, and only to confuse the model. In contrast, patching subspace can be thought of patching meaningful feature groups. Moreover, applying such patching to SAEs requires running evaluation on each of the features, which is too computationally expensive due to the large amount of features.
mutually exclusive).
Because of these fundamental difference, we do not compare with SAES in the main paper. Never- theless, readers might still wonder how would SAES perform under the same evaluation used in the paper. For this, a better way is to derive subspaces from SAE features. We take this up in Section F.2
Qualitative evaluation Our qualitative evaluation has two aspects: (i) interpretability of preimages for subspaces, (ii) consistency the type of information encoded in a subspace across inputs (thus, interpretability of the overall subspace). Regarding (i), we think that a single SAE feature would have interpretability comparable to a single subspace preimage, as we know that they are interpretable most of the time but also some features do not have clear and easy interpretation. However, regarding (ii), this is not applicable to SAEs, as they provide no clear structure between features; rather, to find encodings of the same type of information requires inspecting each feature individually. In contrast,
41

---

Published as a conference paper at ICLR 2026
information in the same subspace found by NDM is of the same "type" across inputs most of the time. Inspecting some of the them can give an idea of the whole feature group, significantly reducing the amount of interpretation work needed.
(2) What about training SAEs within subspaces?
Conceptually, if one trained top-1 SAE (top-k SAE with k=1) within each subspace, we expect that the features would be the cluster centers in the subspace. Inspecting a feature means to find inputs whose activation in the subspace is close to the feature. This is roughly the same inspecting preim- ages in Inversion View, with query activations taking the role of the "feature". Thus, Inversion View can conceptually be thought of "parameter-free" top-1 SAE.
F.2 QUANTITATIVE EXPERIMENTS GPT2 TEST SUITE
In this section, we check whether we can obtain some meaningful subspaces from SAE features and evaluate them with our metrics.
Settings We use the method introduced in Engels et al. (2024) (Sec. 4), where the authors cluster feature vectors and find subspaces. Specifically, the method has three steps:
Cluster features by their pairwise cosine similarity. There are two clustering method, (a) Graph. It connects each feature with its top-k neighbors whose similarity is above threshold, and form clusters by recursively searching for connected features. (b) Spectral. Spectral clustering.
2. For each cluster, run the SAE on model activations. When reconstructing activations, they only use features inside the cluster (i.e., activation value for other features are always zero), we refer them as cluster-specific reconstructions. They remove 7ero reconstructions from this collection, so that at least one feature in the cluster activated.
3. Find the multi-dimensional subspaces spanned by these cluster-specific reconstructions. They apply PCA on these reconstructions and inspect the points in the subspaces spanned by pairs of principle components with large eigenvalues.
In case, aim extract subspaces from clusters of features. Thus slightly change their method for our purpose:
1. Cluster features. Same as above.
2. Sort clusters either by the average cosine similarity between features inside the cluster, or by the cluster size (number of features) in descending order.
For each cluster (in this order). apply the next three steps. Overall. we iteratively build matrix C' x of orthonormal where m is the number of dimensions selected far. The columns of C' are orthonormal columns, representing a basis for the subspaces constructed far. for each cluster, we add another set of columns representing the subspace for that cluster.
(a) Obtain cluster-specific reconstructions, using the same method as Step 2 above in the original method. Denote the reconstructions as X [æd where is the number of data points and d is model dimension.
(b) Remove variance on previous selected directions (skip if no direction has been selected so far). Denote the directions selected so far as E RRd where m the number of selected directions. We then take
B-X-Co"L
(13)
which removes variance on the directions selected so far, or equivalently - proclecttt on the orthogonal complement of the subspaces constructed so far.
(c) Apply SVD on X. We select all left singular vectors (i.e., the same same as the eigenvectors in PCA) whose corresponding singular values are above a threshold. The threshold is set by multiplying the maximum singular value with a hyperparameter rtol. The selected singular vectors, or directions, form a new subspace corresponding to this feature cluster. They added as the columns of C.
42

---

Published as a conference paper at ICLR 2026
Cluster contg
Extract condig
susospaces
test
Vir,
1est
Vir,
Test3
Var,
TesE
Var,
Test
Vir,
Avg
Spectral aua-10
aua-11
avg sim, n000.11
0,003
0.34
1,20
01.1
0.41
1,50
0.13
11:19
00.5
0.74
01311
0.51
0.71
0.30
03.0
11.56
Spectral, num-10
sze, 1ol-4).1
5.7.8.8
0.70
0.61
0.25
0.40
0.32
0.56
0.70
0.30
0.89
0.64
0.22
0.64
0.68
0.50
0.46
0.49
Spectral, 2um-25
29g sm, nol-0.03
6.7.7.7
0.78
0.72
0.72
0.42
0.15
0.36
0.48
0.24
0.53
0.35
0.20
0.30
0:48
0.33
0.41
0.42
Spectral, mmm-25
avg sim, noL.0.1
10,12,12,13
0.52
0.60
0.53
0.50
0.26
0.48
0.60
0:45
0.54
0.60
0.30
0.51
0:59
0.36
0.48
0.49
Spectral, mm--5
9.98.77
0.81
0.67
0.36
0.53
0.57
0.55
0.38
0.31
0:49
0.42
0.59
0.51
0.42
0:50
Spectral, mm
sog-em, mal:0.01
5.6...
0.65
0.67
0.31
0.13
0:11
0:18
0.35
0:27
0.15
0.32
0:33
Spectral, mum-50
sog-aim nal-0.01
7,8.9.8
0.77
0.43
0:39
0.43
0.46
0.42
Spectal, mus-50
aegsim, rld-0.1
15,18.21,19
0.85
0.80
0.53
0.55
0.44
0.53
0.36
0.46
042
0.49
0.49
0.47
Spestall aua-50
7.8.877
0.63
0.49
0.46
0.58
0.50
0.48
Spectal, sum=100
arg-sin, nole0.01
9.8.9
0.70
0.25
8.11
0.19
0.26
1131
0.30
0.31
Speccal, sum=100
ang- nol=0.13
13.12,11.12
0.84
0.33
0.36
0.40
Specual, zum=100
avg ridl-0.1
24,26.26,25
0.86
0.51
041
042
0.41
0.45
Special, sum-100
size, no0-0.11
9,12,12,11
0.54
031
Table 8: Results on GPT-2 Small test suite, higher better. Refer to Table for explanation of the metrics. We include hyperparameters and the number of subspaces produced for each layer used in the tests.
3. We stop once C' becomes full basis, i.e., C' E Rdxd or the clusters are exhausted. In the latter case, we supplement C' with remaining directions to make it an orthogonal matrix.
In sum, we follow the original method from Engels et al. (2024) as much we can, while adapting to construct a full orthogonal decomposition of the space. We test this method on the GPT-2 test suite, with many different hyperparameter configurations. Each configuration is a combination of the following hyperparameters: In step 1, use Graph or Spect For the former, there are two hyperparameters: the k used in defining top-k neighbors and the threshold cutoff. For the latter, num specifies how many cluster there should be. In step 2, sort the feature in two ways, avg-8 8 m or size. In Step 5, rtol that filters out unimportant directions. We search among [2, 3, 4, 6, 8, 16, 32], search cutoff. among [0.1, 0.11, 0.13, 0.14, 0.16, 0.18, 0.21, 0.23, 0.26, 0.3, 0.34, 0.38, 0.43, 0.48, 0.5, 0.55, 0.62, 0.7, 0.78, 0.89, 1.0], search num among [10, 25, 50, 100], search rtol among [0.01, 0.03, 0.1]. For graph, the resulting number of clusters varies lot. We remove those configurations whose number of clusters in any layer is out of the range [5, 200]. This results in 16 configurations for graph. When extracting subspaces, we ignore clusters which contain S 3 features (as the target information we are testing is not likely to be encoded in 3D or smaller subspaces), and further restrict the number of subspace to be inside [5, 50] in order to speed up the evaluation. Unexpectedly. none of the configuration using graph produced a partition satisfying the constraint. We find that this is mainly because it produces very uneven-sized clusters. There are usually a few big clusters, and many small clusters each of which has a few features. Some configurations that use spectral are end up being filtered out.
The results are shown in Table g We can see there are two hest-performing configurations whose average Gini coefficient is 0.56; we put the first row in the main paper, Table It shows that extracting subspaces from SAE feature clusters is significantly better than other baseline methods (0.21-0.38 in Table D.. probably producing somewhat meaningful partition. But there is still significant gap between it and NDM's 's result (0.71). In addition, we can see that many configurations are somewhat successful in test 1, e.g., the third last row (Spectral, num=100 100, avg-sim, rtol=0.03), but not very successful in other tests.
Results
G ADDITIONAL DISCUSSION
G.1 WHY IS INVERSIONVIEW MORE SUITABLE IN SUBSPACES
Compared to whole representation space, we argue that Inversion (Huang al. 2024b) is more suitable to be applied to subspaces. For example, subspace 2 and subspace 7 in Fig. 3 eenodd current token and current position respectively. If we look for the preimage in the joint space of subspace 2 and 7, the inputs will need to have the same token at the same position to be inside the preimage. Thus when the query activation contains multiple pieces of information, or encode multiple aspects of the input, the samples in preimage should be the same in all these aspects, which makes it hard to find such samples. Even if we can find enough, they might simply have the same recent tokens, which does not give much insights.
Importantly, some subspaces might be dominating the joint space while some others are ignored. If the norm of the activations in subspace A is much bigger than those in subspace B, the cosine
43

---

Published as a conference paper at ICLR 2026
dormant feature
direction patch along (V)
ortheggonal complement
activation patch from
activation to patch into
causally disconnected feature
sprmsa
Figure 33: An example illustrating subspace activation patching illusion. Fig. from (Makelov et al. 2023). The 2D space is partitioned into two ID space (red arrrows). Green arrows represent activations in the whole space. Subspace patching modify their projection along the red v direction, resulting new activation (blue arrow). In this setting, suppose x-axis has variance but downstream causal effect, while y-axis has no variance when running model on normal data but has causal effect there is any nonzero value. In this setting, resulting new actívation (blue arrow) is out-of-distribution and will have causal effect, suggesting v is a meaningful direction. However, this 2D space does not play meaningful role model's computation (suppose this 2D space is subspace of a bigger space). In our case this does not raise a severe concern. When applying NDM in this 2D space, partitioning like red arrows in this figure very high MI. Partitioning like the and y-axis is the optimal choice, thus resulting two unimportant 1D space.
likely to happen, because the resulting two ID subspaces have
similarity in the joint space of A and B will mainly depend the cosine similarity in subspace A. In this case, the information read out in preimage of the joint space is only the information encoded in subspace A. In some cases this is fine, as the information in subspace A is much more prominent than the one in subspace B, and Inversion View gives the right overall picture. But in some cases, e.g., in the residual stream of transformers, small subspaces might dominate certain attention heads, as attention heads usually operate in small subspaces (we do not think this only applies to attention heads).
G.2 ON THE RELIABILITY OF SUBSPACE ACTIVATION PATCHING
Our quantitative experiments on language models relies on subspace activation patching. (Makelov et al., 2023) found cases where subspace activation patching produces illusory conclusion. Subspace intervention might change the model's 's output by activating dormant pathway which does not have effect in normal computation, as illustrated in Fig. 33 Here we argue that our experiments are still reliable because of following reasons: (1) Our method finds subspace partition that reduces MI or dependency between subspaces, instead of partition that maximize patching effect. The partition shown in Fig. 33 that causes illusion has very high MI between subspaces, thus our method would not give such partition. In other words, we apply subspace patching on subspaces found by NDM, because of relatively good independency, so activations from different subspaces can be composed more or less freely, without causing out-of-distribute doing subspace patching in activation bottlenecks, especially residual stream, to avoid such illusion. This is exactly what we did in experiments (3) We also provide validation by examining the subspaces qualitatively.
activations. (2) (Makelov al.) 2023) recommend
G.3 ABOUT FUTURE DIRECTIONS
So far we have provided various evidence supporting our claim that we can decompose representation space into smaller and meaningful subspaces, without any human supervision. These subspaces, being
44

---

Published as a conference paper at ICLR 2026
orthogonal to each other, can serve as new basic units mediators for mechanistic interpretability. Compared to neurons, they capture the distributed nature of neural representations. Compared to sparse features, they provide more structure. Each subspace can be thought of as a group of infinitely many features sharing a certain higher-level concept, by interpreting some of them we can understand the whole group. Moreover, it enables to understand the model by directly analyzing the model's s weight matrices. For example, we can understand an attention head by checking which subspace its query, key, and value matrices read from and which subspace it writes to (Elhage et al., 202T). By doing so, we build connections between subspaces across layers, resulting in subspace circuits. We believe subspaces could be more "fundamental" than model components as they are designed to be as independent as possible. Moreover, subspace circuits would, like real transformers, operate in the same manner over the whole input sequence (uniform across token positions) as well as over different input sequences. Such circuits could be predecessors of source-code-lic explanation of the model, since variables in RASP language (Weiss et al. 202I) can fundamentally be treated as subspaces of residual stream. To support this ambitious plan, the current version of subspace partition method might not be good enough. We believe there is plenty of room for further improvement. For example, allowing more flexible searching over dimension configurations (rather than having all dimensions as multiples of fixed number). We also think a hierarchical structure of subspaces could also be helpful. For example, subspace A and B are very independent (e.g., current token and position), and A can be further partitioned into A1 and A2, which are less independent (e.g., the plain token and retrieved knowledge about it). More importantly, there could be potentially different ways achieving the same goal (e.g., by minimax approaches, see App. G.4). We also call for efforts from industry as this unsupervised method should benefit from more data and compute.
G.4 POSSIBLE ALTERNATIVE APPROACHES
In this section, we describe some alternative approaches in order to inspire future research.
Split- approach The high-level idea is as follows:
1. partition the space into two subspaces. We search over n possible configurations (for example, when d=100, we can have [d] = 25, d2 = 75], [d] = 50, d2 = 50). [d, = 75, d2 25]), each configuration has dedicated orthogonal matrix. It turns out that initialization matters a lot and standard basis is kind of special (Fig. [5p. though they are symmetric, empirically we found [dl = 25, d2 75] and [d] 75, d2 = 25] produces quite different results. After certain amount of steps, measure (normalized) MI for each configuration, select the one with the lowest MI.
2. Repeat step for each subspaces already obtained, while restricting the rotation inside the already obtained subspace. For example, assume we have learned a matrix Reev and configuration [dp = 75, detev 25]. we keep (previous) subspace 1 and 2 fixed. For subspace 1, we search over configurations [dl 18, d2 = 57], [d] 36, d2 = 39], [d, = 54, d2 = 21], each associated with a separate R, R2, R3 E 75x75 Similarly do this for subspace 2. Assume in this iteration, after training [d] = 36, d2 = 39] has lowest MI for subspace I, we then merge its orthogonal matrix R2 into Re by multiplying prev it with corresponding rows in Re R (0:75] + R2 (0:73]- On the other hand, suppose none of the configuration for subspace 2 is lower than threshold, then we do not update that subspace. After this iteration, we have a new Re and new configuration [d 36, dgeev = 39, dgrev 25J. Then keep repeating this loop until no subspace is separable (all configurations of all subspaces have MI greater than the threshold), or a pre-defined maximum iteration is reached.
This process increases the number of subspaces exponentially, if all subspace partition attempts successful (MI below threshold). However, a disadvantage compared to merging- based approach in the main paper is increased compute cost, because search over configurations, training Rs and select only one. An advantage is that it gives a tree structure, we can understand subspaces of different granularity. Our code link also contains code for this variant.
We did some preliminary experiments to test this approach, the results are not better than merging- based approach, while it takes longer train. For example, Table shows partition learned for
45

---

Published as a conference paper at ICLR 2026
index
2
3
4
6
8
9
10
dg
36
54
54
36
12
144
108
36
18
54
54
162
Var, (x (x10%)
0.4
1.0
0.5
0.4
0.8
2.7
8.3
0.6
0.4
0.6
0.6
5.8
P. (x10 2)
0.3
1.3
1.1
0.3
0.3
2.8
2.3
0.6
0.4
1.9
0.6
40.7
(x10
0.9
2.5
2.0
0.8
2.1
1.9
2.2
1.6
2.3
3.5
1.1
25.1
Var, (x10
0.9
1.3
2.2
0.7
0.3
1.1
0.3
0.9
1.1
3.2
1.0
7.0
Table 9: Test raw results for split-based NDM. Gini coefficient for row AP, is 0.78, for row d,
AP2
is
0.55, and for row AP is 0.47. Var,
layer and tested on Test 5, it runs 4 outer iteration (each iteration splits subspaces), and each time 3 configuration tested for each subspace. Search number 5 214, totally 40k steps.
However, our negative results do no rule out its usefulness, it could be more computationally efficient there's ' more flexible and efficient way to determine subspace dimensions, and might be suitable in some cases.
Minimax approach Mutual information can also be estimated by neural networks (Belghazi et al., 2018). The Mutual Information Neural Estimator (MINE) is as follows:
Let F (To Jace be the set of functions parametrized by a neural network. MINE is defined as (we simplify the notation to avoid some details, please refer to (Belghazi et al. 2018) for the true definition):
T(x; 2)
sup Exxz [To] - log xPFxxp222 (e)))
(14)
Simply speaking. Pxz means sampling vectors and from joint distribution, and Px Pz means sampling them from their respective marginal distribution. The neural network Tp takes both ær and as input and output scalar, it is optimized with gradient ascent to approach the supremum.
Like split-based approach, we train an orthogonal matrix RR to partition the space into two subspaces, given fixed dimension configuration di, d2. Assume X and Z represent random variables in the two subspaces. In order to find independent subspaces, we can optimize R. to lower MI as follows:
min [(X; Z) = min max E. [T] log (Epx (Bxx R
(15)
R #E0
Empirically, Pxz means that subspace activations are projected from the same activations, i.e., each pair æ and results from rotating and splitting a single activation vector into two parts. On the other hand, Px & Pz means each pair results from projecting two independently sampled activation vectors.
The intuition is that, if X and Z have high mutual information, knowing one predict the other, the neural network can easily discriminate whether x and are projected from the same activation, thus output high value when they "match" and low value when they do not, resulting high estimate of I [[;; Z). R rotates and reflects the space such that two subspace are as independent as possible, such that the neural network cannot easily discriminate, resulting low MI estimate. This is similar Generative Adversarial Networks (Goodfellow et al. 2020).
However, empirically do not find this approach is effective. We estimate MI with KSG estimator (so that comparable to our previous approaches) to evaluate performance during training, and found this does not produce significant decrease in MI.
In addition, based on the same intuition, we try another simpler approach: train two neural networks to reconstruct one subspace's 's activations based on the other's, s, and train R make it hard to reconstruct.
max min Ex MSE( (z)) (T) =)].
(16)
R Ti.Ta
where MSE denotes mean squared error, and T,, T2 are separate neural networks of different input- output dimension. We find this approach is more effective than the based MINE, but resulting
46

---

Published as a conference paper at ICLR 2026
final MI is still much higher than split-based NDM (and harder to train). Again, negative results do not rule out its potential, the results might be because of the difficulties in optimization rather than training objective.
H COMPUTE USAGE
We run all experiments on H100s. We report the sources used when using 25 2 search number (largest option in our experiments). Train a subspace partition for one activation site in GPT-2 Small takes around 2 hours single H100 (stops at around 80k steps). Train partition for one activation site in Qwen2.5-1.5B takes around hours on a single H100 (stops at around 80k steps). For Gemma-2-2B, takes around 8 hours on single H100 (stops around 50k steps). Experiments for subspace patching run quickly.
I
LLMS USAGE
We used LLMs to polish the writing and rephrase some statements to improve the readability, while ensuring that the content continues convey our intended meaning accurately. We also used them to identify related work and to generate code for producing figures.
47


Reviews:
[{"review_id": "acWdWDPFkC", "reviewer": "Reviewer_rZmX", "summary": "The paper describes and evaluates an unsupervised method — neighbor distance minimization — for decomposing representation space into interpretable subspaces.", "questions": "In Section 5.1, you say that “The results of NDM using the best hyperparameters are shown in Table 1…”  This raises the possibility of overfitting because only the “best” hyperparameters are shown. How were the best hyperparameters selected, and is this method reasonable in practice?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Well-written with clear explication:** The paper presents complex concepts with clear language, helpful intuitions, and useful examples.\n\n**Vast amount of work:** The paper, combined with the appendices, reflects a vast amount of experimental work.", "weaknesses": "**Evaluation**: As the authors say, “The key question is NDM’s applicability to real-world neural models.” The evaluation relies on the intuition that “when processing inputs, key intermediate results should ideally lie in a single subspace.” On the one hand, this idea is appealing, partially because it provides a fairly clear evaluation criterion for methods for decomposing representation space into interpretable subspaces. On the other hand, this criterion could both under under-determine and over-determine useful results. That is, essentially useless decompositions could satisfy this criterion, and useful decompositions could violate it. Ultimately, a more convincing evaluation would use an end-to-end criterion (i.e., one that shows that an MI pipeline produces more useful results when it includes NDM rather than some other subspace discovery approach). That’s a very tall order, but one that is far less error-prone than this paper’s current evaluation criteria. This reflects a more general problem with current MI research: whole MI pipelines cannot be easily created because we don’t have all the components, and candidate components cannot easily be created because we don’t have the pipeline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FS7bhEGRGU", "reviewer": "Reviewer_EMRg", "summary": "A core objective in interpretability research is to understand content and geometry of activations. The authors propose a novel, original idea to learn a set of linear subspaces, where each subspace holds mutually exclusive features encoded via superposition. Different subspaces are orthogonal thus independent. The authors propose a method to learn these subspaces unsupervised from activation data alone, by learning a rotation matrix (similar to DAS) such that data points within each subspace have minimal distance. They provide intuitive understanding, show their approach in toy models and prove its applicability in small and medium language models. Specifically, they show for some known tasks that all features of interest lie in the same subspace, and they show that subspaces are interpretable (ie monosemantic).\n\nThe paper is well-written, proposes an original idea, and elegantly combines mathematical intuition, toy models, and translation to LLMs. My main concern is that some assumptions/hypotheses were not empirically validated. For example, it wasn't shown that features within a subspace are mutually exclusive, or that the orthogonality requirement is faithful of real activation space geometry. They critique SAEs but don't use them as baselines.", "questions": "1. Many things are mutually exclusive. For example, if the text is about software licenses, it's not about a fiction novel or a mental health consultation. Features from all these different contexts could be squished into a single subspace. Especially since the number of subspaces is extremely limited. Do you observe this in practice? Doesn't this imply that mutual exclusiveness != interpretability? Doesn't this hurt interpretability of those subspaces and the applicability of this method a lot?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- great presentation, paper is well-written and I found it easy to follow. The paper provides both intuitive understanding and mathematical precision.\n- the paper elegantly combines mathematical intuition, validation in toy models, and translation to LLMs\n- the paper posits an interesting, novel idea for an important problem rather than an incremental improvement\n- I do find the author's work interesting from a slightly different perspective that wasn't as highlighted as it could: Feature independence. When steering with SAEs, editing individual features results in an OOD reconstruction because many SAE features co-occur which is a real problem.", "weaknesses": "Major:\n1. \"Mutual exclusiveness\" doesn't seem like a more fundamental condition, or much different from sparsity at all. Elhage 2022 say that when features are sparse, they can be encoded via superposition. Sparse features are already \"almost mutually exclusive\" but the \"almost\" seems important as strong guarantees are hard to make for neural networks. \nThe authors write that superposition wouldn't work if 2 or more features are active but this is false. Yes, interference and error would grow, but only by a tiny amount in practice. Best proof are SAEs that work well with e.g. 20 features active. \nIn fact, feature co-occurrence is baked into superposition theory: A prediction of their theory and toy models is that features that are mutually exclusive should make heavy use of superposition, while features that often co-occur, should be better separated, ie encoded as orthogonal directions (at least more orthogonal than mutually exclusive ones). So no new theory is needed to predict geometry of feature groups so I don't understand how \"mutual exclusiveness and feature groups\" are fundamentally different from \"sparsity and superposition\". It's good to have methods to decompose this, but it follows from superposition theory, and not new theory/hypothesis has to be invented.\n2. The authors assume that features within a subspace are mutually exclusive but you never test this (in fact, you don't even extract features within a subspace).\n3. The authors for some tasks that most task-relevant features land in a single subspace but they don't show that this subspace only contains those features, aka is interpretable and monosemantic. It could very well be the case that completely unrelated information is stored within the same subspace as many unrelated features are mutually exclusive.\n4. The number of subspaces is strictly bound by the dimensionality of activation space as they must be exactly orthogonal. This might limit the method's potential and interpretability. As we see with SAEs, representation space can be well-approximated with tens of Millions of features and having only few hundred feature groups available at most might be quite limiting, and many different features might be packed into the same subspace (because different features are often mutually exclusive as well). This would limit interpretability. My concern is that in reality, the independence-between-feature-groups assumption doesn't hold 100% and we would be better off with relaxing this restriction a little to allow more feature groups to exist and be more interpretable. A possible approach here could be to use MOLTs (Lindsey, Anthropic 2025).\n5. The authors heavily criticize superposition and SAEs but they never directly compare against SAEs although direct comparisons should be possible. They state that other baseline comparisons aren't possible because their subspaces are learned unsupservised from activations, but SAEs are as well. In fact, I think that their method doesn't disagree with superposition and sparsity at all.\n\n\nOther things that would improve the paper:\n- Insight into LLM computation. This paper mainly proposes a new method and validates it but there's no new mechanistic insight about LLM computation. It would improve the paper a lot if the authors could prove by example that this method can discover things that other methods like SAE, DAS, etc can't\n- More experiments with LLMs. Qualitative examples in Figure 2 lack rigor or quantification. IOI/greater than results don't measure logit difference recovered when only patching an individual subspace, more evidence that subspaces are monosemantic and interpretable, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oUz2YawnGr", "reviewer": "Reviewer_MPxT", "summary": "The paper begins from the premise that mutual exclusiveness could be a fundamental condition for superposition. Building on this idea, it proposes that data may contain groups of mutually exclusive features; for example, features encoding categorical variables such as different subjects, where only one is active at a time (L. 110). Motivated by this observation, the paper introduces an unsupervised method, termed NDM, designed to identify subspaces of mutually exclusive features. The method does so by finding subspaces in which data points are projected to similar locations. The paper demonstrates the effectiveness of NDM both on a synthetic (toy) model and on representations from large language models.", "questions": "- In Eq. (1), why is $h = W x x'$ instead of $h = W x$, as used in the experimental setup of Elhage et al. (2022)? It is possible that the equation is incorrectly formatted and that $x'$ is intended to appear on the right-hand side?\n- How to choose the MI threshold and how sensitive is the method to that threshold?\n- How would sparse autoencoders (SAEs) perform under the proposed evaluation setup, both in quantitative metrics and in qualitative analyses?\n- According to the manuscript, superposition occurs when mutual exclusiveness holds, and the proposed method aims to identify subspaces of mutually exclusive features. However, it is not clear why we should expect these subspaces to be inherently interpretable if superposition still occurs within them. Wouldn’t we, in many cases, need an additional method to disentangle or remove the remaining superposition? Furthermore, how can we identify situations in which such additional disentanglement would be necessary?\n\nDue to the open questions regarding the evaluation and the distinction from sparse autoencoders (SAEs), I would currently lean toward rejecting the paper. However, I believe the work has potential, and if my concerns are addressed convincingly and no major issues are raised by other reviewers, I would be happy to reconsider and improve my score.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- I really appreciate the core idea and intuition behind the argument that mutual exclusiveness may be a fundamental condition for superposition, as well as the proposed method built on this insight.  \n- The results on the toy model effectively illustrate and complement the central narrative.  \n- The paper does a good job explaining the intuition behind the approach and is, for the most part, well written and thorough in its exposition.  \n- Overall, I found this to be one of the more enjoyable papers to read, with a clear take-home message.\n- While the evaluation still leaves considerable room for improvement, the paper does at least include some empirical assessment of the proposed approach.", "weaknesses": "- I did not find the qualitative examples in Figure 2 particularly convincing. The positions shown in panel (c) still exhibit a fairly large range, and for the other examples, it seems plausible that sparse autoencoders could identify similar concepts. Given the narrative developed in the paper, I would have expected the qualitative examples to focus more on illustrating the idea of distinct variables, rather than on specific concepts that might also be captured by SAEs.\n- In Table 1, it is unclear why no comparison to sparse autoencoders (SAEs) is included. A theoretical justification would be sufficient if there is a solid reason for omitting such a comparison. Additionally, it would be helpful to include an analysis of how the number of subspaces affects the results. For instance, if only a single subspace were used, the condition that the high-level \"variable\" lies within the same subspace (L. 261) would trivially hold.\n- The method relies on the mutual information (MI) threshold, but it is not clear how this threshold should be selected in practice. Although the paper includes ablation studies on this parameter, the specific effect of the threshold remains unclear, as the quantitative experiments are relatively small in scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ohid3Sl0KI", "reviewer": "Reviewer_bKFF", "summary": "This paper introduces Neighbor Distance Minimization (NDM) as a method for performing unsupervised identification of interpretable subspaces within the latent spaces of neural models. NDM operates under the hypothesis that groups of mutually exclusive features form subspaces, and the features within these subspaces exist within superposition. Through experiments on GPT-2, the authors demonstrate that these subspaces behave similarly to the “variables” used by the model. This approach provides a new avenue for interpretability research within and between subspaces.", "questions": "1. What are the computational requirements for applying NDM?\n2. How sensitive is NDM to the provided N model activations?\n3. Have you explored an iterative application of NDM? Namely, re-applying NDM to the identified subspaces?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Motivation and Construction: The motivation of NDM and its subsequent demonstration on the toy models of superposition setting provides a clear intuition as to why we’d expect it to work on larger models and how we can interpret its results.\n2. Novel Perspective: The developed perspective offers an alternative avenue for future research in the field of interpretability, specifically by considering the interaction between or within subspaces.", "weaknesses": "1. Lack of Details on the Computational Aspect of NDM: There is a lack of information regarding the computational burden imposed by using NDM. Indeed, 2B parameter models are not large-scale by current standards. This raises questions about the practicality of these methods in real-world settings.\n2. Single Domain Evaluation: NDM is only evaluated on text models. It would also be important to evaluate the effectiveness of exploring image models.\n3. Comparison to other Interpretability techniques: Although the intuitive differences between NDM and other interpretability methods, such as sparse autoencoders, are provided, no practical comparison is made.\n4. Limited Granularity: By construction, NDM can only identify as many subspaces as there are dimensions in the latent space. In fact, it is much less than that, as the subspaces often have relatively large dimensions. Therefore, it is unclear how granular these “variables” corresponding to subspaces are, and it is likely that additional interpretability techniques would have to be applied to the individual subspaces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)