Addressing divergent representations from causal interventions on neural networks
Authors: Satchel Grant
Simon Jerome Han
Alexa R. Tartaglini
Christopher Potts
Venue: ICLR 2026 Oral (2026)
Keywords: activation patching
mech interp
DAS
representational divergence
faithfulness
Primary area: interpretability and explainable AI
TL;DR: We show empirical representational divergence between native and causally intervened latent states, we show that this can be pernicious and propose a solution.

Abstract: A common approach to mechanistic interpretability is to causally manipulate model representations via targeted interventions in order to understand what those representations encode. Here we ask whether such interventions create out-of-distribution (divergent) representations, and whether this raises concerns about how faithful their resulting explanations are to the target model in its natural state. First, we demonstrate theoretically and empirically that common causal intervention techniques often do shift internal representations away from the natural distribution of the target model. Then, we provide a theoretical analysis of two cases of such divergences: "harmless" divergences that occur in the behavioral null-space of the layer(s) of interest, and "pernicious" divergences that activate hidden network pathways and cause dormant behavioral changes. Finally, in an effort to mitigate the pernicious cases, we apply and modify the Counterfactual Latent (CL) loss from Grant (2025) allowing representations from causal interventions to remain closer to the natural distribution, reducing the likelihood of harmful divergences while preserving the interpretive power of the interventions. Together, these results highlight a path towards more reliable interpretability methods.

Published as a conference paper at ICLR 2026
ADDRESSING DIVERGENT REPRESENTATIONS FROM CAUSAL INTERVENTIONS ON NEURAL NETWORKS
Satchel Grant
Simon Jerome Han Stanford University
Alexa R. Tartaglini Stanford University
Christopher Potts Stanford University
Stanford University
ABSTRACT
A common approach to mechanistic interpretability is to causally manipulate model representations via targeted interventions in order to understand what those representations encode. Here we ask whether such interventions create out-of- distribution (divergent) representations, and whether this raises concerns about how faithful their resulting explanations are to the target model in its natural state. First, we demonstrate theoretically and empirically that common causal intervention techniques often do shift internal representations away from the natural distribution of the target model. Then, we provide a theoretical analysis of cases of such divergences: 'harmless' divergences that occur in the behavioral null-space the layer(s) of interest, and 'pernicious' divergences that activate hidden network pathways and cause dormant behavioral changes. Finally, in an effort to mitigate the pernicious cases, we apply and modify the Counterfactual Latent (CL) loss from Grant (2025) allowing representations from causal interventions to remain closer to the natural distribution, reducing the likelihood of harmful divergences while preserving the interpretive power of the interventions. Together, these results highlight a path towards more reliable interpretability methods.
INTRODUCTION
central goal of mechanistic interpretability is to understand what the internal representations of neural networks (NNs) encode and how this gives rise to their behavior. Perhaps the most powerful approach to pursuing this goal is through causal interventions, where methods such as activation patching and Distributed Alignment Search (DAS) directly manipulate internal representations to test how they affect outputs (Geiger et al., 2021; 2023; Wu et al., 2023; Wang et al., 2022; Meng al., 2023; Nanda, 2022; Csordás et al., 2024). Indeed, even correlational methods such Sparse Autoencoders (SAFs) and Principal Component Analysis (PCA) often use causal interventions as a final judge for whether the features they identify are truly meaningful (Huang al., 2024; Dai et al., 2024). Causal interventions thus occupy a central place in making functional claims about neural circuitry (Pearl, 2010; Geiger et al., 2024; 2025; Lampinen et al., 2025; Braun et al., 2025).
The use of causal interventions rests on a fundamental assumption that counterfactual model states created by interventions are realistic for the target model. Despite its pervasiveness, however, this assumption is often untested. For example, some activation patching experiments multiply feature values by up to 15x (Lindsey et al., 2025); these settings, it seems possible that intervened representations diverge significantly from the NN's natural distributions. This raises questions about the reliability of causal interventions for mechanistic interpretability. Do divergent representations change what intervention can say about an NN's natural mechanisms? When, and to what extent, is it okay for such divergences to occur? When it is not okay, how can we prevent them from occurring?
In this work, provide both empirical and theoretical insight these issues. We first demonstrate that divergent representations are a common issue for causal interventions - across a wide range of intervention methods, find that intervened representations often do diverge from the target NN's natural distribution. We then provide theoretical examples of two types of divergence: 'harmless' divergences that can occur from paaiin-deeision-buunaar covariance along causal dimensions or from deviations in the null-spaces of the NN layers, and 'pernicious' divergences that activate hidden network pathways and can cause dormant changes to behavior. We provide discussion on
I Link to public repository: https: //g ht////ttuuu thub. wwt....om/gaaatttbbb/oooo vergenceeeeeeeee

---

Published as a conference paper at ICLR 2026
A
B
Harmless
Input
Hidden
Hidden
Output
Causal Dim
Potched Representation
affirming evidence
Pernicious
refuting
Caussal Dim
f
PC1
PC2
Causal Dim
Natuzd
Naturd
Interpened
Intervened
Natura
Intervased
Hodden
Dorment
puth
20th
path
path
Resresontatio
Represemation
Crout
Benavior
Region
Reg on
Figure 1: Causal interventions can recruit hidden circuits that produce misleadingly confir- matory or dormant behavior. (a) Consider natural pathways (dashed arrows) for two classes A and B that carry activity to different behavioral outputs y. In hypothetical intervention meant find path A, patching h' with divergent representation can activate distinct, hidden pathways (solid arrows) that result in misleadingly confirmatory behavior (orange) and/or undetected behavior (red). (b) Consider 2D projections of the neural activity of ht for different network that classifies states into one of 10 classes (denoted by huc). Suppose that natural representations (dark points) lie within well-defined decision boundaries (dashed lines) and covary along causal axes, and that intervened representations (light points) are constructed by patching the first axis from sampled natural representation. Although these representations diverge from the natural distribution, this can be harmless (top) pernicious (bottom) depending the network's functional landscape. In particular, it can be pernicious if the network has a functional landscape where intervened activity unknowingly recruits hidden circuits (visualized an orange region) crosses dormant behavioral boundaries (red regions), depending on the claims.
how harmless and pernicious cases are not always mutually exclusive, where the harm depends on the specific mechanistic claims. Finally, we provide broad-stroke, initial solution for mitigating pernicious divergences by minimizing all intervened divergences. We show that can use the Counterfactual Latent (CL) auxiliary loss introduced in Grant (2025) to reduce all representational divergence in the Boundless DAS setting from Wu (2023) while maintaining the same behavioral accuracy; and we introduce modified version of the CL loss that targets causal subspaces and show that it can improve out-of-distribun do not propose this method as the final solution to representational divergence, pose it step towards more reliable interventions.
(OOD) intervention performance on synthetic tasks. Although
We summarize contributions follows:
We show theoretical and empirical examples of divergence between natural and causally inter- vened representations for variety of causal methods (Section 3).
2. We provide a theoretical treatment of cases in which divergence can arise innocuously from variation in null-spaces, demonstrating that some divergences even be desired (Section 4.1)
3. We provide synthetic examples of cases where divergent representations can (1) activate hidden computational pathways while still resulting in hypothesis-affrmmi behavior, and (2) cause dormant behavioral changes, together raising questions about the mechanistic claims that can be made from patching results alone (Section 4.2).
4. Lastly, we use the CL. auxiliary loss from Grant (2025) to minimize patching divergence di- rectly the 7B Large Language Model (LLM) Boundless DAS experimental setting from Wu et al. (2023), and introduce stand-alone, modified CL loss that exclusively minimizes divergence along causal dimensions, improving OOD intervention performance in synthetic settings. Together, these results provide initial step towards mitigating pernicious divergences (Section 5).
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND AND RELATED WORK
2.1 ACTIVATION PATCHING
Activation patching generally refers to a process of "patching" (i.e. substituting) some portion of neural activity at intermediate layer into or from corrupted forward pass of network (Geiger et al., 2020; Vig ct al., 2020; Wang et al., 2022: Meng et al., 2023; Zhang & Nanda, 2024). It can be performed at various granularities such as whole layers, attention heads, individual neurons. Many forms of activation patching can be unified under the assumption that subspaces, rather than individual neurons, are the atomic units of NN representations (Rumelhart et al., 1986; McClelland et al., 1986; Smolensky, 1988; Elhage et al., 2022; Geiger et al., 2021; Grant et al., 2024). Activation patching the level of individual neurons axes, and many of its high-level granularities can be understood as specific forms of individual neuron patching (Geiger et al., 2020; Vig al., 2020; Wang et al., 2022; Meng et al., 2023).
be understood as subspace patching along neuronal
2.2 DISTRIBUTED ALIGNMENT SEARCHI
Distributed Alignment Search (DAS) (Geiger et al., 2021; 2023; Wu et al., 2023) can be understood as a form of activation patching that operates in a transformed basis so that specific, causally relevant subspaces be manipulated analogously to high-level variables from causal abstractions (CAs) (e.g. symbolic programs), Many cases of individual neuron (coordinate) patching can be understood as specific cases of DAS that use the identity transform. We use DAS in Sections and 5, so here we introduce its theory and background.
DAS finds alignments between neural representational subspaces and causal variables from Causal Abstractions (CAs) by testing the hypothesis that an NN's latent state vector h E Rdm can be transformed into a vector 2 E R that consists of orthogonal subspaces encoding interpretable variables. This transformation is performed by learnable, invertible Alignment Function (AF), A(h). We restrict our considerations to linear AFs of the form A(h) Wh where W E Rdmm Xtm invertible. This transformation allows to formulate h in terms of interpretable variables and to manipulate encoded values.
For given CA with variables var, E {vari, var2, ... var, J. DAS tests the hypothesis that is composed of subspaces Zvar, E Rd corresponding to each of the variables from the CA. A causally irrelevant subspace Textm Rdam also included to encode extraneous, functionally irrelevant neural activity (i.e., the behavioral null-space).
Eyari
A(h)
(I)
Zvat.
[
Here, each Ævar, E Rdu,; is a column vector of potentially different lengths, where dvar, the subspace size of var,, and all subspace sizes satisfy dextra >E= dwar; dm- The value of single causal variable encoded in h can be manipulated through an interchange intervention defined as follows:
h= A- ((I -var, A(())) + D A(ttt))
Here, Dvar E Rdd Xdm is a manually defined, block diagonal, binary matrix that detines the subspace size dyar, and Xdd.. is the identity matrix. Each Dvar, has set of dear, contiguous ones along its diagonal to isolate the dimensions that make up aar,, se is the source vector from which the subspace activity is harvested, h is the target vector into which the harvested activity is patched, and h the resulting intervened vector that replaces "8 in the model's processing. This allows the model to make predictions using a new value of variable var,.
To train A, DAS uses counterfactual behavior c ~ D as training labels, where e generated from the CA. for given state of a CA and its context, is the behavior that would have occurred had a causal variable taken a different value and everything else remained the same. c is generated by freezing the state of the environment, changing one more variable values in the CA, and using the CA to generate new behavior in the same environment using the new values. We train A on intervention samples while keeping the model parameters frozen, minimizing the following objective
3

---

Published as a conference paper at ICLR 2026
(for non-sequence-basee settings)::
(AA 1 N N
Σ A=1
log PA ((k)) [ h(0))
where N is the number of samples in the dataset, ( is the counterfactual label sample k axk) is the model input data, and p.((- I) is the model's conditional probability distribution given the intervened latent vector, h. We minimize CDAS(AA (AA using gradient descent, backpropagating into A with all model weights frozen. A is evaluated on new intervention data, where the model's s accuracy on following each intervention is referred to the Interchange Intervention Accuracy (IIA).
2.3 PROBLEMATIC CAUSAL INTERVENTIONS
Prior work has implicitly explored issues related to representational divergence from causal inter- ventions. For example, methods such as causal scrubbing or noising/denoising activation patching (Wang et al., 2022; LawrenceC et al., 2022; Meng et al., 2023; Chen et al., 2025; Zhang & Nanda, 2024) intentionally introduce divergent representations to test the sufficiency, completeness, and faithfulness of proposed circuits. Works such as Wattenberg & Viégas (2024), Méloux et al. (2024), and Chen et al. (2025) also implicitly explore the dangers of divergent intervened representations by showing how circuits and features can be redundant have combinatorial effects that are difficult to enumerate given current methodologies, while Zhang & Nanda (2024) and Heimersheim & Nanda (2024) point out easy misinterpretations of patching results. Shi et al. (2024) and Wang et al. (2022) provide criteria centered on faithfulness, completeness, and minimality for evaluating circuits through causal interventions. explored differences between on- on-manifold and off- aff-manifold adversarial attacks (Stutz et al., 2019), and some works have explored constraining counterfactual features to the manifold of the dataset (Verma et al., 2024; Tsiourvas et al., 2024). Our proposed method in Section 5 differs in that it trains a principled alignment to generate counterfactual representations and constrains deviations along causal dimensions. For DAS in particular, Makelov et al. (2023) demonstrate that it is possible to produce an interaction between the null-space and dormant subspaces that affect behavior. Because they define dormant subspaces as those that do not vary across diffeerent model inputs, variation along these directions is, by definition, a form of divergent representation. Finally, Sutter et al. (2025) posit that it is possible to align any causal abstraction to NNs under number of assumptions including sufficiently powered, non-linear alignment function, raising questions about what non-linear causal interventions really tell us.
body of work on counterfactual explanations exists, some of which has
ARE DIVERGENT REPRESENTATIONS A COMMON PHENOMENON?
We begin by demonstrating that divergent representations are common (if not likely) outcome of causal interventions, both in theory and in practice. We do not yet consider its perniciousness, however, and we reserve the question of whether and when divergence is harmful for Section 4.
3.1 FOR MOST MANIFOLDS, COORDINATE PATCHING GUARANTEES DIVERGENCE
We first consider theoretical setting where coordinate- based patching of one or more vector dimensions is performed on a single manifold, similar to what might done in neuron level activation patching (Vig et al., 2020; Geiger et al., 2021). We prove that in this setting, divergent representations are guaranteed to occur if patching is performed exhaustively. For simplicity, we consider minimal version of this proof that involves a circular manifold with two dimensions. A more general proof, which applies to most manifold geometries, can be found in Appendix A.2.
Formally, let MM {eK and radius Fx > 0. Given two native representations TM = Cg + u and h = Cg + v, let us define coordinate patch (onto class K)) that keeps the first coordinate of hee and the second of h
||ulla <TK } C 2 be class-K manifold with centroid cg R²
h =
hyre
CK.1 +
h
CK2 + U2
Then the deviation from cg is
h
(u1; T2)T,
1 cxl
= U2
CK
4

---

Published as a conference paper at ICLR 2026
A
B
30-
Coordinate Patching
Mean Difference Patching
Sparse Autoencoder
Boundless
DAS
EMO
10
Vean
5AG
Natural
esentation
Reponoo tiaiion
Interveeed
END
EMD
Daowiin
Figure 2: Representational divergence is common occurrence across various interventions. (a) Directly replacing a coordinate value in one natural representation (orange) with the value from another will eventually create divergent representations (blue). (b) Top two principal components of natural and corresponding intervened representations, taken from the residual stream at the intervention position and with PCA is performed over the combined set of natural and intervened vectors, for three popular causal intervention techniques: replication of Feng & Steinhardt (2024) for mean difference patching, reconstructed vectors for a single transformer layer using SAEL (Bloom et al., 2024) for sparse autoencoder, and interchange interventions for Boundless DAS (Wu et al., 2023). (c) L2 distance between natural and corresponding intervened representations, and Earth Mover's Distance (EMD) between natural and intervened distributions (with baseline comparing the natural distribution to itself).
Proposition (coordinate patching exceeds the class radius). If he h!3 E MM (i.e., ||ule S rk and ||u||2 S rK), then the patched point à is aff--manifod whenever uf + r In particular, there exist boundary points hee n E 0MK with u = (rx, 0) and v = (0, rx) such that || cx|12- = VPA + + = rK TK-
Proof. Since h - cx = (u1, U2), we have I| cx 12 = of + vg. Choosing u (rx: 0) and v r0,rk) gives the stated violation.
As noted above, this intuition holds for all manifold shapes other than axis-aligned hyper-rectangles (see Appendix A.2). Thus, in these relatively simple theoretical intervention settings, divergent representations are guaranteed to occur with enough intervention samples.
3.2 MANY EXISTING CAUSAL METHODS EMPIRICALLY PRODUCE DIVERGENCE
Stepping beyond the theoretical setting, we also empirically demonstrate that common, real- world causal interpretabilityy methods often produce divergent representations. The notion that these methods can create divergence has been indirectly explored in previous work through attention patterns in Gaussian noise corrupted activation patching (Zhang & Nanda, 2024). Here we extend this view to three popular causal intervention methods: mean difference vector patching (e.g. Feng & Steinhardt (2024)), Sparse Autoencoders (c.g. Bloom et al. (2024)), and DAS (c.g. Wu et al. (2023)).
Figure shows the top two principal components of the natural and intervened representations for each intervention method, distinguished by color. These results demonstrate that divergence is a common phenomenon in practice and is not specific to any one method. Even simple methods that patch along a single mean direction are subject to divergence, despite high behavioral accuracy (Feng & Steinhardt, 2024; Geiger al., 2023; Wu et al., 2023). We quantify this divergence in Figure 2(c) using the Earth Mover's Distance (EMD) ( (Villani, 2009) between the full dimensional natural and intervened distributions, using the corresponding natural-
baseline. We see that the intervened divergence exceeds that of the natural (more metrics and details are in Appendix A. Note that this result does not necessarily imply that the respective methods are invalid or that their claims are incorrect; the panels are only meant to show the presence of divergence.
comparison
4 WHEN ARE DIVERGENT REPRESENTATIONS HARMLESS OR PERNICIOUS?
Having demonstrated that divergent representations are a common phenomenon, we now consider whether divergence is concerning phenomenon. We propose that divergence is harmless many
5

---

distribution, but the behavior will remain unchanged because Zvan, is behaviorally binary and its sign Published as a conference paper at ICLR 2026
functional claims if it exists in the behavioral null null-space, and that it can be pernicious if it recruits hidden pathways or causes dormant behavioral changes. However, we stress that the harm is inherently claim dependent, meaning that these forms of divergence are not mutually exclusive.
4.1 HARMLESS CASES
Here we explore a set of cases for which we might consider divergent representations to be harmless to functional claims. First among these are cases in which divergence is bottle-necked into the null-spaces of the next interacting weight matrices. Formally, we define the null-space of a weight matrix W E Ed' xd as N(W)= {o E R We 0}. Neural activity in the null-space of the weights refers to any changes E R for which (h + 5) = Wh. We propose that divergence E (WW) is harmless to the computation of W because it is equivalent to adding the zero vector W(h + v) = W4+ + We = Wh + = ((h + 0). Notably, however, this harmlessness does not apply to the sub-computations of the matrix multiplication because (W) does not imply that Why + ey) = W (h, + 0) for vector row j and matrix row i. Thus, e in this case is potentially harmful to mechanistic claims about individual aetivation-weiglte sub-computations, while being harmless to the overall matrix multiplication.
L We can generalize this notion beyond matrices to an arbitrary function ub. Let s/ Rd + Rd and let X R We define the behavioral null-space with respect to X as
W x) = {u E Rd Vor E X,
(x))
common case of v in practice for a layer & an NN f consists of all subsequent computations after and including layer l, denoted (h) We propose that behaviorally null divergence is harmless to the overall computation of far because it is equivalent to adding 0 to the input. However, v can be v)) is not guaranteed to be equal to fe+& (f24, (h + 0)) (Sec. 4.2.1). See Appx. A.3 and Algorithm to practically classify harmlessness when N(v, X) characterizes the full space of harmless divergence.
harmful to claims about sublayer e+ k within fze because (f+
Idealized Case Study. We now present an example of harmless divergence in the behavioral null- space by introducing behaviorally binary subspace-a vector subspace which causally impacts the outputs of a future processing layer (e.g., classification labels) only through its sign. Formally,
let f _d + IRd denote a computational unit (possibly consisting of multiple NN layers and functions). Let sign(-) denote the elementwise sign map sign Rd {-1, 1,dd,, and assume fixed alignment function A and subspace selection matrix D E R (Sec. 2.2). A linear subspace Z S Rd is behaviorally binary (with respect to f. Dvan. and A) iff for all A(h), A(r') E Z,
sign A(h)) = sign( D A(h')) =
= s(h')
(5)
Now, suppose we have an NN with two causal subspaces, Zvar, S. Rdda and Zvans S CR with
values =(xx) and s(xx) for model input 24, where the bold, tilde notation distinguishes variables from their values. Further assume that Zvan, is behaviorally binary and co-varies with, Zvar. Using
h(())) and (xx) from Equation under a given input Hy, we use the following definition:
(xx) =(2.)
A(h(to)) =0())
Zaaar Zvar. Zvan
(6)
Zvars
Due to the assumption of covariance in Zvar, and Zvarg it is reasonable to assume that the values =(Zug) systematically distinct for distinct values of Zvar, under some classes of inputs,
((now)) and Evara
Zvars
Now and Thigh. while sign(zuar, =((iie)))) = sign( ( Now, an interchange intervention on žvan, using
source vector from Flow and target vector from Thigh, will produce:
$
Zvaar
Zvara
Zvaro
Zvars
Because we assumed that the value of Zvare is systematically unique due to covariance in Zvara s=(ama))
(((ti))) in Equation will have never existed together in the native and Zvan,, the values Zvar, and Zvars
has not changed. This divergence is thus harmless to the claim of discovering f's causal axes.

---

Published as a conference paper at ICLR 2026
Summary. We propose that divergences within behavioral null-spaces harmless to many func- tional claims about a function s computations when the claim encapsulates (i.e. ignores) the internal, sub-computations of v. However, encompasses the set of all harmless divergences. In general, such an exhaustive set cannot exist without assuming the superiority of some scientific claims/assumptions over others. For example, take the set of all harmless divergences for specific claim, then modify the claim to assume it permissible diverge in a manner previously excluded from the harmless set. The modification to the claim also modifies the harmless divergences. Lastly, we note that behaviorally null divergence not always harmless. Indeed, one could even desire to intervene on the behavioral null-space to causally test that it is null. Thus, we stress that the mechanistic claim for which an intervention is meant to support is important for determining the harmlessness of the divergence.
not suggesting that the behavioral null-space
4.2 PERNICIOUS DIVERGENCE VIA OFF-MANIFOLD ACTIVATION
We now explore pernicious cases of representational divergence involving the concept of hidden pathways, which refer to any unit, vector direction. subcircuit that is inactive on the natural support of representations for given context but becomes active and influences behavior under an intervention. Although hidden pathways can be compatible with harmless divergences (Sec. 4. they can also undermine claims about natural mechanisms and can prime dormant behavioral changes discussed in Sec. 4.2.2.
Formally, let D denote the data distribution over latent representations h' E Rd at layer and let S supp(h ~ DD denote its support with Sx = supp(h ~ D) the support for class K. Denoting the intended class K following an intervention as subscripted + K, an intervened representation K. is said to be divergent if バ_ & SK (e.g. it exists off the natural manifold of class K). We define the convex hull of class-K K representations as conv(Sk) = EE ag a; 2 0, Σ, a; = where the subscript K denotes that hf, was taken from class K inputs. Projecting intervention onto cSnv(Sk) ensures that it remains within the convex interpolation region of class-K.
4.2.1 MEAN-DIFFERENCE PATCHING CAN ACTIVATE HIDDEN PATHWAYS
Patching with a mnean-differenee vector can flip decision by activating a unit that is silent for all natural class inputs.
Setup. Consider two-layer circuit with ReLU nonlinearity. Let ht E I feed into
1 1 RRL.UU.. be),
We R be E R3,
where
T0.75
0.25
0
0.57
0.5 b 0.5
W
0
1
0
1
1
positive score (s > 0) indicates class Suppose class-A and class-B representations at layer & are
[1, 0, 1
T
(case 1) (case 2)
00,0,11 1, 1,
(case 1)
h/
h
[0, 1,
10
(case 2)
Evaluating h yields
h ++ Awsey
10.25,0,0] h "Aae
[0, 0,,,0.
so for class-A, (0.25, 0.5}. For class-B, all outputs are zero, 8B = 0.
Mcan-difference patching. We construct mean difference vector between the classes,
2
8B = MA - MB
Σ
(hhmmn)) 2 i=1
(K = [0.5, 0.5, 0, -0.5]
1=1
Applying this to class-B representations gives
[0.5, 0.5, -0.5
(case) 1) (case 2)
h 1 h/B 1
8
(0.5, 0.5, 1, 0.5]
7

---

Published as a conference paper at ICLR 2026
After propagation through the circuit:
{ [0, 0, 0.5]
Scases 0.5
h +++
B-AA
[0.25, 0, 0J., Seascz 0.25
The intervention flips the decision to class-A (& > 0). However, the third hidden unit becomes active only for A. never for natural h. This new activation is a hidden pathway that was silent under all native samples. Thus the mean- deaa-differeece patch crosses the decision boundary only by activating an offf-aaifood circuit.
we project A onto conv(SA) (or equivalently onto a local PCA subspace of SA), this ReLU state change disappears, and the decision boundary no longer crossed confirming that the original effect was driven by divergence rather than within-manifold causal mechanism. It is unclear what this patching experiment reveals about the natural mechanisms of the model.
4.2.2 DORMANT BEHAVIORAL CHANGES
Divergent representations can also yield dormant behavioral changes: perturbations that appear behaviorally null in one subset of contexts but alter behavior in others. Formally, let s Rd xC -> Rd and let C1 C C subset of contexts. The space dormant behavioral changes relative to X, C1C is V(e, (1,,,,
V(,,X,C, (W(,,,,C),
Illustration. Extend the network from the previous Sec. 4.2.1 by adding one row of zeros to W and be. producing h IR where the final coordinate is always zero. Add context vector E R2 and final affine layer:
1
0.5
ý We+ (1++1 + v) + be+1:
We+1
0
0
0
0
be+1
0.25
[
0
1
1
[-1]
Here, the first argmax index ÿ [ū), f/2, 073]
corresponds to class predictions A, B, and C.
Assume 2 = [0, 0, O, vnl v. For f++1
Ban, +A = [0, 0, 0.5.
όI 0.25,
1/2 0.25
1/3 (0.5 + 04) 1 0.5.
The model predicts class-A when 24 < 0.75 but switches to class-C when 14 > 0.75. Notably, is would not naturally cause class-C prediction below value of 1 due to the bias threshold. Thus the same intervention that was benign in one context (v4 < 0.75) produces a behavioral flip another (1 > U4 0.75) purely due to the latent divergence priming new pathway.
Implications. Dormant behavioral changes highlight that behaviorally "safe" interventions can still introduce hidden context dependencies. Detecting them would require evaluating across all possible contexts, which is infeasible in practice. Therefore, causal intervention based experiments should ideally (1) report any introduced representational divergence outside of the null-space and (2) test causal interventions for context-sensitivee
Summary. Hidden pathways arise when off-manifold representations activate computations that never oceur for natural representations. Such pathways can potentially alter causal conclusions even when behavior appears unchanged. Manifold-preserviin projections and ReLU-patten audits are potentially practical safeguards against these pernicious forms of divergence.
5
HOW MIGHT WE AVOID DIVERGENT REPRESENTATIONS?
We have thus far shown that divergent representations are common and that their harm depends on multiple factors. We now consider the question of how such divergences might be avoided. Some existing methods solve this by projecting counterfactual features (in our case, intervened representations) directly to the natural manifold (Verma et al., 2024). However, we seek a method that generates principled interventions that are constrained to be innocuous. this pursuit, we first apply the Counterfactual Latent (CL) loss from Grant (2025) to the DAS experiments from Wu et al. (2023) and find that we can reduce intervened divergence while preserving accuracy on a Llama based LLM. We then introduce modified CL loss that only targets causal dimensions, and show that it can improve OOD intervention accuracy. We emphasize, however, that minimizing divergence magnitude does not guarantee elimination of hidden pathways; it only reduces the risk surface.
8

---

Published as a conference paper at ICLR 2026
5.1 APPLYING THE COUNTERFACTUAL LATENT LOSS TO BOUNDLESS DAS
To encourage intervened NN representations to be more similar to the native distribution, we first apply the CL auxiliary loss from Grant (2025) to the Boundless DAS setting in Wu et al. (2023). This auxiliary training objective relies on counterfactual latent (CL) vectors as vector objectives. CL vectors defined vectors that encode the same causal variable value(s) that would exist in successful. We can obtain CL
the intervened vector, h, assuming the interchange intervention vectors from sets of natural vectors from situationn and behaviors that are consistent with the values of the CA which are aligning. See Figure for visualization. As example, assume that we have a CA with causal variables vara, varw, and var and following a causal intervention we expect h to have a value of u for variable var, and to for variable var... A CL vector lizz for this example can be obtained by averaging over of m natural representations, hcz. m Σ MCz
(#:)
(zt. has the same variable values: var, and var = w (as labeled by the CA). where each Cz
The CL. auxiliary loss Lci. introduced in Grant (2025) is composed of the mean of an L.2 and a cosine distance objective using CL vectors as labels. Using notation defined in Section 2.2, LCL. for single training sample is defined as follows:
LcL Lai (chh))))== ) 2 heull2
1
h hea
2 11illa ||AczIl2
CCL is combined with the DAS behavioral loss Coss into single loss term using a tunable hyperparameter: Ltotal = eCct. + LDAS- The loss is computed as the mean over batches of samples and optimized using gradient descent (Appendix A.5).
Results: We applied the CL loss to the Boundless DAS notebook from Wu et al. (2024) which reproduces the main result from Wu et al. (2023) (see Appendix A.4). Figure 3A provides qualitative visualizations of the decreased divergence when applying the CL loss, and Figure 3B shows both IIA and EMD function of increasing the CL weight e. For small values of e. the IIA is maintained (and potentially even improved) while EMD decreases, demonstrating that the CL auxiliary loss can directly reduce divergence in practical interpretability settings without sacrificing the interpretability method.
5.2 MODIFIED CL LOSS IMPROVES OOD INTERVENTIONS IN SYNTIETIIC MODEL SETTINGS
We next modify the CL. loss to work independently of the DAS behavioral loss and show that improves OOD intervention performance. We simulate intermediate layer of an NN by constructing a synthetic dataset of f vectors with known feature dimensions and labels that we use to train Multi-Layer Perceptron (MLP). The dataset consists of noisy samples around set of grid points from two feature dimensions with correlational structure. Specifically, we define set of features as the Cartesian product of two values along the x1 -aiaxis {-1, 1} and five values along the azisxis {0, 2,2,3, 3, 4}. resulting in ten unique coordinates that each correspond to one of ten classes. We add noise and covariance to these feature dimensions and concatenate n extra noise dimensions, resulting in simulated vectors h E 2R2+ where n = 16 unless otherwise stated. The feature dimensions of these vectors are shown as the natural distributions in Figures 1(b) and 3. We then train small MLP on these representations predict the class labels using standard cross entropy loss. After training, we perform DAS analyses with either the behavioral loss or the CL loss independently. See Appendix A.5 for further details dataset construction, MLP training, and DAS training.
We modify the CL loss by applying it to individual causal subspaces only (as discovered through the DAS training). This allows to construct Vi vectors specific to single causal variable var;, The modified f for a single training sample is defined as follows:
var. (D A(h)),
h stopgrad
(A (vrce))())))
(9)
( 1h
h yar, h
ca
=
AAa CL
var;
(10)
2 | vare 1
vars
[|2
Results: Figures 3D and 3E provide a qualitative comparison of intervened and native representations for interventions using trained DAS rotation matrix. Each dot in the figures shows the values of the

---

Published as a conference paper at ICLR 2026
A
<
B 10
Source Representation
E
10
Prenscorded Natural
DAS
Representations
-1, x
Target Represontation
DAS
Cross Tack IM
S 0.5.
0.8
xi= -1 =
10
10'
-1,1,,,-
k=-1,,,
PC
Weight
Dmm
Patched Reprasentation
MSE
Counterfactual
FC1
Representation
Katural
Dim DAS cL
Figure 3: The CL loss reduces representational divergence and can improve out-of-distribution generalization. (a) PCA of natural (orange) and intervened (blue) representations in the Boundless DAS setting presented in Wu et al. (2023) for two CL loss weightings with the same final IIA. (b) IIA (orange) and divergence (purple) of intervened representations from Section 5. as a function of CL loss weight (c). (c) Diagram of CL loss; rectangles
model representations and æ1 and D2 are deterministic values of the representations along the two synthetic causal dimensions shown in panels (d) and (e). We patch the æ2 value from source target using DAS and define the CL representation as the average of all natural representations that possess the same variable values as the post-intervention representation. (d) and (e) two causal feature dimensions of representations from a synthetic dataset consisting of ten classes (colors), with both natural (dark) and intervened (light) representations shown. (d) shows results from DAS trained using behavior only, (e) shows DAS trained using only the CL loss. (f) performance of aligmment matrices trained on one task and evaluated on another that uses the same causal dimensions. CL loss leads higher OOD performance.
feature dimensions for a single representation. The native states are displayed in darker colors and the intervened in lighter. Each huc indicates the ground truth class of the state. We the intervened representations using the CL loss. Quantitatively, the DAS loss produces EMD values along the feature dimensions of 0.032 + 0.003 whereas the CL loss produces 0.007 = 0.001 with IIAs of 0.997 + 0.001 and 0.9988 + 0.0005 respectively on training/test sets with held-out classes.
tightening of
What is the practical utility of reducing divergence? We hypothesized that divergence could influence IIA when transferring the DAS alignment to OOD settings. To test this, we partitioned the synthetic task into a dense and a sparse cluster of classes based on their relative spacing (Appx. A.5.4). We then trained an MLP and alignment on each partition and evaluated the alignment on the held-out partition. The CL. loss performed better than the behavioral loss in these OOD) settings (Figure 3F). We then regressed OOD IIA on training EMD to find an anti-correlatio (coef. -.34, R² = -733, F(1,28)=75.28, p < .001), showing that divergence can predict lower OOD performance (Appx. A.6).
DISCUSSION AND LIMITATIONS
In this work demonstrated that variety of common causal interventions can produce representa- tions that diverge from target model's 's natural distribution of latent representations. We then showed that although this can have benign effects for some causal claims, it can also activate hidden pathways and trigger dormant behaviors that can perniciously affect other claims. As a step towards mitigating this issue, we provided broad-stroke solution by directly minimizing the divergence of intervened activity along causal dimensions, mitigating both pernicious and harmless forms of divergence.
remaining gap our work is the failure to produce a principled method for classifying harmful divergence for any claim. Additionally, the modified CL. loss is confined to a narrow set of simplistic settings and is not specific to pernicious divergence. We look forward to exploring ways to classify and mitigate pernicious divergence through self-supervised means in future work.
Where does this leave us with respect to causal interventions in mechanistic interpretability? Given our theoretical findings, any divergence outside of the null-space of NN layers is potentially pernicious. This poses challenges for aspirations of a complete mechanistic understanding of NNs using existing methods alone. However, collecting sufficiently large intervention evaluation datasets, and continued development of methods such the CL loss can reduce the problem even further. We are optimistic for the future of this field.
note that many practical mechanistic projects be satisfied by
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGMENTS
Thank you to the PDP Lab and the Stanford Psychology department for funding. Thank you Noah Goodman and Jay McClelland for thoughtful feedback and discussions. Thank you to the PDP lab and the Stanford Mech Interp community for opportunities to present and for thoughtful discussion.
8 LLM USAGE STATEMENT
We used ChatGPT to generally edit for clarity, as well as to improve notational consistency in the behavioral null-space, hidden pathways. and dormant behavioral changes formalizations in Sections 4.1 and 4.2. We also used ChatGPT to provide an initial layout of the proof offered in Appendix A.2 showing that axis-aligned hyperrectangles are the only manifold shape that do not have divergent source-target vector pairs in coordinate patching settings, and we used it to suggest, implement, and generate the initial writeup for the additional divergence measures in Appendix A.1.2.
REFERENCES
Joseph Bloom, Curt Tigges, Anthony Duong, and David Chanin. Saelens. https://githin com/ jbl obloomAus//// SAELens, 2024.
Lukas Braun, Erin Grant, and Andrew M Saxe. Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks. In Forty- second International Conference on Machine Learning, 2025. URL httpt:://tpeeeeeeeeeeeettt /ppenrevie.. net/ ? YucuAuXMpT.
Hang Chen, Jiaying Zhu, Xinyu Yang, and Wenya Wang. Rethinking circuit completeness in language models: And, or, and adder gates. arXiv preprint arXivv2505..003: 2025.
Róbert Csordás, Christopher Potts, Christopher D. Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations, 2024. URL https: //wpivaaaaiivvv//a///aa/eaaaaiaa org/ abs 2408 10920.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, pp. 2292-2300 2013.
Qin Dai, Benjamin Heinzerling, and Kentaro Inui. Representational analysis of binding in language models. arXiv preprint 240::2499004444 2024.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. https:/ttrassfommer Eicruiispb//2222loymmmmd////ii
model/index.html
Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context?, 2024. URL
https: /tt///axxii arxiv. www.wiv..rr/..ooo alss /2310. 17191.
Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models
partially embed theories of lexical entailment and negation. arXiv preprint wxXiv2204.11662: 2020.
Atticus Geiger, Hanson Lu, Thomas Ieard, and Christopher Potts. Causal abstractions of neural networks. CoRR, abs/2106.02997, 2021. URL https:://aaattt o abs //s///// 201....0066.......
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. Finding alignments between interpretable causal variables and distributed neural representations, 2023.
Atticus Geiger, Duligur lbeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. Causal abstraction: theoretical foundation for mechanistic interpretability. 2024. URL https:// axxv.. orrivvorrgaaa /200 /
11

---

Published as a conference paper at ICLR 2026
Atticus Geiger, Jacqueline Harding, and Thomas Ieard. How causal abstraction underpins computa- tional explanation. arXiv preprint rrXiv:250..11211 2025.
Satchel Grant. Model alignment search. arXiv preprint arXiv:2501.06100 2025.
Satchel Grant, Noah D. Goodman, and James L.. McClelland. Emergent symbol-like number
variables in artificial neural networks. Transactions on Machine Learning Research, 2024. URL
https: /arxiv. org/abs////oooooooosog.gsps.atrr
Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching. arXiv preprint arXiv:2404.15255, 2024.
Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. RAVEL: Evaluating interpretability methods on disentangling language model representations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting the Association for Computational Linguistics (Volume I: Long Papers), pp. 8669-8687 Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: : /aaaaaaaanntoloooooo org/2024. acl- long. 470.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. pmlr, 2015.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Andrew Kyle Lampinen, Stephanic CY Chan, Yuxuan Li, and katherine Hermann. Representation biases: will we achieve complete understanding by analyzing representations? arXiv preprint arXiv:2507.222 16, 2025.
LawrenceC, Adrià Garriga-alonso, Nicholas Goldowsky-Dill, ryan_ greenblla,,,,, jenny. Ansh Radhakrishnan, Buck, and Nate Thomas. Causal Scrubbing: method for rig- orously testing interpretability hypotheses [Redwood Research]. Less Wrong, dec 2022. URL https: http://wwllglmnmme.... agiggetnt forum For.m.org htt.g///pooooo///aaa/aaaaaaeee hwwh//cuu222uyyyyy/////// causal- cuusal-scrubae scwubbinuubb-ceeeeee ----eeooo--- for rwg.ooosloyoeeeeseeeeeeee testing..
Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Transformer Circuits Thread, 2025. URL https:/ htt//taanss/e rme r-winniiiisssts.....momss.. pub/ falb/2225// h/////ttribuuttt won.ggttttnggggggsnsasso graphs /billogolllllo//ooo//o//.ooooool html.
Aleksandar Makelov, Georg Lange, and Neel Nanda. this the subspace you are looking for? an interpretability illusion for subspace activation patching. arXiv preprint 1. 2023.
J. L. McClelland, D. E. Rumelhart, and PDP Research Group (eds.). Parallel Distributed Processing. Volume 2: Psychological and Biological Models. MIT Press, Cambridge, MA, 1986.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023. URL puttps::/arxxt org/ abs///////aa 2202. 05262.
Maxime Méloux, Silviu Maniu, François Portet, and Maxime Peyrard. Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? In Workshop on Mechanistic Interpretability. oct 2024. URL https:tpp//.o.. http:////pe/eviieeee ew. fort///rrmm/ 5 IWJRSt
Neel Nanda. Attribution patching: Activation patching at industrial scale. https: /www. neelnanda. wttp//ehaanitiii--nneeettaaaaaaa 2022.
Judea Pearl. An Introduction to Causal Inference. The International Journal of Biostatistics, 6(2).7, February 2010. ISSN 1557-4679. doi: 10.2202/1557-467.1 1203. URL https:/ http://www/////t///e///// ncbi.
mlm.nih hwt.ahhh..oo/oo/atttiiile/e////s
12

---

Published as a conference paper at ICLR 2026
D. E. Rumelhart, J. L. McClelland, and PDP Research Group (eds.). Parallel Distributed Processing. Volume 1: Foundations. MIT Press, Cambridge, MA, 1986.
Skipper Seabold and Josef Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference, 2010.
Claudia Shi, Nicolas Beltran Velez, Achille Nazaret, Carolina Zheng, Adrià Garriga-Alonso, Andrew Jesson, Maggie Makar, and David Blei. Hypothesis testing the circuit hypothesis in llms. Advances in Neural Information Processing Systems, 37:94539-94567, 2024.
Paul Smolensky. On the proper treatment of connectionism. Behavioral and Brain Sciences, 11(1): 1-23, 1988. doi:
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):199-1195, 2014. URL http:// hmrr./mmr.../ wwtg/papess/vll /v15/
srivastavale 4a htmm.
David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and generaliza- tion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6976-6987, 2019.
Denis Sutter, Julian Minder, Thomas Hofmann, and Tiago Pimentel. The non-linear representa- tion dilemma: Is causal abstraction enough for mechanistic interpretability? arXiv preprint arXiv:2405.05887, 2025. URL https: //axxiv. org/ abs/ 08802.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anm: Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Asterios Tsiourvas, Wei Sun, and Georgia Perakis. Manifold-aligned counterfactual explanations for neural networks. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li (eds.), Proceedings of The 27th International Conference Proceedings of Machine Learning Research, pp. 3763-3771. PMLR, 02-04 May 2024. URL html.
Artificial Intelligence and Statistics, volume 238 of
https: http//ppecceeenneeeeeeeeeeees.ee mlr www.mmmrrr....... press /
Sahil Verma, Varich Boonsanong, Minh Hoang, Keegan Hines, John Dickerson, and Chirag Shah. Counterfactual explanations and algorithmic recourses for machine learning: A review. ACM Comput. Surv. 56(12), October 2024. ISSN 0360-0300, doi: 10.1145/3677119 URL https:
//doi. org 10. 1ttpg//1..11//777111111
Jesse Vig. Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias. arXiv preprint arXiv:2004.12265 2020.
Cédric Villani. The wasserstein distances. In Optimal Transport: Old and New, pp. 94-105. Springer, Berlin, Heidelherg, 2009. doi: 10.10077978-3--54-71000---
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022. URL https: // /aarivvvvvv///////////aaa///av.. org/ abss/ 2211. 00593.
Martin Wattenberg and Fernanda B. Viégas. Relational Composition in Neural Networks: A Survey and Call to Action, July 2024. URL http://axxiv.org/ http:///////bb/0..111466.. McXiv:2407.14662 [cs].
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Timothée Rault, Rémi Louf, Morgan Funtowicz, Jamie Davison, Sam Shleifer, Patrick von Platen, Qi Ma, Mariama Jernite, Julien Plu, Liwen Xu, Teven Lhoest, Giancarlo Maggiore, Pietro Paul, Umar Ruwase, Jacob Devlin, Matt Tomei, Margaret U Gorman, Eric Jurgens, and Alexander M Rush. HuggingFace's ' Transformers: tf-he-of-the-ar Natural Language Processing. arXiv preprint arXiv: 191v:31110 2019.
13

---

Published as a conference paper at ICLR 2026
Zhengxuan Wu, Atticus Geiger, Thomas Ieard, Christopher Potts, and Noah Goodman. Interpretabilitty at scale: Identifying causal mechanisms in alpaca. Advances in neural information processing systems, 36:78205-788888 78226, 2023.
Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah Goodman, Christo- pher Manning, and Christopher Potts. pyvene: A library for understanding and improving PyTorch models via interventions. In Kai-Wei Chang, Annie Lee, and Nazneen Rajani (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technologies (Volume 3: System Demonstrations), pp. 158-165, Mexico City, Mexico, June 2024. Association for Computational Linguistics. URL https::/acclattho..oo wwwphhhwww.ooo..../eeeeeooo/ee/e wttgg/0220000 naacl- demo. 16.
Fred Zhang and Neel Nanda. Towards Best Practices of Activation Patching in Language Mod- els: Metries and Methods, January 2024. URL htwpthtttt/at.iitttettt 26009 16042. 1XXi:22309.16042 [cs] version: 2.
14

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 EMPIRICAL INTERVENED DIVERGENCE METHOODLOGICAL DETAILS
A. INTERVENTION METHODS
We considered three families of interpretability interventions that modify hidden-layer representations. In all three, we visualize the residual stream output from the specified layer:
1. Mean Difference Vector Patching (MDVP) (Feng & Steinhardt, 2024), where an intervention vector 5MD E _Rd is defined as the difference mean activations between two conditions and then added to or subtracted from activations h E Rd. Formally,
h = h + EMD
(11)
We examine the representations h from a sample size of 100 unique contexts across 4 token positions at each individual layer. We compare the representations to the native cases of the swapped binding positions. We used layer 10 of Meta's Meta-Llama-33-m --8-tt-ttt through Huggingface's transformers package for this task and visualization (Touvron et al., 2023; Wolf al., 2019). We selected layer 10 as it had the lowest EMD difference of all layers, although, we note that this measure did not necessarily correlate with the subjective interpretation of divergence in the qualitative visualizations. We report the EMD difference in Figure 2(a) as the average over all model layers.
2. Sparse Autoencoder (SAE) Projections (Bloom et al., 2024), where is projected through trained encoder E' _2 + I and linear decoder D : IR* + Rd:
h D(Eh)).
(12)
SAFs are trained with sparsity penalty ASAE to encourage interpretable basis functions. We offload further experimental details to the referenced SAElens paper and code base. We compare the reconstructed representations to an equal sample size of 2000 vectors from the natural distribution. We used layer 25 of Meta's 's Meta-LLama-3-88 InsIrrcttt through Huggingface's 's transformers package for this task and visualization (Touvron et al., 2023; Wolf et al., 2019). We selected layer 25 appeared to be the only layer available through SAElens' pretrained SAEs.
3. Distributed Alignment Search (DAS) (Wu et al., 2023), where representations are aligned to causal abstraction using a learned orthogonal transformation Q E Rddd See Section 2.2 and Wu et al. (2023) for further detail on the method. We compare the intervened representations to an equal sample size of 1000 vectors from the natural distribution. We used the model and layer specified in Wu et al. (2023) for the visualizations in Figure 2.
A.1.2 MEASURING DIVERGENCE
For each intervened sample, there exists a corresponding ground truth sample that the intervention is meant to approximate. the case of the mean difference experiments, these ground samples consist of the naturally occurring entity attribute in the position which the 8MD is meant to approximate. For the SAEs, each reconstructed vector has a corresponding encoded vector. For DAS, the ground truth vectors equivalent CL vectors.
To quantify distributional differences between original and intervened representations, we approximated the Earth Mover's Distance (EMD) (Villani, 2009) including all vector dimensions using the Sinkhorn loss from the GeomLoss python package with a p and blur 0.05 (Cuturi, 2013). Let H { denote a set of original representations and
Earth Mover's Distance:
H {he LV their intervened counterparts. We computed
EMD(H, Hi = min yETI(a.))
Σ
ni |1/4 hyll2.
(13)
N
where is the number of samples in H. μ and v are the empirical distributions over H and H, and M(,, v/) denotes the of couplings with marginals μ and
15

---

Published as a conference paper at ICLR 2026
Baseline divergence. To ensure that we did not introduce bias from the sampling procedure in the baseline comparison, we use the corresponding ground truth vectors for the intervened vectors when comparing to the baseline divergence. Formally, let H! be the set of ground truth natural vectors corresponding to the set H
DivergenCe EMD(H, H').
(14)
Local PCA Distance: For each reference point 24, we identified its nearest neighbors (2;,) in Euclidean space and computed the local tangent subspace via PCA. Let Ua(z,) Rxx denote the top d principal components explaining at least 95% of the local variance. The projection matrix onto this tangent space is P. U.
For a query point v, the Local PCA Distance is the orthogonal residual between and its projection onto the local tangent space at the nearest reference sample:
DPCA(U) min - - 1111 a;
Small values indicate that lies close the locally linear approximation of the manifold, whereas large residuals reflect departures orthogonal to the manifold surface.
Local Linear Reconstruction Error:
We computed an error inspired by Locally Linear Embed- ding (LLE). Given the same neighborhood (u()) of k reference points Xs {II,.... T2k. we found reconstruction weights = (Wi,..., wx) that best express v as a convex combination of its neighbors:
Σw5=1 =1
min
WjJj
s.t.
small Tikhonov regularizer A/ was added to the local covariance for numerical stability. The Local Linear Reconstruction Error is the residual norm the optimum:
DI.R (v) ||u X 1l2
This metric measures how well y can be expressed as locally linear interpolation of nearby manifold points, poor reconstruction (large DIARR) implies off manifold position or local curvature mismatch.
Kernel Density (KDE) Density Score: We estimated a nonparametric probability density function over the reference set using Gaussian kernel density estimation:
a exp
|
p(x)
DhD
212
where h is the kernel bandwidth determined by Silverman's 's rule of thumb or cross-validation. The KDE Density Score for query point is its log-density under this model,
SKDE (v) log p(v).
which inversely reflects off-manifold distance: lower log-density corresponds to less typical or out- of-distribution points. To express results comparable scale, report the negative log- log-density (i.c., -SKDE), so that larger values consistently indicate greater deviation from the manifold.
A.1.3 VISUALIZATION
We visualized both original and intervened representations by projecting onto the first two principal components of the covariance matrix of H' and H. combined:
71' H
PCs eigenvectors
)).
(15)
The top two principal components were used to plot representations in two dimensions, with colors distinguishing intervention method and condition (Figure 2).
16

---

Published as a conference paper at ICLR 2026
A.2 CLOSURE UNDER COORDINATE PATCHING CHARACTERIZES CARTESIAN PRODUCTS AND HYPERRECTANGLES
In this section, we will show that axis- hyperrectangles are the only convex manifold shape that does not have source-target vector pairs that produce off- of--fooifold intervened representations.
For vector dimensions S S [d] (1....,d} and hT 11T8 E Rd, define coordinate patch
1NT i S, h i S.
Patchs (h 113);;
where refers to the hh vector coordinate. set M C Rd is patch-closed if Patchs E M for all "m h E .M and all 5 S [d].
Let n. Rd + R be the hh coordinate projection and write 4 == T,(M) = [ha h E M).
Theorem A.2 (Patch-closure 1: (M). Then
- product of projections). Let M S IRd be nonempty and let
M is patch-closed
> M 1 x x... la-
Proof. (=) Immediate: if M II, I, then patching replaces coordinates by elements of the corresponding I,, so the result stays in M.
(=) Suppose M is patch-closed. The inclusion M S III 4: is tautological. For the reverse inclusion, fix = (41....,(d La) with 4 E 4.- For each i pick (1()) E .M with n,)) = 44- Define h == ( and for k 2 2 set (k) = Patchtay (f-)),), (())).) Patch-closure gives (k)) e M,, and by construction (k) = 1, for all j S k. Hence (000 = t E M, yielding II, 7, S.M.
Corollary (Convex case = hyperrectangle). If M is also convex, then each L, n, (M) R
is convex, hence an interval. Therefore M = II, 4 is an axis-aligned hyperrectangle (Cartesian product of intervals). Conversely, any axis-aligned hyperrectangle patch-closed (and convex).
Implication. Consequently, any nonempty convex set in Rd that is not axis-aligned hyperrectangle (e.g., a ball, ellipsoid, or a polytope with non-axis-aligned faces) fails to be patch-closed: there exist he h08 e M and S [d] such that Patchs ( ()*) & M.
A.2.1 ACTIVATION PATCHING IN BALANCED SUBSPACESS
Here include an additional example of pernicious activation patching that assumes the existence of balanced subspaces, defined as one or more behaviorally relevant subspaces that are canceled out by opposing weight values. Before continuing, in practical models due to the fact that they would only arise in cases where two rows of a weight matrix W E R are non-zero, scalar multiples of another, assuming however, could arise in cases where the input & is low rank and a subset of the columns of two rows in W are scalar multiples of one another.
note that such subspaces are unlikely exist
Wx. This example,
Consider the case where there exists an NN layer that classifies inputs based on the mean intensity of dimensions and 4 for latent vector / E R where the NN layers that produce h are denoted f(x) with data inputs z sampled from the dataset, constructed weight vector w E 1 where w' is thus defined as follows:
D, and where the layer of interest has a synthetically
[wi
W2
W3
w4
[1 -1
2
1 1 The layer
2
y = w f(x()) = w O 0- 14 (0) 1h
(16)
Here, i denotes the index of the data within the dataset. Further assume that some behavioral decision depends on the sign of y, that hn and /t2 together form balanced subspaces given w (meaning that for all ((6 D, wrh( (a) -wzh? (2) and that they are non-dormant, meaning that for some pairs (z(0) (0) where i # j. then ( * (,))) Under these assumptions, the subspace spanned by [1 0_ and [o
0j is not causally affecting the network's output under the natural distribution of h. However, if we intervene hı by while leaving hy and hg unchanged, the intervened representation å will diverge and potentially cross the decision boundary.
17

---

Published as a conference paper at ICLR 2026
Concretely, if set h [1 1 1] and h(5 13 3 -1 - and then perform
intervention on h2 using (9) as the target and h() the source, we get: h [1 1 This will result in a negative value of y, thus crossing its decision boundary using a non-native mechanism. This intervention could used as experimentally affirming evidence for a mechanistic claim, when in reality we have not addressed the model's s original mechanism.
A.3 PRACTICAL ALGORITHM FOR HARMLESS DIVERGENCE WHEN BEHAVIORAL NULL-SPACE CHARACTERIZES THE FULL HARMLESS SET
In settings where perturbations v E (e), X) are treated as harmless and perturbations (v X) as harmful, the behavioral null-space formalism suggests a practical procedure for testing the harmlessness of given divergence v. Let Xк C Rd be the of natural representations for class K, and let žg E R² be an intervened representation for class K. To approximate the natural manifold Mr locally around KK we first select the nearest neighbors of i Xx:
(²x) = (x
(n) C CK..
Let U E RR be the matrix whose rows are the neighbors u, = (T( and let
μK (0 RRd
denote their mean. Define the centered data matrix
(2) μ(x)
E Raxd
(2!!)) =x)
We compute a rank-r PCA of Ü, obtaining the top principal directions Qr Rddx (columns are orthonormal). The corresponding local projection operator is
(xx(r) μk + Q (x mk).
The local projection the intervened representation is then
proj 11 ($x),
and we define the divergence vector
Pproj-
We now provide Algorithm practical method for classifying divergence as harmless or perni- cious. We note, however, this algorithm only approximates harmlessness and is not guaranteed to be successful. This especially the case for situations prone dormant behavioral changes (Sec, 4.2.2).
A.4 CL LOSS APPLIED TO BOUNDLESS DAS
To perform the Boundless DAS experiments, used the Boundless DAS tutorial provided in the pyvene python package (Wu et al., 2024) which reproduces a main result from Wu et al. (2023), and included the CL loss a weighted auxiliary objective described in Section 5.1. The exact task used in this tutorial one involving continuous valued features, which resulted in few occurrences of valid CL vectors in the provided dataset. In order obtain exact CL vectors, we generated token sequence sample that contained a valid CL vector for each intervention sample in the dataset. We left hyperparameter choices the same across all trainings except for the CL. loss weight
A.5
CL LOSS IN SYNTHETIC SETTINGS
Here continue Section 5 with experimental details and additional experiments and results.
18

---

Published as a conference paper at ICLR 2026
Algorithm Classifying the harmlessness of divergence vector when the behavioral null-space characterizes harmlessness
Require: Intervened representation ž E R for class K; natural class representations X5 C R evaluation set exval IR; function R + R neighborhood size nı local dimension r; tolerance 2 0.
Ensure: Classification of the divergence vector e as harmless harmful.
(Local manifold estimation for class K) Let (+(x) {a(") ,..., [m)) Xx be the nearest neighbors of ž in XK.
2:
Compute the mean μk
3: Form the centered matrix E Roxd with rows (x px)
4: Perform rank-r PCA on Ù to obtain the top principal directions Q, E Rdxr,
μk + Q, (x μk) and Eproj k II ($x). 6: Define the divergence vector t IK- Eproj
5: Define the local projection 11
7: (Behavioral fest over a broader context) For each E Xeval, compute A x)) |/b(x
e) (-
8: if maxxe (xx) S then
9: return v is HARMLESS.
10: else
11:
return is HARMFUL.
12:
end if
A.5.1 SYNTHETIC DATASET CONSTRUCTION
The default synthetic task reported in the results section of Section 5.2 was constructed as a dataset of simulated intermediate-layee representations h R18 with known ground-truth labels y {1...., 10} and two causal feature dimensions, where 18comes from 2 feature dimensions plus 16 concatenated noise dimensions is the total feature dimensionality. We split these classes into partition I and 2, each consisting of 8 of the 10 classes. The held out classes for partition 1 were contained in partition 2 and visa versa. See Figure 5 for a visualization of the task.
Base feature coordinates. We first defined grid of base coordinates the Cartesian product of {-1,1) 1} along the first feature axis and {0, 1, 2, 3, 4) along the second feature axis:
G 1, {0, 3,
(17)
This procedure yields 10- = [g] unique base coordinates, each corresponding to a distinct class label.
Noise and correlation structure. For each base coordinate (x,, E G, we generated N noisy samples by adding Gaussian noise with variance 0.12 and covariance parameter 0.2. Specifically, each sample was drawn as
T
I
To.1²
0.2
(18)
F2
0.2
0.1
Additional noise dimensions. We augmented each 2D noisy base vector with 16 independent Gaussian noise features, each sampled from (om,((0, (0, 1), producing final representations h e R2+16
A.5.2 MLP TRAINING
We trained a feedforward Multi-Layer Perceptron (MLP) classifier to predict the class label y from the synthetic representations h. The MLP was parameterized with:
input dimensionality d, defaulting to d 18 as previously described a ID batch normalization of d dimensions (Ioffe Szegedy, 2015),
one hidden layer of width 128,
activation function ReLU.
dropout with probability 0.5 to drop (Srivastava al., 2014).
1D batch normalization of 1 288 dimensions,
19

---

Published as a conference paper at ICLR 2026
output layer with 10 logits.
Training was performed with a standard categorical cross-entropy loss:
LCR
log po(yi ht),
(19)
where B the batch size and pe denotes the MLP's predictive distribution over class labels.
We perform the the MLP training on both partitions combined for the default dataset split. Then we perform an alignment function training each partition independently and test the alignment on the untrained partition. We report the average IIA over both data partitions for DAS analyses over 5 seeds. Note that we use an independent MLP for each partition in the OOD experiment (Appendix A.5.4.
Optimization used stochastic gradient descent with learning rate 0.01 for 300 epochs with early stopping using an Adam optimizer (Kingma & Ba, 2017). The code was implemented in PyTorch.
A.5.3 DAS TRAINING
Following MLP pretraining, applied Distributed Alignment Search (DAS) with varying intensities of the behavioral loss and contrastive learning (CL) loss terms. Specifically, the DAS objective was
LDAS Cbebavior Lbebavior CCL CCL,
(20)
where Cbehavior and ECL are tunable coefficients controlling the strength of each term. We only use values of 0 for Ebchavior and explore values of ECL referring to as the CL epsilon in figures. We default to an overall learning rate of 0.01 and subspace size of unless otherwise specified. Details of these loss functions are provided in Section 5.
Importantly. we stopped training after loss convergence with a patience of 400 training epochs, and we kept the best DAS alignment matrix decided by IIA, and the best CL aligmment matrix by EMD. Furthermore, for these trainings, we used symmetric invertible linear weight matrix as our alignment function as introduced Grant et al. (2024). Namely, the linear alignment matrix X constructed as X (MM] + X where M E Rd Xdm. is matrix of learned parameters initially sampled from a centered gaussian distribution with a standard deviation of 7. Rd xdm. is the identity
matrix À 0.1 to prevent singular values equal to 0, and S R. xdm is diagonal matrix to learn a sign for each column X using diagonal values 8; = Tanh(a,) + X(sign (Tnh( (a))) where each a is learned parameter and À 0.1 to prevent values. We perform the the alignment function training on both partitions for each synthetic dataset and test each DAS alignment on the untrained partition. We report the average IIA over both data partitions for DAS analyses.
A.5.4 OUT-OF-DISTE DISTRIIUDITIIITIOONN EXPERIMENTAL DETAILS
To perform the OOD CL loss experiments, we partitioned the classes into 2 non-overlapping groups. Two of the 10 classes were excluded entirely. The groups were chosen so that the Sparse set had strictly greater spacing than the Dense set. See a visualization of the Dense and Sparse partitions in Figure 5. A separate MI. training was performed on each partition individually. Then an alignment function was trained on each partition/classifer tuple using the settings specified in Appendix A.5.3. The alignment functions were then tested on the untrained partition. We report IIA values averaged over the performance on each partition.
A.5.5 FURTHER CL Loss EXPLORATIONS
In these explorations, we explore DAS learning rate and the number of extra noisy dimensions for the OOD experiments. We show accuracies. EMD divergences, and EMD divergences restricted the causal dimensions. The EMD values are scaled by the number of extra noisy dimensions. We refer the EMD measurements along causal dimensions only as the Row EMD. We do this for both the trained partitions and held-out partitions for various DAS trainings.
A.6 LINEAR REGRESSION
In an effort to establish a more general, concrete relationship between intervened divergence an out-of-distribation (OOD) intervention performance, we performed linear regression trained
20

---

Published as a conference paper at ICLR 2026
alignments using EMD along causal axes (as discovered through the alignment training) as the independent variable and interchange intervention accuracy (IIA) as the dependent variable. We performed these regressions independently training consisting of two partitions with Straining seeds and 3 types of alignment trainings: DAS behavioral loss only, CL. loss only, and DAS+CL loss, creating 30trainings total. We used the statsmodels python package (Seabold & Perktold, 2010) to perform the regression.
the Default task and the OOD task trainings, each
Dep. Variable:
IIA OLS Least Squares
R-squared:
0.729
Model:
Adj. R-squared:
0.719
Method:
F-statistic:
75.28
Date:
Wed, 26 Nov 2025
Prob (F-statistic):
2.00e-09
Time:
11:54:42
Log-Likelihood:
76.575
No. Observations:
30
AIC:
-149.2
Df Residuals:
28
BIC:
-146.3
DI' Model:
Covariance Type:
nonrobust
coef
std err
P>
[0.025
0.975]
Intercept
0.9885
0.004
243.666
0.000
0.980
0.997
Training EMD
-0.3424
0.039
-8.677
0.000
-0.423
-0,262
Omnibus:
33.330
Durbin- Watson:
1,903
Prob(Omnibus))
0.000
Jarque-Bera (JB):
98.629
Skew:
-2.235
Prob(JB):
3.83e-22
Kurtosis:
10.676
Cond. No.
11.1
21

---

Published as a conference paper at ICLR 2026
0.08
Nearest Cos Distance
0.35
Natural
inteveneed
g 0.07
intervened
Ratural
0.30
8 0.06
0.25
08
0.05
0.20
0.04
0.15
0.03
d 0.02
0.10
0.05
0.01
0.00
0.00
DAS
Mean Diff
SAE
DAS
Mean Diff
SAE
0.40
Netural
Natural
Min Cos Pairing
0.35
Intervened
Min L2 Pairing
0.08
Intervamed
0.30
0.25
0.06
0.20
0.04
0.15
0.10
0.02
0.05
0.00
0.00
DAS
Mean
Diff
SAE
DAS
Mean Diff
SAE
35
Local PCA Distanc
12
30
Intervened
10
25
0
20
6
15
4
10
2
5
0
DAS
Mean Diff
SAE
DAS
Mean Diff
SAE
14
500
Netural
Local Reconstruction sntttuuutiion
12
Natural
intervened
400
10
300
8
6
200
100
2
0
DAS
Mean Diff SAE
DAS
Mean Diff
SAE
Figure 4: A number of additional divergence measures to demonstrate the difference between the natural and intervened distributions. Each is labeled by its y-axis. Each metric is computed over random sample natural vectors to simulate the natural manifold, and a sampled set of intervened or natural vectors for which to measure the distance from the natural distribution. We refer this distribution as the "compared" distribution. The sampled intervened and natural vectors are always the "ground-truth pair" described at the beginning of Appendix A. Nearest Cosine Distance: refers to the cosine distance to the nearest sample in the natural manifold. Multiple sampes in the compared distribution can share the same natural sample. This value is averaged over all compared samples. Nearest 1.2 Distance refers to the cosine distance to the nearest sample in the natural manifold. Multiple sampes the compared distribution can share the same natural sample. This value is averaged over all compared samples. Min Cos Pairing refers to the lowest cost pairing where cost the cosine distance between two samples. Vector pairs are exclusive. This value is normalized by the number of samples. Mín L2 Pairing refers to the lowest cost pairing where cost is the L2 distance between two samples. Vector pairs are exclusive. This value is normalized by the number of samples. Local PCA Distance refers to the distance the manifold created using local PCA of the nearest neighbors. See Appendix 1.1.2, EMD refers to the Earth Mover's Distance. See Appendix A.1.2. KDE refers the Kernel Density Estimation score. See Appendix 1.2. Local Linear Reconstruction refers to the local linear reconstruction error. See Appendix A.1.2.
22

---

Published as a conference paper at ICLR 2026
Default Task
OOD Task
Partition
Sparse
Causal Dim 1
Dense
Partition One Two
Causal Dim
Causal Dim
Figure 5: Visualization of the different synthetic tasks used for Figure 3. The Default Task is split into two partitions, both withholding two classes that are contained in the other partition. The OOD task also split into two partitions, both consisting of classes. The Dense partition consists of tighter cluster than the Sparse.
23

---

Published as a conference paper at ICLR 2026
extra dims Ir 0.005
extra dims Ir 0.01
extra dims 0 0.05
1.0
Cross Task IIA
0.8
0.6
extra dims 16
0.005
extra dims 16 0.01
extra dims 16 Ir 0.05
1.0
Cross Task IIA
0.8
0.6
extra_dima
64
0.005
extra_dims 0.01
ditr 64 Ir 0.05
1.0
Cross Task IIA
CL Epsilon
0.8
1
10
0.6
CL
DAS+ CL
CL
DAS+ CL
CL
D CL
Figure 6: Out of distribution hyperparameter search showing the DAS IIA on OOD validation data for the trained task partition. We see the DAS loss learning rate (Ir) and extra concatenated noisy input dimensions (extra dim) across the panel columns and rows. The DAS+CL reported values include the behavioral loss whereas the CL label excludes the behavioral loss. The pink dashed lines represent DAS trained with the behavioral loss only.
24

---

Published as a conference paper at ICLR 2026
extra dims Ir 0.005
extra dims Ir 0.01
extra dims 0 0.05
1.0
Trained Task llA
0.8
0.6
extra dims 16
0.005
extra dims 16 0.01
extra dims 16 Ir 0.05
1.0
Trained Task llA
0.8
0.6
extra extra_dims
64
0.005
extra extra_dims 0.01
dims 64 Ir 0.05
1.0
Trained Task IIA
CL Epsilon
0.8
1
10
0.6
CL
DAS+ CL
CL
DAS+ CL
CL
D CL
Figure 7: distribution hyperparameter search showing the DAS IIA on in-distribution validation data for the trained task partition. We see the DAS loss learning rate (Ir) and extra concatenated noisy input dimensions (extra dim) across the panel columns and rows. The DAS+CL reported values include the behavioral loss whereas the CL label excludes the behavioral loss. The pink dashed lines represent DAS trained with the behavioral loss only.
25

---

Published as a conference paper at ICLR 2026
extra dims Ir 0.005
extra dims 0.01
extra dims 0.05
0.150
Cross Row EMD
0.125
0.100
0.075
0.050
0.025
0.000
extra dims 16 Ir 0.005
extra dims 16 Ir 0.01
extra dims Ir 0.05
0.150
Cross Row EMD
0.125
0.100
0.075
0.050
0.025
0.000
extra dims
64
0.005
extra ditra 64 0.01
extra dims Ir 0.05
0.150
Epsilon
Cross Row EMD
0.125
1
10
0.100
50
0.075
100
0
0.050
0.025
0.000
CL
DAS+ CL
CL
DAS+ CL
CL
DAS+ CL
Figure 8: Out of distribution hyperparameter search showing the DAS row-space EMD on OOD validation data for the trained task partition. We see the DAS loss learning rate (Ir) and extra concatenated noisy input dimensions (extra, _atrra_dmmmmaaaa across the panel columns and rows. The DAS+CL reported values include the behavioral loss whereas the CL label excludes the behavioral loss. The pink dashed lines represent DAS trained with the behavioral loss only.
26

---

Published as a conference paper at ICLR 2026
extra dims 0.005
extra extra_dims 0| Ir 0.01
extra extra_dims- 0.05
0.150
Trained Row EMD
0.125
0.100
0.075
0.050
0.025
0.000
extra dims
16 Ir 0.005
extra dims 16 0.01
extra dims Ir 0.05
0.150
Trained Row EMD
0.125
0.100
0.075
0.050
0.025
0.000
extra dims
64
0.005
extra_dims 64 0.01
extra dims Ir 0.05
0.150
Epsilon
Trained Row EMD
0.125
1
10
0.100
50
0.075
100
0
0.050
0.025
I CL
0.000
CL
DAS+ CL
CL
DAS+ CL
DAS+ CL
Figure In distribution hyperparameter search showing the DAS row-space EMD on validation data for the trained task partition. We see the DAS loss learning rate (Ir) and extra concatenated noisy input dimensions (extra dim) across the panel columns and rows. The DAS+CL reported values include the behavioral loss whereas the CL label excludes the behavioral loss. The pink dashed lines represent DAS trained with the behavioral loss only.
27


Reviews:
[{"review_id": "Wjr6RLhpA4", "reviewer": "Reviewer_6Hkv", "summary": "This paper investigates whether causal interventions in mechanistic interpretability create out-of-distribution (divergent) representations that compromise the faithfulness of resulting explanations. The authors: (1) empirically demonstrate that common intervention methods (mean difference patching, SAEs, DAS) produce divergent representations; (2) provide theoretical analysis distinguishing \"harmless\" divergences (null-space, within-boundary covariance) from \"pernicious\" divergences (hidden pathways, dormant behavioral changes); (3) propose adapting the Counterfactual Latent (CL) loss to mitigate divergence. The work is motivated by the fundamental assumption in interpretability that counterfactual model states should be realistic, yet experiments are limited to synthetic settings.", "questions": "Q1. Real model validation (relates to W1, W3): Can you provide results on at least one real LLM (e.g., GPT-2 Small) showing that CL loss maintains or improves causal identification accuracy while reducing divergence? Even preliminary results would significantly strengthen the paper's claims. In Figure 2(a), more sophisticated causal methods (DAS, SAE) show 2-4× higher divergence than mean-difference patching. Do these methods also achieve higher counterfactual accuracy? \n\nQ2. Example I justification (relates to W2): In Example I, the balanced subspace contributes zero to outputs under natural conditions (as noted in lines 279-280). Can you clarify what makes intervention on such dimensions a case of \"pernicious divergence\" rather than simply \"intervening on wrong (non-causal) variables\"? How is this different from general causal identification errors?\n\nQ4. Practical detection (relates to W4): Beyond collecting large evaluation datasets, can you suggest even a heuristic method for practitioners to detect when divergence is likely to be pernicious versus harmless in their specific applications?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper addresses an important yet underexplored question about whether causal interventions preserve distributional faithfulness, a property essential for claims of mechanistic interpretability.\n2. Section 3 systematically examines divergences arising from three distinct intervention methods (mean difference vectors, SAEs, and DAS), demonstrating that the problem is both broad and pervasive.\n3. The “harmless vs. pernicious divergence” framework introduced in Section 4 offers a clear conceptual vocabulary for distinguishing when distributional shifts are consequential.\n4. The paper is well-structured and logically coherent, progressing naturally from Problem Definition to Empirical Evidence (Section 3), Theoretical Analysis (Section 4), and finally to the Proposed Solution (Section 5).", "weaknesses": "1. The proposed solution (CL Loss) is only validated on a synthetic 2D task. However, the problem itself is demonstrated on large models (Llama-3-8B). The gap between these synthetic toy problems and real transformers is enormous. This complete lack of validation on actual LLMs models undermines the practical applicability and credibility of the proposed solution. \n\n2. The \"pernicious\" case in Example I (Sec 4.2.1) assumes \"balanced subspaces\" where $w_1h_1 = -w_2h_2$ always holds. If this were true, this subspace would be in the behavioral null-space (its net contribution is always zero) and should not be identified as \"causal\" by any reasonable method in the first place. The example therefore demonstrates \"intervening on wrong variables fails\" rather than \"divergence on causal variables is pernicious.\" This conflates choosing incorrect intervention targets with distributional divergence problems, undermining the credibility of the pernicious divergence category. \n\n3. The example in Sec 4.2.2 requires precisely engineered weight matrices, ReLU boundaries, and specific data distributions. The probability of such a fragile and exact configuration occurring in a real, trained network is vanishingly small. This makes the example feel practically irrelevant and weakens the argument that \"pernicious\" divergence is a common, real-world problem.\n\n4. Even on the simple toy task where the solution should work perfectly, the CL Loss provides minimal benefits. The results show that while EMD (divergence) was reduced, the IIA (task accuracy) actually dropped from 1.0 to 0.998. This suggests the method is weak, ineffective, and may even introduce negative side effects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tIUD269MbB", "reviewer": "Reviewer_PwXp", "summary": "The paper studies mechanistic interpretability approaches that manipulate model representations via targeted interventions and what could go wrong if those interventions result in out-of-domain values that cause unexpected behavior in the model. The paper shows that these divergences are quite common and differentiates them between “harmless” divergences and “pernicious” divergences. A new objective based on counterfactual latent loss is presented to mitigate these issues.", "questions": "4. The distinction between harmless and pernicious cases are not well defined, and the paper only provides examples of each, rather than providing a concrete definition of what a divergent representation is, and whether it is harmless or pernicious.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper studies mechanistic interpretability approaches that manipulate model representations via targeted interventions and what could go wrong if those interventions result in out-of-domain values that cause unexpected behavior in the model. The paper shows that these divergences are quite common and differentiates them between “harmless” divergences and “pernicious” divergences. A new objective based on counterfactual latent loss is presented to mitigate these issues.", "weaknesses": "1. The paper addresses an important issue in mechanistic interpretability that is often not considered. If the data that was used to train a model is limited, then the model may be poorly defined on unseen interventions. The interpretability of such cases are therefore potentially unreliable, and this should be taken into account.\n\n2. The points are clearly established with examples, improving the clarity of the paper.\n\n3. The CL loss provides a potential solution to mitigating this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jagcn7Yz5V", "reviewer": "Reviewer_Rf6s", "summary": "This paper examines distributional divergence in mechanistic interpretability interventions. The authors argue that such interventions can push hidden representations off the model’s natural manifold, leading to pernicious effects such as spurious circuit activations or dormant behaviors. They distinguish harmless from pernicious divergences and propose a modified Counterfactual Latent (CL) loss to constrain interventions toward causal subspaces while maintaining interchange intervention accuracy (IIA). The topic is timely and important, and the paper offers useful illustrative examples and intuitions. However, the treatment remains largely informal, with vague definitions, unstated assumptions, and limited quantitative validation. The work would benefit from clearer formalization, stronger theoretical grounding, and more rigorous empirical evaluation to solidify its contribution to addressing divergent representations from causal interventions.", "questions": "Please refer to the `Weaknesses` section for detailed questions. \n\n---\n\nThe following points are minor and do not impact the overall evaluation:\n\n**1. Notation consistency.**  \nThroughout the analytic sections (Sec. `4`–`5`), ensure that all notation is defined before use, and that it is applied consistently and unambiguously to improve clarity.\n\n**2. Style.**  \nCitations should be properly linked, e.g., \"Grant (2025)\" in the abstract (`L22`). Avoid multiple acronym introductions, such as \"causal abstractions (CAs)\" (`L126` already introduced, repeated at `L131`). Replace colloquial or vague phrases (e.g., \"in causal subspaces is okay\", `L280`) with precise language.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "**1. Problem statement is timely and important.**  \nThe field of mechanistic interpretability relies heavily on causal interventions, while systematic study of representational divergence has been lacking. The harmless/pernicious dichotomy offers useful language for describing what practitioners often observe intuitively.\n\n---\n**2. Useful negative examples.**  \nThe \"balanced subspaces\" and ReLU examples are compact and concrete; they effectively illustrate how confirmatory behavior can arise from incorrect or misleading activation pathways.\n\n---\n**3. Simple mitigation with plausible effect.**  \nRestricting the CL penalty to causal subspaces aligns with the intended goal of keeping interventions \"on-manifold\" with respect to identified causal variables.", "weaknesses": "**1. Missing formal statements or guarantees.**  \nThe harmless/pernicious taxonomy is presented through narrative examples rather than formal propositions (Sec. `4.1`–`4.2`, `L213`–`L377`). No formal statements, e.g., theorem or proposition, specifies necessary or sufficient conditions under which divergence is guaranteed harmless (e.g., confined to a null-space) or pernicious (activating hidden pathways).  \nKey terms such as *\"behaviorally binary subspace\"* (Eq. `4`, `L248`–`L254`) and *\"behavioral null-space\"* (`L141` and many later references) are invoked but never **formally defined** or tied to measurable properties (e.g., local Jacobians or decision-region invariance). As a result, these claims remain intuitive or conceptual rather than analytical.\n\n---\n\n**2. Under-specified core objects.**  \n*(1) Behaviorally binary subspace:* The \"behaviorally binary subspace\" (Sec. `4.1`, Eq. `4`, `L248`–`L254`) is defined via an elementwise sign function mapping to {$-1, 0, 1$}. Including $0$ makes the decision boundary ill-posed: infinitesimal perturbations near $0$ can flip the behavioral label, so the binary characterization becomes unstable unless neighborhoods around $0$ are excluded or measure-zero sets are explicitly treated.  \n\n*(2) Null-space:* The \"null-space\" of downstream layers (Sec. `4.1`, `L213`–`L216`) is repeatedly invoked (e.g., `L481`–`L484`) but never operationally defined under nonlinear composition, normalization, or residual connections. It remains unclear whether this refers to a local Jacobian null-space at $h$ or to a global manifold of invariant directions across contexts.\n\n---\n\n**3. Argument relies heavily on hidden assumptions.**  \nSeveral strong but unstated assumptions are embedded in the narrative exposition. For instance, the discussion of \"harmless divergence\" in Sec. `4.1` presumes a clean factorization between causal subspaces (e.g., $\\tilde{\\mathbf z}_{var_a}, \\tilde{\\mathbf z}_{var_b}$) and their covariance structure. This effectively assumes approximate orthogonality and cross-context stability of these subspaces, which is rarely satisfied in deep, nonlinear architectures such as Transformers. Additionally, the notion of harmless covariance within decision boundaries presumes stable, convex class regions (e.g., assumptions and suppositions from `L246`–`L269`), yet the later \"dormant behavioral changes\" example (Sec. `4.2`) admits that minor contextual shifts can cross those boundaries. The conditions distinguishing truly benign intra-boundary covariance from merely undetected behavioral drift are left unstated. Such implicit assumptions substantially weaken the generality of the reasoning.\n\n---\n\n**4. Evaluation concerns.**  \nDivergence is visualized using 2D PCA and summarized by a single EMD score (e.g., Fig. `2`). The reviwer concerns whether such low-dimensional projections may distort pairwise distances, and averaging in EMD may mask high-magnitude divergence along select axes.  Additionally, subtracting the baseline EMD of $h$ against itself is reasonable, but since EMD is not linear, simple subtraction might obscure axis-specific drift. Have the authors explored distributional or coordinate-free divergence metrics, reporting with confidence intervals estimated across random projections?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sWSio5GcE7", "reviewer": "Reviewer_YbiP", "summary": "This paper investigates a critical, often-overlooked assumption in mechanistic interpretability: that causal interventions (like activation patching) produce latent representations that are ”faithful” or ”in-distribution” for the\ntarget model. The authors argue that interventions can create ”divergent,” out-of-distribution states, which may\nundermine the validity of the resulting explanations.\nThe paper’s contributions are threefold:\n\n- It empirically demonstrates that common intervention methods, including Mean Difference Vector Patching, Sparse Autoencoders (SAEs), and Distributed Alignment Search (DAS), do create representations that\ndiverge significantly from the model’s natural distribution.\n\n-  It provides a theoretical framework for classifying these divergences into ”harmless” (e.g., in the null-space\nor within existing decision boundaries) and ”pernicious” (e.g., activating hidden, non-native computational\npathways or causing dormant behavioral changes). The authors use synthetic examples to show how pernicious divergences can lead to misleadingly ”affirming” results.\n\n- To mitigate this issue, the authors propose an adapted Counterfactual Latent (CL) loss (based on Grant\n(2025)). This loss regularizes interventions by encouraging the intervened latent state to remain close to an\naverage of native states that share the same intended causal properties . In a synthetic setting, they show this\nmethod reduces divergence and improves out-of-distribution (OOD) generalization.", "questions": "-  The CL loss mitigation seems promising but raises practical questions. How do you\nenvision applying this to large language models where the ground-truth causal variables (needed to find\nmatching native states for $h_{CL}$) are not known a priori? Doesn’t this create a circular dependency where\nyou need the causal abstraction to find the CL vectors, but the intervention (which you’re trying to fix) is\nwhat you use to find the abstraction?\n\n- You note that the CL loss is a ”broad-stroke” solution. Do you\nhave any initial thoughts on how one might distinguish pernicious from harmless divergence, perhaps without a full causal abstraction? For instance, could one use a measure of ”local faithfulness” or analyze\nactivation changes in all output dimensions (as hinted at in 4.2.3 ) to flag interventions that recruit ”hidden”\npathways?\n\n- The concept of ”dormant behavioral changes” (Section 4.2.3, Appendix A.2) is particularly concerning. You mention that an ”infinitely expansive” dataset could detect them. In practice, how\ncould a researcher gain any confidence that their intervention hasn’t created such a dormant vulnerability?\nDoes the CL loss’s reduction in OOD error (Fig 3d) suggest it is mitigating these, or is that a separate issue?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "-  By questioning the\n”faithfulness” of intervened states, the work highlights a critical potential failure mode for many mechanistic\ninterpretability claims.\n•  The theoretical categorization of divergences into ”harmless” (e.g., nullspace, within-boundary covariance) and ”pernicious” (e.g., hidden pathways, dormant behavior) is clear,\nintuitive, and  valuable.\n-  The synthetic examples in Section 4.2 are particularly strong. They offer\nconcrete, simple-to-understand illustrations of how an intervention can be misleading; for instance, by\nbreaking a ”balanced subspace” or activating a pathway via a mean-difference vector that is not used by\nany native data point . The concept of ”dormant behavioral changes” is also a very insightful and worrying\nfailure mode.\n-  It also proposes a concrete (if\npreliminary) mitigation strategy. The adaptation of the CL loss to ”anchor” interventions to the native data\nmanifold is a good approach.", "weaknesses": "- The paper successfully categorizes divergences but does not offer a method\nto detect or classify whether a divergence observed in a practical (non-synthetic) setting is harmless or\npernicious. This makes it difficult to know when to be concerned about the results of their\ninterventions.\n-  The primary weakness of the proposed solution, acknowledged by the authors,\nis that the CL loss is a ”broad-stroke” approach. It penalizes all divergence, rather than selectively targeting only ”pernicious” divergence. This might be overly restrictive, as some ”harmless” divergences (e.g.,\ninterventions that explore the null-space) could be desirable for making certain causal claims.\n\n- The empirical validation for the CL loss mitigation is confined to a synthetic, small-scale MLP setting. It is unclear how this approach would scale to modern,\nlarge-scale models. For example, generating the ”Counterfactual Latent” (CL) vectors requires averaging\nover a pre-recorded set of native states with specific causal properties. This seems computationally challenging and, more importantly, dependent on having a correct, known causal abstraction, which is often\nwhat is being searched for in the first place.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qTVlo7aBwK", "reviewer": "Reviewer_Ahsj", "summary": "This paper shows in synthetic settings how causal intervetions, a widely used tool in interpretability, may introduce divergent representations in NNs. The authors studies 2 kinds of such divergences: \"harmless\", which are in the null space of successive projections; and \"pernicious\" that may active unintended behaviors downstream. The authors then propose an approach training DAS projections ([Geiger et al, 2023](https://arxiv.org/abs/2303.02536)) incorporating Counterfactual Latent loss ([Grant, 2025](https://arxiv.org/pdf/2501.06164)) to get rid of pernicious divergences.", "questions": "1. Figure 2: You show divergences introduced by adding mean diff, patching SAE and DAS. But what about activation patching from a single source sample ([Prakash et al, 2025](https://arxiv.org/abs/2505.14685), [Feucht et al, 2025](https://arxiv.org/pdf/2504.03022), inter alia)? Do you assume such interventions to also introduce divergences?\n\n2. Section 4.2.2: Very interesting observation. Unless I am understanding it wrong, a layernorm right after the intervention can potentially fix this issue, right? You probably should mention this point as most architectures have them nowadays.\n\n3. The situations simulated showing harmless and pernicious divergences are plausible and I do think some of the inteventions will be susceptible to such divergences. However, I am not entirely convinced about how prevalent pernicious divergences are in practice. And usually it is considered a good scientific practice to evaluate proposed hypotheses on 100s (if not 1000s) of examples covering a distribution of scenarios. Do you assume such cases may hold across entire evaluation sets? \n\n4. (related to previous question) Training DAS projections to address this seem quite expensive, even might be just infeasible in practice. The setup makes sense and is convincing in synthetic settings. But do you see any way this can be scaled to real models and datasets? And is it worth the effort [referring to the previous question]? I will be just blunt: is this paper more of an intellectual exercise? Or do you see any practical applications?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Solid theoretical discussion on how divergences may arise from causal interventions.\n* Great presentation and framing of the problem.", "weaknesses": "* Simulated in scenarios with very strong assumptions which may not hold in practice. \n* The proposed approach seems unrealistically expensive to be applied in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)