The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?
Authors: Denis Sutter
Julian Minder
Thomas Hofmann
Tiago Pimentel
Venue: NeurIPS 2025 spotlight ()
Keywords: interpretability
mechanistic interpretability
causal alignment
distributed alignment search
linear representation hypothesis
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We show in theory and practice that by allowing non-linear transformations in causal abstraction, any neural network (even random ones) can be perfectly aligned to any algorithm, rendering this interpretability approach meaningless if unconstrained.

Abstract: The concept of causal abstraction got recently popularised to demystify the opaque decision-making processes of machine learning models; in short, a neural network can be abstracted as a higher-level algorithm if there exists a function which allows us to map between them. Notably, most interpretability papers implement these maps as linear functions, motivated by the linear representation hypothesis: the idea that features are encoded linearly in a model's representations. However, this linearity constraint is not required by the definition of causal abstraction. In this work, we critically examine the concept of causal abstraction by considering arbitrarily powerful alignment maps. In particular, we prove that under reasonable assumptions, any neural network can be mapped to any algorithm, rendering this unrestricted notion of causal abstraction trivial and uninformative. We complement these theoretical findings with empirical evidence, demonstrating that it is possible to perfectly map models to algorithms even when these models are incapable of solving the actual task; e.g.,
on an experiment using randomly initialised language models, our alignment maps reach 100\% interchange-intervention accuracy on the indirect object identification task. This raises the non-linear representation dilemma: if we lift the linearity constraint imposed to alignment maps in causal abstraction analyses, we are left with no principled way to balance the inherent trade-off between these maps' complexity and accuracy. Together, these results suggest an answer to our title's question: causal abstraction is not enough for mechanistic interpretability, as it becomes vacuous without assumptions about how models encode information. Studying the connection between this information-encoding assumption and causal abstraction should lead to exciting future work.

The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?
Denis Sutter,5TH Julian Minder, 5PFL Thomas Hofmann, ETH Tiago Pimentel ETH
ETH Zürich EPFLEPFL
densutter@ethz.ch, jullan.minder@eppllcch, sthomas.hofmane
Bago.pimentel@@nneeth... deenssttrrioo///naarrreeeeenntnt
Abstract
The concept of causal abstraction got recently popularised to demystify the opaque decision processes of machine learning models; in short, a neural network can be abstracted as a higher-level algorithm if there exists function which allows us to map between them. Notably, most interpretability papers implement these maps linear functions, motivated by the linear representation hypothesis: the idea that features are encoded linearly in a model's representations. However, this linearity constraint is not required by the definition of causal abstraction. In this work, we critically examine the concept of causal abstraction by considering arbitrarily powerful alignment maps. In particular, we prove that under reasonable assumptions, any neural network can be mapped to any algorithm, rendering this unrestricted notion of causal abstraction trivial and uninformative. We complement these theoretical findings with empirical evidence, demonstrating that it possible to perfectly map models algorithms even when these models are incapable of solving the actual task; e.g., on an experiment using randomly ini- tialised language models, our alignment maps reach 100%
accuracy on the indirect object identification task. This raises the non-linear rep- resentation dilemma: we lift the lincarity constraint imposed to alignment maps in causal abstraction analyses, inherent trade-off between these maps' complexity and accuracy. Together, these results suggest answer to our title's question: causal abstraction is not enough for mechanistic interpretability, as becomes vacuous without assumptions about how models encode information. Studying the connection between this information- encoding assumption and causal abstraction should lead to exciting future work.
are left with principled way to balance the
Introduction
The increasing popularity of machine learning (ML) models has led to surge in their deployment across various industries. However, the lack of interpretability in these models raises significant concerns, particularly in high-stakes applications where understanding the secision-making process is crucial (Goodman and Flaxman, 2017; Tonekaboni et al., 2019; Zhu et al., 2020; Gao and Guan, 2023). Unsurprisingly, this opacity has motivated multitude of research mechanistic (or causal) interpretability, which tries to analyse and understand the hidden algorithms that underlie these models (Olah al., 2020; Elhage al., 2021; Mueller al., 2024; Ferrando et al., 2024; Sharkey al., 2025).
promising approach to address this challenge is causal abstraction (Beckers and Halpern, 2019; Geiger et al., 2024a), which tries to map the behaviour of model to higher-level (and conceptually simpler) algorithm which solves the task. At the core of this concept is the idea that if an intervention is found to change model's behaviour in way that aligns with specific algorithm, then that algorithm can be considered implemented by the model. Recent research, however, has raised consid- erable issues with this approach (e.g., Makelov et al., 2024; Mueller, 2024; Sun et al., 2025). Among those, Méloux et al. (2025) notes that a model's causal abstraction is not necessarily unique, showing
that many algorithms can be aligned to the same neural network. Additionally, most work on causal
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Neural Frue/Fatse Classifier
411
Acc.
Iderthy
Identiy
0.8
Onj
Non-linear
Arnumayy-1.0 1.0
Hor- Soneiinttr
Cousol Absttaaction
Linear
Hierarchical Equality Algorithi
02
Ty Camebo
roeed
Non-liuure O
9n:
Datepoirta::
Intervntions:
Ditisprinnt wilt labsll Tras Datspointwith label Pails
O- (4400)0a0) (d07))
GoCol (Hlat) (illa,
X-m (a.5)b088 (000000 x- (pm...0000002.--2
-olyg, (igg) a o- (nlue)
Figure A visualisation of what happens when analysing causal abstractions with increasingly complex alignment maps Φb The more complex & is, the higher the intervention accuracy- consequently, the stronger the algorithm-DNN alignment. In Theorem I, we show that given arbitrar- ily complex alignment maps, we can always find perfect alignment (under reasonable assumptions).
abstraction (Wu et al., 2023; Geiger et al., 2024b; Minder et al., 2025; Sun et al., 2025) implicitly assumes information linearly encoded in models' representations, relying on the linear representa- tion hypothesis (Alain and Bengio, 2016; Bolukhasi et al., 2016). Linearity, however, is not required by the definition of causal abstraction (Beckers and Halpern, 2019) and increasing evidence suggests that not all representations may be linearly encoded (White et al., 2021; Olah and Jermyn, 2024; Mueller, 2024; Csordás et al., 2024; Engels et al., 2025a,b; Kantamneni and Tegmark, 2025).
In this paper, we first prove that, once we drop the linearity constraint, any model can be per- fectly mapped to any algorithm under relatively weak assumptions e.g. hidden activation's 's input- injectivity and output- surjetttureeciiitty. which we will define formally. This renders causal abstraction vacuous when used without constraints. If we restrict alignment maps only consider, c.g., linear functions, this problem does not arise though. It follows that causal abstraction implicitly relies on strong assumptions about how features are encoded in deep neural networks (DNNs), and becomes trivial without such assumptions. This puts us at an impasse: we may want to rely on stronger notions of causal abstraction which may leverage non-linearly encoded information, but this may make our analyses vacuous; we call this the non-linear representation dilemma (schematised in Fig. 1).
To empirically validate theoretical results, reproduce the original distributed alignment search (DAS) experiments (Geiger et al.. 2024b), but while leveraging more complex alignment maps. We find that key empirical patterns they observed-such as the first layer being easier to map to the tested algorithms in a hicrarchical equality task-vanish when we use more powerful maps. Additionally, we find that we can achieve over 80% interchange intervention accuracy (IIA) using non-linear alignment maps in randomly initialised models. Extending our experiments to language models from the Pythia suite (Biderman et al., 2023), we show that near-perfect maps can be found for randomly initialised models in the indirect object identification (IOI) task (Wang et al., 2023); notably, as training progresses, the complexity of the alignment maps needed to achieve perfect IIA in this task decreases. Overall, our results show that causal abstraction, while promising in theory, suffers from a fundamental limitation: without a priori constraints on used alignment maps, becomes vacuous as a method for understanding neural networks.
2 Background
In this section, we formally define algorithms ($2.1) and deep neural networks ($2.2). These will then be used to define causal abstraction ($3). First, we formalise task function T X - }, where x E X' represents a set of input features and y E y denotes the corresponding output.
2.1 Algorithms
Given a task T, we may hypothesise different ways it can be solved. We term each such hypothesis an algorithm! A, which we represent deterministic causal model-a directed acyclic graph that
I "Algorithm" here need not match formal definition as, e.g., the considered functions may be uncomputable.

---

implements function fA X + V.² These causal models have set of nodes 7a11 which can be decomposed into three disjoint sets: (i) input nodes 7/x representing elements x, (ii) output nodes My representing elements in y, and (iii) inner nodes 7/mnne representing intermediate variables used in the computation of fa- As we focus on acyclic causal models, the edges in this graph induce a partial ordering on nodes 7 X inner < 7y. Let v, denote the value held by node 7), and let V,, denote values taken by the set of nodes 7). The set of incoming edges to a node 1) represent a direct causal 2 (1), denoted: Vy = (V (V))),())) We can compute algorithm fx by iteratively solving the value of its nodes while respecting their partial ordering:
relationship between that node and its parents par,
Vip 7/e
def
Un (V(rr,r,(,)))
fx(x) def
V
T/e Jtane
where we define Vn to be the input value x, and take the value of the output nodes Vay as our algorithm's 's output. Importantly, for an algorithm A to represent task T, its output under "normal" operation must be fx(x) = T(x). For an example of a task and related algorithms, see App. I.1.
These causal models, however, allow us to go beyond "normal" operations and investigate the behaviour of our algorithm under counterfactual settings. We can, for instance, investigate what its behaviour would be if we enforce a node 1/ value to be a constant By = G, which we write as: fa(x, (uy + c)) Vny (2)
1/ =X,
Uy =
A THE Unt
By = fR (V(yaar
Now, let f (x) represent a function which runs our algorithm with input until it reaches node 17, outputting its value Vyy We can use such interventions to investigate the behaviour of algorithm under input x. when node n/' is forced assume the value would have under x' as: fx (Uy + fA (xx))). Now, let I be a multi-node intervention, e.g., I, = (V.r + c') where r/' = [f/" and c' = U (x'), c]. We can observe how our model operates under those interventions by running fx(x,I)) See App. B for pseudo-code implementation.
2.2 Deep Neural Networks
Deep neural networks (DNNs) are the driving force behind recent advances in ML and can be defined as a sequence of functions f/ : Hare + HHapes where the denotes the set of neurons in layer & and
Hap is the corresponding vector space. A DNN N with L layers can be specified as follows:
h+p fi(x,,
h f (haee)
Pn(y x)= = (hwz)
(3)
where has, denotes the vector of activations for neurons the We focus on DNNs with real-valued neurons and probabilistic outputs, so that H.p = X. Ha,, = Rll and Har.. = AD1-1 We define 21 (x") as the function that computes the activations of the subset of neurons vp when the network fü
is evaluated on input x'. In particular, f (x) returns the activations at layer for input x. Thus, the standard computation of the DNN corresponds to evaluating Pa x) f (x). This formulation allows us to instantiate common architectures, such as multi-layer perceptrons (MLPs) or transformers, by specifying the form of each J and the structure of the neuron sets 1/.. The parameters of these models are typically optimised to minimise the cross-entropy loss.
Notably, similarly to the algorithms above, DNN's architecture induces partial ordering on its neurons, respecting the order in which they are computed vh < T/ < ubL. We can thus analogously define DNN intervention as follows: given set of neurons sb in the network and corresponding set of values Cg. we denote the intervention by ft pee (x. (he + c.)). This notation means that, during the forward computation of the DNN, the activations of the neurons in zb are fixed to the specified values cy. while the rest of the network operates as usual.
3 Causal Abstraction
To define causal abstraction, we will base ourselves the definitions in Beckers and Halpern (2019) and Geiger et al. (2024a). Let an abstraction map be defined as 7 7 + N, where H and N are, respectively, the Cartesian products of the hidden state-spaces in neural network N (i.e., whan the inputs uh and 77x In words, an abstraction map translates the inner states of neural network into an algorithms' inner states. Now, consider the DNN intervention I, hy + CD and the algorithm
der 1)1)). and the node value-spaces in an algorithm A (i.e., 7nt Mimer Unyy). both excluding
Geiger et al. (2024a) also considers eyclic deterministic causal models and Beckers and Halpem (2019) considers eyclic and stochastic causal models. We leave the expansion of work such models for future work.
3

---

intervention I, V, t Cn.. Further, let =ny=ey be the set of states in DNN for which he cy holds, and equivalently for N v. Under abstraction map T, we can define an intervention map as:
wr (b t c+) { {
Vn Cn undefined
if N = (T(h) h E H else
Intuitively, w, maps a DNN intervention Iy to an algorithmic one I, if the sets of states they induce on N and A, respectively, are the same. Further, let I, be set of interventions L. which can be performed on algorithm A. We can use wi to derive a set of equivalent DNN interventions as:
Iy= = (h f cw Vn + Cn E w (h f- C.)) Vn f cn Given these definitions, now put forward first notion of causal abstraction.
Definition 1 (from Beckers and Halpern, 2019). An algorithm is a T-abstraction of a neural network N iff: T is surjective; IA = Wy (I)); and there exists a surjective Tıx such that:
V XXX LEZZ
T(fN (x. (x.
I,) fA f (x), where I w- (Ix)
(6)
In words. the first condition in this definition enforces that all states in an algorithm are needed to abstract the DNN, while the second and third enforce that interventions in the algorithm have the same effect as interventions in the DNN. We further say that A is a strong 7-abstraction of N if it is r-abstraction and I, is maximal, meaning that any intervention allowed on algorithm However, while strong r-abstractions give
notion of equivalence between algorithms and DNNs, the maps may be highly entangled and provide little intuition about the DNN's behaviour. To ensure algorithmic information is disentangled in the DNN, we say is constructive abstraction map if there exists partition of N's neurons {w) 17 E 7im U {w1} [ where ab, are non-empty-aaee and there exist maps T, such that 7 is equivalent to the block-wise application of Tr (h..). ). In other words, constructive ab- straction maps compute the value U, of each node 7) E 7/tn in A using non-overlapping sets of neurons bb,, from N, with set t/i being left unused. We now define second notion of causal abstraction.
Definition 2 (from Beckers and Halpern, 2019). An algorithm is constructive abstraction of neural network N iff there exists an T: for which A is a strong T-abstraction of N; and T is constructive.
As we will deal with algorithm -NN pairs which share the same input and output spaces, we will impose an additional constraint r-one that is not present in Beckers and Halpern's 's (2019) definition. Namely, we restrict: &/,,, to be the neurons in layer zero and Tyy to be the identity; and abny to be the neurons in layer I. +++ and Thy to be the argmax operation."
3.1 Information Encoding in Neural Networks
The definition of constructive abstraction above maps non-overlapping sets of neurons in N, i.e., bh,, to nodes in A, i.e., 77. However, much research in ML interpretability highlights that concept information is not always neuron- aeluned and that neurons are often polysemantic (Olah et al., 2017, 2020: Arora et al., 2018; Elhage et al., 2022). In fact, there a large debate about how information is encoded in DNNs. We highlight what we see as the three most prominent hypotheses here.
Definition 3. The privileged bases hypothesis (Elhage el al., 2023) argues that neurons form privileged bases to encode information in neural network.
Most evidence in favour of this hypothesis comes from indirect evidence: i.c., the presence of neuron-alije aligned outlier features or activations in DNNs (Kovaleva et al., 2021; Elhage et al., 2023; He et al., 2024; Sun et al., 2024). Going back to 2015, Karpathy (2015) already showed that single neuron in a language model could carry meaningful information. Importantly, this hypothesis is consistent with the notion of constructive abstraction above, as argues each node 17's information should be encoded in separate, non-overlgp sets of neurons. Several researchers, however, question the special status of neurons assumed by this hypothesis, assuming instead that information is encoded in linear subspaces of the representation space, of which neurons are only special case.
Definition& 4. The linear representation hypothesis (Alain and Bengio, 2016) argues that information is encoded in linear subspaces of a neural network.
e We overload function wi here, with W, (Iz) simply applying wh elementwise to the interventions set Ig. We note that this implies algorithm A and network N must have the same outputs on the input set X..

---

A large literature has developed, backed by the linear representation hypothesis, including: concept erasure methods (Ravfogel et al., 2020, 2022), probing methodologies (Elazar et al., 2021; Ravfogel et al., 2021; Lasri et al., 2022), and work on disentangling activations (Yun et al., 2021; Elhage et al., 2022; Huben et al., 2024; Templeton et al., 2024). Some, however, still question this idea that all infor- mation must be encoded linearly in DNNs: as neural networks implement non-linear functions, there is no a priori reason for why information should be linearly encoded in them (Conneau et al., 2018; He- witt and Liang, 2019; Pimentel et al., 2020b,a, 2022). Further, recent research presents strong evidence that some concepts are indeed non-linearly encoded in DNNs (White et al., 10 Pimentel et al., 2022; Olah and Jermyn, 2024; Csordás et al., 2024; Engels al., 2025a.b; Kantamneni and Tegmark, 2025).
The non-linear representation hypothesis (Pimentel et al., 2020b) argues that infor- mation may be encoded in arbitrary non-linea subspaces of a neural network.
Definition 5.
3.2 Distributed Causal Abstractions
Following the discussion above, the definition of constructive abstraction may be too strict, as assumes 7 must decompose across neurons-and thus that node information is encoded in non-overiapping neurons. With this in mind, Geiger et al. (2024a.b) proposed the notion of distributed interventions: they expose the subspaces where node information is encoded in a DNN N by applying a bijective function to its hidden states; this function's 's output is then itself constructive abstraction of algorithm Here, we make this notion a bit more formal.
We define T as a distributed abstraction map if the following two conditions hold. First, there exists bijective function d-termed here an alignment mht-that maps the inner neurons ihin of N block-wise to an equal-sized set of latent variables v in manner that respects the partial ordering of computations in the network. Specifically, for each layer É, there exists a bijection og : Wel > Rl on its neurons such that ø is defined as the concatenation of these layer-wise bijections. Similarly to the neurons' activation hy, we will denote latent variables as has @(h)).
Second, there exists a partition (155) 1) E 7mm } U {vb]} of the resulting latent variables vb where 2b are non-empty- and a set of maps T, such that 7 is equivalent to the block-wise application of Tr (hae) A using non-overlapping partitions of latent variables vbg from N, with partition bp remaining unused.
we) ). In words, distributed abstraction map computes the value 6, of each node E 7/nt in
Given an aligmment map o, we can perform distributed interventions: h ventions are performed by first mapping the hidden state h& to the latent variables h = o(h.). intervening on subset 4h by replacing h,s with desired values c c... and then mapping these
e f C.,s. These inter-
intervened latent variables back to the original neuron base via h'g Φ ( (h's.) Thus, interventions applied in the latent space defined by o. generalising privileged-base bases interventions to arbitrary (possibly non- non-linear) subspaces. We are now in a position to define distributed abstractions.
Definition 6. An algorithm A is a distributed abstraction of a neural network N iff there exists an T: for which A is strong T T-abstraction of N; and T is distributed abstraction map.
Finally, we note that the set of all possible interventions I, and Iy may be hard to analyse in practice. Geiger et al. (2024b) thus restrict their analyses what we term here input-restricted interventions: the set of interventions which are themselves producible by a set of other input-restricted interventions. In other words, we restrict interventions ha + Cujy (where rb' C iiin or rb' C V and V + Cay' to C.p. and Cip' which are a, p.oruuuct of other input-restricted interventions, e.g., Cupy f (x') or
C,y =A (x', (V,," + fi" (x")).. This leads to the definition of input-restricted T a-abstraction: a weakened notion of strong abbstract,on, where intervention sets are restricted input-restricted interventions. Finally, we defne an analogous version of distributed abstraction, which is input- restricted; this the notion typically used in practice by machine learning practitioners.
Definition 7 (inspired by Geiger et al., 2024b). An algorithm & is an insut-restricted distributed abstraction of a neural network N iff there exists an T: for which A is an input-restricted T-abstraction Nf N: and T a distributed abstraction map.
A visual representation of how these causal abstraction definitions are related is given in App. D as Fig. 6. Finally, we further introduce input-restricted V-abstractions: input-restricted distributed abstractions for which we restrict alignment maps é to be in a specific variational family V. The case of linear alignment maps, will be particular important here-as as it relates to the linear representation hypothesis- and we will thus explicitly label it as insut-restricted linear abstraction.
5

---

3.3 Finding Distributed Abstractions
How do we evaluate if an algorithm is an input-restricted distributed abstraction of a DNN? Geiger et al. (2024b) proposes an efficient method to answer this, called distributed alignment search (DAS). Before applying DAS, one must assume a partitioning {10 I E 7int] } U { which remains fixed during the method's 's application; we term |2b6] the intervention size. The principle behind
DAS is then to leverage the constraint on Try. which is fixed as: Vny yrgmaxyey Px(y x) since px(y x) = haz Given this constraint, we can initialise parametrised function Φ, which we train to predict this equality under possible interventions; this is done via gradient descent, minimising the cross-entropy between the DNN and the algorithm. Specifically, we first select a of nodes to be intervened
PP(7 ), where P is a function that takes the powerset of set, along with corresponding counterfactual inputs x,, E X' for each 1) E 7j and a base input X8 E X. We then define the following two interventions:
I the men + [S (xx)) men and I = [v,) T < [ (x,, nen
(7)
Finally, we run our algorithm under base input X2 and intervention I, to get a ground truth output: y ff((x L.). which we can train the alignment map o such that the DNN matches the algorithm:
Repeating this process N times, we build a dataset D = {(x}) (m) I(())
(r))) on
C ==
Σ
log Pu
m2)
(m) (x)) where p xs, In) fx(x,, In)
(8)
(xc 11 J(U)
Notably, DAS mostly ignores how function T is constructed, relying solely on the assumed definition of Thy Finding low-loss alignment map é is then assumed as sufficient evidence that A is an input-restricted distributed abstraction of N.
Unbounded Abstractions are Vacuous
In this section, we provide our main theorem: can be shown to be an input-restricted distributed abstraction of any DNN N. making this notion of causal abstraction vacuous. To show that, we need a few assumptions (for their formal definition, see App. F). Our first assumption (Assump. 1) is that we have countable input-space X. While this may not hold in general, it holds for common applications such as language modelling (where the input-space is the countably infinite set of finite strings) or computer vision (where the input-space a countable union of pixels, which can assume a finite set of values). The second assumption (Assump. 2) is that DNNs are input-injective in all layers: i.e., f is injective for all layers. This guarantees that no information about a DNN's input is lost when computing the hidden states h This assumption is also present in prior work (e.g.. Pimentel et al., 2020b) and we show in App. G- assuming real-valued weights and activations- this is almost surely true for transformers at initialisation." 5 Due to floating point precision and neural collapse (Papyan et al., 2020), it is likely not to hold fully in practice; however, it still seems to be well-approximated in many empirical settings (Morris et al., 2023, and App. H). The third assumption (Assump. 3) is strict sutput-surjectivtty in all layers. This assumption guarantces that in each layer there is at least one choice of h that will produce the desired output. Notably, this assumption may not hold in theory, due to issues like the bittax--ottleneck (Yang et al., 2018). In practice, however, even with large vocabulary sizes, seems that almost all outputs can still be produced by language models (Grivas et al., 2022) which is sufficient for these DNNs to be abstracted by many algorithms. Our fourth assumption (Assump. 4) is that the algorithm A and DNN N have matchable partial-orderings, meaning that there is a which would match the partial-ordering of nodes A; this is likely to be the case for most reasonable algorithms given the size of state-of-the-an art deep neural networks. Finally, our last assumption (Assump. 5) is that the DNN N solves the given task We believe this assumption to be reasonable, as it would be impractical in practice to evaluate a neural network that does not perform the task correctly. Given these assumptions, we can now present our main theorem.
that under reasonable assumptions, any algorithm
partitioning of neurons
Theorem 1. Given any algorithm A and any neural network N such that Assumps. I to 5 hold, we can show that A is an input-restricted distributed abstraction of
Proof. We refer to App. F for the proof.
AIss Nikolaou et al. (2025), who show almost sure injectivity holds for transformers throughout training. 'If the model does not solve the task, perfect IIA is impossible since non-intervened inputs yield incorrect out- puts. Thus, assuming the model solves the task is necessary. In practice, however, even when the DNN is imper- feet, alignment map could produce correct outputs for all intervened inputs, achieving perfect IIA scores.

---

5 Experimental Setup
Building on the previous section's proof that alignment maps between DNNs and algorithms always exist, we now demonstrate their practical learnability and how increasingly complex alignment maps reveal various causal abstractions for different tasks, even on DNNs that do not solve them.
Alignment Maps. To assess how complexity impacts causal-abstractio analyses, we explore three ways to parameterise éb First, we will consider the simplest identity maps: (1 = h. This is the least expressive & we consider, and if we find that A abstracts N under this map. we can say that A is a constructive abstraction of N; further, this map implicitly assumes the privileged bases hypothesis. For Ø²ad, we greedily search for the optimal partition { & 17 E Mint] (instead of keeping it fixed) by iteratively adding neurons to them. For all inmer simultaneously, one neuron is added at time for each 26., up to maximum allowed intervention size; these neurons are chosen to minimise the loss in eq. (8). Second, we will consider linear maps: (hin(h) W ortah,, h, where W Worth E [R xde is an or- thogonal matrix. This the type of alignment map originally considered by Geiger et al. (2024b),' and implicitly assumes the linear representation hypothesis, evaluating input-restricted linear abstractions. Finally, we consider non-linear maps: doonlin(h) (h) = revnet[Lrm, (h), where revneti drn] is reversible residual network (RevNet; Gomez et al., 2017) with Len layers and hidden size den- We can modulate the complexity of this final map by increasing Len and dmm assuming the non-linear representation hypothesis. We note that all three maps are bijective and easily invertible.
Evaluation Metric. We evaluate the effectiveness of an alignment map e using the interchange intervention accuracy (IIA) metric proposed by Geiger et al. (2024b). For a held out test set Drest with the same structure as the training set D defined in $3.3, we compute the accuracy of our model (i.e.. argmaxy'E> @ (y' x3 I I!"))) )) when predicting the intervened y(m) = fx(xx ).). We compare this to the DNN's accuracy the test set Drest without interventions.
5.1 Tasks, Algorithms, and DNNs.
Hierarchical equality task (Geiger et al. (2024b)). We will showcase our results primarily on this task. Let x X1 o X2 o X3 o X1 be a 16-dimensional vector, and x1 to X, each be 4-dimensional vectors, where o represents vector concatenation. Further, let X' = [-.5, 5] This task consists of x4). As our DNN N. we investigate a 3-layer multi-layer perceptron (MLP) with hidden size 16, trained to perform this task; we describe this DNN, and its training procedure in more detail in App. I.1. Finally, we explore three algorithms for this task. The both equality re relations algorithm first computes the two equalities (Vi,, (x1 == x2) and Ung = (x3 = x4)) separately; it then determines whether they are equivalent as a second step. The left equality relation intionn algorithm first computes the left equality (Vn, = (x) = X2)), and then determines a single step this is equivalent to (X3 = X4). Finally, the idetitt of first argument algorithm assumes we copy the first input to node (V,,, x1) and then compute the output directly. These three algorithms are more rigorously defined in App. I.I.
evaluating: y = (X1 = X₂)
(x3
Indirect object identification (IOI) task. In a second set of experiments, we explore this task, inspired by Wang et al. (2023) and using the dataset of Muhia (2022). This task is more real- istic and relies on larger (language) models. Here, inputs x E X are strings where two peo- ple are first introduced, and later of them assumes the role of subject (S), giving or say- ing something to the other, the indirect object (IO). The task is then to predict the first token of the IO, with the output set y containing the first token of each person's 's name. E.g., "Friends Juana and Kristi found a mango at the bar. Kristi gave it to" and "Juana"". As our DNN, we models from the Pythia suite (Biderman al., 2023) across different sizes (from 31M to 410M parameters) and training stages. We evaluate the ABAB-ABВA algorithm where, given two names A and B. an inner node Uny captures if the sentence structure is ABAB (e.g.. "A and B ... A gave to B²") or ABBA (e.g.. "A and B ... B gave to A"), and the algorithm outputs prediction B if 2y is ABAB and A otherwise. This algorithm is more rigorously defined in App. I.2.
Experiments and Results
We now proceed with our empirical study, applying alignment maps of varying complexity on both "toy" and real neural networks to evaluate their effects the causal abstraction method DAS.
We note that, while Geiger et al. (2024b) describe the used & as rotation, their pyvene (Wu et al., 2024a) inplementation orthogonal matrices. This, however, makes no difference in the power of the alignment map.
7

---

Recality Rebitions
Left Equulity Relation
llerrtty First Arguneet
1.00
litert
idlexiity
Accimar
linear
0.50
0.25
IND
2
122 L3
1.2
L3
01 1.2
13
Figure 2: IIA in the hierarchical equality task for causal abstractions trained with different alignment maps Φ. The figure shows results for all three analysed algorithms for this task. The bars represent the max IIA across 10 runs with different random seeds. The black lines represent mean IIA with 95% confidence intervals. The | denotes the intervention size per node. Without interventions, all DNNs reach almost perfect accuracy (>0.99). The used goenlin uses Leu = 10 and d.a = 16.
1.0
1.0
0.9
Accurey
Accuracy
0.8
0.8
FNN
0.7
Aes 3. S. dea
0.6
0.6
10,
11,
0.5
1 2
16
TM
Full
den
DNN Training Steps
Figure 3: IIA of alignment between the both equality relations algorithm and an MLP, with interventions at layer 1. Left: Mean IIA over 5 seeds using gmonlin (Lnn 1) on the trained DNN. Performance improves with larger hidden dimension den and intervention size |xb Right: Maximum IIA across 5 seeds using معقn and gonnli with | = 8. Complex alignment maps achieve high IIA even with randomly initialised DNNs, while simpler maps gradually improve as training progresses.
Hierarchical equality task, main results. 8 Fig. 2 presents IIA results across different alignment maps e for all three algorithms. As expected, the identity map øهd generally results in the worst per- formance. Using linear alignments (dlin), ,, we observe patterns consistent with Geiger et al. (2024b): IIA for both equality relat ions and left equality relaticn decreases substantially in the third layer, indicating information becomes difficult manipulate using linear transformations at deeper layers. With the non-linear alignment (dmonlin), this layer-dependent degradation vanishes, yielding near-optimal IIA across all layers. Consequently, while assuming linear representations seems to enable us to identify the location of certain variables in our DNN, many of these insights fail to generalise when more powerful non-linear alignment maps are employed. The ident ity of fi rst argument algorithm's 's IIA consistently hovers around 50% for oa4a Øهli and ogonnin Additional experiments (App. H) suggest this is caused by insufficient capacity of the used revnet model, as the identity of x1 seems to be encoded in the model's 's hidden states.
Hierarchical equality task, exploring ghenlin's complexity. Fig. 3 (left) illustrates how varying the hidden size d and intervention size [ affects IIA with the both equality relation: algorithm on layer 1 of our MLP. Fig. 3 (right) shows IIA evolution alignment complexity increases throughout the MLP's training (evaluated on its layer 1). Remarkably, even with randomly initialised DNNs, we achieve over 80% IIA using the most complex alignment map. As training progresses, simpler alignment maps gradually attain higher IIA values. Additional results in App. 1.1.3 extend these findings to other MLP layers, intervention sizes, and algorithms, consistently revealing similar patterns that reinforce our conclusion about the impact of alignment map complexity on IIA dynamics.
Indirect object identification task, main results. Fig. (left) presents the results of trying to find causal abstractions between the ABAB-ABBA algorithm and Pythia language models, exploring how model size affects alignment capabilities. Notably, despite only the larger models (160M and 410M parameters) successfully learning the IOI task, we can align the algorithm to models of all sizes- including the 31M and 70M parameter models that fail to learn the task. Further, and somewhat surprisingly, this alignment is perfect even for randomly initialised models across all sizes; smaller fully trained models (31M, 70M), though, show slightly reduced alignment accuracy. This reduction As an additional task similar hierarchical equality, also explore the distributive law task in App. L.3.
8

---

1.0
1.00
Accurucy
Accuracy
DNN
0.75
0.5
DNN [Pall) + non-lin. (Full) -0- En. (Pa)
DNN (Juit.) -a- non lin. (Iutt.) -a- Im. [nit.)
0.50
31mms
70mps
160mne
410ma
0
1k
2k
3k
4k
Fiill
DNN Size
DNN Training Step
Figure 4: IIA of alignment between ABAB algorithm and Pythia language models. Left: IIA across model sizes at initialisation (Init.) or after full training (Full), with intervention at the middle layer. Right: IIA with increasingly complex alignment maps during Pythia-410m's training. Results show complex alignment maps yield near-perfect IIA. All gmanlin use d,n = 64.
may stem from these smaller models saturating late in training (Godey et al., 2024), becoming highly anisotropic and making it harder for grenlin to access the information needed to match the algorithm.
Indirect object identification task, exploring Jpealin' complexity. Fig. 4 (right) illustrates the interplay between model training progression and algorithmic alignment for the Pythia with 410M parameters, Notably, while this model begins to acquire task proficiency only around training step 3000 (as indicated by model accuracy), employing an 8-layer ggenlan as alignment map yields near- perfect IIA across all training steps, including for randomly initialised models. This pattern partially extends to a 4-layers dionlin configuration; however, there is noticeable dip in IIA at step 1000 for this configuration. which may be due to the model over- werr-itting to unigram statistics (Chang and Bergen, 2022; Belrose et al., 2024) at this weint-thereby making context (and hidden states) be mostly ignored when producing model outputs. Interestingly, as training advances, even less complex alignment maps (1- and 2-layer gienllin) eventually attain perfect alignment. contrast, linear maps only approximate perfect alignment in the fully trained model, following a similar trend to the DNN's performance.
7 Discussion
Our results show that when lift the assumption of linear representations, sufficiently complex alignment maps can achieve near-perfect alignment across all models-regardlles of their ability to solve the underlying task. This provides compelling evidence for the non-linear representation dilemma, suggesting causal alignment may be possible even when the model lacks task capability. We now discuss our results in the context of prior literature, with additional related work in App. C.
Causal Abstraction is not Enough. Causal abstraction (Geiger et al., 2024a) has gained traction as theoretical framework for mechanistic interpretability, promising to overcome probing limitations by analysing DNN behaviour through interventions: if you intervene on a DNN's representations and its behaviour changes in a predictable way, you have identified how the DNN "truly" encodes that feature (Elazar et al., 2021; Ravfogel et al., 2021; Lasri et al., 2022). Recent critiques of causal abstraction (c.g.. Mueller, 2024) highlight practical shortcomings, including the non-uniqueness of identified algorithms (Méloux et al., 2025) and the risk of "interpretabiiity illusions" (Makelov et al., 2024). Despite counterarguments to some of these critiques (Wu et al., 2024b; Jørgensen et al., 2025), concerns have emerged that methods based on causal abstraction may introduce new information rather than accurately reflect the behaviour of the DNN (Wu et al., 2023: Sun al., 2025); as an example, causal abstraction methods applied to random models sometimes yield above- chance performance (Geiger et al., 2024b; Arora et al., 2024). By examining the implications of assuming arbitrary complex ways in which features may be encoded in DNN, we show that nearly any neural network can be aligned to any algorithm. Together, our results thus suggest that the shift in interpretability research to causal abstractions does not, by itself, resolve the core challenge of understanding how representations are encoded. Additionally, we note that early causal abstraction methods (Geiger et al., 2021) implicitly rely on the privileged bases hypothesis, while recent advancements (Geiger et al., 2024b) rely on the linear representation hypothesis instead.
Balancing the Accuracy vs. Complexity of Φ. Diagnostic probing was previously popular method for interpretability research (Alain and Bengio, 2016), where probe was applied to the hidden representations of DNN and trained to predict specific variable. Notably, the architecture chosen for this probe implicitly reflected assumptions about representation encoding, and the absence of a universally accepted model for representation encoding precluded a theoretically founded

---

choice of probe architecture (Belinkov. 2022). The debate regarding the trade-off between probing complexity and accuracy (Hewitt and Liang, 2019: Pimentel et al., 2020b.a; Voita and Titov, 2020) underscores the risk of complex probes merely memorising variable-specific relations, instead of revealing which information the DNN "truly" encodes and uses. In this paper, we revive this debate by showing clear analogue in causal abstraction methodologies: the effect of ø complexity on IIA. Unfortunately, this debate was never solved by the probing literature, and solutions ranged from: controlling for the probe's memorisation capacity (Hewitt and Liang, 2019).' explicitly measuring probe's s complexity accuracy trade-off (Pimentel et al., 2020a), training minimum description length probes (Voita and Titov, 2020), or leveraging unsupervised probes (Burns et al., 2023). 10
The Role of Generalisation. We now highlight that Theorem 1 provides an existence proof for perfect abstraction map (thus guaranteeing perfect IIA) between a DNN and an algorithm. This existence proof, however, leverages complex interactions between the intervened hidden states and the DNN's structure, requiring perfect information about both and thus representing a form of extreme overfitting. Crucially, this theorem offers no guarantees regarding the learnability of the alignment map & from limited data or its generalisation to unseen inputs. This gap between theoretical existence and practical Icarnability becomes evident in practise. For instance, in an additional experiment on the IOI task (in App. I.2.3), we show that when training and test sets contain disjoint sets of names, the learned alignment map fails to generalise, resulting in low IIA on the test set. This suggests that generalisation should play crucial role in causal abstraction analysis, the ability to learn abstraction maps that transfer beyond training data seems fundamental to interpreting a model, distinguishing genuine understanding about its inner workings from mere training pattern memorisation.
Investigating Representation Encoding in DNNs. How neural networks encode variables/concepts is a long-standing question in interpretability, with three main hypotheses standing out: the privileged bases, linear representation, and non-linear representation hypotheses (see $3.1) One way to try to distinguish between these hypotheses is with causal abstraction analyses, but what can we learn about these hypotheses if our methods themselves rely on them as assumptions? One solution could be to compare results using ø with different architectures. Our Fig. 4 (right), for instance, shows that while o granlin achieves consistently near-perfect results throughout model training. ddnn accompanics the actual DNN's ' performance more closely. Intuitively, may thus be inclined to support the linear representation hypothesis here. We (the authors), however, cannot make this intuition formal to justify why we believe this is the case. Furthermore, gasn still manages to sometimes achieve IIA higher than the DNN's s accuracy, implying it may also "learn the task". We expect future work will propose novel methodologies analyse information encoding and try answer these questions.
8 Conclusion
This paper critically examines causal abstraction in machine learning, when assumptions imposed on how representations are encoded. We show that, under mild conditions, any algorithm can be perfectly aligned with any DNN, leading the non-linear representation dilemma. Empirical validation through experiments on the hierarchical equality and the indirect object identification tasks corroborate our theoretical insights, demonstrating near perfect IIA even randomly initialised DNNs. So, what should you do if you want to perform causal analysis of your DNN? We believe that it must be decided on byse-by-case basis. If you have reason to believe the linear representation hypothesis holds for the features you wish to extract, constraining & to linear functions may be advised. If you do not, however, you may face the non-linear representation dilemma, and be forced to investigate some kind of trade-off between Ø's accuracy and complexity.
Limitations. Our proof that any algorithm can be aligned with any DNN (Theorem 1) relies on a form of overfitting, Yet, our experiments show that the learned alignment maps ó generalise to unseen test data; studying the factors behind this generalisation would be valuable. Further, our theorem relies on two strong assumptions: input-injectivit (Assump. 2) and strict output-surjectivity (Assump. 3) in all layers. While we justify both, there are settings settings-relater to, e.g., the softmax bottleneck where they may fail; studying these failure modes could clarify our assumptions' limitations.
Notably, this method was previously applied to causal abstraction analysis by Arora al. (2024). The complexity accuracy trade-off in probing arises mainly in supervised settings, where more complex probes can extract richer features from model representations. Unsupervised probing avoids this, lacking the supervision that enables such "gerrymandered" mappings.
10

---

Contributions
Denis Sutter led the project, implemented the base version of the DAS code, conducted the MLP experiments and derived the base proof of Theorem 1 as well as the proof of Theorem 2. Julian Minder implemented and ran the language model experiments, produced all plots, and helped refine the proof of Theorem Thomas Hofmann provided guidance throughout the project. Tiago Pimentel supervised the project, giving initial intuitions for the proofs in both Theorem I and Theorem refining the proof of Theorem 1, and defining the main notation in the paper, integrating feedback from Denis and Julian, All authors wrote the paper together.
Acknowledgments
This work was mostly done in the Data Analytics Lab at ETH Zürich. We would like to thank Pictro Lesci, Julius Cheng, Marius Mosbach, Chris Potts, and Atticus Geiger for their thoughtful feedback. We would also like to thank Frederik Hytting Jørgensen for bringing to our attention a mistake in our original Definition and for his feedback on manuscript. We thank Zhengxuan Wu and Kevin Du for early discussions related to the ideas presented here. We are grateful to the Data Analytics Lab ETH for providing access to their computing cluster. Julian Minder is supported by the ML Aligmment Theory Scholars (MATS) program. Denis Sutter gratefully acknowledges the financial support of his parents, Renate and Wendelin Sutter, throughout his graduate studies, during which this work was carried out, as well as the technical support of Urban Moser and Leo Schefer.
References
Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes. arXiv.
Aryaman Arora, Dan Jurafsky, and Christopher Potts. 2024. CausalGym: Benchmarking causal interpretability methods on linguistic tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pages 14638-14663, Bangkok, Thailand. Association for Computational Linguisties.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495.
Sander Beckers and Joseph Y. Halpern. 2019. Abstracting causal models. Proceedings of the AAAI Conference on Artificial Intelligence, (3(01) 268-2265.
Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219
Nora Belrose, Quintin Pope, Lucia Quirke, Alex Mallen, and Xiaoli Fern. 2024. Neural networks learn statistics of increasing complexity. Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org.
Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 2397-2430.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems, volume 29.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations.
Tyler A. Chang and Benjamin K. Bergen. 2022. Word acquisition in neural language models. Transactions of the Association for Computational Linguistics, 10:1-16.
11

---

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann. Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayant Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Heler-Hellstemn, Douglas Eck, Pa Scaling language modeling with pathways. Jeff Dean, Slav Petrov, and Noah Fiedel. 2023. J. Mach. Learn. Res., 24(1).
Alexis Conneau, German Kruszewski, Guillaume Lample, Loie Barrault, and Marco Baroni. 2018. What you can cram into a single $&.#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136, Melbourne, Australia. Association for Computational Linguistics.
Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. 2024. Recurrent neural networks learn to store and generate sequences using non-linear representations. In Pro- ceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks NLP, pages 248-262, Miami, Florida, US. Association for Computational Linguistics.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of superposition. Transformer Circuits Thread.
Nelson Elhage, Robert Lasenby, and Christopher Olah, 2023, Privileged bases in the transformer residual stream. Transformer Circuits Thread, page 24.
Nelson Elhage. Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac HatfieldDodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread.
Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. 2025a. Not all language model features are one-iimensionally linear. In The Thirteenth International Conference on Learning Representations.
Joshua Engels, Logan Riggs Smith, and Max Tegmark. 2025b. Decomposing the dark matter of sparse autoencoders. Transactions on Machine Learning Research.
Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R. Costa-jussà. 2024. A primer on the inner workings of transformer- based language models. arXiv.
Lei Gao and Ling Guan. 2023. Interpretability of machine learning: Recent advances and future prospects. IEEE MultiMedia, 10(4):105-118.
Atticus Geiger, Duligur lbeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas leard. 2024a. Causal abstraction: A theoretical foundation for mechanistic interpretability. arXiv.
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. 2021. Causal abstractions of neural networks. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA. Curran Associates Inc
12

---

Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. 2022. Inducing causal structure for interpretable neural networks. In Pro- ceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 7324-7338. PMLR.
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. 2024b. Finding alignments between interpretable causal variables and distributed neural representations.
arXiv.
Nathan Godey, Éric Villemonte de la Clergerie, and Benoft Sagot. 2024. Why do small language models underperform? Studying language model saturation via the softmax bottleneck. In First Conference on Language Modeling.
Satvik Golechha and James Dao. 2024. Challenges in mechanistically interpreting model representa- tions. arXiv.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. 2017. The reversible resid- ual network: Backpropagation without storing activations. Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Bryce Goodman and Seth Flaxman. 2017. European Union regulations on algorithmic decision making and a "right to explanation". Al Magazine, 38(3):50-57.
Andreas Grivas, Nikolay Bogoychev, and Adam Lopez. 2022. Low-rank softmax can have unargmax- able classes in theory but rarely in practice. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6738 6758, Dublin, Ireland. Association for Computational Linguistics.
Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, and Thomas Hofmann. 2024. Under- standing and minimising outlier features in transformer training. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 2733-2743, Hong Kong, China.
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey, 2024. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Leurning Representations.
Frederik Hytting Jørgensen, Luigi Gresele, and Sebastian Weichwald. 2025. What is causal about causal models and representations? arXiv.
Subhash Kantamneni and Max Tegmark. 2025. Language models use trigonometry to do addition. arXiv.
Andrej Karpathy. 2015. The unreasonable effectiveness of recurrent neural networks.
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky, 2021. BERT busters:
Outlier dimensions that disrupt transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3392-3405, Online. Association for Computational Lin- guisties.
Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell. 2022. Probing for the usage of grammatical number. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 8818-8831, Dublin, Ireland. Association for Computational Linguistics.
Aleksandar Makelov, Georg Lange, Atticus Geiger, and Neel Nanda. 2024. Is this the subspace you are looking for? An interpretability illusion for subspace activation patching. In The Twelfth International Conference Learning Representations.
13

---

Maxime Méloux, Silviu Maniu, François Portet, and Maxime Peyrard. 2025. Everything, everywhere, all at once: mechanistic interpretability identifiabl?? In The Thirteenth International Conference on Learning Representations.
Julian Minder, Kevin Du, Niklas Stochr, Giovanni Monea, Chris Wendler, Robert West, and Ryan Cotterell. 2025. Controllable context sensitivity and the knob behind it. In The Thirteenth International Conference on Learning Representations.
John Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander Rush. 2023. Text embeddings reveal (almost) as much as text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12448-12460, 12460, Singapore. Association for Computational Linguistics.
Aaron Mueller. 2024. Missed causes and ambiguous effects: Counterfactuals pose challenges for interpreting neural networks. arXiv.
Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Amab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. 2024. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. arXiv.
Brian Muhia. 2022. ioi (revision 223da8b).
Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML' 10, page 807-814, Madison, WI, USA. Omnipress.
Giorgos Nikolaou, Tommaso Mencattini, Donato Crisostomi, Andrea Santilli, Yannis Panagakis, and Emanuele Rodolá. 2025, Language models are injective and hence invertible. Preprint, arXiv:2510.15511
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill. http://ssl...pub22220/vvrri///am
Chris Olah and Adam Jermyn. 2024. What is linear representation? What is a multidimensional feature? Transformer Circuits Thread.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. 2017. Feature visualization. Distill. http:/isii...pp222/0///////eiiae
Vardan Papyan, X. Y. Han, and David L. Donoho. 2020. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):2452-24666
Tiago Pimentel, Naomi Saphra, Adina Williams, and Ryan Cotterell. 2020a. Pareto probing: Trading off accuracy for complexity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3138-3153, Online. Association for Computational Linguistics.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020b. Information-theerrtic probing for linguistic structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4602-4622, Online. Association for Computational Linguistics.
Tiago Pimentel, Josef Valvoda, Niklas Stoehr, and Ryan Cotterell. 2022. The architectural bottleneck principle. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11459-11472. Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Alee Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl blog.
14

---

Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, Online. Association for Computational Linguisties.
Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. 2021. Counterfactual interventions
reveal the causal effect of relative clause representations agreement prediction. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 194-209, Online. Association for Computational Linguistics.
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. 2022. Linear adversarial concept erasure. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 18400-18421, PMLR.
Lee Sharkey, Bital Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Fric J. Michaud, Stephen Casper, Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, and Tom McGrath. 2025. Open problems in mechanistic interpretability. arXiv.
Jiuding Sun, Jing Huang, Sidharth Baskaran, Karel D' D'Oosterlinck, Christopher Potts, Michael Sklar, and Atticus Geiger. 2025. HyperDAS: Towards automating mechanistic interpretability with hypemetworks. arXiv.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. In First Conference on Language Modeling.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers. Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. 2024. Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread.
Sana Tonekaboni, Shalmali Joshi, Melissa D McCradden, and Anna Goldenberg. 2019. What clinicians want: Contextualizing explainable machine learning for clinical end use. arXiv.
Oskar van der Wal, Pietro Lesci, Max Müller-Eherstein Naomi Saphra, Hailey Schoelkopf, Willem Zuidema, and Stella Biderman. 2025. PolyPythias: Stability and outliers across fifty language model pre-training runs. In The Thirteenth International Conference on Learning Representations.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 1 International Conference on Neural Information Processing Systems, NIPS' 17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
Elena Voita and Ivan Titov. 2020. information-theoree probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 183-196, Online. Association for Computational Linguisties.
Kevin Ro Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. 2023. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference Learning Representations.
Jennifer C. White, Tiago Pimentel. Naomi Saphra, and Ryan Cotterell. 2021. non-linear structural probe. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 132-138, Online. Association for Computational Linguistics.
Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah Goodman, Christo- pher Manning, and Christopher Potts. 2024a. pyvene: A library for understanding and improving Py Torch models via interventions. In Proceedings of the 2024 Conference of the North American
15

---

Chapter the Association for Computational Linguistics: Human Language Technologies (Voluee 3: System Demonstrations), pages 158-165, Mexico City, Mexico. Association for Computational Linguisties.
Zhengxuan Wu, Atticus Geiger, Jing Huang, Aryaman Arora, Thomas Icard, Christopher Potts, and Noah D. Goodman. 2024b. A reply to Makelov al. (2023)'s "interpretability illusion" arguments. arXiv.
Zhengxuan Wu, Atticus Geiger, Thomas leard, Christopher Potts, and Noah Goodman. 2023. Inter- pretability at scale: Identifying causal mechanisms in Alpaca. In Advances in Neural Information Processing Systems, volume 36, pages 78205-78226. Curran Associates, Inc.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018. Breaking the softmax bottleneck: A high-rank RNN language model. In International Conference on Learning Representations.
Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann LeCun. 2021. Transformer visualization via dictionary learning: contextualized embedding linear superposition of transformer factors. In Proceedings of Deep Learning Inside Out (DeeLiO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 1-10, Online.
Wenwu Zhu, Xin Wang, and Wen Gao. 2020. Multimedia intelligence: When multimedia meets artificial intelligence. IEEE Transactions on Multimedia, 22(7):1823-1835
16

---

A Reproducibility
We provide the code to reproduce experiments in https: / com/ densutter/ non non-linear- noo-Iinnea--egreeentttttooo---
Refer to the README. md for instructions.
B
Pseudo-code for Running an Intervention on an Algorithm
In Fig. 5, we present pseudo-code that demonstrates how algorithms execute under our intervention framework.
def f (()
def fa (x, [,NNoe)
3
VTx x
4
for in A. topoloogical soott (Minner U U7y
if L and in I::
By [
else:
8
Un (V
return Vay return fA
Figure 5: Pseudo-code implementation of an algorithm with interventions, where interventions I5 are specified as a Python dictionary mapping nodes to their intervened values.
C Additional Related Work
The concept of causal abstraction was ported to deep neural networks by Geiger et al. (2024a), providing a generalised framework for understanding how neural networks can be abstracted to higher-level algorithms. Early work by Geiger et al. (2021) explored direct interventions on neuron- aligned activations, laying the groundwork for Geiger et al. (2024b) introduced distributed aligmment search (DAS), which uses an alignment map align distributed representations in neural networks with causal graphs. Several improvements to DAS have been proposed: Sun et al. (2025) developed HyperDAS, which automates the search for node information using hypemetworks, while Wu al. (2023) introduced Boundless DAS, which automatically determines intervention size through gradient descent, scaling to larger models.
sophisticated approaches. Building this,
However, recent work has raised important critiques of causal alignment methods. Méloux al. (2025) demonstrated that multiple algorithms can be causally aligned with the same neural network, and conversely, a single algorithm can align with different network subspaces. Mueller (2024) identified fundamental limitations in counterfactual theories, showing they may miss certain causes and that causal dependeneies neural networks are not necessarily transitive. Makelov al. (2024) showed that subspace interventions such as those used in DAS can lead to "interpretability illusions" cases where manipulating a subspace changes the behaviour of the model through activating parallel pathways, rather than directly controlling the target feature. In their response, Wu et al. (2024b) argued these illusions may be artefacts of specific evaluation approaches rather than fundamental flaws, and that they depend on the definition of causality being used, a point also made by Jørgensen et al. (2025).
Recent work has also raised significant challenges to the linear representation hypothesis. White et al. (2021) demonstrated that syntactic structure in language models is encoded non-linearly, showing that kernelised structural probes outperform linear ones while maintaining parameter count. Similarly, Csordás et al. (2024) found that recurrent neural networks use fundamentally non-linear representations for sequence tasks. Engels al. (2025a) provided concrete examples of non-linear feature representations in language models, such as days of the week being encoded on a circular manifold. While Golechha and Dao (2024) argued that some language modelling behaviours may be represented linearly due to next-token prediction and LayerNorm folding, Mueller et al. (2024) advocated for exploring non-linear mediators to uncover more sophisticated abstractions. Additional evidence comes from Kantamneni and Tegmark (2025), who found that language models represent numbers on a helical manifold. Olah and Jermyn (2024) offered an important clarification: linear representation hypothesis is not about dimensionality but rather about features behaving mathematically linearly through addition and scaling, allowing for multidimensional features with constrained geometry. This represents a relaxation of the strongest form of the hypothesis.
17

---

D Schematic of the Relation Between Notions of Causal Abstraction
Restricting
Spptt-aatticaas Corer ucthertttttt...m Ausrmcttonn
Covemnation Aberiction
Constructive
& Faaddable SC
Poue-Rentized Dlnnituted Avtraction
Distributed
C
Strong Abbrraction
Unconstraines
Restricting Interventions
4 Sama
Age
Any Set Interventions
All Input- input--estricted Interventions
All Interventions
Figure 6: A schematic of the definitions of causal abstraction $3. The axes represent an increase in how restricted the notion of causal abstraction is based on: y-axis, constraints placed on and æ-axis, constraints placed on the set of allowed interventions. Grey arrows symbolise superset- relationship: if an A-N pair fulfils the conditions in the subset, it also fulfils them in the superset.
E
DNN Definitions
E.1
MLP
A multi-layer perceptron (MLP) consists of sequence of linear transformations interleaved with non-linear activation functions.
Submodule 1.
We can define multi-layer perceptron (mlp) by choosing:
hap (x() Wox,
hap-1 = =hh,)) We (orha))) b
where Wo E BR blll We E 1x1e1,) and b, E gl are trainable parameters, and a is non-linearity like ReLU, For this model, Hi Rlldl for e < L and H,, = RRyy
In this work, we focus on MLPs used for classification tasks, whose final layer includes softmax transformation.
DNN 1. A classification multi-layer perceptron (MLP) defined like Submodule 1 but with softmax on the last layer:
Pa xx) = f (hp,)) sof teax W. h.p))
(10)
where W. & E R a trainable parameter.
E.2 Transformer Language Model
In this section, we provide a definition of decoder-only autoregressive language models (Radford et al., 2019; Vaswani et al., 2017). While many variations of transformer architectures have been developed, we focus on the original GPT-2 architecture (Radford al., 2019). We highlight that the Pythia models explored in our experiments are slightly different from the original GPT-2 and use parallel attention (Chowdhery et al., 2023); however, we do not expect this change to strongly affect results. We now define the different submodules that compose a transformer.
18

---

Submodule 2. The Embedding layer in a transformer maps input tokens to vectors:
[*(x) ex: where ex IR x|xx11
(11)
In this equation, e E R. [ is a learned parameter matrix and x indexes into its rows. II
Submodule 3. Multi- Self-Aftention with heads is defined as:
attn(h) concat(mn (h)....: Mh (h))
(12)
where each head operates in dimension dn < (abel and computes:
ns nh( softmax
hw
(13)
vdk
with learned parameters W E X W W W. E Rldd
Submodule 4. Layer Normalization applies per-feature normalization:
LN(h) = 7 h
(14)
where p. and are the mean and standard deviation across all features for a single input, and 7, B are learned parameters.
Using these submodules, we define transformer block.
Submodule 5. A Transformer Block chains together attention and MLP layers with residual
connections:
h h attn(LN(h)),
(b) = =h'+mmp(LX((h))
(15)
where mlp is applied each token activations separatly defined in Submodule 1.
Finally, we define the complete transformer language model.
DNN 2. A transformer language model consists of an embedding layer, transformer blocks, and an output layer:
h+g 1x(x) ex
(16a)
h fb(W)w))
(16b)
Pa x) - (bbp) = sof tettxa(L((hhp))) WL)
(16c)
where LN(h,,) selects the final token's 's position after layernorm is applied. For this model, Hop = [ for S l & I. and Hopits = A1
F Proof of Theorem 1
In this section, we prove our main theorem. For notational simplicity, write in this section:
Gef p
f
and
pf: def
....o f layer
(17)
up
layer
We start by formally stating our assumptions.
Assumption (Countable input-space). We assume that the space of inputs (i.e., X) is countable.
Assumption 2 (Input-injectivity in all layers). We assume that fil is injective for all layers.
Assumption 3 (Strict ourput-surjectivity in all layers). We assume that the composition of f and Tny is strictly surjective for all layers (we define strict surjectivity in Definition 10).
Assumption 4 (Algorithm and DNN have matchable partial-orderings). We assume that there exists a partitioning { E T U ( of neurons 2/2 int where respects the partial-ordering of algorithm A, i.e., 1) + n/ = Un - 1h Further, for each layer at least one neuron is left unused this partitioning, i.e., sp. the +
where vh,, are single neurons- neurons-which which
simplicity, as they do not affect our proofs of injectivity in App. Note that, in that section, we show injectivity on an entire layer's activations h Position embeddings would be needed to show injectivity on conjecture should also hold. Further, note that position embeddings used in experiments.
We ignore positional embeddings
single position, e.g., the last token's position (hw,)-1r - property which we
19

---

Assumption 5 (DNN solves the task). We assume that for any input E X, the neural network solves the task correctly, satisfying T(x) argmaxyey Pn(y x).
We provide a longer discussion about why we think these assumptions are reasonable in App. F.1. For convenience, we also put self-contained version Definition (input-restrictdd distributed abstraction) in App. F.2. Now, we restate our theorem and present its proof.
Theorem 1. Given any algorithm A and any neural network N such that Assumps. I to 5 hold, we can show that A is an input-restricted distributed abstraction N.N.
Proof. To show that algorithm is input-restricted distributed abstraction of neural network N, we must show (according to Definition 7) that there exists a T for which: A is an input-restricted T-abstraction of N; and
distributed abstraction map. For to be a distributed abstraction map, we need a partition of hidden variables which allows us to independently compute it per node. Further, we need the partitioned hidden variables wh int layer-wise decomposable. We thus have:
to be the output of an alignment map ø which
hao (h) = Ldbe (ébe(hap (hh
= { 1 E 77me U (v1)) T (h) = (h v. ( (18)
Align
Partition hidden vartables
Compute abstraction
Therefore, to define distributed abstraction map т, we must define the following three terms: (i) set of layer-wise alignment maps { (note that the alignment maps do and Φ1+11 are fixed by definition); (ii) partition of hidden variables T and (iii) a set of per-node functions {T.) To prove this theorem, then, we must show that there exists way to define these terms while ensuring that A an input-restricted r-abstraction of N.
We now note that given Assump. 4 and independently of our choice of alignment map e there exists at least one partition Te of the hidden variables 2h in N for which:
V Vn Enm :7 1f = b
1 11
VOUISL 3, g
U 4 (19)
respect parsial ordering
per partition
no layer is folly occupled
where we define & as the latent variables given when applying de on the To facilitate our proof, we choose one such partition & which we will keep fixed independently of our choice of alignment map é. Given partition ف we assign each node 7) to a specific layer as vh, contains single
hidden variable and therefore trivially belongs to a single layer. We therefore can define T)e as all nodes associated with layer &
E ne + 2 S 4
(20)
We now consider the application of interventions on as layer-wise on vb C v for v) E me. Let us therefore define I the set of all interventions on 6 for E Me, where we note that I also includes an empty intervention (i.e., no intervention). For notational convenience, we will write the set of all interventions up to layer as I and the set of all nodes associated with those layers as n:e
I = - I
x I n = U U... Une
(21)
where x denotes a Cartesian product. We analogously define I and I
Finally, we get to an induction proof that will complete this theorem. We will iteratively construet abstraction and alignment maps for each layer such that it holds that:
1ez mene : Tryy (h' = f2 (x. XEX
A
where h' =
and I, = wr (Ix)
Lested condition
pre-in:. hidden veriable, Lez
where int. stands for intervention. Note that this holds for all layers, we have proven that is an input-restricted r-abstraction of N. as we can perfectly reconstruct the behaviour of algorithm A from I's states under any intervention." Also note, however, that our definition of abstraction map restricts Tny (h$r4:) argmax h so special care must be taken to guarantee that this last identity will
The attentive reader may note condition ① only guarantees we can reconstruct the behaviour of algorithm A from pre- ineiieeevention hidden variables. Lemma shows the same holds for post-intervention hidden variables.
20

---

be preserved. We thus also require an additional condition to hold at each step:
b
*Tn (1),
where H fi Ix), and I Wr (Ix)
LEZ
XEX
tested condition
int. seurons, LEZ
Finally, for convenience, we add a third condition to our inductive proof which will make the other two conditions easier to guarantee:
3gn
: 9h worg fI( IN), where h' 40 = = delff (x, Ix)), and 1 w, (Ix)
4EZ xEX
tested condition
pre-int. hidden variable, LEZ!
This condition guarantees that information about previous nodes (i.e., 1) E 77--1) is preserved in
each layer's non-intervened neurons (i.e.. 25 n ff.) This final condition will be useful to guarantee conditions and are preserved in future layers.
Statement. Conditions ①. ② and 3 hold for all layers & DNN.
Base Case (e = 0). For layer e = O, we have 7) = 7/x- We also have both Φi and Ty as the identity function. Further, we consider I intervention-ann and consider fip to be the identity Now, it is easy to prove our base case:
{0} and I [@]-where where symbol @ here denotes an empty
(Note that layer fü is not applied in fi²)
1 follows trivially, as (x, Ix) = x and fi (x) = X.
follows from Assump. 5.
follows trivially given 7)-1 an empty set.
Induction Step (given l - 1, then L). Now, due to the inductive hypothesis, assume that !. 2 and 3 hold for layer ()- Given this, must now prove that these conditions also hold for layer We will consider two cases: 7)e is either empty or not. Before doing so, however, we note that ! and 3 hold for layer (E 1)'s pre-intervention hidden variables. In Lemmas and 2, we show that the same applies for the post-intervention hidden variables.
Let's consider the case where is empty.
In this case, can simply define de the identity map. Further, given an empty 7/8 we know that there interventions in this layer, i.e., I (0), and, as such, we have that: I I: I I We can now prove the induction step for this case.
1 is true trivially, since Te is empty.
follows using the inductive hypothesis. Let I E IV and e X. Now, let h',
Jif Iu), h'
he - (x, Iu), and IA wr (Iu) We can now show that:
Tny $ $ (1bb))
1 f x In))
definition of h',, (22a)
Tny
Tny
fk (ft (fi (x, (xx,I)))
no intervention at layer & (22b)
Tny
r (ost (x,Ixt) Inn
definition of fo
cf-1:
(22c)
definition of h (22d)
Tny
[x(x, I.)
inductive hypothesis on (22e)
This shows 2 holds for layer é when ne empty.
3 follows using the inductive hypothesis. Let In E I E X, and E 7-8-1 Now, let h Cf fif (x, In)), h' 22 = (fitt 1x, In)), and I, (In). Further, let
21

---

9n 1-1 (h'
f2 (x, In); we know such function exists due to the inductive hypothesis on together with Lemmas and 2. Finally, since fif is injective (by Assump. 2) and -
I and 3
since
f we know that each h # is mapped to a unique h' in the next we
layer. We can thus define function 1-11 on the domain formed by these hidden variables which, given a hidden variable h' #;
returns its "parent" h' -
in other words, - is an
partial inverse of f'-1 defined only on image. Defining now function gf = - M 1 we can show:
9 (h' (g = (h #.
is empty (23a)
d0 1/ 26
no intervention at layer & (23b)
d
fh (
Φ is the identity function (23c)
11 6-1
(h 1
definition of (23d)
9n (h' 1
definition of é-1
-f (23c)
= f (x, In)
inductive hypothesis on ! and 3 (23f)
Let's now consider the case when 77 is not empty.
To show 1
2 and for layer l we need find suitable bijective de. We now show careful way to construct this map which satisfies these conditions. To do so, we will again split this step of the proof into parts. We will first take of the case in which interventions layer é, guaranteeing that the model behaves correctly in those cases. In that case, we must handle the set of input-restricted pre-intervention hidden states layer É, which define as:
applied to
H
p (x. Ix)
X I
(24)
pre-int.. as we do not include
Notably, instead of defining the entire alignment map Φe at once, we will first define its behaviour only on those hidden states. We will denote this Nomain-restricter function as of Given this function, we will be able to define set of input-restricted pre-intervention hidden variables in layer l as:
daf {0r (hwe) hibe E H t
H
(25)
where represents the non-intervened hidden states and variables, and we will use + to represent the intervened instances. Note that H 3
is the of representations output by alignment map of
The second case we will consider will then handle interventions this layer, and will again guarantee that the model behaves as expected in those cases. We thus define the set of input-restricted post- intervention hidden variables as:
H
(x, 1
x X, In E }
(26)
pest-int.. include
Notably, what intervention layer l does is re-combine the representations in H. 402
We can thus
write H Hyp in terms of H 4 as:
Hogg
(er (baree b E H }
(bree hyg E H
(27)
T/É
pre-int. h.v., projected to w.)
pre-int. a.v., projected to wf
We further define the set of input-restricted intervention-only hidden variables as:
7
H 3
H 1
(28)
1
22

---

By carefully defining the behaviour of de on this set, we can guarantee the conditions above to hold. In particular, we will define this part of the function via its inverse of which maps these hidden variables back to hidden states. We therefore have Φe and its partial inverse defined as:
(b), Φe h) = h))
if h bz
hh), if h H
$5 (h) =
(29)
if h E H
(h). if he H
We now define og.
Definition 8. Partial map of H the dimension, Le., (10)) V bu h1 * h of (h1); + 07 (h2);
- R w0 some fixed function that is injective on each
Such a function exists, because H,, is countable (Lemma 4) and R is uncountable. Further, its partial inverse o defined on the image 7- wi exists because of is injective.
We can now prove that conditions ! and 3 hold. We also prove that 2 holds when I E E i... when there is no intervention layer L.
① follows using the inductive hypothesis. Let E I x E X, and me. Further, let (f:f-1 (x, In)). and I, = wr (Ix). Now, note that there exists function In, for which e, - (V .). as the parents of are subset of 77 It now suffices to show that h' encodes information about v [ (x. [.)eene- By #
h' &e(fi (x. In), h' 3 (fi
th
the inductive hypothesis on and together with Lemmas I and 2, we know that h' 22
encodes information about Vijas let In.e-1 be a function that extracts this information, i.e., -n.-. (h' Vn----- Now, since fif is injective, and of is injective on each output dimension, we know that h'g o (1/ ( (h' ))) Wut contains the same information as h' #2:
We can thus construct (partial) inverses 1/-1 L
and define T, as E which concludes this step of the proof: definition of h' (30a)
the composition 9., -: de- 1
Ty [h' wy (+pl = T, (1o - (h M)}
(9 (1 (01-1
(lo ( 22 111))))111
(30b)
= -9 22
(30c)
(x,
(30d)
2 when ly I follows using the inductive hypothesis. Let Ig E I and x X.
Now, let h'ye =fi (x, Lu). Db = fx, Iy), and L, wr([x), We can show that:
Tny $ (((b))))
-ny
f (f (f
definition of h' (31a)
Try
c 1 (fit (x, II)
no intervention at layer l (31b)
Tnv
f= Fiz (xc In)
definition of ff11:
(31c)
E-1 (1' (Dbren)
Try
definition of h'
(31d)
fa(x, Ia)
inductive hypothesis on (31e)
This shows 2 holds for layer l when there is no intervention in layer l.
follows using the inductive hypothesis. Let Iy E I x E X, and 1 E 176-1- 221 -ch- (f:t-1 (x, In)), and Ia w- (In). Further, let
Now, let h' #/ Ge(sif (x, Ix)), h'
23

---

(x, we know such function exists due to the inductive hypothesis on 9 (h' 1 ) (x,,
! and 3 together with Lemmas and 2. Finally, since f and o are injective and Φx- is bijective, we can define the partial inverse function tu f-1 of their composition of o f o Φ (applied only to their image) which, given the hidden variable h'
a returns its "parent"
h' P Defining now a function T = gh - Va
which and g (h' wynw? 9 (h' we
exists, of is injective each dimension we can show:
g (h' 10 (grpee = 32 (h'
is injective on all dimensions (32a)
Th
J1 +2
18-1
(0 (h'
48 1))
no intervention at layer & (32b)
Ihh
in (0t (fh ( (h'
definition of g (32c)
Un f-1 wE
definition of 1 {-1 (32d)
= - In)
inductive hypothesis on 1 and
(32e)
We have now proved 1 and 3 We have also partially proved 2 for cases where there is no intervention in layer f. 13 We now finish our proof by considering cases where there an intervention in this layer f. In the second case, we need handle intervention-only representations He We will
now define of -1 on this domain H to fulfil 2
Definition 9, Partial map of
H
49 is some fixed function such that it holds:
1. of maps to the set H.v H
2. ot is an injective map
3. Let In E I I and x E X. Now, let h' 48 S (x, and I = =wr(T)) We have
that ( ( (h)) = L.(
Where the first two conditions ensure the necessary bijectivity of oe and the last characteristic ensures
2/8 (x, Iy), and 2. Now, let & be any input and In I any intervention. Further, let h' a =
I = wr (Ix). We now note that that-given 1 and 3, and Lemmas and 2-the value un, = f (x, L.) for all nodes 7) ne are encoded in h' we This is enough information to determine the algorithm's '
output y = fx I.). Now, define a function 9" which maps an element h E 7
the output
algorithm expects. Further, by Lemma 6 there exists an uncountably infinite set of hidden states Hl>") such that:
vh H(Y") the
y argmax [S (har)ly y'Ey
(33)
We define A((" = H H
by which- as H., is countable-is still uncountably infinite. We can
now map any h E H to an element in 7i(g" (h)) fulfilling the third characteristic of Definition 9, th
"
That such a mapping exists adhering to the first and second characteristic of Definition 9 is ensured by the fact that R((x") is uncountable and 7 is countable (shown in Lemma 5). Further, as of v
is injective, its partial inverse of on its image H.S exists.
Thiis also proves the result for any input x E X and intervention 1, E I' where E 2... By 3 at layer there exists E I constructed by applying only the interventions from on layers such that fi (x, Is) = fif (x, I, Since both encode the same values on nh according to 3 which fully determine the output of /.. and since 2 holds for (x, 12) at layer É, must also hold for Is).
24

---

def extract_ extract_unused (h, H 20g wp):
U H
2
rep-h
3
while repé H UH
wy:
rep-0 (rep) return rep
Figure 7: Pseudo-code for extract unused rep. This function returns an unique element in (H to UH 4,,
) (HH UH$ for each h (H UH* (HH UH* u,, ensuring bijectivity of (h).
The attentive reader may have noticed that we defined de only over the domain H. U H instead over Bl We note that it is simple to extend by to an d' defined over Rlvel, Let id be the identity function and extract _uccctd__uuudd__ee rep be defined by the algorithm given in Fig. 7. A bijective function &y over givl mapping to glvel can be defined as:
(b)
if h E H. UH (H UH
Uy
((h))
extract sttpauuuuuuuuu__eeeeee H of
UH
w
if h E (H
UH+
w/
h
else
(34)
which completes the proof.
O
F1 Discussion about Assumptions
Assump. (Countable input-space). While this assumption cannot be made on all neural networks like MLPs, it holds for models working language and images. The of all images with specific resolution is finite, as it considers finite number of pixels where each pixel has a finite number of channels (e.g. values for red, green and blue) and each channel is number between 0 and 255. The set of all sequences in a language model is also countably infinite, as each set of sequences of some length finite given finite tokens; so we have set made of the countable union of finite sets, which is still countable.
Assump. 2 (Input-injectivity in all layers). Neural network layers (e.g.. MLP blocks) are not necessarily injective. The usage of learnable weights, activation functions like ReLU (Nair and Hinton, 2010) and information bottlenecks makes it possible to have a non-injective model. However, we prove App. G that transformers, at least, are almost surely injective at initialisation on their inputs. Further. Nikolaou et al. (2025) recently published a proof-as well as empirical evidence- that transformers are almost surely injective the hidden states of their last token, both at initialisation and after training. We also see in our empirical experiments in App. H that the MLPs we analyse are also, practice, injective- or close enough to it that we observe no collisions in embedding space.
Assump. 3 (Strict output-surjeecivity in all layers). Surjectivity can be defined the output distribution fh: Hos - A AD1-1 but that is rather strong assumption. For proofs, will rely on strict surjectivity the classification space instead (Tny fti Hab, + (!),, such that every class can be predicted. However, surjectivity on the classification space still does not necessarily hold for DNNs. LLMs have problems like the softmax bottleneck (Yang al., 2018), which can lead to model having insufficient capacity to predict all possible tokens. Grivas et al. (2022) also evaluate and find this problem, but show that surjectivity the tokens is still likely in practice, making this reasonable assumption in LLM settings.
Assump. (Algorithm and DNN have matchable partial-orderings). We assume this since, for a neural network N to be abstracted by the algorithm A, need it to have this minimal width and depth.
Assump. 5 (DNN solves the task). We assume this because, if a neural network does not solve the given task, it will also not be abstracted by an algorithm which implements it.
F.2 Detailed Version of Definition 7
Definition 7 can also be written without referring to previous definitions as following:
Alternative Definition (Equivalent to Definition 7). An algorithm is an input-restricted distributed abstraction of a neural network Ni iff there exists T, Ia. and Ig such that
25

---

T is distributed abstraction map. Lee, there exists alignment map é. partition {apa E 7am U [bb]] of &h (with non-empty 2b%), and subabstraction maps {T,, 17 E Tint such that T is equivalent to computing the value of each node block-wise with Tyy (h bg) An alignment map à is a bijective function that maps the inner neurons innt of N onto an equal-site sized set of latent variables Vine. with & respecting the network ss computational order by being the combination of layer-wise bijections de IR ael - Rlvel applied to the neurons of each of the DNN's layers (E);
I, and In are a maximal input-restricted intervention set. maximal input-restricted intervention set is composed of all interventions produced from other input-restricted interventions, i.e., it is set with hgo +- Cupé (where up v Vn f- Cry (where 1j S
M/mt) where Cipe Cn arise from valid input-restricted computations (e.g., Cupa Cn f Cx, C. fi"(x))
pbb (x)
T is surjective;
IA = w, (Zn);
There exists a surjective Typ such that
=T(h (x, Is)) =fa - (Tri (x), where I, w (Ix)
(35)
XEX
GZ
F.3 Useful Definitions and Lemmas for Theorem
We say the composition of function f Rdd - ADMI with argmax is strictly surjective if. for any output y" E E,, there exists an input h E IRd for which outputs y no matter how ties are broken in the argmax. Formally:
Definition 10.
Vy" y, 3h R Vy' EV EyXyy'> [f(h]ly- [f(h)ly
(36)
Lemma 1. Let N be a DNN and A be an algorithm. Further, let t be a distributed abstraction map with partition V and {T, memss If. for all 1 Th Ty satisfies the conditions in I (defined in Theorem I's proof) applied on layer l's pre-intervention hidden variables, i.e., if:
V. 1 X
(h' 4 fxx, (x),
where h'
22 Ix)) and I = w.y (In)
(37)
47
tested condition
pre-int. hiddon variable, I,I
where we note that fx de o fif when no intervention is applied to layer f. Then Tú also satisfies this condition when applied to layer l's post-intervention hidden variables:
= Ix),,,
where h' = S 26 (x,Ix)) In)), and I, = wIn (38)
XEN
tested condition
post hidden variable, LEZ
Proof. Let Ty be the abstraction map of 1) me- By assumption, condition 1 holds for all pre- intervention hidden variables, i.e., hidden variables of the form h vee show the same function applies to post-intervention hidden variables, i.e., hidden variables of the form:
[de(fi [1)g We can
{ { (oot In)
if h' 4
In
h
(39)
else
Now let E Ií be any intervention and x E X' be any input. Further, let wir (In). If I then the post-intervention hidden variable is identical to a pre-intervention one, and the conditions in I still hold, i.c..: h' : = [ o (x, k))]pp and T, is such that 7,' 6 f2 (x, I.). If & T:4-1
for each node's 's hidden variables v might not intervene it. If we do not intervene node 7) then we still have the case h' v Le.. Tr (h' we)
[fe( (x. Iy)] & and thus T, still gives the correct solution,
(x It). If intervene on 7), then we know there exists intervention of form
26

---

D + fif((' [2) in I,, for which I I as our interventions are input- We also know (by eq. (4)) that there exists an equivalent intervention Cn t TT, (T [.) ) in L.. We thus have that Tyy (h' 1 (x(x,I,)
Lemma 2. Let N. be a DNN and à be an algorithm. Further, let be distributed abstraction map with partition V. If. for all E 7 there exists a function Oy which satisfies the conditions in (defined in Theorem I's proof) applied on layer l''s pre-intervention hidden variables, i.e., if:
V 4 1 XEX
9 Verue) II),
where
h h VE
fu f (x, In), and I, = Wi (Ix)
(40)
bested condition
gre-int.
hidden variable, as LEZZ;
where note that Ja de l' when no intervention is applied layer l. Then In also satisfied this condition when applied to layer E's post-intervention hidden variables:
V
9g (gnne)) fx(,
where h' 1 fn # (x, and I w. (Ix)
(41)
XEN
tested condicion
post-Int int. hidden variable, as LE
Proof. Let 17 7a--1 and 9n be function that satisfies condition 3 for it. Note that condition 3 holds for all pre-intervention hidden variables, i.e., hidden variables of the form h'
[de(Ji (x, 10p7pp We show the same function On also applies post-intervention hidden variables, i.e., hidden variables of the form:
C Cog [fe(fi [)) 2/
i if h' vingg
C
wone Iu else
h WEnNT -
(42)
Now, let IN E I x E X. Further, let I (In)... If In E I then the post-intervention
hidden variable is identical to a pre-intervention one, and the conditions in 3 still hold, i.e.;:
h #/ [f [x]]] [
gggnng nas and 9,, is such that gn (h' mgn fx I.). If & I 8-1
means that we intervene on at least one hidden variable in this layer 2b. However, we never intervene on neurons in Th meaning that for those still have the case h' wonwr [obeSi (x, Iu)) ( fnwf
and thus the same function 9n still satisfies our condition 9,, (h' vgrwe) Ix)
Lemma 3. Under Assump. 1 and given a fixed o, the set of input-restricted interventions Iy is countable.
Proof. This can be shown by induction. More specifically, we show that for any layer l, the set of input-restricted interventions I is countable for a specific
Base Case (L = 0). The base can be proved trivially, I {0}.
Induction step (l given 1). By the induction hypothesis. I:f-1 is countable. Now, note that I can be decomposed as:
I = I
(43)
As the Cartesian product of two countable sets is itself countable, and as I is countable by the inductive hypothesis, we only need to show that I is countable to complete our proof. This set I defined as the set of all insut-restricted interventions layer é. Given set of neurons hidden variables in this layer æb', we are thus dealing with interventions of the form: hy + f (x, I,), where: (i) 2// C d. 2/ C u (ii) X; and (ii) I, E I The set of all input-restricted interventions in this layer thus bounded in size by the Cartesian product: X X I
These three sets are countable, and thus so is I This concludes our proof.
Lemma 4. Under Assump. and given a fixed o, the set of input-restrittdd pre-intervention hidden states in layer E, i.e. H is countable.
27

---

Proof. The set of input- input-restricted hidden states Hag is formed by hidden states h = fx (x, In). which we can write as:
H { (x, In) x I I
(44)
We thus have that the size of H bounded by the size of the Cartesian product X I As both of these sets are countable (by Assump. and Lemma 3, respectively), H.pe is also countable. This completes our proof.
Lemma 5. Under Assump. I and given a fixed Ø, the set of input-restricted intervention-only hidden variables in layer f, i.e, H. is countable.
Proof. A similar proof to Lemma applies here. In short, have three relevant sets for this proof. First, the set of input-restricted pre-intervention hidden variables:
H #
{ h E Hope
(45)
Second, we have the set of input-restricted post-intervention hidden variables:
H
X
(or(hue)
ha E H
x (oe(hre) hig E H
(46)
EEE
Pre in.. h.v.r projected to wi)
Pre-int. h.v., projected to 4
Both sets above are countable, since H. is countable (by Lemma 4), and me is finite. Third, we have the set of input-restricted intervention-only hidden variables, defined as:
H
H
7 2
(47)
Since H. wg is countable, H #/
is clearly also countable. This completes the proof.
Lemma 6. Under Assump. 3 and given target output y E y we know that there is an uncountably infinite set H(y") which predicts it, i.e.;
(x)) h E b1 yy* = argmax (h)]y
(48)
y'Ey
Proof. Under Assump. 3, we know that for any target output y* E y -heteee is at least one hidden state h² E RRvvel which predicts it, i.e.:
y argmax [f (h")ly
(49)
y'Ey
where we note that / (h*) outputs a probability distribution over v, i.e., Paly' h²).
To show that we have uncountably infinite set, let us first notice that
Vh' e gled. (b) fl (h')lla
m.
m2 = y
argmax Y'EY'
[f (h')ly
(50)
for my be the max value of f (h) and my the second highest value of (h). m, my follows by the strict subjectivity mentioned Assump. 3. Eq. (50) follows by the definition of the euclidean
norm (II-112), argmax and ff (h') E A l/2 -1 as m, has to be lowered at least mi m2 to increase 2
Whs ---ma for those two values to be the same. Increasing any other value in (h) would m2 by 2
require mi being lowered more than m-=ma or any other value increased by more than m. m2 Now, given continuity of neural networks, we know that:
Ve > 0, 38 > 0, 0, E Rlvel 0 < | -hll22 <
1 (h) (b'lla <.
(51)
28

---

Therefore, we see that:
36 > Vh E ER 0 < ||h - 1 <
6,
115 (h) f (h') 12)1 m1
(52a)
2
- y argmax f (h')ly
(52b)
y'Ey
We notice that ||h 12 < 6 for > denotes a continuous region in Blld which therefore includes uncountably infinite points.
G Transformers at Initialisation are Almost Surely Injective on each Layer
Theorem 2. Transformers like DNN 2 with randomly independent initialised from a continuous distribution (riicd.) weights are almost surely injective at initialisation up to each layer 0 S & L.
Proof. To show injectivity up layer l' in a transformer, it suffices to show that fh is injective on any countable subset H its domain for all layers é (0 S E S ()). This suffices as we assume the set of inputs X' is countable, and the composition of injective functions is injective. Let @ be the random variable representing the transformer's 's weights. To show (almost sure) injectivity on layer é for any fixed imput set H., need that (because of Lemma 10):
pe (Vh1, h2 H. h1 F h2 f # ((
(53)
Since the transformer operates over sequences of tokens, any element h E H. has its first dimension indexing the sequence length. Let [h| denote the sequence length and [h], refer the t-th element in h. Let T be the set of token positions T = {I...., min(miin))))))) h1), [b2))} For injectivity, it suffices to show that:
pe (vh1, h2 E H,t E T, [bi]: * [ba]: : (h.)) * (b2()))) =1
(54)
Note that eq. (54) only ensures injectivity when [h, [b2]. However, this is sufficient because when h1 o [h2), eq. (53) follows trivially: since [fr)) A (b2)), we immediately have ((h)) ) * / (h2). When |h|| = [hz), we can show that eq. (54) implies eq. (53) as follows: if hi of h2, then there exists at least one token position l' E T where [hi]e- o [ha]er- By eq. (54), this implies (hr)) + (hz))er almost surely, and therefore fi(h;) + f(2)) almost surely.
We observe that a transformer's input set X' consists of all sequences formed from a finite token vocabulary, which is countably infinite. Since transformers are deterministic functions, the input set H encountered at any sublayer is also countably infinite. Therefore, it suffices to prove eq. (54) for any fixed countably infinite input set H.
We show that Eq. (54) holds for any fixed countably infinite subset H of the layer's domain. This is established for the embedding layer (f& (h) en). the MLP layer (f (h) h h+mlp(LN(b))), and the attention layer (
attn(LN(h)) (h)) ) by Lemma 7, Lemma 8, and Lemma 9, respectively. Li
The 3 theorems facilitating the proof above are:
Lemma 7. Lets assume we have an embedding layer randomly independent initialized from continuous distribution (riicd.) weights and any countably infinite input sets (in embeddings token indexes). We denote the set of random variables over the weights as Q. We then can show for any fixed countably infinite input set H that this Layer injective almost surely.
(a ha H. T, [hi]: # [b2]a: [en,]e le # [eng]t) =
(55)
Proof. See App. G.2.
Lemma 8. Lets assume we have sub-block consisting of MLP with residual connection and layer norm (i.e., h + (mlp(LN(h))) with riied. weights. We can show that, for any fixed countably
WWe note that pz (Vz Z 2). where Z is of events, is the same pz (7aez (=}) formally.
29

---

infinite input set H. this layer is injective almost surely:
pe(yh,, h2 E H, 1 E T, [hi]e + [ba]e:
(56)
h1 + mlp((M(h:))) # [h2 mlp(LM(b2)))))
Proof. See App. G.3.
Lemma 9. Lets assume we have a sub-block consisting of a self-attention with a residual connection and layer norm (i.e., h+ (+aat((LN(h))) with ricd.. weights. We can show that, for any fixed countably infinite input set H, this layer is injective almost surely:
pe(, h2 E H, E T, [hi]e + [ba]e:
(57)
[ + attn(LM(bi))l # [hg attn(LM(hg))l)
Proof. See App. G.4
G.1 Fundamental Lemmas
In this section we present some fundamental lemmas used to prove G.2 to G... Lemma 10. For layers function f/ to be injective on its input set H. it has to hold that:
pe(vh1 h2 E H. h1 * h2 (h1) # f (h2)) = 1
(58)
This can equivalently be written as:
pe (vh1, he E H. h1 # h2 (bh)) # (b2)) =
(59)
Proof. We can derive eq. (59) from eq. (58): pe (Yh1, b2 H. h1 + h
f (h1) + f (b2))
pe
hi + h2 + ((b1) + (b2))
(60a)
h1.
pe
n
{(hh + h2) ^ (bhi) # f(h22) v (hi =ha)]
h2)}
(60b)
pe
n b,, #h2
(hi # h2) A fā(hi) + A(bba)3 n
n
((h) = h2)}
hb))
(60c)
by E W. =h
always true
(hi (hi h2) A/i())) 1(((b)) })
pe
(60d)
by by
always
= pe
(f((h)) + f((b))
(60c)
hy b
pe (Vhi, ha E H, hi + h ((bb) # f((h))
(60f)
Lemma 11. we have countable set Z of almost sure events know that their intersection is also almost surely. Formally:
Vz 2:pz) =1) (
(61)
30

---

Proof. First, observe that
-1=)=1-p((
((==)))))
((
( -1 LEZ
p
U :
(62)
SE
se2
where z the complement of event z, Since p(=) 1 for all = Z, follows that p(=5) 0 for all E 2 By the countable subadditivity of probability measures:
p =
-(=)= = -
(63)
:EZ
ZEZ
2E2
Therefore,
p(Vz E 2:2) =
2
= =1-0=1n
(64)
:EZ
G.2 Proof of Lemma 7
In this section, we will prove Lemma 7 which states that the embedding layer is almost surely injective on countably infinite inputs.
Lemma Lets assume we have an embedding layer randomly independent initialized from continuous distribution (ricc..) weights and any countably infinite input sets (in embeddings token indexes). We denote the set of random variables over the weights as O. We then can show for any fixed countably infinite input set H that this Layer is injective almost surely.
pe (v1, h2 E H,t E.E.P T, [ha]: + [22]: [cn,) * (cha]a) =1
(55)
Proof. We can apply Lemma 11 three times (on h1 h2 and t) to show that eq. (55) is equivalent to, for any h1 b2 E H and E T for which [hale * [b2],, it holding that:
Pe([en, le + (c),.r)
(65)
We further note that, by the definition of embedding block (Submodule 2):
pe (e * [eh = + (e * elbai,) = 1
(66)
We can thus apply the law of total probability by defining O' as all the random variables @ except the one for the first element ep, i.e. except [em and (b as the embedding [hi]e without the first element:
/ pe * Ohull
Cjaje C(ha), \eer (c/nn). U (d(hh) U((())))) = (67)
therefore suffices to show that, for any e/m,), and bali
pe ee # e/azle (fnnle:
(68)
This holds trivially when any embedding dimension other than the first of obh.). and @[bz]e differs. When all dimensions except the first are equal, we apply:
Peter ((e * [ep e/n,je (halt)=
(69)
Pe Pe),er(om).). [e)n))]11 c/n.). ((ha), })
The right-hand side [e]na]_h l a constant while the left-hand side [em.)..1 ]1 is a random variable over continuous region; this event has measure 0, resulting in probability 0.
G.3 Proof of Lemma 8
In this section, we will prove Lemma 8, which will show that the block consisting of an MLP, residual connection and layer norm is almost sure injective its countably infinite inputs.
Lemma 8. Lets assume we have sub-block consisting of an MLP with residual connection and layer norm (i.e,, h + mh)) with riced. weights. We can show that, for any fixed countably
By this refer to the first element of the embedding vector of [ha]e.
31

---

infinite input set H. this layer is injective almost surely:
pe(yh,, h2 E H, 1 E TT, [hi]e + [ba]e:
(56)
h + mlp(LM(h:))) # [h2 mlp(LN(h2))l)) =
Proof. For notational convenience, let m, = ml,(LN(h.).. Given Lemma 11, it suffices to prove that for any h1 h2 E H and E T, where [hı]e * [ha]e, we have:
pe (br + m. + [he + ma]c) =1
ma)+)=1
(70)
Without loss of generality, fix one such h1 h2 H and E T. We can manipulate this probability distribution as:
pe [h] + milt + [h2 + male)
= Pe) [hi]: [mi] * [ba): + [m2l:)
-pe
[mi)e * (m2]e) pe [bi]) + [mi]: + [h2] + [ma]e [mi]+ * (m22.))
(71a)
+ +p
[mi]] = (m²]))) pe [bi] + [mi]e * [h2]e + [mz] [mma]e = [male]
since [hile>
pe [mmz]e * [mm21]] pe [halle + jmale * [bale + [m2] [ma]e of [mele]
(71b)
+ pe [myle = m22)))
Therefore, suffices to show that:
pe
[hi]e + [m,] * [hale + [ma]e [male * [mele)
(72)
since pel [mi) * (m²].) + pe [mi]e (ma):)) is trivially 1. We now unfold the last layer of
the MLP as m, Wz(m!)) + by, where m (fit- (LNhh))). We can rewrite eq. (72) as:
pe [h1]: + [W, mm + bu]e # [b2] + Wumm + bL.l [mi]e * [ma].)
-p [hi]. + Wz[m<l + [bo]: + Wz|mále [mi]e * [ma]]]
(73a)
@ od
[h.). W Wy [m/] * bbl: Wz|mble [mi] * [mb][e.))
(73b)
2 pe
(hilk.1) + W m| | * [2l| + [W w.mbltaaj
[mile * [mblt..)
(73c)
[bale
[hald
pe
[brtk.1] +
W Whb mm m/ * [alka1] +
W W Wehsslme (mblaal [mfla. * [mbl(t.3]
(73d)
(3)
{ per
[bi]aaa] +
( veluaajmm # [h2]e.1]+
(73e)
ee
|lbalel
1=1
Σ
Wellaa
[mb]
[m]] [.. * [má](a.]] 0) pe [mi] A [m (r..) )d0
where equality (1) holds since [myle * [mz]. implies there exists some index such that [m]] [p.4] * [mb] (2) holds because if the inequality is satisfied for a single component of the vector, must also be satisfied for the entire vector, In (3), we define O' as the random variable responsible for the value of [W tln, and 0 as realisation of the random variables θ O'. Therefore, to prove
16 ([mí]]:.) represents two dimensional indexing. referring to the i-th element of the representation of the 4-th token.
32

---

eq. (72), it suffices to show:
[hild
[ W W.ll |
per (Dilea.) +
Wrl # [hala..]] +
[mi]]t.]] * [mb]lc.,, 0)
= 1
(74)
For brevity, we omit repeating the conditions in the following probabilities as they remain unchanged to the previous equation:
Daald
hall
per
[bilea11]
+
[W ['lp [mi * [b2]
+
W ( l...)
(75a)
==1
[[b
hasld
+ pe
(bal +
W Welaaaii [mil] 1elea.11 +
W pelaammmlee ....) = 0 (75b)
3=1
+ per
(m/))
I1.a - [
Welu.glmblee
= [ba]t.. - [hi]p.1]
(75c)
Σ
[Wilp.)
[mb]
Σ
[W p.p.apmmlee I...) O
j=1,
1=1
+ per (Welpa,
[belaa]] [hi]p.1]
(75d)
[mile.. - [mb]re.
[[hald
[hald Σ
Σ
W (mile.))))))))) 1...) -0
J=1,jşi
f=1,A
where the last step follows from the condition [mn] A [mm]je.s which ensures the denominator is non-zero. Now, Eq. (75d) holds because the right-hand side is constant (since its elements are fixed given the conditions of the probability) while the left-hand side is random variable drawn from continuous distribution (since the weights are riicd.). Therefore, the probability that this equality holds is zero, the event has measure zero.
G.4 Proof of Lemma 9
In this Section, we prove Lemma 9, which establishes that the self-attention sub-block (consisting of attention, residual connection, and layer normalisation) is almost surely injective on countably infinite inputs. The proof structure parallels that of Lemma 8, so we highlight the key differences and necessary adaptations without repeating the full derivation.
Lets assume we have sub-block consisting of a self-attention with a residual connection and layer norm (i.e., h attannN(hht)t with riccd. weights. We can show that, for any fixed countably infinite input set H. this layer is injective almost surely:
Lemma 9.
pe(vh h2 E H,t E T, [br]e # [b2):
(57)
h + attn(LM(hi))} + [ha + attn(LN(b>))l)) =
Proof. We follow proof strategy analogous to that of Lemma 8 in App. G.3. Following the same steps up to eq. (72), suffices show for this lemma that for any h1, h2 H and E T, where [br] * [h2].. we have:
pe [bs]e [attn(LN((;))) # [h2]: + [attn(LM(h:))) (attn(LN(h;)) )) * |attn(LX(b>)))]
=1
(76)
We can write this according to the definition of an attention block (Submodule 1):
pe [bi]e + (W") [hi]e * [b2] (W hgle [attn(LN(b,))) * [attn(LN(b2)))) (77)
where h and hé are the hidden states after concatenation in the self-attention mechanism (see eq. (12)). The remainder the proof follows the same approach as the proof of Lemma 8 App. G.3. starting from eq. (73a).
33

---

All Pairs
Same Output
Not Same Output
Same Variables
Not Same Variables
Input
8.5e-2ss 1:le=2
8.5e 2 Lile22
1.Te- 1 1.0e-2
8.5e 2= 1.le=
1.6e- = 1.00-2
Layer
5.7e-4 44
5.7e- 4 4.6em4
2.4e 200-2=0.00 20.003
5.7e 4 4.5emd
1.1e-2=400
Layer
4.5c 113.6-44
4.5c
3.
3.2c 1.20-22
4.5c 1 3.s
1.2c 2.
Layer
3.3c 4 2.80-
3.3c 1.2.0-4
8.9c
1:136-22
3.3c 2.80-
1.5e 2 2.100--2
Table I: An approximation of the minimal Euclidean distance of the trained MLP model in the hierarchical equality task using 1,280,000 samples. The minimal Euclidean distance computed between all samples to randomly selected subset of 10,000 samples. We compute it over 10 different random seeds and present the mean and standard deviation (the number after =).
H MLP Injectivity in Hierarchical Equality Task
We see in Fig. 2 that the IIA remains low for the i identity of first argument algorithm a fully trained model even when using a granlin alignment map (based on revnet). A reasonable assumption for why would be that the fully trained model does not fulfii some assumption required by our proof of Theorem (any algorithm is an input-restricted distributed abstraction for any model) given in $4. In this section, present follow-up experiments investigating the reason for this disagreement between our empirical results on the identity of first argument algorithm and the theoretical result of Theorem 1.
Let us first note that to prove Theorem we rely on an existence proof: showing there exists a function 6 which satisfies the conditions for DNN to be abstracted by an algorithm. It says nothing, however, about this function being learnable in practice. Our experiments, however, measure IIA on an unseen test set-which requires gmonlin to not only fit training set, but generalise to new data. Therefore, following our proof of Theorem we explore the IIA on the train set. However, on the normal training set (with 1,280, 1,280,000 000 samples), we still do not get an IIA over 0.55. On the other hand, we repeat the experiment with only 1,000 training samples, we see d achieves an IIA of over 0.99 the training set. Therefore, it is likely that the revnet used when defining nnonlin does not have enough capacity to fit the overly complex function our proof describes.
To further analyse why the capacity of the used revnet is not sufficient, we analyse the injectivity of the evaluated MLP by investigating its hidden representations. We first evaluate 1,280,000 0000 randomly sampled inputs and their hidden states, checking if they are all unique. In these 1,280,000 samples (and repeating this experiment with 10 different random seeds), no collisions were found, implying the evaluated MLP is (at least close to) injective.
We now examine the supposition that the model finds it more difficult to distinguish between hidden states that share the same values for the variables in both equality relations than between those that do not. To this end, we compute the minimal Euclidean distance between hidden states across the entire set of 1, 1,280,000 samples to randomly selected subset of 10,000 samples. Specifically, we measure the minimal pairwise Euclidean distance among: (i) all sample pairs, (ii) sample pairs sharing the same output, (iii) sample pairs sharing the same values for both equality variables both equal lity relati inn,, (iv) sample pairs that do not share the same output and (v) sample pairs that do not share the same values for both equality variables. The results are presented in Table 1. We observe that the minimal Euclidean distances are smaller for pairs sharing the same output the same varialee values compared to pairs that do not. This suggests that, although injectivity is preserved, a RevNet likely will find it more challenging to separate hidden states that share variable values.
I Additional Experiment Details
In this section, we present additional details about our hierarchical equality task (in App. I.l) and indirect object identification (in App. I.2) experiments. We also present details and results on the distributive law task (in App. I.3).
L.1 Hierarchical Equality Task
Task 1 (from Geiger et al., 2024b). The hierarchical equality task is defined as follows. Let x = X1 o X2 o X3 X be a 16-dimensional vector, where each x; R2. for E (1,2,3, 4), and denotes vector concatenation. The input space is X [-0.5, 0.5] 16, and the output space is
34

---

y' = (false, true). The task function is:
T(x) = ((x1= -- X2) (X3 = X1));
(78)
where the equality (x; == xj) holds if and only if x; and xj are equal as vectors in RR.
L1.1 Algorithms
We define the following three candidate algorithms in detail.
Alg 1. The both equality relations alg. to solve Task has Tinner (/x/==xa//xx===>> and:
fla-- (Vm) (Unxs = Unia
fx f (Vpar, (ear,,(ee (0xg--x-) = (Unees = Unyes
S (pprr((n)) = (Uim---
Ciing-maa
Alg 2. The left equality relation alg. to solve Task has V/innar (1xx== and:
f (Vpar, (1x = (UNKs = Unnu
fh (Vpar, (Une)=
(Uiies Uryma))
Alg 3. The identity of first argument alg. to solve Task has Minner- {1/x3) and:
fT (par, (1x.) =X1
fT (par, (Tp) = ((1g)) == X2) == (X3 == xx))
L1.2 Training Details
|ubyl 16. The model trained using the Adam optimiser with learning rate 0.001 and cross-entropy loss. We use a batch size of 1024 and train on 1,048,576 samples, with 10,000 samples each for evaluation and testing. Training runs for maximum of 20 epochs with early stopping after epochs of no improvement.
For the hierarchical equality task, we use 3-layer MLP with 1
= |
For the training progression experiments, we use the same configuration but limit training to 2 epochs.
When training the alignment maps d, we with early stopping after 5 epochs of no improvement (using threshold 0.001 for the required change, compared to 0 for MLP training). We use the Adam optimiser with learning rate 0.001 and cross-entropy loss. To generate the datasets for DAS, for Alg. we intervene with a probability of 1/3 on 7x, 1/3 ->,;==x, and 1/3 both variables. The samples for the base and source inputs are generated such that (x) == X2) and (X3 = x4) each hold 50% the time. For Alg. and Alg. 3 we intervene !x1 == and 7x, for all samples, respectively. For each algorithm, sample 1,280,000 interventions for training, 10,000 for evaluation, and 10,000 for testing.
batch size of 6400 and train for up to 50 epochs
I.1.3 Additional Results
We present results for the three candidate algorithms for the hierarchical equality task, analysing the effect of hidden size d., and intervention size | across all MLP layers.
For the both equality relations algorithm, Fig. 8a, and demonstrate that the hidden size experiment aligns with previously reported trends, while also showing how alignment maps & of increasing complexity perform across training epochs, layers, and intervention sizes.
For the fe equality ity ielation on algorithm, as shown in Fig. 8b, 9c and 9d, we observe similar patterns: increasing hidden size and intervention size improves performance, and alignment is generally more successful in later layers during early training.
For the iccntity of first argument algorithm, Fig. 8c, 9e and 9f reveal that, interestingly, some alignment is achieved esseciallespeciallyl in layer 3 during the first half of training, but this effect diminishes in the second half.
Overall, these results demonstrate that the hidden size experiment is consistent with the findings reported in the main paper. They also show that it is easier, in untrained models, to find an alignment map for later layers, and that transient alignment can occur specific layers and algorithms during the initial stages of training.
35

---

Layer
Lager 2
Layer
des
dm
des
(a) both equality relations
Layer
Accuracy
4e
de
(b) left equality relat
Layer
Laver
Layer
Auquzecy
8:30
0:40
Am
(c) identity of first argument
Figure Mean IIA over 5 seeds using daonlin (I.m = 1) on the trained DNN. Performance improves with larger hidden dimension d,n and intervention size ( Each subplot corresponds one of the three candidate algorithms for the hierarchical equality task, showing how the model's representational capacity influences performance.
I.2 Indirect Object Identification Task
Task 2. The Indirect Object Identification (10I) task involves predicting the indirect object in sentences with specific structure. Each input X consists of text where subject (S) and indirect object (10) are introduced, followed by the S giving something to the I0. For example:
"Friends Juana and Kristi found a mango the bar. Kristi gave it to" > "Juana"
Here, "Juana" " and "Kristi" are introduced, with "Kristi" (S) appearing again before giving something to "Juana" (IO). The output set y consists of the first tokens of the two names:
y = (tirst_tokenSS) first_tokentt
(79)
L2.1 Algorithm
For this task, evaluate the ABAB-ABBA algorithm. Denoting the names in the story A and B, this algorithm determines whether the sentence follows an ABAB pattern (c.g., "Friends Juana and Kristi found mango the bar. Juana gave it to Kristi") or an ABBA pattern (c.g., "Friends Juana and Kristi found mango at the bar. Kristi gave it to Juana"), If the pattern is ABAB (where the indirect object IO), the algorithm predicts the first token of B. Conversely, for an ABBA pattern, it predicts the first token of A. In our experiments, we intervene on whether an input follows the ABAB pattern or not.
36

---

Fall
Full
DVX Stepe
DVN Stope
DXN
Strpe
DVV
Strpe
DXN Strpe
DXN Trining Shrpe
(a) Max IIA for both equality relations
(b) Mean IIA for both equality relations
82
Tomnng Stego
Toining Stepe
Thinnng Stepe
Toining Stope
Tuinng Stepe
Trinigg Stepe
Max IIA for left equality relation
(d) Mean IIA for Left equality relation
8x
88
DSN Toming Strpe
DSN Troning Stepe
DXX Trining Stape
DSN Thoinng Strpe
DXN Trinnng Stepe
DXX Triinng Stepe
IIA identity of first argument
(f) Mean IIA for identity first argument
Figure 9: IIA over 5 seeds for each combination of MLP layer (rows) and intervention size (columns) during training progression for the tested algorithms.
Alg 4. The ABAB-AMES algorithm for the 1OT task has inner node Tinner as follows:
{111 and is defined
f (par, (m1)) = check is abab wwwppprptttttatt.
(80a)
f (par ffirst_tokenneeee name b(Un)))
if if true
(80b)
fffirsttokeenfe name a(Un=))
if false
Here, get_name_ale extructs the first name (denoted A, e.g. Juana in our example) and get_nane sii_nae_bb(x extracts the second name (denoted B, e.g. Kristi in our exumple) from the input sentence x. The function check is _i___s__aa_______ abab patllpptttttetitee returns true if the sentence follows an "ABAB" structure (e.g.. "A and B A gave to B" meaning B is the I0) and false if it follows
37

---

an "ABBA" structure (e.g. "A and B B gave to A", meaning A is the I0). first token token(Name) returns the first token of the specified name. The output y is the first token of the indirect object.
1.2.2 Training Details
We models from the Pythia suite (Biderman al., 2023) to evaluate the IIA performance of the different o the IOI task. Specifically, we employ the Pythia 31M, 70M, 160M, and 410M parameter models. We also examine different training checkpoints provided by these models analyse how IIA evolves during training. To assess robustness, we replicate a subset of ex- periments using alternative Pythia model seeds from (van der Wal et al., 2025).
Thaining
Thinnigg Staps
We train all alignment maps on 2 epochs of 10t interventions based on data from Muhia (2022),
Figure 10: Cross Entropy loss (y-axis) during train- ing (π-axis) of the alignment map for the randomly initialised Pythia-3Im 3m. The loss plateaus between 4k and 6k steps, and suddenly drops after 6k steps. mension. For smaller models (31M, 70M), we train using float64 precision and a learning rate of 10-3, these adjustments proved crucial for convergence. We also note that observed quite severe grokking behaviour, where models had low IIA for a long time, which quickly jumped to high IIA values at certain point of training (see Fig. 10; wandb run).
with batch size of 256 and learning rate of 10- For all experiments, we set the interven- tion size | to half of the DNN's hidden di-
I,2.3 Additional Results
Robustness across random seeds. In Fig. 11, we examine how our main results from $6 generalise across multiple training seeds of the Pythia model. The key trends hold consistently across all seeds we can find perfect alignments using qonllin in most cases. However, we observe two notable exceptions. For one seed, the DNN fails to learn the IOI task even after full training. For another seed, we cannot find an alignment using even complex alignment maps under our current sctup. All other seeds achieve perfect alignment under orenlin, We hypothesise that the alignment failure case is primarily due to suboptimal training of the alignment map. Due to computational constraints, we did not perform extensive hyperparameter tuning that might have achieved convergence.
0.8
0.4
DNN
loos
0.3 Inft
Full
DNN Thaining Step
Figure 11: IIA of alignment between the ABAB-ABBA algorithm and the Pythia-410m model across multiple seeds (seeds 5 from van der Wal al., 2025), with interventions layer 12. We evaluate the IIA of both in and nmnnlin (with d2 64, K = 1) randomly initialised (Init.) and fully trained (Full) DNNs.
Generalisation across distinct name sets. In the main paper. we split the dataset from Muhia (2022) by ensuring that no two sentences appear in both the training and evaluation sets. However, this splitting strategy does not guarantee that the names themselves are distinct between training and evaluation sets. In Fig. 12, we examine the results when using completely different sets of names for training and evaluation. The results differ substantially: we cannot find an alignment using even complex alignment maps for the randomly initialised DNN. This suggests that IIA on the randomly initialised DNN may depend critically on overlap between the specific entities encountered during
17 These failures occur in different seeds: seed 3 shows poor IIA despite learning the task, while seed 4 fails to
learn the 1O1 task.
38

---

training and evaluation. For the fully trained DNN, we observe perfect alignment using @ reasonably high alignment using Ølin.
and
Accuracy
0.75
DAN
linerr
0.50
Full
DNN Training Step
Figure 12: IIA of alignment between the ABAB-ABBA algorithm and the Pythia-410m model using a different set of names for training and evaluation, with interventions at layer 12. We evaluate the IIA of both G and daoulin (with d = 64, K = 1) on randomly initialised and fully trained (Full) DNNs.
1.3 Distributive Law Task
We now study a similar task to the hierarchical equality (in App. I.1), based on the distributive law of and (A) and or (V).
Task 3. The distributive law task is defined follows. Let x o 0x3 0x X6 be 24-dimensional vector, where each x, E 1-0,5, 0.5] for E 3, 4,5,6), and denotes vector concatenation. The imput space X 1-0.5, 0.5) and the output space is y ffalse, true). The task function is
T(x)=
((x) == X2) A (X3 == x4)) V ((x3= = x4) (Xs = xx));
(81)
where the equality (x; =- Xj) holds if and only if and Xj are equal as vectors in R.
1.3.1 Algorithms
We define the following two candidate algorithms. Alg 5. The And-Or-And alg. to solve Task 3 has
iinnr (x 7xxs M
and it is defined as follows:
f (0s ( pV (Mi()) (xx)))))))A0)))))30)
(Unms Unno ) (Unns Unna
Sh Maa (1xg==x))1n
(Vpar, (01(gg (010kg--x)) AA0s -xo))) (Urag == Unnee A (ErAA = Unxo
f7 (VPaar, (mo))
U Uni ( ( V Unice (atx)===x)n(tt (g)
Alg 6. The And-Or alg. to solve Task 3 has
Minner {1/xa = x4y xxx =xxx==
and it is defined as follows:
M (Veg==x)))) (Uing Unnaa
f1 -gg--agiv (V "ypar, mag mao1)) (UiTze)
Unpes ) V (UTTa = Uniee
f (Vpar,(nn)) Uney==e, Uniar -xxg---gggioo wxg/vias -mx))
I.3.2 Training Details
For the distributive law task,
3-layer MLP (see App. E. with input dimensionality of 24, hidden layers of dimensionality | = //2 |/b = 24, and an output dimensionality of
2. The model is trained using the Adam optimiser with learning rate of 0.001 and cross-entropy loss. We use batch size of 1024. The datasets are generated by randomly sampling input vectors = X1 .... X5 such that the target label ÿ T(x) is true 50% of the time. We sample 1,048,576 samples for training. 10,000 for evaluation, and 10,000 for testing. Training runs for a maximum of 20 epochs with early stopping after epochs of no improvement.
For training d we use batch size of 6400 and train for up epochs of no improvement (using threshold of 0.001 for the required change). We use the Adam
to 50 epochs with early stopping after
39

---

optimiser with learning rate 0.001 and cross entsopy loss. To generate the intervened datasets: For Alg. 5, we intervene with a probability of 1/3 on =xx=/))) x4), 1/3 on 1](xx==x))AX =x)) and 1/3 on both variables. For Alg. 6, we intervene with a probability of 1/3 on /xx===x- 1/3 on =/xx==x)VV(x===>>
and 1/3 on both variables. For both algorithms, the samples for the base and source inputs are generated such that the output of the intervention changes compared to the base input 50% of the time. We sample 1,280,000 interventions for training, 10,000 for evaluation, and 10,000 for testing for each algorithm.
1.3.3 Results
In this section, we discuss the results on the distributed law task using the And-or-And and And-Or algorithms. Our findings corroborate the results presented in the main paper. As shown in Fig. 13, using linear and identity alignment maps reveals distinct dynamics. The And-Or algorithm achieves higher IIA using Tt particularly in later layers where the IIA of datn on the And-Or algorithm approaches 0.5. However, these dynamics completely vanish when using a more complex alignment map like @ where we achieve almost perfect IIA everywhere.
AndOr
AudOrAnd
1.00
0.75
Accuracy
0.50
linear
identity
0.25
non hnear
1921
1
2
12
2 12 2
12
2
12
2
12
2
12
L1
L2
L1
L2
L3
Figure 13: IA in the distributive law task for causal abstractions trained with different alignment maps 6. The figure shows results for both analysed algorithms for this task. The bars represent the max IIA across 10 runs with different random seeds. The black lines represent mean IIA with 95 % confidence intervals. The ( denotes the intervention size per node. Without interventions, all DNNs reach 100% accuracy. The used gaonlin uses Lyn 10 and d, 24.
Fig. 14a and 14c present the evaluated IIA throughout model training. These training progression plots show that randomly initialised models often achieve IIA above 0.8 with non-linear alignment maps, supporting our insight that when the notion of causal abstraction is equipped with gioulin it may identify algorithms which are not necessarily implemented by the underlying model. In Fig. 14b and 14d, we plot the mean IIA over 5 seeds instead of the maximum IIA.
The hidden size experiments (Fig. 15a and 15b) show that even RevNets with small de of 4 achieve near-perfect IIA for And-Or-And, while And-Or never reaches perfect IIA in the second layer, regardless of the dg. The training progression plots suggest possible explanation: IIA for And-Or- And initially increases in the last two layers but then decreases, while RevNets maintain near-perfect IIA. This may indicate that And-Or-And is first implemented with simple encodings detectable by linear 6s, before evolving into non-linear encodings that only RevNets can detect. The fact that And-Or never achieves high IIA in later layers further suggests it may not be true abstraction of the model's 's behaviour, though we note this remains hypothesis requiring further investigation.
And-Or-And Training. In this section, we analyse a DNN when this model is trained specifically to rely on the And-Or-Am algorithm (and, consequently, to encode the values of its hidden nodes). We do so with the method from Geiger et al. (2022), training the DNN to encode And-Or-An -And''s hidden nodes' values in its second layer, with an intervention size of 12. This method is similar to how we train (see App. I.3.2), but é is fixed to the identity function, and the DNN itself is trained; further, the training dataset is composed of 1/4 con-intervened samples, 1/4 samples with interventions on 77xx mmxx, Ax))A(x= =xx)) 1/4 on 77[x;==m (x) mxx)) and 1/4 on both variables. We then evaluate if this DNN abstracts both the And-Or-Are and And-Or using different á (as before, after freezing the DNN). The IIA performance of these @ is presented in Fig. 16. We can see here that, when using identity and linear alignment maps Φb IIA scores suggest that the And-Or-And algorithm seems to be implemented perfectly given the second layer, where we have only around 0.75 IIA for
40

---

140
Tall
Full
DNN Steps
DNN Strpe
DXN Strpe
DVV Strpe
DXN Strpe
DNN Triiinn Shrge
(a) Max IIA for And-Or
(b) Max IIA for And-Or
19
088
Tomnnn Stepo
Toitinn Stepp
Triing Stepe
Toining Stope
Toining Stepp
Trining Stepe
(c) Max IIA for And-Or-And
(d) Max IIА for And-Or-And
Figure 14: IIA seeds for each combination of MLP layer (rows) and intervention size (columns) during training progression for the evaluated algorithms.
the And-Or Or algorithm. However, these differences vanish almost completely using nonlln alignment map.
And-Or Training. this section, we report an experiment similar to the above, but we train our DNN to rely the And-Or algorithm instead. These results shown in Fig. 17. In this figure, again see that, when using identity and linear as alignment map ø, IIA performance suggests that the And-Or algorithm seems be implemented perfectly given the second layer, where have only around 0.65 IIA for the And-Or-And algorithm. These differences however vanish when using guonlin alignment map, which leads perfect IIA scores with either algorithm.
Computational Resources
The experiments on MLP were executed on CPU (10 computers with i7-4770 or newer) over weeks, noticed that DAS small MLPs are faster CPU than GPU. The experiments the Pythia models were executed on a single A 100 GPU with 80GB of memory using approximately 30 GPU hours, including the hyperparameter tuning.
41

---

Lagre
Accuracy
Am (a) And-Or
dau
Layer
Layer 2
Layer 3
Aecavry
des
dm
des
(b) And-Or-Ance
Figure 15: Mean IIA over 5 seeds using @onlln (Lm 1) on the trained DNN. Performance improves with larger hidden dimension den and intervention size |bb Each subplot corresponds of the two candidate algorithms for the distributed law task, showing how o' representational capacity influences performance.
AndOr
AndOrAnd
1.00
0.75
Accuroo
0.50
linrer
idlentity
0.25
non- linear
2
2
2
2
Figure 16: IAA in the distributive law task for causal abstractions trained with different alignment maps and a DNN trained to use the And-Or-Am algorithm. The figure shows results when evaluating if the DNN encodes either of the analysed algorithms for this task. The bars represent the max IIA across 10 runs with different random seeds. The black lines represent mean IIA with 95% confidence intervals. The | denotes the intervention size per node. All DNNs reach >99.9% accuracy after training. The used monnln L,n 10 and dra 16.
AndOr
AndOrAnd
1.00
0.75
Accaracy
0.50
linear
identity
0.25
non- lineer
2
2
Figure 17: IIA in the distributive law task for causal abstractions trained with different alignment maps and DNN trained to use the And-Or algorithm. The figure shows IIA results when evaluating if the DNN encodes either of the analysed algorithms for this task. The bars represent the max IIA across 10 runs with different random seeds. The black lines represent mean IIA with 95% confidence intervals. The | denotes the intervention size per node. Without interventions, all DNNs reach 99.9% accuracy. The used geenlnn uses Lyn 10 and dyn 16.
42

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The theoretical and empirical results mentioned in the abstract given in the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide limitations section and clearly label our theorem's 's assumptions, discussing when they might not hold in practice.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
43

---

Answer: [Yes]
Justification: Our proofs are based on assumptions given in the main paper and detailed proof' and assumptions for our theorems given in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We discuss the experimental settings and data used in the main paper well as providing detailed information in the appendix. We additionally link to the code in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
44

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We link to the code used generate our results in the paper with description of how to reproduce it.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines
cc/
pub 1 c gui des CCodeSu'mm ss issionolll cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.. cc/ public ic gu des CooeeSoeeeSSeeeeeee sumii s onnool clycyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental setting is discussed in the main paper and details provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For some of our results provide, besides the mean, 95% confidence intervals. For our experiments on Pythia, due to computational requirements we only ran one seed per configuration; the consistent trends in these results across model sizes and training steps. however, still enable us to have confidence in their robustness.
Guidelines:
The answer NA means that the paper does not include experiments.
45

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mention the used computation in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://neurrsspp httppppblcc//tthhcccss...eeeee
Answer: [Yes]
Justification: Our work does not provide new method, but rather analyses already published research methodology. Therefore, we believe it adds no additional ethical concerns the current research field.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
46

---

Justification:
Our method analyses an already published research methodology and we believe it adds no additional societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not publish any new method, model or dataset.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The code implemented by us (except standard libraries) and the only used external dataset is referenced.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
47

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assett are released, the license, copyright information, and terms use the package should be provided. For popular datasets, papers paperswitt sechoode com/ dat datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We link to the created code the paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human subjects were involved in our research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects were involved in research.
Guidelines:
48

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were only used for writing, editing, or formatting purposes and did impact the core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer LLM policy (https: : / http:iip.ccc//oooeeeeeeeeece /005// LLM) for what should or should not described.
49


Reviews:
[{"review_id": "DxxVnBN4DZ", "reviewer": "Reviewer_x7mN", "summary": "The most popular formalization of interpretability so far is the theory of _Causal abstractions_. They define a neural network (NN) as a complex computational graph, and an interpretation (G, m) as a simple (abstract) computational graph G, plus a map m of the graph's nodes and edges map to the NN's. Then, the interpretation (G, m) is a good one if there exists a mapping \\tau that maps values of the NN's nodes to their corresponding abstract graph nodes; and causally intervening in both NN and abstract graph A in the same way provides correct results.\n\nIt is well known that if \\tau is trivial, then causal abstraction is pretty useless. For example, if \\tau_n(x) = 0 for all nodes, all causal interventions produce the desired result and the graph is considered good.\n\nThe authors of this forcefully argue for the notion that, if we allow \\tau to be arbitrarily powerful, causal abstractions are also vacuous. The theoretical argument centers on \\tau partitioning NN activations into arbitrary groups (gerrymandering, if you will) that correspond to the algorithm's abstract values, which can be done regardless of whether the NN implements the algorithm A, so long as the NN has sufficiently many states. The practical argument centers on fitting a couple of well-known interpretability 'found' algorithms into completely unrelated or even randomly initialized networks, by using an up-to-8-layer NN as the map \\tau.\n\nThe authors then conclude that restricting \\tau (to be, for example, linear) works well.", "questions": "# Section 4, Theorem 1's assumptions\n\n## Appendix F, proof for the injectivity assumption: Lemma 8\n\nBecause I was skeptical of transformers being input-injective, I dug into the proof, which line 237 claims exists in Appendix F.\n\nThe proof of this Theorem 2 uses Lemma 8 (\"MLPs are injective almost surely\") also does not assume linearities are injective.\n\nMy question is about the final line of the proof in section F.3. Each line seems to check out, except the last part in words in lines 1408-1411: \"Now, eq. 87 holds because the right-hand side is a constant while the left-hand side is a random variable drawn from a continuous distribution\". The left-hand-side is a row of the L layer's weights W_L, the right-hand-side is an expression involving the weights of other rows of W_L. **How is the RHS a constant?**\n\n## Assumption 5: The DNN solves the task T\n\nYou state in 247-248 that \"it would be impractical in practice to evaluate a NN that does not perform the task correctly. But that is exactly what you do in experiments! Fig 3, line 328-329 state that you're using random NNs.\n\n**Why does the theorem need this assumption?**\n\n## Assumption 3: output-surjectivity\nYou state that \"in practice, all outputs seem to still be producible by language models\" . This is directly contradicted the phenomenon of \"glitch tokens\" found by Rumbelow and Watkins (2023, https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation), where some token's unembedding has smaller norm than every other token in its direction, so there is no activation that will produce it as the maximum.\n\nAgain this shows that the theorem is false in practice, but it's *approximately* true in many circumstances and is a useful intuition pump. Please present it as such.\n\n# Figure 1: right plot, no perfect IIA?\nin the right-most plot in Figure 1, none of the alignment maps are able to get perfect IIA. Why do you think this is the case?\n\n# Arguments and remarks from page 9\n\nL379: claims that Burns et al. 2023 \"leverag[es] unsupervised probes\" in order to control the probe's complexity. In what way does unsupervised learning control the probe's complexity? Does that not come from the fact the probes are linear?\n\nL388-389, where the authors state they cannot formalize why Fig. 3 (right) inclines them to support the linear probe. That's OK, but what about an informal justification? One could say that it's provided in the previous sentence (\"\\phi^lin accompanies the actual DNN performance much more closely\"). If so, please just remove this sentence.", "limitations": "mostly. The authors addressed some of the limitationsI found in the final section. But I think they're problems with critical assumptions of the main theorem, so the assumptions should be addressed **when they're stated** and not at the end where readers might not find it.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "# Strengths\n\nThe paper points out a critical problem with causal abstractions that many people that work with them do not think of, but that can pose problems. In practice basically all papers assume that the mapping \\tau is simple in some way, so it does not come up much. However, it's still an assumption that practitioners are making without being aware of, which could go away soon; for example if causal abstraction is used as a metric for automatic interpretability. For this reason, I think this is a fundamentally significant and original paper and all of its weaknesses are about presentation or about mistakes in the argument that make it not quite go through; but which can fixed easily.\n\nHere's my own experience reading the paper: I highlighted several chunks of the abstract, scribbling notes in disbelief, like \"Surely it's impossible to map [arbitrary] models to [arbitrary] algo[rithm]s\", implying that instead it's only possible to map arbitrary algorithms to NN models, because algorithms are simple whereas models are much complicated. I had also forgotten that causal abstraction only specifies that you can map high-level to low-level model, but not necessarily the opposite. If you were forced to map low-level to high-level model, then surely that map doesn't work for arbitrary low-level models? I think that _is_ true and made me believe this paper relatively quickly, but for the wrong reasons; I then thought that this paper spent too long on proofs for a relatively trivial statement. I read it again and realized the statement was not so trivial.\n\n# Weaknesses\n\n**My score is low because I want my comments to be addressed, but they should be easy to address; so I am hopeful that I will be able to recommend this paper to be accepted.** Here are the weaknesses, all fixable:\n\n## Section 4, Theorem 1's assumption: input-injective NNs\n\nI think the discussion in lines 234-237 of the \"DNNs are input-injective\" assumption is pretty misleading. First, this is clearly untrue: transformers use SiLUs or GeLUs which hit a negative peak at zero, and then go positive on both sides. ReLUs are also popular. Second: the citation of Morris et al. 2023 which is used to claim the assumption is \"well-approximated in practice\", only recovers 92% of 32-token inputs -- a far cry from 100%. Third, the authors themselves later write in lines 318-319 that this is not true in practice for their setup, and say this again in lines 407-410. Please be more upfront with this!\n\nAll of this needs to be said in Section 4 when discussing the assumptions. Input-injectivity is just not true. Which kind of invalidates the theorem. But in reality, NNs are ~close to input-injective, so they can be mapped to ~a large amount of algorithms that they don't implement. The theorem is still a good intuition builder.\n\n## Clarity: give intuition for Theorem 2\nIt would be nicer to give an intuition for the theorem. That is, if the NN is injective (it has a different state for every input sequence) and the algorithm is allowed to be arbitrarily complicated, then we can just gerrymander the hidden states to map to the abstract model. The hidden states need to be countable so there exists a computable algorithm that maps them to an abstract algorithm. (I haven't checked if this is the case thorougly, but the theorem does make use of partitions).", "was_revised": true, "final_justification": "My concerns were minor to begin with, and the authors addressed all of them. I believe the paper provides significant insight into causal abstractions, its mathematics are correct and has a solid empirical evaluation section.", "rebuttal": "Thank you for reviewing our paper in such detail. Your engagement with our work is greatly appreciated! We agree that highlighting the limitations of our assumptions early on (when they are first stated) is important, and we implemented this change in our paper.\n\nThe main changes we thus implemented, following suggestions, try to highlight that input-injectivity and output-surjectivity are likely not to hold exactly, but are close to true in practice, which is enough for many algorithms. These are the specific changes:\n* Input-injectivity: We changed “This assumption is also present in prior work (e.g., Pimentel et al., 2020b) and seems to be well approximated in practice (Morris et al., 2023). Further, we show in App. F that this is almost surely true for transformers at initialisation—assuming real-valued weights and activations (and thus ignoring floating point precision).” to “This assumption is also present in prior work (e.g., Pimentel et al., 2020b) and we show in App. F—assuming real-valued weights and activations—that this is almost surely true for transformers at initialisation. Due to floating point precision and neural collapse (Papyan et al., 2020), it is likely not to hold fully in practice; however, it still seems to be well-approximated in many empirical settings (e.g., see Morris et al. (2023) and App. H).” App. H (newly added) contains new experiments on input-injectivity for the hierarchical equality task (see details below).\n* Output-surjectivity: We changed “Notably, this assumption may not hold in theory, due to issues like the softmax-bottleneck (Yang et al., 2018). In practice, however, even with large vocabulary sizes, all outputs seem to still be producible by language models (Grivas et al., 2022).” to “[...] even with large vocabulary sizes, it seems that most outputs can still be produced by language models (Grivas et al., 2022) which is enough for these DNNs to be abstracted by many algorithms.”\n\nWe hope these adjustments clarify the limitations sufficiently, and we'd be happy to further iterate based on the reviewer's feedback.\n\n> Input-injectivity is just not true. Which kind of invalidates the theorem. But in reality, NNs are ~close to input-injective, so they can be mapped to ~a large amount of algorithms that they don't implement. The theorem is still a good intuition builder.\n\nAs suggested (and as mentioned above), we made this shortcoming of our assumptions clearer in the paper.\n\nWe previously hypothesised (lines 318-319) that \"perfect alignment remains elusive because the MLP is not actually injective or surjective at initialisation due to the ReLU activation function.\" However, after further investigation, we ruled out input-injectivity as the main cause of low IIA in the hierarchical equality task. We sampled 1.28M inputs, passed them through the MLP to get hidden states, and found no collisions across 10 random seeds. Thus, input-injectivity held for these inputs and is unlikely to have caused the observed issues. We added an appendix with this analysis.\n\nThis reopens the question:  why didn’t we achieve perfect (100%) IIA in all our experiments, particularly for the \"identity of first argument\" intervention (Fig. 1, right), as also noted by the reviewer. Here are some thoughts. First,  output-surjectivity is satisfied here, since the model is able to perform the task (an “identity of first argument” intervention does not critically change the possible model outputs). High IIA for the “Both Equality Relations” and “Left Equality Relations” further suggest that other assumptions hold well enough. We believe the lower \"identity of first argument\" IIA results stem primarily from two factors:\n\n1. **Generalisation Gap.**\nTheorem 1 presents an existence proof: there exists a function such that a DNN is abstracted by an algorithm. However, it does not discuss the learnability of this function and can be seen as a form of “strong overfitting”, providing no guarantees that a learned map would generalise to unseen data. In our experiments—where training and test sets are separate—the generalization gap becomes apparent. \n\n2. **Model Capacity Limitations.**\nWhile Theorem 1 assumes arbitrarily complex models, a RevNet is relatively limited in capacity.  Our RevNet appears insufficiently expressive to fully overfit a large dataset (1.28M samples), never exceeding a 0.55 training IIA. But with just 1,000 samples, the RevNet achieved >99% training IIA.\n\nGiven these results, we added an extra paragraph to our discussion about the role of generalisation in interpretability and causal abstraction analyses.\n\n> Clarity: give intuition for Theorem\n\nWe agree with the reviewer, and we added the following proof sketch for our theorem:\n\n“We prove this Theorem by induction. We fix an arbitrary partitioning of the DNN $\\mathtt{N}$'s neurons. Then, for each layer, we show that we can create a suitable alignment map that: (i) preserves the information of its and preceding nodes, and (ii) enforces that the DNN's output matches the output of the intervened algorithm $\\mathtt{A}$ under any intervention up to that layer. These two conditions are sufficient to guarantee that an accompanying $\\tau$ exists, and thus that $\\mathtt{A}$ is an input-restricted distributed abstraction of $\\mathtt{N}$. We enforce (i) by ensuring that $\\phi_{\\ell}$ is injective on each dimension—which is possible due to, in practice, its inputs only covering a countable set. We then leverage $\\mathtt{N}$'s surjectivity (Assump. 3) to enforce (ii).”\n\nAdditionally, we have added a new figure to our paper that offers a visual representation of how increasing alignment complexity can lead to higher or perfect IIA, which conveys intuition about the non-linear representation dilemma.\n\n> Appendix F, proof for the injectivity assumption: […] How is the RHS a constant?\n\nIn Equations (82–83), we constrain the probability space by fixing the values of all weights of the MLP except one—specifically, the weight from node $i$ in the second last layer to node 1 in the final layer. This random variable is denoted $W^′$. The other parameters are treated as constants in the following equations, given their realisations in the conditions.\nTherefore, the RHS of Equation (87) does not depend on $W^′$, but only on fixed parameters and the input, and is thus constant. We tried to make this clearer in the paper.\n\n\n> Assumption 5: The DNN solves the task T [...] Why does the theorem need this assumption?\n\nOur theorem states that, under the given assumptions, any algorithm is an input-restricted distributed abstraction of any model. By definition, this abstraction only holds if the model’s behaviour exactly matches that of the algorithm—requiring IIA of 1.0. If the model does not solve the task, perfect IIA is impossible, as the *non*-intervened inputs yield incorrect outputs. Thus, assuming the model solves the task is required. In practice, however, even if the DNN does not perfectly solve the task, there still exists an alignment map which achieves correct outputs on all intervened inputs.\n\nWe use random models in our experiments not only to provide empirical examples of the theorem, but to illustrate its relevance in a practical setting where not necessarily perfect IIA is required or reachable. A random neural network clearly does not implement the intended algorithm, yet the alignment methods can still report high IIA.\n\n> Assumption 3: output-surjectivity [...]  This is directly contradicted the phenomenon of \"glitch tokens\" found by Rumbelow and Watkins [...]. \n\nWe appreciate the comment about glitch tokens. While the blog post notes that “many of these tokens were among those closest to the centroid of the entire set of 50,257 tokens,” this alone does not confirm that these tokens are non-generatable given some hidden activation. Due to this unclear connection, we opted not to reference this explicitly. Still, based on blog comments, some glitch tokens may indeed be non-generatable. \n\n> in the right-most plot in Figure 1, none of the alignment maps are able to get perfect IIA. Why do you think this is the case?\n\nSee answer to input-injectivity above.\n\n\n> L388-389, where the authors state they cannot formalize why Fig. 3 (right) inclines them to support the linear probe. That's OK, but what about an informal justification? One could say that it's provided in the previous sentence [...]. If so, please just remove this sentence.\n\nWe removed this sentence.\n\n\n> L379: claims that Burns et al. 2023 \"leverag[es] unsupervised probes\" in order to control the probe's complexity. In what way does unsupervised learning control the probe's complexity? Does that not come from the fact the probes are linear?\n\nWe added this example following the intuition that the complexity–accuracy trade-off of probing arises only in supervised settings, where increasing probing complexity allows increasingly more complex “features” to be extracted from the model’s representations. Unsupervised probing would, in this respect, be immune to such problems, since without supervision these “gerrymandered” maps would not be found. We acknowledge, however, that such reasoning was not made explicit in our paper, and we have now added this explanation to our paper.\n\nAgain, we thank the reviewer for such a detailed reading of our paper. Please don’t hesitate to let us know if we’ve misunderstood any part of your feedback or if further clarification is needed. We’d be happy to address any additional questions.\n\n### References\n\nPimentel et al. (2020b). Information-theoretic probing for linguistic structure\n\nMorris et al. (2023). Text embeddings reveal (almost) as much as text\n\nPapyan et al. (2020). Prevalence of Neural Collapse during the terminal phase of deep learning training\n\nGrivas et al. (2022). Low-rank softmax can have unargmaxable classes in theory but rarely in practice"}, {"review_id": "ZYmWCdPogJ", "reviewer": "Reviewer_XbMd", "summary": "This paper studies the application of causal abstractions for mechanistic interpretability, where the idea is to explain black box ML models (like neural networks) with high-level algorithms that serve as causal hypotheses of the model’s behavior. The hypotheses are tested by learning an abstraction mapping between the ML model and the algorithm such that the properties of a causal abstraction hold. The main result of the paper is a theorem that proves that, without constraints on the abstraction map such as linearity, an abstraction map can be constructed between any ML model and any high-level hypothesis. The ensuing discussion emphasizes the implication that in the general case, this approach of explaining ML models can be vacuous. Experiments demonstrate that with nonlinear abstraction maps, interchange intervention accuracy remains consistently high.", "questions": "7. To what extent does Thm. 1 imply the problem is caused by the linearity assumption of the alignment rather than other factors? For example, perhaps IIA is an imperfect metric of measuring alignment since IIA does not consider semantic meaning of the algorithm nodes.\n\n8. How does the topic of alignment relate to disentangled causal representation learning? As in, is the idea of learning the alignment map similar to the idea of learning a representation map that disentangles the inputs into latent causal factors?", "limitations": "Limitations are discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. Thm. 1 is an impactful result. Indeed, with growing literature in the space of causal abstractions for mechanistic interpretability, the non-linear representation dilemma really strikes at the heart of the issue of what assumptions are meaningful for determining causal hypotheses and what they imply in the behavior of neural networks.\n2. Assumptions are clearly stated and justified in Sec. 4.\n3. The paper is very well-written, with all notation clearly defined and well-labeled. The proof of Thm. 1 is carefully organized. The paper also neatly organizes all of the important background information, including the general process of using causal abstraction for mechanistic interpretability, the discussion of how information is encoded in deep neural networks, and some new definitions surrounding distributed abstractions that grounds prior work in the context of this paper.\n4. The experimental results show some interesting analyses. It is notable that in linear alignments, IIA decreases in deeper layers, a problem that goes away with nonlinear alignments. The fact that IIA increases with simpler alignments following more training is also interesting.\n\n**Weaknesses:**\n\n5. The paper could benefit from having figures that describe the causal abstraction for mechanistic interpretability procedure and highlight the impact of Thm. 1.\n6. The paper does not offer many new novel contributions and in some ways serves as a review or position paper. All content until Sec. 4 is background information (although Sec. 3.2 offers new definitions that portray existing work in new perspectives). The main contributions of the paper are Thm. 1 (with proof in the appendix) and the experimental results. The contributions are impactful, but they are mainly adding to discussion points surrounding the efficacy of this mechanistic interpretability approach rather than offering new solutions.\n\nOverall the paper is quite strong, and I recommend it for acceptance. I do not believe that the weaknesses I have discussed warrant a lower score.", "was_revised": true, "final_justification": "I will maintain my score and recommend the paper for acceptance. The paper addresses an important problem and is well-written, providing a promising avenue of future research. The rebuttal has further reinforced this, and the promised changes will improve the paper even further.", "rebuttal": "Thank you very much for your thoughtful and constructive feedback. Below, we respond to your points and questions.\n\n\n\n> 5. The paper could benefit from having figures that describe the causal abstraction for mechanistic interpretability procedure and highlight the impact of Thm. 1.\n\nWe have added two new figures to the revised version of the paper that directly address this point:\n* A visual illustration of the non-linear representation dilemma, visualising how causal abstraction works and demonstrating how increasing the complexity of the alignment map ϕ can improve interchange-intervention accuracy (IIA). This figure is in the main body of the paper.\n* A schematic overview of the causal abstraction definitions introduced in Section 3. This figure highlights the subset–superset relationships between different definitions, illustrating how constraints on the allowed abstraction map and interventions shape these relationships. This figure is in the appendix.\n\n\n> 6. The paper does not offer many new novel contributions and in some ways serves as a review or position paper. [...] The main contributions of the paper are Thm. 1 (with proof in the appendix) and the experimental results. The contributions are impactful, but they are mainly adding to discussion points surrounding the efficacy of this mechanistic interpretability approach rather than offering new solutions.\n\nThank you for this perspective. We agree that a central focus of the paper is to mathematically and empirically demonstrate the non-linear representation dilemma. While this contribution is more diagnostic than prescriptive, we believe it fills a gap in the current literature. In our view, clarifying and naming this issue is a necessary step before more targeted solutions can be developed. Further, we explicitly discuss potential next steps, such as incorporating complexity constraints into causal abstraction, drawing parallels to similar concerns raised in the probing literature.\n\nBeyond this, we also added a new paragraph to emphasise the role of generalisation in causal abstraction:\n\n**The Role of Generalisation.** We now highlight that Theorem 1 provides an existence proof for a perfect abstraction map (thus guaranteeing perfect IIA) between a DNN and an algorithm. This existence proof, however, leverages complex interactions between the intervened hidden states and the DNN’s structure, requiring perfect information about both and thus representing a form of extreme overfitting. Crucially, this theorem offers no guarantees regarding the learnability of the alignment map ϕ from limited data or its generalisation to unseen inputs. This gap between theoretical existence and practical learnability becomes evident in practice. For instance, in an additional experiment on the IOI task [added to the appendix in the revised version], we show that when training and test sets contain disjoint sets of names, the learned alignment map fails to generalise, resulting in low IIA on the test set. This suggests that generalisation should play a crucial role in causal abstraction analysis, as the ability to learn abstraction maps that transfer beyond training data seems fundamental to interpreting a model, distinguishing a genuine understanding of its inner workings from mere training pattern memorisation.\n\nAs a brief note—should it be of interest—we are currently building on the analysis in this paper to further explore how features are encoded in models. Specifically, we are evaluating the Carry One variable from the Mechanistic Interpretability Benchmark (MIB) to better understand this variable’s encoding. So far, we have observed that a linear (but non-orthogonal) alignment map achieves >0.96 IIA in that task—yet, this alignment map does not appear to reflect the underlying mechanism and seems to also “hack” the DNN. We hope this ongoing work will lead to deeper insights and, eventually, more practical guidance for causal abstraction.\n\n\n\n\n\n> 7. To what extent does Thm. 1 imply the problem is caused by the linearity assumption of the alignment rather than other factors? For example, perhaps IIA is an imperfect metric of measuring alignment since IIA does not consider semantic meaning of the algorithm nodes.\n\nWe’re not entirely sure we fully understand this question. Could you give an example of how a metric that considers the semantic meaning of the algorithm nodes should look?\n\nAs a minor clarification: the assumption of linearity is not the root cause of the non-linear representation dilemma; instead, the dilemma arises from the use of unrestricted non-linear alignment maps. However, relying exclusively on linear maps is also problematic, as noted in recent work (e.g., Kantamneni and Tegmark, 2025). This creates the tension: unrestricted non-linear maps are too flexible and can yield misleading alignment; linear maps may miss important aspects of the true mechanism. This tension reflects the non-linear representation dilemma.\n\n\n\n\n> 8. How does the topic of alignment relate to disentangled causal representation learning? As in, is the idea of learning the alignment map similar to the idea of learning a representation map that disentangles the inputs into latent causal factors?\n\nYes, there is a meaningful connection. Causal abstraction—under distributed or constructive $\\tau$—requires disentangling the representations of different variables. This is due to the inherent requirement for partitioning the latent variable space in these types of causal abstraction, which should disentangle inputs into such latent causal factors.\n\n\n---\n\nPlease don’t hesitate to reach out if we have misunderstood any part of your feedback or if additional clarification would be helpful. We’d be happy to continue the discussion and address any remaining questions.\n\n### References\n\nKantamneni and Tegmark. (2025). Language models use trigonometry to do addition"}, {"review_id": "7kpFntBvxt", "reviewer": "Reviewer_9y8j", "summary": "This paper examines a fundamental limitation in causal abstraction methods for mechanistic interpretability. The authors prove that when alignment maps between neural networks and algorithms are allowed to be arbitrarily complex (non-linear), any neural network can be mapped to any algorithm under reasonable assumptions, making causal abstraction vacuous. They demonstrate this through Theorem 1 and empirical validation showing that complex alignment maps achieve perfect interchange intervention accuracy (IIA) even on randomly initialized models incapable of solving target tasks. The work identifies the \"non-linear representation dilemma\" - the inherent trade-off between alignment map complexity and interpretability validity - and argues that causal abstraction implicitly relies on strong linearity assumptions about neural information encoding.", "questions": "1. Can you provide a more systematic analysis of when the key assumptions (particularly input-injectivity and output-surjectivity) hold in practice? What are the concrete implications when these assumptions fail, and how common are such failures across different architectures and tasks?\n    \n2. Given that you've identified the non-linear representation dilemma, what specific recommendations would you provide to practitioners currently using causal abstraction methods? How to choose appropriate alignment map complexity? Relatedly, what are the most promising research directions for addressing the identified limitations?\n    \n3. Do your findings extend to other interpretability methods beyond causal abstraction? Are there fundamental insights here that apply more broadly to the interpretability research program?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\ntechnical\n- theoretical contribution is mathematically rigorous with well-stated assumptions that are reasonable for practical neural networks (countable input spaces, input-injectivity, output-surjectivity)\n- proof structure is sound and the empirical validation effectively demonstrates the theoretical predictions across multiple tasks\n- experimental design systematically explores alignment map complexity using identity, linear, and non-linear (RevNet) parameterizations, providing evidence that complex maps can achieve high IIA regardless of model capability.\n\npresentation\n- makes inherently complex mathematical concepts accessible to the NeurIPS audience\n- progressive introduction of concepts from basic causal models to distributed abstractions is well-structured\n-  hierarchical equality and IOI tasks provide concrete illustrations of abstract theoretical points\n- figures effectively communicate the key findings\n\nnovelty and significance\n- focus on alignment map complexity in causal abstraction is novel\n- non-linear representation dilemma provides a framework that could redirect research toward principled constraints for alignment maps.\n\n### Weaknesses\n- key assumptions (particularly input-injectivity) may fail in practice (as the authors admit) - \"identity of first argument\" experimental results also show this → somewhat limits the theoretical result\n- empirical validation limited to relatively simple tasks and specific model architectures\n- for Pythia, single-seed reduces statistical confidence\n\n- math in Section 3 is quite dense and could benefit from more intuitive explanations\n- notation becomes complex and inconsistent in places (particularly the intervention notation in equations 4-6)\n- practical implications for practitioners could be articulated more clearly (what is the concrete guidance?)\n\nsignificance and novelty\n- limited to causal abstraction methods\n- complexity-accuracy trade-off, while formalized in a novel context, echoes similar debates in the probing literature (but actually the authors were fairly thorough in citing the complexity-accuracy trade-off literature from probing)\n\nminor: \n - throughout the paper, in figures and text, I'd advise to use different colors/patterns that are colorblind-friendly - you are using red-green contrast, in general the color scheme doesn't seem coherent to me (but well done on the different shadings in figure 1 to distinguish the categories without color). I would probably swap the red-green for blue-orange (given that you use blue in the plots). Overall this is minor though as in the figures you made sure to distinguish the elements without color.\n - figure captions could better explain what readers should conclude from each result\n - l. 99: The notation \"Δ|Y|-1\" is unclear - clarify this is the probability simplex\n - l. 182: Ensure the bijection notation is consistent with later usage\n - Add more intuitive explanations alongside formal definitions (especially Definitions 1-2, 6-7)\n - maybe include a glossary of key terms in the appendix, \n - maybe include a notation table in the appendix listing all symbols\n - proof sketches in the main text for Theorem 1, even if full proof is in appendix\n - consistent subscript/superscript conventions throughout (some inconsistencies in ω vs ε notation)\n - In the introduction, better motivate why this problem matters for the broader ML community", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and your helpful suggestions. We address your points below.\n\n> key assumptions (particularly input-injectivity) may fail in practice (as the authors admit) - \"identity of first argument\" experimental results also show this → somewhat limits the theoretical result\n\nWe agree that the assumptions in Theorem 1 generally do not hold exactly. However, we aimed to convey that input-injectivity and output-surjectivity often hold approximately in practice, which is sufficient for many algorithms. To clarify this point, we revised the following parts of Section 4:\n* Input-injectivity: We changed “This assumption is also present in prior work (e.g., Pimentel et al., 2020b) and seems to be well approximated in practice (Morris et al., 2023). Further, we show in App. F that this is almost surely true for transformers at initialisation—assuming real-valued weights and activations (and thus ignoring floating point precision).” to “This assumption is also present in prior work (e.g., Pimentel et al., 2020b) and we show in App. F—assuming real-valued weights and activations—that this is almost surely true for transformers at initialisation. Due to floating point precision and neural collapse (Papyan et al., 2020), it is likely to not hold fully in practice; however, it still seems to be well-approximated in many empirical settings (e.g., see Morris et al. (2023) and App. H).” App. H (newly added) contains new experiments on input-injectivity for the hierarchical equality task.\n* Output-surjectivity: We changed “In practice, however, even with large vocabulary sizes, all outputs seem to still be producible by language models (Grivas et al., 2022).” to “[...] even with large vocabulary sizes, it seems that most outputs can still be produced by language models (Grivas et al., 2022) which is enough for these DNNs to be abstracted by many algorithms.”\n\nAs noted, we had hypothesised that imperfect alignment from \"identity of first argument\" might stem from the MLP not being injective or surjective at initialisation due to the ReLU activation. However, further analysis (newly introduced App. H) ruled out input-injectivity as the main issue: across 1.28M inputs and 10 seeds, we found no hidden state collisions, suggesting that input-injectivity holds in practice for this setting.\nThis raises the question: why didn’t we achieve perfect IIA, particularly for the “identity of first argument” intervention (Fig. 1, right)? Since the model successfully performs the task, output-surjectivity appears satisfied. Moreover, high IIA in the Both Equality Relations and Left Equality Relations settings suggests other assumptions are also met. We believe two factors explain the lower IIA:\n\n1. **Generalization Gap.**\nTheorem 1 is an existence proof, does not discuss the learnability of this function and can be seen as a form of “strong overfitting”. In our experiments—where training and test sets are separate—the generalization gap becomes apparent. For instance, with only 1,000 training examples, the RevNet could overfit and reach >99% training IIA.\n\n2. **Model Capacity Limitations.**\nTheorem 1 assumes arbitrarily expressive models, but our RevNet has limited capacity. It fails to fit large “identity of first argument” datasets (e.g., 1.28M samples, max training IIA ≈ 0.55), yet overfits smaller ones (training IIA of >99% on 1,000 examples).\n\n> empirical validation limited to relatively simple tasks and specific model architectures\n\nWe agree that broader empirical validation would strengthen the work. However, in line with prior work, we focused on a diverse but tractable set of tasks to enable in-depth analysis. While most tasks are simple, we also include results on a real-world language modelling task—Indirect Object Identification (IOI)—using Pythia models. Despite its simplicity, IOI offers meaningful insights into the non-linear representation dilemma in LLMs.\n\n> for Pythia, single-seed reduces statistical confidence\n\nWe agree that this is an important point. To clarify: we do provide multi-seed results in **Appendix G.2.3** for a subset of the experiments. We made this more explicit in the main text in the revised version. Due to computational constraints, we couldn’t rerun all Pythia experiments for all seeds.\n\n> practical implications for practitioners could be articulated more clearly (what is the concrete guidance?)\n\nThis is an important point. While our paper focuses on theoretical foundations, we agree it is valuable to begin identifying early takeaways for practice. To that end, we have added the following paragraph to the discussion section:\n\n**The Role of Generalisation.** We now highlight that Theorem 1 provides an existence proof for a perfect abstraction map (thus guaranteeing perfect IIA) between a DNN and an algorithm. This existence proof, however, leverages complex interactions between the intervened hidden states and the DNN’s structure, requiring perfect information about both and thus representing a form of extreme overfitting. Crucially, this theorem offers no guarantees regarding the learnability of the alignment map ϕ from limited data or its generalisation to unseen inputs. This gap between theoretical existence and practical learnability becomes evident in practice. For instance, in an additional experiment on the IOI task [added to the appendix in the revised version], we show that when training and test sets contain disjoint sets of names, the learned alignment map fails to generalise, resulting in low IIA on the test set. This suggests that generalisation should play a crucial role in causal abstraction analysis, as the ability to learn abstraction maps that transfer beyond training data seems fundamental to interpreting a model, distinguishing a genuine understanding of its inner workings from mere training pattern memorisation.\n\nWe are further continuing to research this topic. In particular, we are evaluating the Carry One variable from the Mechanistic Interpretability Benchmark (MIB) to better understand variable encoding. So far, we have observed that even a linear (but non-orthogonal) alignment map achieves >0.96 IIA on this task, while appearing to *not* reflect any true mechanism used by the DNN.\nAs this research is still preliminary, we refrain from making definitive recommendations and aim to offer clearer insights in future work.\n\n> throughout the paper, in figures and text, I'd advise to use different colors/patterns that are colorblind-friendly - you are using red-green contrast\n\nThank you for the suggestion. We tested the plots with a colorblindness simulator and ensured the red is sufficiently dark relative to the green (in grayscale). That said, we're happy to revise the colors further if the reviewer has more insights.\n\n> math in Section 3 is quite dense and could benefit from more intuitive explanations\n> Add more intuitive explanations alongside formal definitions (especially Definitions 1-2, 6-7)\n\nWe agree, and to address this we have added two new figures to our paper:\n* A visual illustration of the non-linear representation dilemma, visualising how causal abstraction works and demonstrating how increasing the complexity of the alignment map ϕ can improve interchange-intervention accuracy (IIA). This figure is in the main body of the paper.\n* A schematic overview of the causal abstraction definitions introduced in Section 3. This figure highlights the subset–superset relationships between different definitions, illustrating how constraints on the allowed abstraction map and interventions shape these relationships. This figure is in the appendix.\n\n\n> Can you provide a more systematic analysis of when the key assumptions (particularly input-injectivity and output-surjectivity) hold in practice? What are the concrete implications when these assumptions fail, and how common are such failures across different architectures and tasks?\n\nWe touched on this in our reply to the first point with regard to the tasks used in this paper. We agree that it would be valuable for future work to systematically analyse when these assumptions hold. In practice, however, these assumptions seemed to be well-approximated in practice in our experiments, and we did not find either of them to affect our analyses.\n\n> What specific recommendations would you provide to practitioners currently using causal abstraction methods? How should alignment map complexity be chosen? What are the most promising directions for future research?\n\nWe touched on this in our reply to the first point.\n\n> Do your findings extend to other interpretability methods beyond causal abstraction? Are there broader insights for the interpretability research community?\n\nYes, we believe the non-linear representation dilemma is relevant to a range of interpretability methods. For instance, the **complexity–accuracy trade-off**—central to our dilemma—has also been observed in probing (e.g., Hewitt and Liang, 2019). This trade-off arises in any interpretability method that aims to identify features within hidden representations:\n* **Loosely constrained** methods may yield higher accuracy but risk capturing *spurious* correlations.\n* **Tightly constrained** methods may promote simplicity but fail to recover *important non-linear* structure. \n\n---\n\nWe also worked on clarifying the paper in response to the suggestions that were not explicitly addressed above. We hope this response addresses your questions and concerns. Please let us know if there are any points that remain unclear—we would be happy to elaborate further.\n\n### References\n\nPimentel et al. (2020b). Information-theoretic probing for linguistic structure\n\nMorris et al. (2023). Text embeddings reveal (almost) as much as text\n\nPapyan et al. (2020). Prevalence of Neural Collapse during the terminal phase of deep learning training\n\nGrivas et al. (2022). Low-rank softmax can have unargmaxable classes in theory but rarely in practice"}, {"review_id": "wjlEcKQMy2", "reviewer": "Reviewer_vv44", "summary": "This paper critically examines the framework of causal abstraction for mechanistic interpretability. The authors prove that, under reasonable assumptions, allowing for arbitrarily powerful (non-linear) alignment maps renders the notion of causal abstraction vacuous, as any neural network can be perfectly mapped to any algorithm. This theoretical claim is supported by strong empirical evidence on both synthetic tasks and language models, where near-perfect interchange-intervention accuracy (IIA) is achieved even for randomly initialized models that cannot solve the task. The paper frames this finding as the \"non-linear representation dilemma,\" highlighting the fundamental trade-off between the expressive power of an alignment map and the meaningfulness of the resulting explanation.", "questions": "In your conclusion, you state an \"intuition\" to support the linear representation hypothesis (LRH) but acknowledge that you \"cannot make this intuition formal.\" This difficulty may arise because your work seems to make the general LRH intractable from both a falsification and verification standpoint:\n* To falsify the universal LRH, one must demonstrate a feature that is provably non-linear. This requires showing both (i) the failure of linear maps and (ii) the success of a non-linear one. However, (i) might never be finished, and for (ii) your theorem shows that a successful alignment with a powerful nonlinear map is a method-induced artifact, providing no evidence that the network actually represents the feature non-linearly. Thus, a required counterexample can never be convincingly established.\n* Simultaneously, any verification by a linear alignment map is weakened. It can always be challenged as a mere approximation of a more complex, unfalsifiable non-linear ground truth with perfect IIA.\n\nGiven that the universal LRH is rendered practically unprovable and unfalsifiable within this framework, shouldn't the paper's main takeaway be a stronger recommendation to abandon it as a useful scientific hypothesis? Instead of relying on intuition, shouldn't the community be guided to formulate and test a portfolio of more modest, but strictly falsifiable, hypotheses? For example:\n* (a) a local, circuit-specific claim (e.g., \"The IOI circuit in GPT-2 Small uses a linearly decodable representation for the 'ABAB pattern' variable\").\n* (b) An instrumental, Pareto-optimal claim (e.g., \"Within a fixed complexity budget, a linear decoder provides the best explanation on the Pareto front of IIA vs. complexity\").", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper tackles a foundational and timely problem in interpretability research. The central theorem, proving the vacuity of unconstrained causal abstraction, is a rigorous and significant contribution.\n2. The empirical results provide compelling support for the theoretical claims. The demonstration that complex alignment maps can find perfect abstractions even in randomly initialized or incapable models is a strong piece of evidence.\n\nWeaknesses:\n1. The paper's conclusion feels somewhat tentative and does not fully embrace the strong implications of its own findings. The authors state an \"intuition\" to support the Linear Representation Hypothesis (LRH) but claim they cannot formalize it. This seems to miss an opportunity to propose more nuanced, falsifiable versions of the LRH as a path forward, as I will detail in the questions.", "was_revised": true, "final_justification": "The authors basically addressed all my questions. I think the paper is solid and well motivated. Therefore I keep the score.", "rebuttal": "Thank you very much for your thoughtful and constructive review. We greatly appreciate your detailed feedback and suggestions. Please find our response below.\n\n\n> Summary of Feedback:\n\n> The paper's conclusion feels somewhat tentative and does not fully embrace the strong implications of its own findings. The authors state an \"intuition\" to support the Linear Representation Hypothesis (LRH) but claim they cannot formalize it. This seems to miss an opportunity to propose more nuanced, falsifiable versions of the LRH as a path forward, as I will detail in the questions.\n\n> Given that the universal LRH is rendered practically unprovable and unfalsifiable within this framework, shouldn't the paper's main takeaway be a stronger recommendation to abandon it as a useful scientific hypothesis? Instead of relying on intuition, shouldn't the community be guided to formulate and test a portfolio of more modest, but strictly falsifiable, hypotheses? \n\n\nWe would like to clarify that—while we believe some of our results suggest the linear representation hypothesis to produce more intuitively reliable interpretability insights—we do not fully endorse this linear hypothesis as a strong or sufficient hypothesis for causal abstraction. Further, we agree that more modest and falsifiable hypotheses can be more scientifically productive. \nAs we see it, researchers would ideally—as the reviewer suggests—formulate and test a portfolio of modest, but strictly falsifiable, hypotheses (relying on local or instrumental claims). However, by themselves these modest hypotheses do not directly answer the core question: *What has the model actually implemented?* Based on a large-ish battery of such tests, it may then be reasonable for researchers to speculate about grander claims (such as the linear representation hypothesis). Many great interpretability papers already follow this format, and we see no harm in this—as long as there is a clear separation between factual and speculative claims.\n\n\n\n> (ii) your theorem shows that a successful alignment with a powerful nonlinear map is a method-induced artifact.\n\nRegarding this statement,\nit is true that successful alignment maps (under arbitrarily complex mappings) are guaranteed by our theorem, and thus they are indeed a method-induced artifact.\nHowever, this interpretation is not always entirely accurate. Our theorem demonstrates that there **exists** an **arbitrarily complex** alignment map such that any algorithm can be shown to be an input-restricted distributed abstraction of any model (given some assumptions). However, our theorem:\n\n* offers **no generalisation guarantees** to unseen data.\n* **does not explain** the empirical success of nonlinear alignment maps that **are not complex enough to overfit** a specific task.\n\nIn our experiments, we use train–test splits, evaluating IIA on held-out data.\nThus, when we observe high IIA using such non-linear alignment maps, we cannot simply dismiss them as artifacts. Therefore, we believe focusing on generalisation to be a promising step for future research. We expanded on this in the discussion section of the revised paper with the following paragraph:\n\n **The Role of Generalisation.** We now highlight that Theorem 1 provides an existence proof for a perfect abstraction map (thus guaranteeing perfect IIA) between a DNN and an algorithm. This existence proof, however, leverages complex interactions between the intervened hidden states and the DNN’s structure, requiring perfect information about both and thus representing a form of extreme overfitting. Crucially, this theorem offers no guarantees regarding the learnability of the alignment map ϕ from limited data or its generalisation to unseen inputs. This gap between theoretical existence and practical learnability becomes evident in practice. For instance, in an additional experiment on the IOI task [added to the appendix in the revised version], we show that when training and test sets contain disjoint sets of names, the learned alignment map fails to generalise, resulting in low IIA on the test set. This suggests that generalisation should play a crucial role in causal abstraction analysis, as the ability to learn abstraction maps that transfer beyond training data seems fundamental to interpreting a model, distinguishing a genuine understanding of its inner workings from mere training pattern memorisation.\n\nWe also believe that generalisation is necessary but not sufficient for faithful abstraction. Therefore, we are actively investigating now (as future work)  complex alignment maps that achieve high IIA, and whether they actually uncover meaningful mechanisms implemented by the DNNs, or whether they “hack” the model to achieve high performance.\nTo continue this research, we are studying the carry one variable from the Mechanistic Interpretability Benchmark (MIB). This ongoing preliminary work illustrates the complexity of defining faithful alignment maps. So far, we've observed that a linear (but non-orthogonal) alignment map achieves IIA above 0.96 on this task, but does not appear to reflect any true mechanism implemented by the model. These preliminary findings raise more questions than they resolve—highlighting the early stage of this line of research. As such, we are hesitant to offer definitive recommendations just yet, but we aim to do so in future work.\n\n\n\n\n\n\n\n\n\nTo conclude, the main message of this paper is therefore: alignment maps must be selected with care, and causal abstraction—though a powerful framework—should not be treated as a silver bullet.  While we offer some practical insights based on our findings, this work represents (to the best of our knowledge) the first systematic evaluation of the nonlinear representation dilemma in causal abstraction. We intentionally avoid making broad or strongly prescriptive claims at this stage. Instead, we see this paper as laying the groundwork for more practical, grounded guidelines, which we aim to develop in future research as our understanding of alignment mechanisms evolves.\n\n---\n\nPlease feel free to let us know if we have misunderstood any part of your feedback or if any clarification is needed. We would be happy to continue the discussion or elaborate further on any point."}]

Author rebuttal:

Meta-review: This paper presents the non-linear representation dilemma: if we allow arbitrary alignment maps in causal abstraction, there is an apparent paradox wherein we can map neural networks to algorithms even when they are incapable of solving a task. This has shades of the no free lunch theorem and various impossibility results when trying to interpret models without making any assumptions. Although not surprising, causal abstractions have recently gained traction and this work suggests caution is needed when adopting them in practice. 

During the discussion, several minor concerns were raised that were addressed by the authors. In the end there was unanimous agreement to accept this thought provoking paper.
Decision: Accept (spotlight)