Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?
Authors: Maxime Méloux
Silviu Maniu
François Portet
Maxime Peyrard
Venue: ICLR 2025 Poster (2025)
Keywords: AI interpretability
mechanistic interpretability
causal consistency
explanatory algorithms
circuits
Primary area: interpretability and explainable AI

Abstract: As AI systems are increasingly deployed in high-stakes applications, ensuring their interpretability is essential. Mechanistic Interpretability (MI) aims to reverse-engineer neural networks by extracting human-understandable algorithms embedded within their structures to explain their behavior. This work systematically examines a fundamental question: for a fixed behavior to explain, and under the criteria that MI sets for itself, are we guaranteed a unique explanation? Drawing an analogy with the concept of identifiability in statistics, which ensures the uniqueness of parameters inferred from data under specific modeling assumptions, we speak about the identifiability of explanations produced by MI.

We identify two broad strategies to produce MI explanations: (i) "where-then-what", which first identifies a subset of the network (a circuit) that replicates the model's behavior before deriving its interpretation, and (ii) "what-then-where", which begins with candidate explanatory algorithms and searches in the activation subspaces of the neural model where the candidate algorithm may be implemented, relying on notions of causal alignment between the states of the candidate algorithm and the neural network. 

We systematically test the identifiability of both strategies using simple tasks (learning Boolean functions) and multi-layer perceptrons small enough to allow a complete enumeration of candidate explanations. Our experiments reveal overwhelming evidence of non-identifiability in all cases: multiple circuits can replicate model behavior, multiple interpretations can exist for a circuit, several algorithms can be causally aligned with the neural network, and a single algorithm can be causally aligned with different subspaces of the network.

We discuss whether the unicity intuition is necessary. One could adopt a pragmatic stance, requiring explanations only to meet predictive and/or manipulability standards. However, if unicity is considered essential, e.g., to provide a sense of understanding, we also discuss less permissive criteria. Finally, we also refer to the inner interpretability framework that demands explanations to be validated by multiple complementary criteria. This work aims to contribute constructively to the ongoing effort to formalize what we expect from explanations in AI.

Published as a conference paper at ICLR 2025
EVERYTHING, EVERYWHERE, ALL AT ONCE: IS MECHANISTIC INTERPRETABILITY IDENTIHIABLE?
Maxime Méloux, François Portet, Silviu Maniu, Maxime Peyrard Université Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France (me 1 ouxm, portet.f, mani us, peyra ---ngeenee--rreooeeeeeee-- fr
ABSTRACT
As Al systems are increasingly deployed high-stakes applications, ensur- ing their interpretability is essential. Mechanistic Interpretability (MT) aims to reverse-engineer neural networks by extracting human- auman-understandable algorithmss embedded within their structures to explain their behavior. This work systemati- cally examines a fundamental question: for a fixed behavior to explain, and under the criteria that MI sets for itself, are we guaranteed unique explanation? Draw- ing an analogy with the concept of identifiability in statistics, which ensures the uniqueness of parameters inferred from data under specific modeling assumptions, we speak about the identifiability of explanations produced by MI. We identify which first detects a subset of the network (a circuit) that replicates the model's ' behavior before deriving its interpretation, and (ii) "what-then weihat-then-wheee then-where", which begins with candidate explanatory algorithms and searches in the activation subspaces of the neural model where the candidate algorithm may be implemented, relying on no- tions of causal alignment between the states of the candidate algorithm and the neural network. We systematically test the identifiability of both strategies us- ing simple tasks (leariing Boolean functions) and multi-layer perceptrons small enough to allow a complete enumeration of candidate explanations. Our experi- ments reveal overwhelming evidence of non-identifiability in all cases: multiple circuits can replicate model behavior, multiple interpretations can exist for a cir- cuit, several algorithms can be causally aligned with the neural network, and a single algorithm can be causally aligned with different subspaces of the network. We discuss whether the unicity intuition necessary. One could adopt a prag- matic stance, requiring explanations only to meet predictive and/or manipulability standards. However, unicity considered essential, e.g., to provide sense understanding, we also discuss less permissive criteria. Finally, we also refer to the inner interpretability framework that demands explanations to be validated by multiple complementary criteria. This work aims to contribute constructively the ongoing effort to formalize what we expect from explanations AI.
two broad strategies to produce MI explanations: (i) wwwere-then-whaee
INTRODUCTION
Interpretability in machine learning spans diverse goals and methods (Molnar 2022 Carvalho et al., 2019 Teney al. 2022), from creating inherently interpretable models to applying post hoc tech- niques to explain model decisions. Mechanistic interpretability (MI) aims to reverse-engineer mod- els reveal simple, interprratte algorithms embedded neural network structure (Olah et al. 2020). MI is focused on generating what we call computational abstractions, where complex neural networks' behaviors are explained by simpler algorithms that track the internal computations (Olah et al. 2020). A computational abstraction a mechanistic explanation - has two components: (a) what is the explanatory algorithm, and (b) where in the computational structure is this algo- rithm embedded? Given the intractability of exhaustively searching all possible algorithms across all subsets of a neural network, researchers have developed methods with different assumptions and trade-offs. We categorize these methods into two broad strategies. The first, which we call whhere-then-whar focuses on finding a subset of the network a circuit - that captures most of the information flow from inputs to outputs. Once this circuit is identified, typically using heuristies,

---

Published as a conference paper at ICLR 2025
the next step is to interpret its components (features) to derive the explanatory algorithm (Dunefsky et al. 2024 Davies & Khakzar 2024 Conmy et al. 2023). The second approach, which we name where,,, starts by identifying candidate al algorithms and then searches subspaces in the neu- ral network where the algorithm may be implemented. This is performed using causal alignment between the explanatory algorithm's 's states and the network's internal states and typically requires approximation algorithms (Geiger et al. 2022a b). Each strategy relies on specific criteria to assess candidate explanations. For instance, circuits can be evaluated by their circuit error, which quan- tifies how closely the circuit's predictions match the full model ones (Conmy et al., 2023). In the wikttthhe-whhrr strategy, candidate algorithms are compared based on causal alignment measures like intervention interchange accuracy (IIA), which assesses how well the algorithm's is states remain aligned with the network's 's internal states after counterfactual manipulations of the states.
In this work, we question a property of explanation that appears to be tacitly taken for granted: do MI criteria guarantee a unique explanation of fixed behavior? The concept of identifiability is well-established in statisties, where model is identifiable if its parameters can be uniquely inferred from data under a given set of modeling assumptions (e.g., Rothenberg. 197T). By analogy, we extend this terminology to interpretability, defining the identifiability of explanation the property where, under fixed assumptions of validity, a unique explanatory algorithm satisfies the criteria.
Specifically, we ask the following questions: In the where-thee then-what strategy, (i) is the circuit (the "where") unique? (ii) Is a given circuit's grounding interpretation (the "what") unique? In the what- then-where strategy, (iii) is the aausall--aligned algorithm (the "'what") unique? (iv) For given algorithm, is there unique subspace of the neural network (the "where") that is causally aligned?
We stress-test the identifiability properties of current MI criteria by conducting experiments in controlled, small-scale setting. Using simple tasks like learning Boolean functions and very small multi-layer perceptrons (MLPs), we search for Boolean circuit explanations aiming to discover which succession of logic gates is implemented by the MLPs. This setup allows us to exhaustively enumerate incompatible candidate explanations and test them with existing criteria. Our experiments reveal con-identifiability at every stage of the MI process. Specifically, we find that: (i) Multiple circuits can perfectly replicate the model's behavior (with a circuit error of zero), (ii) for given circuit, multiple valid interpretations exist, (iii) several algorithms can be perfectly causally aligned with the neural computation (IIA of one), and (iv) for a given causally aligned algorithm, multiple subspaces of the neural network can be equally aligned (IIA 1).
In the discussion, we revisit whether unicity is necessary. We discuss alternative criteria and per- spectives that do not require modifying existing criteria, such as requiring explanations only to meet predictive and/or manipulability standards. However, if unicity is considered essential, e.g., to pro- vide sense of understanding, inner interpretability framework (Vilas et al., 2024) that requires an explanation to be validated by multiple complementary criteria. We hope our work contributes constructively to the ongoing effort to develop rigorous definitions for what it means to explain a complex neural network.
also discuss less permissive criteria. Finally, also refer to the
BACKGROUND
2.1 MECHANISTIC INTERPRETABILITY
Mechanistic interpretability rests on the key assumptions that a neural network's behavior can be explained by simpler algorithm than the full network, and that sparse subset of the network executes this algorithm. Previous research has given support to these assumptions: pruning studies (Gale et al. 2019 Ma et al.) 2023 Sun et al. (024) and the lottery ticket hypothesis (Frankle & Carbin 2019 Liu al. (2024) suggest that networks are often overparameterized, and only a fraction of neurons and connections are critical to the final performance. Training sub-networks (Yuan al. 2019) to approximate the full model (Liao & Kyrillidis) 2022), similar to dropout (Srivastava et al, 2014 supports the idea that sub-networks can approximate the full network's behavior well.
This search for interpretable circuits is inspired by neuroscience, which has long sought to uncover neural circuits that explain observed behaviors (Yuste) 2008). Once the neural circuit is discovered, researchers focus on interpreting the functional roles of each component in the brain (Yuste) 2008). Research in computer vision has already shown that some nodes within neural networks compute
2

---

Published as a conference paper at ICLR 2025
interpretable features (Olah al. 2017). Connections between such features, also called circuits, can be compact explamations of model behavior (Olah et al. 2020) Carter et al. 2019; Dreyer et al. 2024). Finally, recent work has applied mechanistic interpretability to LLMs (Elhage et al. [022]] especially in transformer models (Templeton et al.] 2024, Bricken et al. 2023 Vilas et al. 2023). For
example, Wang al.] (2022) identified a eircuit responsible for Indirect Object Identification (IOI) in transformers, highlighting the potential for mechanistic explanations of complex LLM behaviors.
2.2 DEFINITIONS
A satisfactory mechanistic explanation of a model's behavior consists of two com- ponents: the what, high-level algorithm that closely approximates the model's be-
Base Neural Network
1. Circuit
Computational Abstraction
havior and tracks its internal computation, and the where, specifying how and where this algorithm is embedded in the model's low-level neural computation.
We refer to the combination of an explana- tory algorithm and the mapping between the high- and low-level states as a compu- tational abstraction. This is abstraction as it simplifies the neural network's com- putation, focusing on subset of the com- putational graph and abstracting neural ac- tivations into simpler, high-level features. For example, consider the mechanistic ex- planation of how vision algorithm recog- nizes rectangles. We might identify a com- putational abstraction where certain mod- ules perform edge detection, others detect right angles, and a final component applies an AND logic gate to confirm the pres- ence of four right angles. This abstraction specifies the algorithm and how and where low-level neural activations correspond to high-level features of the algorithm. In this work, we interchange the terms expla- nation and computational abstraction.
Mapping neural activations Jeature values
Figure 1: Illustration of the computational abstraction components within a neural network. The circuit rep- resents subgraph, and the mapping specifies the high- level features computed by the circuit, detailing how their values arise from low-level neural activations. To- gether, these form the computational abstraction (ex- planation of the neural network). Here, feature F2 has three possible values and is defined within the 2D ac- tivation space of two neurons. Features Fo and Fi are binary variables, each assigned to single neuron. Fo covers the entire activation space and Fl only maps specific intervals, leaving some activations unassigned.
Formally, we define a computational ab- straction as tuple (S, r). where S is the circuit, the subset of the neural network's computational graph responsible for the behavior of interest, and J is the mapping between the states of the circuit and the states of the variables of the algorithm, which specifies how to interpret the computational function of the circuit's ' components. We now proceed to define the circuit and mapping formally.
Definition 1 (Circuit). Let G (V.E) represent the computational graph of neural network, where V is the set of nodes (neurons) and E S V x V is the set of edges (connections between neurons). A circuit $ = (Vs, Eg)) subgraph of G that contains at least one path from subset of input nodes to a subset of output nodes.
Definition 2 (Mapping (T)). A mapping between low-level values taken by neurons and high-level values taken by the variables of the explanatory algorithm consists of set of K surjective maps, one for each high-level variable. Each associates the neural network activations with the values of the corresponding high-level variable. For a group of neurons V, in the neural network, mapped to high-level variable A, with possible values ff.. the mapping : VR + {fo...... fm assigns vector of activations one of the possible values of Aj. Each mapping should be surjective (Vf,, 3h RIV3 (h) = fi) and with non-empty pre pre-image (Vf,, 7 (ff) # 0)
fm
These conditions ensure that all high-level values can be realized by some set of low-level activa- tions.
3

---

Published as a conference paper at ICLR 2025
In practice, are interested in mappings that satisfy consistency requirement. Intuitively, consis- tency means that we first perform part of the computation using the neural network and then apply the mapping to get the state of high-level variable, the outcome should be identical to applying the mapping and then performing the computation of the high-level algorithm. The computations in the neural network and the high-level algorithm should align consistently according the mapping.
Definition 3 (Consistent Mapping), Let T be a mapping between groups of low-level neurons (V;) and their corresponding high-level variables (Az). The mapping is said be consistent if for any high-level variable Ag, with parents PAj. the following diagram commutes:
Alg.
PAj
A
TPA,
R
NN,
RIKI
Here: TPA, represents the application of T to each variable in PA5; N refers to the computation between the low-level neural nenwork states; and Alg. refers the computation between high-level variables governed by the explanatory algorithm.
Previous works have explored various types of high-level features and representational abstractions, including mappings based on "directions in activation space" or specific points within activation subspaces (Olah et al. 2020 2018; Bereska & Gavves 2024). This work focuses explana- tory algorithms represented as Boolean circuits, where high-level features are binary (0 or 1). The mappings specify which activations correspond to 0 and 1. Boolean circuits computationally universal and thus sufficient to demonstrate identifiability issues in existing MI criteria.
2.3 APPROACHES TO CIRCUIT DISCOVERY
We identify and describe strategies for reverse-engineering neural networks: the where-then- what and what-then-wher approaches.
WIERE-THEN-WWIN
Methods from this strategy first aim to identify circuit that replicates the behavior of the full model well. Once circuit is found, the next step is to interpret its components to uncover the high-level algorithm being implemented (Dunetsky et al., 2024, Davies & Khakzar 2024). The evaluation criteria for circuits is how well they replicate the full model's behavior for the input of interest.
Definition (Circuit Error). Let be the function computed by a circuit and the function computed by the model which the circuit is defined. For the input set the error of the circuit S is: [ ExEx 1[S(x) = 9(x)]
In the case of perturbed inputs. it can also be defined via the KL divergence between the logits of the circuit and the model (Conmy al. 2023).
In practice, circuit search relies on causal mediation analysis, which seeks to isolate the subset of the network that carries the information from the inputs the output. Since is computationally intractable to enumerate all possible circuits in complex models (Adolfi et al.] (2024b). existing
methods focus on computing mediation formulas for individual components to decide their inclusion
in the circuit (Vig et al. 2020; Meng et al. 2022 Monea et al. 2024 Kramár et al. 2024 Conmy
al. 2023 Geva et al.
2023 Syed al. 2023).
A combination of data analysis and human input is typically used to interpret candidate circuits.
For example, activation maximization identifies inputs that maximally activate a component, which helps clarify its function (Zhou et al. 2016 Zeiler & Fergus, 2014] [imonyyan et al. 2014). This technique has been extended to modern LLMs (Peyrard et al. 2021 Jawahar et al. 2019 Dai et al.)
2022). However, polysemantic neurons, which encode multiple concepts simultaneously (Templeton et al. 2024 Bricken et al. 2023). complicate the interpretation of LLMs components. For a broader overview of these challenges, ct al. (20TT) In this work, we use the concept of consistent mapping as the objective evaluation of the quality of an interpretation.
refer readers to the following surveys: Sajjad et al. (2022); Khakzar
4

---

Published as a conference paper at ICLR 2025
WHAT-THEN HEAN-THEN-WHER
Methods from this strategy first hypothesize candidate high-level algorithm and then search for mappings between the states of this algorithm and subspaces of the neural activations. The goal is identify mappings where the high-level and low-level states are causally aligned, meaning they respond similarly under interventions.
Given a candidate high-level algorithm A. neural activations H, and a mapping defined between them, counterfactual interventions are performed on the inner variables of A, and corresponding interventions are applied to H via r. Intervention interchange accuracy (IIA) (Geiger al. (022b) is then defined for each high-level variable and measures the similarity of outputs in A and H after intervening (metric for causal aligmment). We give in Appendix [A]a complete, formal definition.
A perfect IIA score (1) for all variables indicates that all possible interventions produce the same ef- fect in low-level and high-level models. In practice, exhaustive enumeration is often impractical, and IIA is approximated using randomly sampled inputs (Geiger et al., (002b)). Similarly to the map- ping consistency defined above, perfect causal alignment requires diagram commutation between low- and high-level models under interventions.
Searching for causal alignment between high-level models and neural activations can be computa- tionally expensive, it often requires testing many potential mappings. The Distributed Align- ment Search method (Geiger et al. 2024) addresses this challenge by employing gradient de- scent search for alignments efficiently. This approach also allows for distributed representa- tions, where multiple neurons represent a single high-level variable. Indeed, an underlying assump- tion of IIA that the neural activations corresponding to distinct high-level variables are disjoint: Vx, y E V, (z) nt 1 (y) = 0, which may not occur in real-world examples (Olah et al. 2020).
Currently, no systematic method exists for choosing which candidate algorithms to test. Previous work (wu et al. 2023) has manually proposed a few candidates, but the vast space of possible algorithms makes this an open challenge.
2.4 EXPLANATION "IDENTIFIABIIITY"
The assumption of explanatory unicity the idea that there exists a single, unique explanation for a given phenomenon not only implicit in the practice of mechanistic interpretability (see rel- evant citations in Appendix C but also rooted in human cognitive and psychological tendencies (Trout) 2007] Waskan 2024 Gopnik 2000). Humans demonstrate a cognitive preference for coher- ent explanations that integrate disparate observations into a unified narrative (c.g.. Friedman 1974] Kitcher 1962: 1981 Schurz 1999; Kveraga et af.] 2007). This preference aligns with the psy- chological need for cognitive closure, defined as the desire for definitive conclusion (Kruglansk) 1989). Multiple incompatible explanations disrupt coherence, leading to ambiguity and a sense of unresolved understanding.
Conversely, in the philosophy of science, explanatory pluralism acknowledges that the world is too complex be fully described by single comprehensive explanation (Kellert et al.) (2066] Potochnik 2017]. Multiple explanations often coexist without conflict because they address different explana- tory goals (c.g., explaining distinct behaviors) or employ different simplification strategies (c.g., differing levels of abstraction Marr & Poggio 1976). However, in this work, we deliberately search for conflicting explanations by fixing both the explanatory goal and the simplification strategies, as the ones defined by MI criteria.
Identifiability and incompatible explanations.
As mentioned in Section [ we borrow the term of identifiability from the field of statistics (Rothen- berg 1971), defining identifiability of explanation as the property where a unique explanation valid under fixed standards of validity. An MI strategy is not identifiable if its standards of validity do not discriminate between two incompatible explanations.
We define two explanations as incompatible conflicting if they share the same explanatory goal and simplification strategy, but posit different computational abstractions. In our context, the ex- planatory goal is fixed: explaining the specific input-output behavior of trained MLP. The simplifi- cation strategy is also fixed, corresponding to one of two predefined strategies to find computational abstractions: the what-then--whre the where-then--what defined above.
5

---

Published as a conference paper at ICLR 2025
Cundidate Algorithms
Computational Abstractions
Example perfect mappings alpottti
2 candidates are tesell
What-then-wreere scurdhing los diflerent candidate algndimm cauaally nigned in the activatioro of the lusc nearal nowwrk
Bise Noural Netwook
CB
Tiin NIP to implement XOR
A
TM peffet ouppings (lAw3)
B
Circuits
Computational Abstractions
unique perfect acornyy
unique distraction cceei grouddig
Am XIRO
teural gate iefintions
B-aj (rauF)
resselty
Where-then--ee what searching cinills that sugpent behavcen and thes intepecinng thes Sanaro vis gronding
Figure 2: Illustration of identifiability problems using the XOR example. We train small MLP with two hidden layers of size 3 to compute the XOR function perfectly. The figure shows the outcome of stress-testing the two reverse-engineering strategies: Top: For the what-then-wher strategy, we enumerate all subsets of neurons searching for subsets causally aligned with interme- diate variables of candidate algorithms, with alignment measured by IIA. Even testing only two candidate algorithms, we find perfect implementations of both in the model. Multiple mappings (localizations) for each algorithm were identified, showing that neither the algorithm (what) nor its location in the network (where) is unique. Bottom: For the where-then-whal strategy. we enumerate circuits (sub-networks) and test whether each computes the XOR independently. For each circuit, search for possible feature interpretations of the selected neurons, identifying intermediate logic gates whose values can be mapped consistently with the neurons' activations. Consistency is defined in 3 We find many different perfect circuits (the where is not unique) and for any given circuit, we find multiple valid interpretations (the what is not unique).
Incompatibility of two explanations can occur if (1) the two explanations posit different algorithms for the same behavior, or (2) the same algorithm is embedded in different subspaces of the neural network. Both scenarios entail different internal representations and causal pathways linking inputs to outputs. In Figure we report examples of incompatible explanations in trained MLPs.
Our experiments show that even the strict causal criteria of MI allow many incompatible computa- tional abstractions. In the discussion (Section we revisit whether this expectation of unicity is necessary or even achievable.
ILLUSTRATING POTENTIAL IDENTIFIABILITY ISSUES
This section highlights identifiability counter- examples for small MLP trained to compute the XOR function. is well-known that an MLP requires least two layers to compute the XOR function. Once the network can do so, the interpretability exercise becomes: how is the XOR implemented? For mechanistic explanation, the answer must have two components: what algorithm is being used, such which combination of logic gates transforms the inputs into the XOR truth table, and where these intermediate logic gates are located within the neural network's computation i.c., where the algorithm is executed within the MLP.
counter-examples
To stress-test the two main MI strategies (where-then- small enough to allow exhaustive enumeration of all circuits and extensive search over mappings. The MLP is trained on binary inputs with single logit output to produce the XOR behavior. The inputs are 0 or and can have randomly sampled Gaussian noise of a fixed standard deviation.
and what wheeerr) we chose an MLP

---

Published as a conference paper at ICLR 2025
Our methods to test the different criteria defined in the previous section are as follows:
Circuit search: We enumerate all possible circuits, and then execute the validation data of the XOR on each circuit as if it were standalone neural network, effectively removing from the computation each node and edge that is not part of the circuit. If a circuit achieves perfect accuracy (zero circuit error), we label it a perfect circuit, it exactly replicates the model's behavior. This search tests the identifiability property of the circuit error criteria.
Interpretation search: For each perfect circuit, we attempt to interpret the activations of the in- cluded neurons based on XOR validation data. As the scope of interpretations is limited to logic gates, We search, for each neuron, logic gate whose values are consistent with that neuron's acti- vation. The method proceeds recursively. layer by layer, based on a given neuron's 's relationship with its parents in the circuit. The parents already have an interpretation (mapping their activations or 1). We enumerate all possible inputs from the parents and examine how they are mapped into the neuron's is activation by the model. We then list all possible ways to separate these inputs and label the resulting logic gate. If we find no valid interpretation for given neuron (e.g., all inputs overlap in the output activation and no separation is possible), we end this candidate interpretation of the cir- cuit. If we find multiple, we expand the tree of possible candidate interpretations for the circuit. To avoid trivial over-counting, we ignore value relabeling (c.g.. swapping and 1) and, by convention, assign I to the larger intervals and 0 to the smaller ones. The outcome is computational abstrac- tion, a Boolean circuit computing the XOR function whose internal logic gates are mapped to some neural network components. Note that this method undercounts possible interpretations because it does not consider cases where the high-level logic gates are mapped on multiple low-level neurons. This search tests the identifiability property of the mapping consistency criteria.
Algorithm search: We enumerate all algorithms implementing the XOR gate by considering Boolean formula parse trees. We assume that each neuron's activations encode either the iden- tity, AND, or OR gate, a network of depth d can implement only formulas with a parse tree of depth d. We therefore recursively enumerate all unique formulas of depth d or less using AND, OR, and negation. For d = 3, this results in 56 XOR-equivalent algorithms.
Mapping search: For a given candidate algorithm with specified intermediate logic gates, we ex- plore all possible neuron subsets and mappings between these subsets and the algorithm's 's interme- diate gates. We then measure the causal alignment of the mapping using IIA. a mapping achieves perfect IIA, we call it a perfect mapping. If there is no other mapping with larger images (set inclusion-wise) wise), we also call this mapping minimal. In the example described in this section, we manually test two candidate algorithms, while the next section enumerates algorithms that imple- ment the target function, excluding trivial variations (e.g., negating gates). This search tests the identifiability property of the IIA criteria.
We depict in Figure FigureZcounter counter- examplees for each criterion in one small MLP. In this example, for the whet-then-where strategy, only test two candidate algorithms but find 159 perfect minimal map- pings within the neural network activations, with perfect mappings for both algorithms. Therefore, the algorithm is not unique and, for given algorithm, its localization is not unique. For the where- then-what strategy, we find 85 unique circuits with perfect accuracy, with an average of 535.8 logic gate interpretations (consistent mappings) per circuit. Therefore, the localization not unique, and for given circuit, the interpreted algorithm is not unique. Overall, in this example, we obtain 159 45,543 computational abstractions, most of which are incompatible. This is serious identifiability problem as there is no clear and consensual criterion to decide among all these explanations.
4 EXPERIMENTS
4.1 QUANTITATIVE ANALYSIS
We now repeat the experiment used for the XOR example with different seeds, while varying the architecture size and the complexity of the global behavior.
The basic setup is consistent across all experiments. We choose n 2-input logic gates L Ln generate multilayer perceptron (MLP) N with layer sizes (2, k. kt, n)), and train N to implement the gates L.... Ln- Similarly to the previous section, training is performed on binary samples with added Gaussian noise and continues until the network' mean squared loss is lower than x 10 10-3
7

---

Published as a conference paper at ICLR 2025
We then quantify the identifiability issues again. For the circuit- search (where-then-we strat- egy), we count perfect circuits for L and valid interpretations for each circuit. For the algorithm-first search wheere strategy), N and perfect minimal mappings for each algorithm.
count perfectly aligned algorithms for L in
In the circuit-first search, only search for circuits containing two inputs and output for each target gate. Furthermore, due to the combinatorial explosion in circuit enumeration, we only test circuits with sparsity greater than 0.3, where sparsity is measured as the fraction of components excluded from the circuit. This number is chosen as the smallest sparsity that remains manageable for the size of MLPs that we consider. As a result, the reported number circuits should be consid- ered lower bound. Furthermore, the number of potential interpretations for single circuit grows exponentially with its size. Since we count interpretations for only the sparser circuits, the reported number of interpretations is also significantly lower than exhaustive search would yield.
4.1.1 ARCHITECTURE SIZE
Increasing the neural network's size may impact the number of computational abstractions found in networks. Although larger architecture may create more computational abstractions due the increased search space, it could also lead to greater overparameterization. meaning smaller subset of the network may suffice implement the target gate. This, in turn, could reduce the number of valid abstractions if most of the network is inactive during inference.
In the left side Figure 3 we report the total number of explanations found when the architecture size ranges from k = 2 to k = 5. We exclude from this figure the networks for which no valid mapping or interpretation was found, which we give in Appendix D.2 along with additional plots.
30"
nga
Inte precations
307
20
30
200
307
304
1.7
307
20a
K-3 8-44 K-5
1.0025-1 0,00005-2
0.92000-3
1 T,COROGHO 1.9005--0
Figure 3: Number of computational abstractions found in the circuit-firs: approach (circuit interpre- tations) and the algorithm--ing first approach (perfect minimal mappings), as a function of architecture size is (left) or the number of gates the model trained on (right, averaged over all target gates). One point per neural network.
In both cases, observe that the number of computational abstractions found significantly in- creases with network size, with median values growing from 38 to 910,000 in the circuit-fire first method and from 8 to 3,700 in the algorithm first approach. Less than 2% of the trained networks contain exactly one valid minimal mapping, and no network contains exactly one circuit interpretation.
4.1.2 MULTIPLE TASKS
We also investigate the effect of global behavior complexity. The model is trained to implement single logic gate in the basic setup. What happens when the network is trained in multi-task setting? As the number of target tasks increases, we expect the network to use its activations more efficiently, possibly relying on smaller subset of its structure for each task. To explore this, we fix 3 and vary n from to 6, sampling n logic gates (without replacement) from the same list as above, extended to include the negation of the gates (NOR, NAND, NIMP, and XNOR). The neural network N is then trained to implement these gates in parallel, using two shared input neurons and output neurons (one per gate). We repeat this procedure using different random seeds.
8

---

Published as a conference paper at ICLR 2025
The right side of Figure contains the total number of computational abstractions obtained when training each neural network on a different number of logic gates in parallel, ranging from to 6. More detailed plots are available in Appendix D.3
In both approaches, the number of interpretations significantly decreases with the number of training tasks (p 0.05), up 4 tasks. Past that point, the variation is longer statistically significant.
4.2 TRAINING DYNAMICS
A possible explanation for the high number of computational abstractions we find in trained net- works is that our networks do not perfectly implement the target logic gates, since training stopped when a low but non-zero loss value is reached. We explored this effect by varying the loss cutoff in the basic setup. The results are given in Appendix D.4 More generally, the influence of the training distribution on the number of valid computational abstractions is investigated in Appendix D.6) In addition, we find that adding noise to binary samples during training has no significant effect on the results obtained in the algorithm-first method, but decreases the number of circuits while increas- ing the overall number interpretations found in the circuit-first method (results are described in Appendix D.5). Training dynamics and generalization abilities of a model may therefore reduce the number of available abstractions, but this effect alone is unlikely to mitigate the issue entirely.
4.3 TOWARDS LARGER MODELS
While enumerating circuits mappings is infeasible in large networks, it is still possible to find counterexamples in which multiple circuits exist. For example, we trained a larger MLP on a subset of the MNIST dataset (Deng) 2012). filtered to contain only the digits 0 and 1. We obtained regression model with layer sizes (784, 128, 128, 3, 3, 3, 1). After training, we extracted the last layers of the model to form two sub-networks: one of size (784, 128, 128, 3) and one of size (3, 3, 3, 1). We fed the training samples through the larger sub-network, generating new dataset comprised of partial computations the overall model.
Applying the circuit search method to the smaller sub-network using this new dataset yielded 3,209 valid circuits. While we cannot enumerate circuits in the first half of the network, any such valid circuit can include one of the circuits of the second half its continuation. Two situations may arise: If the first half of the MLP does not contain any valid circuits, then no valid circuit exists for the full network; if valid eircuits exist in the first half of the MLP, then minimum of 3,209 valid circuits exist in the full network. This shows, at least in the case of circuits, that the problem does seem to disappear with significantly larger scale and more complex data distributions.
WHAT DOES IT MEAN FOR INTERPRETABILITV..
Our findings challenge the strong intuition that a unique mechanistic explanation exists for given behavior under fixed explanatory goals and validity criteria. Even when employing the strict causal requirements of MI, find that many incompatible explanations can coexist. While predictive of behavior and causally aligned with the neural network's states, these explanations differ in the computational algorithms they postulate how they are embedded in the network's subspaces. We now discuss several ways to move forward from this striking observation.
5.1 DOES LACK OF UNICITY MATTER?
Whether multiple "valid" explanations pose real problem is worth considering. From a pragmatic stance. one could argue that unicity is not essential if the explanations meet functional goals such as predictivity, controllability, or utility in poecision-makinn This perspective emphasizes crafting practical criteria to evaluate explanations based on their utility, rather than their ontological closeness to the truth. Stating explicitly the pragmatic goals of an explanation can also clarify what is expected of an explanation (Woodward & Ross 2021). For example, in the recent debate about interpretability illusion, Makelov et al. (2023) mention problems about interventions that can potentially activate dormant pathways leading the resulting explanation to misrepresent the mechanisms at play. In their response, one of the arguments advanced by Wu et al. (2024) is point that the explanation produced by their method (DAS), still meets the
(Van Fraassen 1988 AAchnnttenn 1984).

---

Published as a conference paper at ICLR 2025
pragmatic goals of predictivity and manipulabilityy ensuring its usefulness. The debate is resolved by clarifying the epistemic goals of the explanation.
5.2 IF YES, HOW CAN WE AIM TO RESOLVE IT?
If we decide that identifiability of explanations is important, our work demonstrates that current MI criteria are insufficient to guarantee it. One potential approach to resolving this issue involves introducing additional heuristics, such as prioritizing the sparsest circuits. However, Occam's 's razor alone is unlikely to solve the problem. Should we dismiss an entirely different candidate explanation simply because it involves one additional node than another? In experiments, simplicity or sparsity cannot single out one explanation.
To address these challenges, we believe that ideas from causal abstraction (Beckers Halpern 2019 Beckers et al. 2020, RRubenstein et al., 2017) can be helpful (Geiger et al.) (022a).. Although IIA directly inspired by causal abstraction, does not fully implement it in current form. Unlike current MI frameworks, causal abstraction requires that all lower-level model states are accounted for in higher-level representations. Furthermore, components are excluded from an explanation, their absence must be justified causally. This intuition has been formalized recently through the concept of faithfulness, which evaluates how well a circuit replicates the model's behavior and the (lack of) impact of excluded elements (Hanna et al. 2024).
Alternatively, one can look at broader approaches and not focus on searching for explanations that optimize single criterion. Interestingly. Vilas et al. (2024) propose an inner interpretability frame- work based on lessons from cognitive neuroscience. In this framework, the authors emphasize the importance of building multi-level mechanistic explanations and stress-testing these explanations with proper hypotheses testing. An explanation validated by many criteria and which exhibits differ- properties (e.g., invariances) becomes trustworthy. This framework provides promising path to establish MI as a natural science akin to neuroscience or biology.
Vilas
5.3 Is IDENTIHABBILTYY EVEN ACHIEVABLE?
In domains of science, competing theories coexist despite being ontologically incompatible. For instance, the Lagrangian and Hamiltonian formulations of classical mechanics posit different underlying entities but yield identical experimental predictions. Such scenarios are examples of con- trastive underdeterminnatio, philosophical debate about whether empirical evidence can or should uniquely determine scientific explanations (Stanford 2023). The intrinsic computational hardness of interpretability queries (Adolfi et al., (024b)a) a) suggests that MI may have fundamental limits, leaving it possibly underdetermined. More broadly, the search for MI explanations can seen as type of causal representation learning, where identifiability may depend on strong constraints that might not be realistic Morioka Hyvärinen (2024).
5.4 FROM TOY MODELS TO REAL MODELS
Our experiments focus on toy MLPs trained on toy tasks, which differ drastically from large lan- guage models (LLLMs) trained on vast, complex datasets using Transformer architectures. This raises the possibility that the issues in small models may not apply to larger, more sophisticated models. However, if this is true, why the problems would disappear at larger scales must be demonstrated. Understanding why current criteria function well in some regimes but not in others would also lead to refined criteria and definitions.
5.5 CONCLUSION
Our results should encourage the community to reflect the role of unicity when searching for and communicating about mechanistic explanations found neural networks. We believe that exploring stricter criteria based causal abstraction, explicitly formulating pragmatic goals of explanations and embracing broader frameworks such as the inner interpretability one are all promising directions.
The code and parameters used conduct this paper's experiments can be found GitHuub
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was conducted within French research unit UMR 5217 and was partially supported by CNRS (grant ANR-22-CP1--033-601) and by MIAI Areeoble-AApes (grant ANR-19-P3IA-0003,, It was granted access to the HPC resources of IDRIS under the allocation 2025-AD01 1014834 made by GENCI.

---

Published as a conference paper at ICLR 2025
REFERENCES
Pete Achinstein. The pragmatic character of explanation. In PSA: Proceedings the Biennial Meet- ing of the Philosophy of Science Association, volume 1984, pp. 274-292. Cambridge University Press, 1984.
Federico Adolfi, Martina G. Vilas, and Todd Wareham. The computational complexity of circuit discovery for inner interpretability, 2024a.
Federico Adolfi, Martina G. Vilas, and Todd Wareham. Complexity- Limits the Promises of Artificial Neural Network Geverse-Engineering. Proceedings of the Annual Meet- ing of the Cognitive Science Society, 46(0), 2024b. URL https://escoolarshpppooo uc/ mts/item22h78mm/
Sander Beckers and Joseph Y. Halpern. Abstracting causal models. Proceedings the AAAI Confer- ence on Artificial Intelligence, 33001)))677--285. Jul. 2019. doi: 10.1609/aaai, v33i01 1..00.311268
URL Sattps://oo3 aaai www.aaii..r/iiixxp.pi.i. httpp///ww.aaaaaaeeeaeaaaaaeeee/ httpa/aattceeeee./e/eeeeee
Sander Beckers, Frederick Eberhardt, and Joseph Y. Halpern. Approximate causal abstractions. In Ryan P. Adams and Vibhav Gogate (eds.), Proceedings of The 35th Uncertainty in Artifi- cial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pp.
606-615. PMLR, 22-25 Jul 2020. URL https::t/////tt htctps://rroceedigg: wwl.gggsmll.....mmm /ttps/// beckers20a.. html
Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety review, 2024. wtt.oorgg/bb//////././/// 14082
URL
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https:/trrnssfrmmer- circuits. http://00///mooommmmmmaacaaasse
Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve Circuits. Distill, 6()):e00024.006, January 2021. ISSN 2476-0757. doi: 10.23915/distil..000 006.
http:/iiccttttutttteoe curve ci circuits
Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas. Distill, 2019. doi: 10.2391 013391//disiill.00011 https://distill http:p/p////aaaiatiiioooo-aa--a 9/activation-aalas
Diogo V. Carvalho, Eduardo M. Percira, and Jaime S. Cardoso. Machine learning interpretability: A survey on methods and metrics. Electronics, 8(8), 2019. ISSN 2079-9292. doi: 10.3390/ clectronics8080832. URL https: //ww: mdpi com wttpcc/////929/////2220030
Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. In A. Oh, T. Naumann, A. Globerson, K. Saenko. M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 16318-16355 Curran Associates, Inc., http://www..ccppppeeeeeeeee/eeee fflees/papeer/222// 2023. URL 2eee3eeeeebbeS3333eeeeaaabbbbbee her.s//eeeeeepppeeooooeeeeeeeeee Conference.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Anmual Meeting the Association for Computational Linguistics (Volue 1: Long Papers), Dublin, Ireland, May 2022. Association for Computational Linguistics. URL https: ac antho ooy.oor..
Adam Davies and Ashkan Khakzar. The cognitive revolution in interpretability: From explaining behavior to interpreting representations and algorithms, 2024. URL https: htap:s///axxa ///r//aavvv.///////////////aav/v org/ abs/ /20/2448...2 05859
12

---

Published as a conference paper at ICLR 2025
Li Deng. The mnist database of handwritten digit images for machine learning research. 1EEE Signal Processing Magazine, 29(6):141-142, 2012.
Maximilian Dreyer, Erblina Purelku, Johanna Vielhaben, Wojciech Samek, and Sebastian La- puschkin. Pure: Turning polysemantic neurons into pure features by identifying relevant circuits, 2024. URL httttt://axxiiiiii http://aaa.aa...r/aa//2/444444. 06453
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable Ilm feature circuits, 2024. URL.https://aaxixv. org/ abs/2406.... 11944
Nelson Elhage. Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/teansfoomee-ccrriiitt http://asssppppb/2////////moo///
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neu- ral networks. In ICLR. OpenReview.net, 2019. URL ptttp://dblp.. uni-trier de/db/
Michael Friedman. Explanation and scientific understanding. Journal of Philosophy, 71(1):5-19, 1974. doi: 10.2307/2024924.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, 2019.
URL httpt:tt//xxxi...ooo org/
Atticus Geiger, Zhengxuan Wu, Karel D' D'Oosterlinck, Elisa Kreiss, Noah D. Goodman. Thomas Icard, and Christopher Potts. Faithful, interpretable model explanations via causal ab- straction. Stanford AI Lab Blog, 2022a. URL https: .stnnford... wdw.bloogggooooo causal-abatract..
Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Good- man, and Christopher Potts. Inducing causal structure for interpretable neural networks. In Ka- malika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 7324-7338. PMLR, 17-23 Jul 2022b. URL Setps://prrceeet: ngs ml www...........mmmmmmmmmmmmm 62 gewger22......oo html
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Find- ing alignments between interpretable causal variables and distributed neural representations. In Francesco Locatello and Vanessa Didelez (eds.). Proceedings of the Third Conference Causal Learning and Reasoning, volume 236 of Proceedings of Machine Learning Research, pp. 160-187. PMLR, 01-03 Apr 2024. URL https: geiger24a. html
mwl.ggsmm.......m mlr. peill.prees/v233/
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 12216 12216-12235. Singapore, December 2023. Association for Computational Linguis- tics. doi: 10..1863/v11// wttp:022..mmlmmmmme main.751. /
eemrlp ma imin. 751
Alison Gopnik. Explanation as orgasm and the drive for causal knowledge:
The function, evolution, and phenomenology of the theory formation system. In Explanation and cognition., pp. 299-323. The MIT Press, Cambridge, MA, US, 2000. ISBN 0-262-1 1249-3 (Hardcover).
Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms, 2024. URL https: //tp///aavvv///////////e/.//.wa/ wwg/abg....... 2403. 17806
13

---

Published as a conference paper at ICLR 2025
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? In Anna Korhonen, David Traum, and Lluís Marquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics. URL httl.ttts::// ac way.hoooo...o........o....
Stephen H. Kellert, Helen Longino, and C. Kenneth Waters. Introduction: The pluralist stance. In Stephen H. Kellert, Helen Longino, and C. Kenneth Waters (eds.), Scientific Pluralism, pp- vii-xxix. University of Minnesota Press, 2006.
Ashkan Khakzar, Soroosh Baselizadeh, Saurabh Khanduja, Christian Rupprecht, Scong Tae Kim, and Nassir Navab. Neural response interpretation through the lens of critical pathways. In 202/ IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13523-13533,
2021. doi: N0.1109/CVPR444437.2222 2012.00066477.2200111333....
Philip Kitcher. Explanatory unification and the causal structure of the world. Philip Kitcher and Wesley C. Salmon (eds.), Scientific Explanation, pp. 410-505. Univ of Minnesota Pr, 1962.
Philip Kitcher. Explanatory unification. Philosophy of Science, 5774)3307- 1981. ISSN 00318248, 1539767X. URL httt:ht:///wwee http://www...ooooooe/tat//ae////
János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. Atp": e and scalable method for localizing I1m behaviour components, 2024. URL attpsp///axx..oor////bb.
00745
Arie W. Kruglanski. Lay epistemics and human knowledge: Cognitive and motivational bases. Lay epistemics and human knowledge: Cognitive and motivational bases. Plenum Press, New York, NY, US 1989. ISBN 9780306430787.
Kestutis K Kveraga, Avniel S. Ghuman, and Moshe Bar. Top-down predictions in the cognitive brain. Brain and Cognition, 65(2):145 168, 2007. ISSN 0278-2626. doi: http://doi.org//010016// Randc.2007..0 007. URL https: www.sciieneeireee..oooo com/ http://t...ccccceceecrreritaaaa
p11/5007822260700CC
Fangshuo Liao and Anastasios Kyrillidis. On the convergence of shallow neural network training with randomly masked neurons, 2022.
0266688.
Bohan Liu, Zijie Zhang, Peixiong He, Zhensen Wang. Yang Xiao, Ruimeng Ye, Yang Zhou, Wei- Shinn Ku, and Bo Hui. A survey of lottery ticket hypothesis, 2024. URL httpp:://rra:t
orq/ hbgg/ab//aat 2403.443..0 04861
Xinyin Ma. Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems, 2023.
Aleksandar Makelov, Georg Lange, and Neel Nanda. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching, 2023. URL URR/pttpp://aaxxivoorgg abs/ 2311 00211.17033 7030
Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs language models, 2024.
org. abs/ /2t///24433 146.199677
D. Marr and T. Poggio. From understanding computation to understanding neural circuitry. Techni- report, USA, 1976.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations in GPT. Advances in Neural Information Processing Systems, 36, 2022. arXiv:2202.05262.
Christoph Molnar. Interpretable Machine Learning. 2 edition, 2022.
URI. https://
Giovanni Monea, Maxime Peyrard, Martin Josifoski, Vishrav Chaudhary. Jason Eisner, Emre Kıemman, Hamid Palangi, Barun Patra, and Robert West. A glitch in the matrix? ? locating and de- tecting language model grounding with fakepedia, 2024. URL https:/ // //rxivv www.wii.orgaaaaaa
2312. 02073
14

---

Published as a conference paper at ICLR 2025
Hiroshi Morioka and Aapo Hyvärinen. Causal representation learning made identifiable by group- ing of observational variables. In Proceedings of the 41st International Conference on Machine Learning. ICML'24, JMLR.org. 2024.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.23991/diittll.00000
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. doi: 10.23915/
distill.00010. https::/distill. pubp:/00lllpp//000000/ bouulddigg-books..
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Distill, 2020. doi: 10.239 2/2991//iisilll00004.0000
Zoom in: An introduction circuits.
https://distill http://ssllllluo222222ccccuooooo
Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009, ISBN 052189560X.
Maxime Peyrard, Beatriz Borges, Kristina Gligorić, and Robert West. Laughing heads: Can trans- formers detect what makes sentence funny? In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 3899-3905. International Joint Conferences on Artificial Intelligence Organization, 2021. doi: 0..24663ijcci.22221/53.. URL
ijcai 2021/537 /
Main Track.
Angela Potochnik. Idealization and the Aims of Science. University of Chicago Press, Chicago, 2017.
Thomas J. Rothenberg. Identification in parametric models. Econometrica, 39(3)5777-591, 1971. ISSN 00129682, 14680262. URL http:: http://www.tttrroooo.oo.oooooott wtw.://t..or///aae////111113331.
P. K. Rubenstein, S. Weichwald, S. Bongers, J. M. Mooij. D. Janzing, M. Grosse-Wentrup and
B. Schölkopf. Causal consistency of structural equation models. In Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence (UAI), pp. ID 11, August 2017. URL *equal contribution.
//auai. wttp://aa2001//gooeeeeiiggg httpe///ddnnn//pppp/rrrr..e
Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level Interpretation of Deep NLP Models: A Survey. Transactions of the Association for Computational Linguistics, 10:1285-1303, 11 2022. ISSN 2307-387X. doi: 10.1162/tac...000511 URL
tacl a 00519
Gerhard Schurz. Explanation as unification. Synthese, 120(1):95-114, 1999. doi: 10. 1023/a:
1005214721929.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014. URL https://arxiv.
prg/abs/Dn
abs aaaaaas111 1311.31166 6034
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 19956):199--195,, 2014. URL
srivastaval4a. html
Kyle Stanford. (eds.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University. Summer 2023 edition, 2023.
Underdetermination of Scientific Theory. In Edward N. Zalta and Uri Nodelman
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models, 2024. URL
11695
Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery, 2023. URL http:ss:/aaxxw..rog///////o 10348
0348
15

---

Published as a conference paper at ICLR 2025
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Trans- former Circuits Thread, 2024. URL
pub/2024/
scaling monosemanticity: index. html
Damien Teney, Maxime Peyrard, and Ehsan Abbasnejad. Predicting not understanding: Recogniz- ing and addressing underspecification in machine learning. In European Conference on Computer Vision, pp. 458 476. Springer, 2022.
Trout. The psychology scientific explanation. Philosophy Compaxs, 2(3):564-591, 2007. doi: http://o..or//001111117777779999 000.17477779991220000000088... onlinelibrary http:/awww...lle...../o///aaaaaa
URL /tttps::/ccmppsstt
111111777999111 2007.0008...
Bas Van Fraassen. The pragmatic theory of explanation. Theories of explanation, 8:135-155, 1988.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances Neural Information Processing Systems, volume 33, pp. 12388-12401. Curran Associates. Inc., 2020. URL https: htttp:://proeedingg.eeeooooo... wwtppppe......eperrrreeepeeeeeee files/ paper/ 2000://220///20 file//iee///// 011.:/11000622222711111111111130
Martina G. Vilas, Timothy Schaumlöffel, and Gemma Roig. Analyzing vision trans- formers for image classification in class embedding space. In Advances in Neu-
Information Processing Systems, volume 36, pp. 40030-40041, 2023. URL
new.:iinnnn.euuuusppep.......prs nwwrips. http://pppppppepeeee//eaeaaaaeaa files//app::2222//E 0at.a//00000000000000----46----0
Martina G. Vilas, Federico Adolfi, David Poeppel, and Gemma Roig. Position: An inner in-
terpretability framework for ai inspired by lessons from cognitive neuroscience, 2024. URL
https::/arxiv. wwwpxii.org//as... /406 01352
Kevin Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: circuit for indirect object identification in gpt-2 small, 2022. URL
atttss://arxi http///axx.v...//abb..a./e /ttaas//2111... 00593
Jonathan Waskan. Experimental Philosophy of Science: Scientific Explanation, pp. 237-262. De
Gruyter, Berlin, Boston, 2024. ISBN 97831 16311 doi: do::100.511/99781107766933
URL https:://oi.. ort.loorg........ http:/1111111199111111
011
James Woodward and Lauren Ross. Scientific Explanation. In Edward N. Zalta (ed.), The Stan- ford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2021 edition, 2021.
Zhengxuan Wu, Atticus Geiger, Thomas Ieard, Christopher Potts, and Noah Goodman. In- terpretability at scale: Identifying causal mechanisms in alpaca. In A. Oh, Nau- mann, A. Globerson, K. Saenko, M. Hardt, and Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 78205-78226. Curran Associates, Inc., fitpse///eeeeeeee/22/3333 File/ hal.e/aaaeeeaarteeeeeeeaeaaaaaaa
2023. URL |nttps://prrceeellnggs.. wwwp:l/ww...uiiiiiioooco
Zhengxuan Wu, Atticus Geiger, Jing Huang, Aryaman Arora, Thomas Icard, Christopher Potts, and
Noah D. Goodman. A reply to makelov et al. (2023)'s "interpretability illusion" arguments, 2024.
URL MtRLhttt:::/aamm wrw.vvorrggg.. http//bs/244111 12631
Binhang Yuan, Anastasios Kyrillidis, and Christopher M. Jermaine.
Distributed learning of deep neural networks using independent subnet training. CoRR, abs/1910.02120, 2019. URL http:
//axxiv..rr/aaa//11000 02120
Rafael Yuste. Circuit neuroscience: the road ahead. Frontiers in neuroscience, 2:1038, 2008.
16

---

Published as a conference paper at ICLR 2025
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David J. Fleet, Tomás Pajdla, Bernt Schicle, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich. Switzerland, September 6-12, 2014, Proceedings, Part volume 8689 of Lecture Notes in Computer Science, pp. 818-888888880 Springer, 2014. doi: 10.1007/ 978-3-319- 10.50-11_53. 1,.3.
URL https: /ttp://///0...rr1111100000111110 3-1
53
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for dis- criminative localization. 2016 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 2921-2929, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society. doi: 10.1109/CVPR..01..222 URL 1109/CVPR.C 2016.319
17

---

Published as a conference paper at ICLR 2025
A FORMAL DEFINITION OF IIA
The definitions in this section are adapted from Geiger et al. (2022b). We begin by setting notation conventions.
Let N be a neural network, and A be high-level algorithm. Let mapping between low-level neuron groups {Vi) of N and the values of their correspond- ing high-level variables [Aj} in A.
Let Vín (resp. Vout) be the neuron groups corresponding to the imputs (resp. outputs) of N. Let Ain (resp. Aour) be the variables in A with no parents (resp. no children).
Notation (Value reading). Let Unn E IR
Val be some possible input values of the network N. We note N [umn Vj] the values of the activations of Vj that are obtained when setting the values of Vin to and running the computation graph of N.
Similarly, let Uin E RlAal be some possible values of the variables of A with parenis. We note A Aam: Aj] the values of the variable A₃ that are obtained when setting the values of Ain to am and running the algorithm A.
Notation (Intervention). Let Vi E [R network N. We note Nv,+2, copy of N. in which the activations of V, are forcibly set to the value during the computation.
some possible activations of the variable Vj in the
Similarly, let 0j E R Blaa be possible value of the variable Aj in the algorithm A. We note A the value ag when the algorithm is run.
copy of A, in which the value of Aラ is forcibly
This notion is aligned with the do-operator (Pearl 2009).
Definition (Intervention interchange). Let basetin sourcetin R VV51 be some possible input values of the network N. We call low-level intervention interchange the quantity:
Hoo (N, basel,in sourcet,nn Vx) = (Nvve-Noonne Mssourccan Vo) (baser.m: Voul
Similarly, let base sourcen, parent. We call high-level intervention interchange the quantity:
BlA be some possible values of the variables in A with
Hhigh (N, N, basenin, source Av) = (A Moor-eos [base Aourl
This corresponds to the notion of counterfactual intervention: After running the algorithm (or net- work) on a set inputs (source) and recording the value of a given variable, we execute the algo- rithm again a different set of inputs (base) but restore the value of the variable from the first run during the computation. The system now in counterfactual state, and we measure its new output.
Definition 6 (IIA). Let Va((Ag) be the set of possible values of Aj, and Val( = IIvev, Val(V) be the possible combinations of values of the variables in A with no parents. Lef Ay be high-level variable of A.
The intervention interchange accuracy of the mapping T for the variable Ay the quantity:
HA(N, A, AR,7)
[high (A, Ax) = 11 N, b, 8, Vx)
(aleV)((A..)
B OUTPUT EXAMPLES
This section contains additional examples of computational abstractions found by both strategies. Specifically, we report abstractions found a single neural network trained on the XOR gate with k = 3 and with loss cutoff of 10-3
18

---

Published as a conference paper at ICLR 2025
B.1 CIRCUIT-FIRST APPROACH
An exhaustive pass of the circuit-first approach (with minimal sparsity threshold) yielded 59 circuits (#where*). We depict in Figure T the 12 most sparse circuits found.
Figure 4: The 12 most sparse circuits found in the example network.
When searching for interpretations (#what*) in all 59 circuits, we find 114,230 interpretations. We then focus circuit 8 (second row, last column in Figure 4 chosen for illustration purposes yields 24 valid interpretations, which we fully list Table Each of these interpretations leads to different explanation of the network; for example, interpretation corresponds to the formula -(-AAA B) + -(AV B)), while interpretation 2 corresponds to -((-(AA (Av 7 (A(ABB) (AV B))).
B.2 ALGORITIIM-FIST APPROACII
In the algorithm-fin first approach, exhaustive enumeration yields 56 possible logic formulas for the XOR gate with a depth of (excluding commutative-invart formulas). Four of these formulas produce valid mappings for the neural network. Those mappings are listed in Table 2
C IDENTIFIABILITY IN CIRCUIT LITERATURE
Identifiability is, the best of our knowledge, never stated explicit assumption in existing works about circuits. In this section, we list examples from the literature that indicate that it is nonetheless typically taken for granted:
Cammarata et al.
2021 "the curve circuit" (multiple occurrences)
Wang al. (2022)
"discover the circuit", "discovering the circuit", "uncover the circuit"
Kramár
2024) "we investigate the circuit underlying multiple-choic question-
answering"
Conmy al. (0033):
"Choosing clearly defined behavior means that the circuit will be easier to interpret than a mix of circuits corresponding to a vague behavior". "ACDC [...] fully recovers the circuit of toy model".
Hanna al. (204):
"We next search for the circuit responsible for computing this task"
Marks al.
2024 "The circuit for agreement across prepositional phrase (Figure 12y"
19

---

Published as a conference paper at ICLR 2025
Neuron (1, 0)
Neuron (1, I)
Neuron (2, 0)
Neuron (2,
Neuron (3, 0)
#
Gate
Sep.
Gate
Sep.
Gate
Sep.
Gate
Sep.
Gate
Sep.
-0.003
NAND
0.693
NOR
0.151
IMP
0.777
OR
0.5
NOT A
2 -0.003
NAND
0.693
NOR
0.151
IMP
0.777
OR
0.5
RNIMP
-0.003 3
NAND
0.693
NOR
0.151
IMP
0.777
A
0.5
NOT A
-0.003 4
NAND
0.693
NOR
0.151
IMP
0.777
A
0.5
RNIMP
5
-0.003
NAND
0.693
NOR
0.151
IMP
0.987
NIMP
0.5
IMP
6 -0.003
NAND
0.693
NOR
0.151
IMP
0.987
NIMP
0.5
NOT A
7
-0.003
NAND
0.693
NOR
0.151
IMP
0.987
NIMP
0.5
B
8 -0.003
NAND
0.693
NOR
0.151
IMP
0.987
NIMP
0.5
RNIMP
9 -0.033
NAND
0.693
NOR
0.397
NOT A
0.987
NIMP
0.5
B
10
-0.003
NAND
0.693
NOR
0.397
NOT
0.987
NIMP
0.5
RNIMP
11
-0.033
NAND
0.693
NOR
0.397
NOR
0.987
NIMP
0.5
B
12
-0.003
NAND
0.693
NOR
0.397
NOR
0.987
NIMP
0.5
RNIMP
13
0.321
NOR
0.230
NAND
0.151
RIMP
0.777
OR
0.5
NOT A
14
0.321
NOR
0.230
NAND
0.151
RIMP
0.777
OR
0.5
RNIMP
0.321 15
NOR
0.230
NAND
0.151
RIMP
0.777
B
0.5
NOT A
0.321 16
NOR
0.230
NAND
0.151
RIMP
0.777
0.5
RNIMP
17
0.321
NOR
0.230
NAND
0.151
RIMP
0.987
RNIMP
0.5
IMP
18
0.321
NOR
0.230
NAND
0.151
RIMP
0.987
RNIMP
0.5
NOT A
19
0.321
NOR
0.230
NAND
0.151
RIMP
0.987
RNIMP
0.5
20
0.321
NOR
0.230
NAND
0.151
RIMP
0.987
RNIMP
0.5
RNIMP
21
0.321
NOR
0.230
NAND
0.397
NOT B
0.987
RNIMP
0.5
22
0.321
NOR
0.230
NAND
0.397
NOT B
0.987
RNIMP
0.5
RNIMP
23
0.321
NOR
0.230
NAND
0.397
NOR
0.987
RNIMP
0.5
24
0.321
NOR
0.230
NAND
0.397
NOR 0.987
RNIMP
0.5
RNIMP
Table 1: The list of interpretations found for circuit 8 of 59 in the example network. For each interpretation, the four intermediate neurons and the output one are assigned a logic gate and separation boundary. Each neuron is represented by its layer and position in the layer (indexed from 0), as read from left to right and from top to bottom Figure IMP refers the implication gate (A implies B), NIMP to its negation, and RIMP and RNIMP refer to the reversed implication gate (B implies A) and its negation.
Formula ((A B) (AVB)
Formula -((AA() (AV B))
Mapping
AVB
(AA M B)
Mapping A B
-(A B)
Neuron (1, 2)
Neuron (1, 1)
Neuron (1, 1)
Neuron (1, 2) Neuron (1, 0)
2
Neuron (1, 0)
Neuron (1,
2
Neuron (1,
Formula 39:
-((A B) (AV B)) (AV A B
Mapping
AV B
-((A B) V-AV B))
-(Av B)
Neuron (1, 1)
Neuron (1, 2)
Neuron (2, 0)
Neuron (1, 0)
2
Neuron (1, 1)
Neuron (1, 2)
Neuron (2, 1)
Neuron (1, 0)
3
Neuron (1, 1)
Neuron (1, 0)
Neuron (2, 0)
Neuron (1, 2)
4 Neuron (1, 1)
Neuron (1, 0)
Neuron (2, 1)
Neuron (1,
Formula 40: -((AA () (AV -(AV B))
Mapping (A B) V (AV B)
A B
-(A V B) (left)
-(AV B) (left)
Neuron (2, 1)
Neuron (1, 1)
Neuron (1, 0)
Neuron (1, 2)
Neuron (2,
Neuron (1, 1)
Neuron (1, 2)
Neuron (I,
Neuron (2, 0)
Neuron (1, 1)
Neuron (1, 0)
Neuron (1, 2)
Neuron (2, 0)
Neuron (1, 1)
Neuron (1. 2)
Neuron (1,
Table 2: The list of valid minimal mappings for the example network. For each intermediate node of each formula, we specify which neuron corresponds to that node.
20

---

Published as a conference paper at ICLR 2025
D ADDITIONAL PLOTS
D.1 TARGET GATE VARIATION
Figure contains the total number of computational abstractions obtained after fixing A and
and sampling the target gate from the following list: AND, OR, XOR, IMP.
mappangs
pet Indeepratttinn pttaaaai
10
107
20-
OATEHAND
gateRIMP
Figure 5: Total number of interpretations found in the circuit-first approach and mappings found in the algorithm-ffre first approach, grouped by target gate.
Figure 6 contains the results of same experiment but displays separate plots for the number of circuits and interpretations per circuit (resp. algorithms and mappings per algorithm) for each network.
Grouts
mappings per alg. lave)
per circ. (avg)
10
20a
36-
Figure 6: Left: Number of circuits and average interpretations per circuit found in the circuit-fisst approach. Right: Number algorithms and average mappings per algorithm found in the algorithm- first approach (right). Results are grouped by target gate over 100 experiments.
D.2 ARCHITECTURE SIZE
Figure 7 contains additional plots for the experiment described in 4.1.1 in which we vary the archi- tecture size.
D.3 MULTI-TASK TRAINING
We report in Figure additional plots for the experiment in which we vary the number of gates the model is being trained (described in 4.1.2.
21

---

Published as a conference paper at ICLR 2025
Aggetthme intal mwppingadggurillimttt. (ong)
derpretationnncc lavg)
10
10a
104
107
107
101
104
100
109
K=2 150531
km3 (58/8))
t:4 (60/50)
km (50,03)
k=2 (17734
km3 (3425)
km4 155/73
k:55 (63,82)
Figure 7: Number of abstractions found in the circuit-first approach (left) and the algorithm-first approach (right) as function of the architecture size.
Agarthrr
dreuss
Minimal
aaprraaaiione cite. (avg)
10
30'
1.00005-1
0.g3005-2
n_00005-3
0
n
n,93005-2
n.g0025-3
nLgatos-6
R_gatts-8
Figure 8: Number of abstractions found in the circuit-first approach (left) and the algorithm-fing approach (right) function of the number of training tasks.
22

---

Published as a conference paper at ICLR 2025
D.4 LOss CUTOFF
We report in Figure plots for the experiment in which we apply the basic setup with = 1 and k3 on a of networks, trained while varying the loss cutoff from 10 10 For the algorithm firstt approach, two-sample t-test indicates modest but significant decrease in the number of algorithms found when the loss cutoff lower or equal 10-5. In contrast, the number of mappings per algorithm does not statistically vary. For the circuit-first approach, significantly fewer circuits and interpretations per circuit are found when the loss cutoff is high (0.1), but values do not otherwise vary for lower loss values. In addition, we found that multiple computational abstractions can still be identified in randomly initialized networks that happen to implement a logic gate (i.e. without training).
10a
Circuts
10
Aqpettme
109
104
100
0.1
0.01
0.901
0.0001
18-03
Ie-00
0.00L
min
0.000L
Figure 9: Number of abstractions found in the circuit-first approach (left) and the algorithm-firs: approach (right) as a function of the neural network's 's training loss cutoff.
D.5 EFFECT OF NOISE
We report in Table 3 the number of abstractions found in the basic setup (without noisy inputs) for both approaches, and compare it to results found when repeating the same setup while adding Gaussian noise from W 0.1) to the inputs at training time. We also report the p-value obtained with Welch's t-test.
Criterion
Basic setup
Noisy setup
p-value
Algorithms
2.70
2.56
0.766
Minimal mappings/algoritmm (avg)
2.84
3.05
0.617
Total mappings
9.86
9.89
0.959
Circuits
52.3
28.6
< 0.001
Interpretatioossccrceui (avg)
2,270
16,400
0.001
Total interpretations
107,000
285,000
< 0.001
Table 3: Number of abstractions found for both approaches each setup (averaged over all net-
works).
D.6 TRAINING DISTRIBUTION
The influence of the training distribution was investigated through the following procedure:
1. Sample a random neural network N N and target gate with n = 1 and 3
2. Draw Tı,-.., from Ujo.1)
3. Train NN on skewed input distribution, with weights for each input and loss cutoff of 10 10-3
4. Exhaustively enumerate all circuits, interpretations, algorithms, and mappings as in the basic setup.
23

---

Published as a conference paper at ICLR 2025
5. Repeat from step 1.
We repeated those steps 100 times, resulting in varying training distributions with joint entropy varying from 0.8 to 2.0 bits. We then performed linear regression of the resulting counts as function of the distribution's s joint entropy. We give in Taele results for this experiment, which show that the results of the algorithm-first approach do not significantly depend on the training distribution. On the other hand, having an unbalanced distribution causes an increase in the number of circuits and total interpretations found the circuit-firs: approach, but a decrease the number of interpretations per circuit.
Criterion
Slope
Intercept
p-value
A Algorithms
-0.21
2.86
0.857
Minimal mappingsaalgorihm (avg)
-0.14
1.42
0.807
Total mappings
-0.17
8.82
0.974
Circuits
-326
556
0.001
Interpretations/cireuii (avg)
1,235
139
0.029
Total interpretations
-174,000
367,000
0.018
Table Linear regression performed on the number of computational abstractions function of the training distribution's 's joint entropy (in bits).
24


Reviews:
[{"review_id": "5s5o9XhrOH", "reviewer": "Reviewer_ExBt", "summary": "A summary of mechanistic interpretability is provided. Two approaches are proposed that attempt to interpret a “simpler” algorithm that emulates the behavior of a trained neural network in terms of a circuit. One approach focuses on modelling the behavior of the full network before finding a subset of related nodes, while the second approach focuses on finding an “important” sub-network of the full network, whose behavior is then interpreted. Both approaches are showcased in simple toy examples.", "questions": "If the training set is drawn from a distribution with certain biases, there may be correlations that essentially encourage multiple “conflicting” interpretations of a network. Can we resolve some of the issues that arise by putting conditions on the training distributions?\n\nWhat would (be expected to) happen if, in a simple toy example, an experiment was repeated with a perfect training error, with or without overfitting? Would we see a qualitatively different distribution of explanations?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The manuscript is clearly written, with a good explanation of the context the work is placed in. The toy examples provided are illustrative. The overarching conclusion is that uniqueness of an explanation should generally not be expected within the context of mechanistic interpretability, and while a similar analysis cannot be conducted on large-scale models, it is likely that the same behavior could be expected. This insight is important in many practical applications where network interpretations are required (post-training) by a practitioner.", "weaknesses": "The issue of uniqueness of an explanation is addressed in the context of mechanistic interpretability. However, the “incompatibility” of different explanations is not substantially addressed. A more formal framework in which incompatibility can be “measured” would be very interesting, along with analyzing questions on differentiating between equivalence classes of explanations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CFpE6yGnhs", "reviewer": "Reviewer_J9MJ", "summary": "This paper investigates the \\emph{identifiability} of mechanistic explanations from interventional evaluations on toy neural networks. The authors find clear non-identifiability at multiple stages of the interpretability pipeline: multiple interpretations can exist for a circuit, multiple circuits can generate the same behavior, each algorithm can be aligned to multiple activation subspaces, etc.\nThis non-identifiability persists regardless of whether MI explanations are generated first by localizing a subset of the network, then deriving an interpretation, or first generating a candidate algorithm and trying to then find an activation subspace corresponding to that algorithm.", "questions": "1. Could you elaborate on the meaning of “incompatible” in line 355? The paper would benefit from a clear example of two incompatible explanations, ideally in the main body.\n\n2. Please include in the appendix random examples of some of the circuits found so that they can be qualitatively assessed by readers.\n\n3.Could you comment on how these results relate to the discussion of Makelov et al. (2023) and Wu et al. (2024), cited above in section “Strengths”?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "A rare case of compelling and relevant deconfusion experiments which doesn’t devolve into over-claims. The authors do well in the discussion highlighting that identifiability may not be needed for all applications of mech interp, and that the non-identifiability observed here is on toy-models, so might not extend to larger models trained on multiple tasks.\n\nThe “what-then-where” strategy implemented here appears to solely utilize an approach based on Distributed Alignment Search (DAS), which makes the results about non-identifiability quite relevant to recent discussion:\n\nMakelov, A., Lange, G., & Nanda, N. (2023). Is this the subspace you are looking for? An interpretability illusion for subspace activation patching. arXiv. https://arxiv.org/abs/2311.17030\n\nWu, Z., Geiger, A., Huang, J., Arora, A., Icard, T., Potts, C., & Goodman, N. D. (2024). A reply to Makelov et al. (2023)'s \"Interpretability Illusion\" arguments. arXiv. https://arxiv.org/abs/2401.12631", "weaknesses": "Primary weakness: The most novel contribution of this paper are its experimental results, which don’t have enough description: Appendix B only contains aggregated results about the number of circuits and average interpretations per circuit found, but e.g. lacks examples of said circuits for qualitative validation. This significantly undercuts my ability to validate the correctness of the experiments.\n\nIn addition, the authors acknowledge appropriately that identifiability may not be necessary if the goal of MI is merely to steer a model. However, much MI work is driven by a desire to simply further scientific understanding of language models. What types of scientific inquiries require computational identifiability, and which do not? The paper could be strengthened by further discussing how much identifiability matters if the goal is scientific understanding, rather than just the model steering mentioned in lines 520-527.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wKxMjuewbA", "reviewer": "Reviewer_S8oD", "summary": "The authors investigate the potential issue of identifiability in mechanistic interpretability through experiemnts on small MLPs where (isolated) circuits are ennumerated and assessed fairly exahustively. They find identifiability is an issue at all levels: in the number of subcircuits functionally aligned with the full network, in the number of algorithms consistent with the behavior, and in the mappings between algorithms and circuits. This problem gets worse as architecture size increases, and training on a greater number of tasks only mitigates this issue to some extent.", "questions": "Minor comments, questions, and suggestions:\n\nLine 403 states that “larger architecture…could also lead to greater overparameterization”. This could benefit from elaboration; in particular, how larger architecture could lead to a reduction in the number of valid abstractions.\n\nOn Line 065, “Given the near impossibility of exhaustively searching all possible algorithms across all subsets of a neural network”, I might suggest to reframe this not as impossibility but as intractability, infeasibility or implausibility. Certain interpretability queries might have large search spaces that could nevertheless be searched efficiently. The relevant property is the complexity of the interpretability query, not merely the size of the search space. For computational complexity analyses of circuit discovery, see Adolfi et al 2024.\n\nOn Line 067, the authors state “researchers have developed approximation methods with different assumptions and trade-offs”. It seems to me that the circuit discovery methods that are typically developed are heuristics for circuit finding, not approximation algorithms with any proven guarantees. In any, case it would be useful if the authors can distinguish between these two categories in their descriptions.\n\nCitation to Van Rooij on Line 046 does not seem to fit with the corresponding sentence, as that paper does not deal at all with interpretability, as opposed to Lindsay, 2024, which is indeed an approapriate citation. For examples of studying the fundamental properties of (inner) interpretability queries see Adolfi et al., 2024.\n\nSection 2.1 mentions interpretability work on transformer models but only in language. An example from vision transformers can be found in Vilas et al. 2023.\n\nPlease clarify the notation in Definition 4.\n\nLine 229 makes an implicit statement about computational complexity but provides no citation. See Adolfi et al. 2024 for relevant complexity analyses. This is also relevant to the statement on Line 257. Here it would also be useful to clarify how uniform random sampling “approximates” the desired measure, as this seems non-obvious. Perhaps the authors mean random sampling is a heuristic with unknown properties?\n\nLine 494 states that current MI methods can only approximate their targets because exhaustive enumeration is impossible for large models. This is technically incorrect, as even for some problems with exponential serach spaces, efficient search algorithms that find optimal solutions are possible. The relevant notion is the computational complexity of the interpretabililty queries, not simply the size of their search space (see Adolfi et al., 2024).\n\nSection 2.1 describes a parallel between AI interpretability and neuroscience. A framework that draws from lessons grounded in this parallel is described in Vilas et al. 2024. This framework provides a nice embedding for the what-where distinction, corresponding to the algorithmic and implementational levels, respectively.\n\nThe problem of identifiability interacts in interesting ways with the computational complexity of circuit finding. Adolfi et al. 2024 analyses circuit queries that are relevant to the authors’ points on identifiability. See, for instance, counting problems which ask for the number of circuits in a train neural network that have a certain property (e.g., they are sufficient for a behavior). Furthermore, if the number of sufficient circuits is typically large, heuristics for otherwise intractable problems (e.g., sufficient circuit) could seemingly find their targets in a feasible amount of time. In this scenario, non-identifiability is an important catch.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* Well written\n* Well referenced\n* Adresses an issue of interest to the interpretability community\n* Provides exhaustive experiments with well-understood ground truth.\n* Investigates the effects of various variables (architecture size, number of tasks, noise) on the identifiability problem.\n\nThis kind of study is very much needed.", "weaknesses": "I first summarize the most important points and then elaborate.\n\n* It is unclear whether the conclusions about identifiability problems can be stated generally or only in particular for circuits as isolated objects (circuit discovery through zero-ablation), which might mischaracterize the functioning of the network as a whole.\n* The possible improvements advertised in the abstract/introduction are rather only sketched in section 5.1.\n\n\nIs the large number of circuits/possible explanations due to looking for circuits in isolation (e.g., via zero ablation) rather than working in context with the rest of the network (e.g., via activation patching)?\n\nLine 320 describes the circuit isolation procedure. This is equivalent to zero-ablation and the criterion is equivalent to the definition of suficcient circuit.\nHow would identifiability look like if we chose to define circuits as they function in the context of the full network? See for example the definition of circuit via activation patching in Adolfi et al., 2024.\n\nIsn’t it possible that many of the isolated circuits discovered through zero-ablation are mischaracterizations of the in-context functioning of the circuits as they are embedded in the full network?\n\n\nLine 080: “a model’s behavior should have a single, well-defined explanation”. There is no citation here and it is unclear where this intuition is coming from, what is it’s theoretical support, etc. To offer a counter-intuition: consider a circuit that is sufficient on its own to mimic the behavior of the full network over some input domain; such a circuit need not be unique. Trivially, the circuit plus additional neurons to form the full network is another such circuit. But there is no contradiction in intuiting that multiple such circuits of different sizes, with partial or no overlap exist in the network and, in principle, offer alternative (perhaps incompatible?) ‘explanations’ (see Adolfi et al. 2024 for theoretical analyses).\n\nOn Line 091: the authors mention “the identifiability properties of current MI criteria”. The criteria of interest that define circuits leave open the possibility that these circuits are not unique. So the definition of these circuits does not preclude non-identifiability unless the uniqueness property is trivially appended to the definition. This leads one to suppose that uniqueness under the typical definition of circuits is a property left to be determined empirically. It could, in principle, be motivated theoretically, but I see nothing in that direction here. Is it possible to provide some theoretical motivation for uniqueness that is not trivially stipulated but justified from first principles?\n\nIf a network implements the same functionality for different inputs through different circuits and algorithms, does this really make mechanistic interpretation hopeless? (i.e., in this case, is only a functional explanation capable of unifying all the existing ‘explanations’?). It would be useful to have any assumptions about satisfactory explanations made explicit in the manuscript.\n\nLine 489: “the challenge lies in defining criteria that distinguish valid explanations from misleading ones.”\nIt seems to me that, conceptually, identifiability does not pose a problem for distinguishing misleading from valid explanations. The problem arises only if an explanation is presented as unique or valid for a full input domain when this is not so. This issue might warrant some clarification.\n\nLine 490: “According to MI, the explanatory algorithm should be unique, meaning multiple competing explanations should not exist.” But this statement is made without citation. This assumption seems ill-founded to begin with, for the reasons mentioned above. Where does the criterion come from?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XEtxjQ8iWM", "reviewer": "Reviewer_Q4uc", "summary": "- The objective of the paper is to answer the question, do current criteria in mechanistic interpretability (MI) guarantee the identifiability of the explanation?\n- Authors sort MI methods into two broad strategies: \n    - _where-then-what_ focuses on finding a subset of the network – a circuit – that captures most of the information flow from in- to outputs. Once this circuit is identified, the next step is to interpret its components (features) to derive the explanatory algorithm.\n    - _what-then-where_ starts by identifying candidate algorithms and then searches subspaces in the neural network where the algorithm may be implemented, using causal alignment between the explanatory algorithm’s states and the network’s internal states. \n- They stress testing both methods with toy models: small MLPs trained on logic gates. They performed three main types of searches to test different interpretability criteria: \n    - Circuits search: Looking for subnetworks that perfectly replicate the model's behavior\n    - Interpretations search: Trying to map neurons to logical gates in a way that's consistent with their activations\n    - Mappings search: Testing different ways to map logical gates to groups of neurons\n- Key findings in toy models: they found multiple valid interpretations for the same network.\n    - 85 different circuits that achieved perfect accuracy\n    - An average of ~536 possible logic gate interpretations per circuit\n    - 159 perfect minimal mappings between algorithms and neurons\n    - In total, over 45,000 different possible computational abstractions\n- Experiment larger NN trained on a subset of MNIST revealed similar dynamics\n    - The circuit search found over 3000 valid circuits", "questions": "(From the weakness section)\n- What is in the case of the MNIST NN the \"valid circuit\"?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors introduce their own taxonomy and formalization (e.g. where-then-what/what-then-where, Circuits, Mappings, etc) for important concepts discussed in the paper. While I haven’t fully wrapped my head around the usefulness of the taxonomy, I appreciate the effort to deconfuse different interpretability methods. I think this is the strongest suit of the paper and I wish they had focused the paper on the taxonomy and less on the experiments. \n- I enjoyed the writing style, and it was easy for me to follow, particularly Sections 2 and 3. I also found Figure 2 to be helpful. For the _what-then-where_/_where-then-what_ split. \n- The paper touches on an important topic within interpretability – a lack of quality in the discourse around helpful metrics for interpretations.", "weaknesses": "- While I think the taxonomy could bear some relevant fruit, for experimentally testing the metrics using NN this small, it seems challenging to generate insights that are relevant to interpretability as a total. From my understanding, we are not close to having high-confidence circuit interpretations (“perfect circuits”) in the first place, so working under this assumption might be several steps ahead.\n- Continuing on the larger NN experiments: I struggled to understand the points in line 479 ff. What is in this case the \"valid circuit\"? I hoped this section could have bridged a gap toward showing how this metric could be used in the future but failed to show its limitations in this instance. (But maybe I simply oversaw that.)\n- Lastly, I think there is also too little emphasis on existing literature that clearly touches on the underlying problem: disentanglement. I would consider modeling the paper around the taxonomy and then focusing on existing research and problems in relation to known circuits, such as IOI.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)