Open Problems in Mechanistic Interpretability
Authors: Lee Sharkey
Bilal Chughtai
Joshua Batson
Jack Lindsey
Jeffrey Wu
Lucius Bushnaq
Nicholas Goldowsky-Dill
Stefan Heimersheim
Alejandro Ortega
Joseph Isaac Bloom
Stella Biderman
Adrià Garriga-Alonso
Arthur Conmy
Neel Nanda
Jessica Mary Rumbelow
Martin Wattenberg
Nandi Schoots
Joseph Miller
William Saunders
Eric J Michaud
Stephen Casper
Max Tegmark
David Bau
Eric Todd
Atticus Geiger
Mor Geva
Jesse Hoogland
Daniel Murfet
Thomas McGrath
Venue: Accepted by TMLR ()

Abstract: Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks' capabilities in order to accomplish concrete scientific and engineering goals. Progress in this field thus promises to provide greater assurance over AI system behavior and shed light on exciting scientific questions about the nature of intelligence. Despite recent progress toward these goals, there are many open problems in the field that require solutions before many scientific and practical benefits can be realized: Our methods require both conceptual and practical improvements to reveal deeper insights; we must figure out how best to apply our methods in pursuit of specific goals; and the field must grapple with socio-technical challenges that influence and are influenced by our work. This forward-facing review discusses the current frontier of mechanistic interpretability and the open problems that the field may benefit from prioritizing.

Published in Transactions on Machine Learning Research (09/2025)
Open Problems in Mechanistic Interpretability
Lee Sharkey"
Apollo Research
Bilal Chughtai"
Apollo Research
Joshua Batson
Anthropic
Jack Lindsey
Anthropic
Jeff Wu
Anthropid
Lucius Bushnaq
Apollo Research
Nicholas Goldowsky-Dill
Apollo Research
Stefan Heimersheim
Apollo Research
Alejandro Ortega
Apollo Research
Joseph Bloom
Decode Research
Stella Biderman
Fleuther AT
Adria Garriga-Alonso
FAR AI
Arthur Conmy
Google DeepMond
Neel Nanda
Google DeepMind
Jessica Rumbelow
Leap Laboratories
Martin Wattenberg
Haroard University
Nandi Schoots
King College London and Imperial College London
Joseph Miller
MATS
William Saunders
METR
Eric J. Michaud
MIT
Stephen Casper
MIT
Max Tegmark
MIT
David Bau
Northeastern University
Eric Todd
Northeastern University
Atticus Geiger
Pr(AIfr group
Mor Geva
Tel Avic University
Jesse Hoogland
Timaeus
Daniel Murfet
University of Melbourne
Tom McGrath
Goodfire
Reviewed on OpenReview: https: openreview. net/
net/ forum? id= 91H7769999
"Correspondence leedsharkey@gmail. and brehughtaii@gmmiil.com Work done prior to joining Authropic.

---

Published in Transactions on Machine Learning Research (09/2025)
Abstract
Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks" capabilities in order to accomplish concrete scientific and engineering goals. Progress in this field thus promises to provide greater assurance over AI system behavior and shed light on exciting scientific questions about. the nature of intelligence. Despite recent progress toward these goals, there are many open problems in the field that require solutions before many scientific and practical benefits can be realized: Our methods require both conceptual and practical improvements how best to apply our methods in pursuit of specific goals; and the field must grapple with socio-technical challenges that influence and are influenced by our work. This forward-facing review discusses the current front ier of mechanistic interpretability and the open problems that the field may benefit from prioritizing.
reveal deeper insights; we must figure out
2

---

Contents
Introduction
4
1.1
The focus of this review: Open problems and the future of mechanistic interpretability
4
1.1.1
Why 'mechanistic' interpretability?
4
2
Types of open problems)
5
2
Open problems in mechanistic interpretability methods and foundations
7
2.1
Reverse engincering: Identifying the roles of network components
7
2.1.1
Reverse engineering is necessary because Al and humans use different representations and perform different. tasks)
2.1.2
Reverse engineering step 1: Neural network decomposition.
Networks do not naturally decompose into architectural components.
8
Decomposition by dimensionality reduction methods..)
9
Decomposition by sparse dictionary learning (SDL).
9
Current decomposition methods lack solid theoretical foundations.
13
Intrinsic interpretability: Building more easily decomposable models
2.1.3
Reverse engincering step 2: Describing the functional role of components
Explanations for what causes components to activate
15
Explanations for the downstream effects of components
16
2.1.4
Reverse engineering step 3: Validation of descriptions
18
"Model organisms' facilitate hypothosis validation.
19
Validating interpretability methods using benchmarks.
19
2.2
Concept- interpretability; Identifying components for given roles
20
2.2.1
Concept-based probes
2.2.2
Probes need carefully chosen data for well-dehned concepts
20
2.2.3
Probes detect. correlations, rather than causal variables, hidden activations
21
2.2.4
Concept-based intrinsic interpretability .....
21
2.3
Proceduralizing mechanistic interpretability into circuit discovery pipelines: A case study
21
2.4
4
Automating steps in mechanistic interpretability research
23
3
Open problems in applications of mechanistic interpretability
25
3.1
Axes of mechanistic interpretability progress,
25
3.2
Using mechanistic interpretability for better monitoring and auditing Al systems for po-
tentially unsale cognition
25
3.2.1
Mechanistic interpretabilityy based evaluations could help detect unsafe unethical
AI cognition
25
Enabling real time monitoring of Al systems for potentially unsafe cognition
26
Improving our ability to red-team Al systems and clicit unsafe outsuts
26
3.2.2
Using mechanistic interpretability for better control of Al system behavior
27
3.3
Using mechanistic interpretability for better predictions about Al systems
28
3.3.1
Predicting behavior novel situations
28
3.3.2
Predicting capabilities that arise during training or finetuning
29
3.4
Using mechanistic interpretability to improve our ability to perform inference, improve Irain- ing, and make better use of learned mechanisms
30
3.5
Using mechanistic interpretability for 'microscope AT
31
3.6
Mechanistic interpretability on a broader range of models and model families
31
3.7
Human computer interaction with model internals
32
4
Open socio-technical problems in mechanistic interpretability
33
Translating technical progress in mechanistic interpretability into levers for policy and
governance
33
4.2
Social and philosophical problems in mechanistic interpretability
34
Conclusion
36
3

---

Published in Transactions on Machine Learning Research (09/2025)
1 Introduction
Recent progress in artificial intelligence (AI) has resulted in rapidly improved AI capabilitiis. These capa- bilities are not. designed by humans. Instead, they are learned by deep neural networks (Hinton et al. 2006; LeCun al. 2015). Developers only need to design the training process; they do not need to and in almost all cases, do not understand the neural mechanisms underlying the capabilities learned by an Al system.
Although human understanding of these mechanisms is not necessary for AI capabilities, understanding them would enhance several human abilitics. For example, it would permit better human control over AI behavior and better monitoring during deployment. It would also facilitate trust in Al systems, allowing us to fully realize their potential benefits by enabling their deployment in safety-critical and ethically-sensiteee settings.
Beyond the engincering bencfits, understanding AI systems offers immense scientific opportunities. For the first time in history, can create and study artificial minds with a level of access and control that is simply not possible in biological systems. What new laws of nature governing the mechanisms of minds might we discover from studying the internal workings of Al systems? What new methods for analyzing biological neural systems might neuroscientists be able to glean from understanding artificial ones?
The scientific opportunities are not limited the field of AI. If an AT can putporform tools designed by humans in given scientific field, it suggests that the Al system representing something about the world currently unknown to us. We can develop deeper comprehension of the world by understanding those representations. What can we learn about protein folding from ATs that can successfully predict protein structure? What insights can we glean about discase from radiographer that performs beyond human ability?
Mechanistic interpretability might unlock these benefits. This field of study aims to understand neural networks' dccision-making processes. Here, we define "Understanding a neural network's docision-makiin process" the abilily to use knowledge about the mechanisms underlying network's decision-making process in order to successfully predict its behavior (even arbitrary inputs) or to accomplish other practical goals with respect to the network. Such goals might include more precise control of the network's behavior, or improved network design. Interpretability promises greater assurances for AI systems through better understanding of what neural networks have learned, thus enabling us to realize their potential benefits.
1.1 The focus of this review: Open problems and the future of mechanistic interpretability
Several recent. reviews of mechanistic interpretability research and related topics exist (Rauker et al. 2023
Geiger et al., 2022 Bereska & Gavves 2024 Ferrando et al.] 2024 Rai et al.) 2024 Anwar et al. 2024
Davics & Khakzar 2024 Mosbach et al. 2024; Mucller et al., 2024 Our review takes a more forward-
looking stance. We discuss not only where the frontier is today, but also which directions we might benefit most from prioritizing in the future.
1.1.1
Why 'mechanistic' interpretability?
The distinction between interpretability and mechanistic interpretability is not always clear and is therefore worth clarifying. The motivations and methods used in interpretability work are often diverse (Lipton)
2018 Doshi-Veles & Kim 2017: Jacovi 2023). As a result, there are many ways in which interpretability
research might be categorized. Prior categorizations of interpretability include causal vs. correlational methods, supervised vs. unsupervised methods, bottom-up vs. top-down methods, among others (Geiger et al. 2024b Mueller et al., 2024 Bereska & Gavves 2024 Belinkov 2022a Zou al.] 2023a Davies & Khakzar 2024). This review focuses specifically mechanistic interpretability. But what distinguishes 'mechanistic interpretability' from interprctability in general? It has been noted that the term is used in a number of (sometimes inconsistent) ways (Saphra & Wiegreffe 2024). In this review, we use the term 'mechanistic interpretability" in a technical sense, referring specifically to work that investigates the mochanisms underlying neural network generalization. Mechanistic interpretability represents one of three threads of interpretability research, each with distinct but sometimes overlapping motivations, which roughly reflects the changing aims of interpretability work over time.
4

---

Published in Transactions on Machine Learning Research (09/2025)
The first thread aims to build AI systems that are interpretable by design. Much early interpretability work focused on explaining the sensitivity of machine learning models to inputs and training data. This work typically used small-to-medium sized models designed to be easily interpretable, such as decision trees [Breiman 1984 Hu et al. 2019). lincar models (Rowcis & Ghahramani 1999), and generalized additive models (Hastie & Tibshirani 1986 Agarwal et al. 2021p. These models could be used alongside attribution
methods such as influence functions [Hampel 1974: Koh & Liang. 2017] and Shapley values [Shaplcy] 1997;
Lundberg Lee 2017), which were common techniques used to characterize model decision boundaries with respect to imputs. "Interpretability by design" continues to be an active research area, including architectures
such as Concept-Bottlencee Models (Koh et al. 2020), Backpack Language Models (Hewitt et al. 2023
Kolmogorov-Aras Arnold Networks (Liu et al. 2024c), and sparse decision trees (Xin et al. 2022).
With the rise of larger-scale nonlinear neural networks KKiizhesky et al. 2012 He et al. 2016), another thread grew in importance, driven primarily by the question: Why did my model make this particular decision? However, one challenge in interpreting larger networks was finding attribution methods that could scale to large networks (Zeiler & Fergus 2014). In response, a number of local altribution methods were developed, including grad-CAM (Selvaraju et al. 2011), integrated gradients (Sundararajan et al. 2017). and masking- mosking-based causal attribution (Fong & Vodaldi 2017), SHAP (Lundborg & Lee 2017] LIME (Ribeiro
et al. 20160, and many other methods, including backprop-basee based visualization methods [Simonyan et al. 2014a:
Nguyen et al. 2016b).
Inspired by, for example, Inception (Szegedy et al. 2013) and GPTT3 (Brown et al. 2020 another thread emerged as models became capable of more profound generalization. Focused on the broader subject of generalization, it was driven by the question: How did my model solve this general class of problems? Due to its emphasis the mechanisms underlying neural network generalization, the work in this category is commonly referred to as 'mechanistic interpretability" (in addition to other, more cultural reasons, according to (Saphra & Wiegreffe ).24)). This kind of interpretability work is driven by fundamental hypothesis in deop learning that generalization arises from shared computation (LeCun et al. 2015). Early work in this area, such as feature visualization (Olah et al.] 2017b), or nelwork dissection (Bau el al. 2020), sought "global" explanations for model generalization by investigating the roles of model components across class
of decisions. More recent work in this area looks at "circuits" of components (Wang al.
2023), generalizable
patterns of information flow (Geva et al.) 2023), representation subspaces (Geiger et al. 2024c Zou et al.
2023a and probes (or self-supervised searches via sparse dictionary learning) for representation vectors that
carry information that generalizes across many instances for a particular task or set of tasks (Huben et al.
2024
Bricken 2023
Todd et al. 2024 Hollinsworth et al. 2024
1.2
Types of open problems
The field of mechanistic interpretability ultimately aims to achieve concrete scientific and engineering goals. For instance, we would like to be able to:
Monitor AT systems for signs of cognition related to dangerous behavior (Section 3.2
Modify internal mechanisms and edit model parameters to adapt their behavior to better suit our needs (Section 3.2.2;;
Predict how models will act in unseen situutions or predict when a model might. learn specific abilities (Section 3.3
Improve model inference, training, and mechanisms to better suit our preferences (Section 3.4
Extract latent knowledge from Al systems so we can better model the world (Section 3.5
Despite recent hopeful signs of progress, mechanistic interpretability still has considerable distance to cover before achieving satisfactory progress toward most of its scientific and engineering goals.
To achieve these goals, the field not only needs greater application of current srate-of-the-art mechanistic interpretability methods, but also requires the development. of improved techniques. The first major sec-
5

---

Published in Transactions on Machine Learning Research (09/2025)
tion (Section 2 therefore discusses open problems related to the methods and foundations of mechanistic interpretabilityy (Figure)
We then explore key axes of research progress that will determine how far we can advance toward the goals of mechanistic interpretability (Section p. Section 3 outlines how applications of interpretability methods have made progress toward the field's ' goals, and, for each goal, discusses the specific axes along which progress likely needed to achieve specific objectives.
Finally, note that the goals, applications, and methods of mechanistic interpretability do not exist in vacuum. Like any scientific field, they lie within broader societal context. The final section of this review examines open socio-technical problems in mechanistic interpretability (Section 4. It discusses current initiatives and possible pathways to translate technical progress into levers for Al governance, alongside consequential social and philosophical challenges faced by the field.
Open problems in Mechanistic Interpretability (M))
Open problems applications (53)
Open problems methocs & foundations (52)
Monitoring auditing
Predicting behavior (53 3)
Reverse engineering (' ('Identifying companents') (52.)
Concept- based (" '"Identifying components given roles")
Improving
(53.4)
Microscope
(Knowledge discovery) (53.5)
Concept-besed probes overcoming their challenges
Applying
Methods of nelwork decomposition (52122)
Methods describing roles of companents
Methads
range of models ($3.6)
validating descriptians
Human-campunee interactian
Neurons, layers, dimersionaiity
e
Describing components
Experiments,
model organisms. benchmarks
Concept- basett-aaddd intrinsic
Open socia- technical problems in MI (54)
reduction,
interpretability (52.24)
Translating into levers for governance (54.1)
sparse dictionary leaming, others (52112a-d)
($2.14a-b)
Social and philosophical problems (54
intrinsically decomposable (52.1.2)
Descriiing components' effects
Cross crotincutting appreaches
Proceduralizing methods into pipelines ('Cicuit Discovery' Case Studyl (52
Automating steps MI research (52.4)
Figure 1: Summary of the core sections of this review.

---

Published in Transactions on Machine Learning Research (09/2025)
2 Open problems in mechanistic interpretability methods and foundations
One way of thinking about what neural nelworks do internally is that they learn parameters that implement neural algorithms. These neural algorithms take data as input and, through a series of steps, transform their internal activations produce an output. Different parts of the network learn different stops in the algorithm; mechanistic interpretability aims to describe the function of different network components.
The first approach, often called 'reverse engineering', is to decompose the network into components and then attempt to identify the function of those components (Section 2.1). This approach "identifies the roles of network components". This approach involves three distinct methods for three key steps: first, methods for neural network decomposition to break down the network into simpler parts (Section 2.1.2 second, meth for description of components to formulate hypotheses about their functional roles and interactions, such as max-activating dataset examples, attribution methods, feature synthesis methods, or causal intervention methods (Section 2.1.3): and third, validation of these descriptions test the correctness of the hypotheses, for example through testing doscriptions' predictions, attempting to the descriptions for downstream lasks, or use of benchmarks (Section 2.1.4..
Conversely, the second approach, sometimes referred to as 'concept- based interpretability' proposes a set of (human-derived) concepts that might be used by the network and then looks for components that appear to correspond to those concepts (Section 2.2). This approach thus "identifies nelwork components for given roles". Variants on this approach have been developed in order to overcome some of its issues, such as the issue that the approach often needs carefully chosen data for well-defined concepts (Section 2.2.2) or the issue that simple variants of the method only detect correlational, rather than causal variables (Section 2.2.3 Related methods build human concepts into the training of models such that they are more interpretable by default (Section 2.2.4).
In this section, we will examine both approaches ('Reverse engineering' and Concept-based interpretability" (Figure discussing their methods and open problems. We will also touch on open problems that cut across either approach, including proceduralizing the mechanistic interpretability pipeline (Section 2.3) and its uses in automating interpretability research (Section 2.4). We provide a high-level overview in Table summarizing the principal methodologies that have been used in mechanistic interpretability.
2.1 Reverse engineering: Identifying the roles of network components
2.1.1 Reverse engineering is necessary because Al and humans use different representations and perform different tasks
Large language models produce text that closely resembles human writing, and it is tempting to assume that they generate it through cognitive processes similar to those of human writer:]] However, humans and AI models often solve problems in different ways. For example, a model that is only 1% the size of GPT-3 outperforms humans on next-token prediction tasks (Shlegeris et al.] 2024). Inversely, even state-of-the- art multimodal LLMs struggle with tasks that a four-year-old could easily master, such as learning causal properties of new objects involving simple lights and shapes (Kosoy et al. 2023). An even clearer sign that humans and AI are using different representational processes are cases where humans cannot solve a problem
at all, as in the case of predicting a protein structure from sequence (Jumper et al. 2021 Lin et al. 2023
Even when both humans and Al exhibit comparable levels of competence on a given task, they may use different heuristics. For example, research shows that image models tend to rely more heavily on textural features (such as recognizing elephants by their hide rather than their shape (Geirhos et al.. 2019)) or rely on dataset correlations (as when identifying fish by the fingers that proud fisherman use to hold them (Brendel & Bethge| ) to greater degree than people do. Even simple algorithmic tasks, like modular addition, which humans might solve with simple carries, were solved by small transformer model by learning Fourier transform strategy that rescarchers only understood in retrospect (Nanda et al. 2023a).
1 'However, even if this assumption were true, understanding LLMs would remain challenging, we also lack mechanistic understanding of the cognitive processes invalved in human text creation!
7

---

Published in Transactions on Machine Learning Research (09/2025)
Reverse Engineering
Concept Based Interpretability
Decompose network, describe componencc, cascriptions
Neur al network
Understarding the function of networs components
Human concept
Location of concept network
Figure 2: Two approaches to neural network interpretability. (Teft) Reverse Engineering is characterized by decomposing networks into functional components and describing how those components interact to produce the network's behavior. It thus aims to 'identify the roles of network components' (Section 2.1). (Right) Concept- based interpretability on the other hand attempts to discover human concepts within neural network internals. It thus aims to 'identify the network components for given roles' (Section 2.2
To grasp the polentially alien cognition of these models, must. develop methods uncover and understand the previously unknown concepts and mechanisms implemented within them. In other words, must be able to reverse engineer these models (Olah 2024
Reverse engineering generally involves three steps, whether it is an engine, a piece of software, or a neural network (Figure 3)
1. Decomposition: Breaking down the object of study into simpler parts (Section 2.1.2
Description of components: Formulating hypotheses about the functional role of component parts and how they interact (a process that can be called 'interpretation') (Soction 2.1.3
3. Validation of descriptions: Testing if our hypotheses are correct (Section 2.1.4).
If our hypotheses are invalidated, we must either improve our decomposition or improve our hypotheses regarding the functional roles of its components. We will systematically examine cach step, analyzing current methods and their respective shorlcomings.
2.1.2 Reverse engineering step 1: Neural network decomposition
In mochanistic interpretability, our aim is to decomposo a neural network and study its parts in isolation order explain how neural networks generalize. This aim raises the question of how best to carve neural network "at its joints" for the purposes of interpretability.
Networks do not naturally decompose into architectural components. The naive approach to decompose neural networks involves breaking them down into their architectural components, such as indi- vidual neurons, attention heads, or layers.
Some early attempts to interpret deep artificial neural networks studied the responses or weight structure of individual neurons or single convolutional filters (Erhan et al. 2009 Le et al., 2012 Krizhevsky et al.
2012 Szegedy et al. 2014; Simonyan et al.) 2014b Zhou et al. 2015 Srivastava
2014 Yosinski et al. 2015 Mordvintsev 2015 Olah et al. 2017a Bau et al. 2020 Dalvi et al. 2019 Olah et al.) 2020a
Cammarata et al. 2020). These efforts paid homage to the 'Neuron Doctrine in neuroscience, which posits
that individual neurons are the structural and functional unit of the nervous system (Cajal 1924 Sherrington,
1906). However, researchers discovered that single neurons are 'polysemantic"
they seem to respond lo
multiple kinds features both artificial (Wei et al. 2015 Nguyen et al. 2016c Olah et al. 2017a] and
biological networks (Churchland Shenoy 2007 Rigotti et al. 2013 Mante al. 2013 Raposo et al.
2014). These observations support earlier thcoretical work that suggested representations used by neural
networks do not necessarily align with the activation of individual neurons (Hinton 1981)

---

Published in Transactions on Machine Learning Research (09/2025)
Network don't yet understand
Decomposition
Hypotheses
Validation
components and their actions that analyse individually
about the functions
of and secompositions enisure they explein the network well
components and interactions
Siue -9 rprus purple e gbtla red- (green) neturneed
Decompose
Describe
Validate
X
The steps of reverse engineering neural networks. (1) Decomposing a network into simpler com- ponents. This decomposition might not. necessarily use architecturally--efnnee bases, such as individual neurons or layers (Section 2.1.2p. (2) Hypothesizing about the functional roles of some or all components (Soction ). (3) Validating whether our hypotheses are correct, creating a cycle in which we iteratively refine our decompositions and hypotheses to improve our understanding of the nelwork (Section 2.1.4..
Figure 3:
Interpreting individual attention heads docs not fare better than interpreting individual neurons, as attention heads also exhibit polysemanticity (Janiak et al. 2023). More broadly, research suggests that studying the attention patterns of models can often be misleading (Jain & Wallace 2019 Pruthi et al. 2020).
Some work even suggests that representations in language models might span multiple layers YYun et al, 2021 Lindsey et al. 2024 Ameisen el al. 2025). This chimes with work that edits or intervenes on individual layers, which indicates that this level is too coarse gaarse-grained to robustly carve the network at its joints (Meng ct al.
2022b Wang et al. 2023).
If natural architectural components, such as individual neurons, attention heads, or layers, do not provide a natural way to decompose neural network representations, then what does?
Decomposition by dimensionality reduction methods. If individual neurons are not the right de- composition, perhaps groups or patterns neurons are. Many decomposition methods attempt to identify activation vectors that correspond the basic unit of neural network compulation. One common approach is provide models with a range of unlabeled inputs, collect the resulting hidden activations, and then apply unsupervised dimensionality reduction techniques to these hidden activations. The hope that structure in the hidden activations corresponds to the structure of neural computation. Commonly used dimensionality reduction methods include Principal Component Analysis or Singular Value Decomposition (Hollinwworhh
et al.
2024
Marks & Tegmark 2024 Huang et al. 2024a Bushnaq et al. 2024), tensor factorization (Oldfield
et al.
2023 and non-negative matrix factorization OOlah et al. 20188 Voss et al. 2021 Cammarata et al.
2020
Fel et al. 2023 though these techniques are no longer predominant methods used for mechanistically
decomposing language models (Fricdman al. 2024). Related approaches use singular value dccomposition an initial step, followed by unsupervised clustering step in order to identify semantically unique and distinguishable concepts (Graziani et al. 2023
Decomposition by sparse dictionary learning (SDL). According to the 'superposition hypothesis', neural networks are capable of representing more features than they have dimensions, as long as each feature activates sparsely (Elhage et al., 2021) (Figure 4p. This is key reason that dimensionality reduction mot are not considered state-of aattr-rrttttt because they cannot identify more directions than there are activation dimensions. The superposition hypothesis, coupled with the failure of dimensionality reduction methods to overcome it, motivated the search for methods that can identify more directions than dimensions. Recent work has explored the use of sparse dictionary learning (SDL) to this end (Elaage et al. 2021 Sharkey et al.
2022b Huben et al. 2024 Bricken 2023
Currently, SDL is the most popular set of unsupervised decomposition methods in mechanistic interpretabil- ity. SDL encapsulates family of methods, including Sparse Autoencoders (SAFs) (Gao et al. 2024 Tem-

---

Published in Transactions on Machine Learning Research (09/2025)
Linear Representation Hypothesis
Features represented in superposition
Sparse Dictionary Learning
Featura
Figure 4: Three ideas underlying the sparse dictionary learning (SDL) paradigm in mechanistic interpretabil- ity. (Left) The linear representation hypothesis states that the map from 'concepts' to neural activations is linear. (Middle) Superposition is the hypothesis that models represent many more concepts than they have dimensions by represent ing them both sparsely and linearly in activation spaces. (Right) SDL attempts to recover an overcomplete basis of concepts represented in superposition in activation space.
pleton et al. 2024), Transcoders Dunnfskyy et al. 2024 Ameisen et al., 2025), and Crosscoders Lindsey et al. 2024
2024
Rajamanoharan et al. 2024 Makelov et al. 2024 Kissane et al. 2024b Braun et al.
Ameisen et al. 2025
In SDL, hidden activations are typically passed to a small neural network consisting of only two layers, which correspond to an encoder and decoder respectively, with wide hidden space. The encoder activations represent how active each 'latent' iss, and the decoder matrix corresponds We want to train the dictionary elements align with 'feature directions' in the network's hidden activations. Since we assumee that individual features are sparsely present in the activations, the encoder activations are trained to be sparsely activating. In the case of SAEs, the output is trained to either reconstruct the input or, in the case of transcoders (Dunefsky et al. 2024 Ameisen et al. 2025), to reconstruct the activations of the next layer or layers. Crosscoders (Lindsey et al. 20224] Ameisen et al. 2025) permit a wider class of inputs and outputs, potentially reconstructing the activations of many layers simultaneously. Since the encoder is nonlinear, it thought be able learn to activate latent only if feature is 'active' in the hidden activations and remain 'off' if it is not.
dictionary of latent directions.
Although SDL is considered leading decomposition method for mechanistic interpretability, it has substan- tial practical and conceptual limitations (Figure
SDL reconstruction crrors are too high: Large errors in SDL roconstruction raise the question whether SDL methods can reconstruct the hidden representation sufficiently well such that the latents learned by SDL are adequately faithful to the models being interpreted. To measure this, the model's true hidden activations can be replaced with sparse dictionary reconstructions, then subsequently evaluating the extent to which the model's performance decreases. In practice, the error results in significant performance reductions. When a sparse dictionary with 16 million latents was inserted into GPT-4, the language modeling loss was equivalent to a model with only 10% of GPT--'s pretraining compute (Gao et al., 2024h. Similarly, Makelov et al. (2024) found that using reconstructions from sparse autoencoders decreased GPT-2 small performance by 10% when trained on specifc data, and 40% when trained on the full distribution.
Making sparse dictionarics much larger and sparser to reduce errors a feasible but computationally expen- sive approach. Furthermore, in the limit, this results in merely assigning one dictionary latent per dalapoint, which is clearly less interpretable. One partial solution is using SDL methods with 'error nodes' (Marks et all 2024), to account for the discrepancies between the original and reconstructed activations. However, while the sparse autocncoders identified interpretable latents, the error nodes contain 'everything clsc', making them an inadequate solution to the problem. Engels et al.] (2024b) found that these reconstruction errors are not purely random, as much of the direction of the error and its norm can be linearly predicted from
2Th term latent is often used instead of the word feature refer SDL dictionary elements, since the 'feature' is often used refer to multiple different ileas (Smith 2024b)
10

---

Published in Transactions on Machine Learning Research (09/2025)
SDL Reconstruction errors are too high
Sparsity is not good proxy for interpretability
SDL describes activations but not mechanisms
Feature Splitting
Feature Absorbtion
7
??
begins with except ekephanc
pensar GIM
activation vector
reconstruction
"elephant"
dittoonry
input
output
SDL cannot be straightforwardly applied to architectures
SDL assumes the Linear Representation Hypothesis in non linear models
Practical and Conceptual Limitations of Sparse Dictionary Learning (SDL)
Attention byyer
MLP
layer
ReLU
Heads
Neurans
SDL leaves feature geometry unexplained
SDL latents may not contain the concepts we want
SDL methods are expensive to apply to large models
Role
Jan
relative geo-
metry?
Mar
$
latents
by
Figure 5: Sparse dictionary learning has a number practical and conceptual limitations that cause issues when using reverse engineer neural networks (Section 2.1.2h.
the initial activation vector. This suggests that current SDL methods may systematically fail to capture certain structured aspects of model representations, but also implies potential solutions. To overcome this issue, may be necessary to improve SDL training methods, or develop entiroly new methods of network decomposition.
SDL methods are expensive to apply to large models: SDL involves training a small neural network for every layer of the AI model that we want to interpret. Typically, the sparse dictionaries have more parameters al that layer than the original model does, and consequently will probably be relatively expensive to train compared to the original model. 3 As Al models become larger, scaling costs of SDL also increase, although it remains unclear whether relative scaling costs are sub- or supra-linear. For cost effectiveness, it may be important to develop intrinsically decomposable methods for training models, while remaining at (or close to) state-of-the-art performance (Section 2.1.2 This approach will help avoid incurring the expense of both training and decomposing Al models.
SDL assumes the linear representation hypothesis in nonlinear models: Other problems with SDL arise from the assumptions which SDL is based. One such assumption is the linear representation hypoth- esis. The linear representation hypothesis observes that though neural networks are nonlincar functions and could potentially use highly nonlinear representations Black el. al. 2022 Engels et al. 2024b Kirch et. al.
TThe actual relative is unclear since there public attempts to apply SDL to every vector space in model.
although some work applies SDL various layers Marks al. 2024 Gao 2024 Bloom Tim 2024a Huben al. 2024
2024
Brann al. 2024 Lieberum
11

---

Published in Transactions on Machine Learning Research (09/2025)
2024), they tend to use representations that exhibit strikingly linear behavior (Smolensky 1986 Mikolov Olah et al.] 2020b; Elhage et al.] 2021 Park et al.] 2023a; Guerner et al.] 2024 Gurnee & Tegmark 2024). The hypothesis formalizes this phenomenon by stating that high level concepts are lin- carly represented as directions (vectors) in neural network embeddings. Its two core claims are (i) that the composition of multiple concepts can be represented as the addition of their corresponding feature vectors, and (ii) that the intensity of a concept represented by the scale of its corresponding feature vector (Olah) & Jermyn 2024). Earlier work (Elhage et al., 2021) defined the linear representation hypothesis with the additional assumption of one-dimensionall features, but this definition has since been refuted (Yedidia 2023 Chughtai & Lau 2024 Engels et al. 2024a) and clarified (Olah & Jermyn 2024). Another recently proposed criterion is that the intensity of concepts should be retrievable by a linear function of the embeddings, up
et al., 2013:
to small error (Hänni et al. 2024 Olah & Jermyn 2024].
A weak version of the linear representation hypothesis states that some concepts are linearly represented, while a strong version may assert that all concepts are (Smith) 2024a). Some works have shown that the strong version is false for some models (Black et. al.. 2022 Csordás al., 2024). The weaker version of the linear representation hypothesis is supported by the successes of linear probes (Section 2.2.1), activation stecring (Section 3.2.2), and the success of sparse autocncoders in finding scemingly interpretable latents (Section 2.1.2..
Sparsity is not a good proxy for interpretability: Another of SDL's key assumptions is that feature activations are sparse. Therefore, SDL methods optimize their latents accordingly to be sparsely activating, with the implicit assumption that sparser decompositions are more interpretable than denser ones. However, this assumption may not necessarily hold. The (related) problems of feature splitting (Bricken 2023), feature absorption CChanin et al., 2024 and composition (iill 2024) suggest that with sufficient optimization pressure, sparsity as a proxy for interpretability breaks down (though note that it is debated whether feature splitting is actually a problem). Other proxies, such as minimum description length, might be better optimization targets than sparsity per se (Ayonrinde et al. 2024 It also may be the case that no proxy metric is sulficient.
SDL leaves feature geometry unexplained: SDL decomposes networks into single directions in the activation space, which is reasonable approach only if we consider feature activations akin to 'a bag of features' (Harris 954) without any internal structure. However, the geometric arrangement of features in relation to each other seems to reflect semantic and functional structure (Engels et al. 2024a Gurnee & Tegmark 2024 Park et al. 2024b; Bussman et al. 2024). Understanding the geometric structure of a network means understanding the position of one feature relative to (potentially) many others. Comprehending this may be necessary if we want to know why and how a network treats certain features similarly or differently.
If understanding the global geometry (all-to-all relationships) of features is essential to understand neural networks, this might pose fundamental problem for current approaches to mechanistic interpretability (Hänni et al., 2024 Mendel 2024). However, if only local geometric relationships between features need to be understood, understanding networks with a 'bag of fcatures' approach may be more feasible.
SDL cannot straightforwardly be applied to all architectures: SDL was originally developed to iden- tify features that may be represented in superposition across many neurons within a single layer. However, ropresentations may be spread over other network architecture components besides neurons. In transform- ers, for example, representations may be spread across separale allention heads (Jermyn et al., 2023 Janiak et al.] 2023). and even across different layers (Yun et al.) 2021 Lindsey et al., 2024). However, is not immediately obvious how to decompose representations distributed across attention heads with SDL (Math- win et al.] 2024 Wynroe & Sharkey 2024). Nor is it straightforward to translate cross-layer distributed representat ions into causal descriptions of neural network mechanisms (Lindsey et al. 2024), though some recent work makes progress toward this (Ameisen et al. 2025).
SDL decomposes the input and output activations of network mechanisms, but not the mech- anisms themselves: Ultimately, the aim of mechanistic interpretability is to understand the mechanisms learned by neural networks. The parameters of the network, along with its nonlincarities and other architoc- tural components, implement these mechanisms, which are applied to the input' hidden activations. SDL identifies directions in activation space. Being activations, they only interact with the network's 's mechanisms,
12

---

Published in Transactions on Machine Learning Research (09/2025)
but are not the mechanisms themselves. Describing the mechanisms directly remains unresolved with SDL, Gaining insights about the network's s mechanisms from SDL. latents requires further post hoc analysis, which can be labor intensive, computationally expensive, or data set dependent (Huben et al., 2024 Bricken 2023; Riggs et al.) 2023 Marks et al.) 2024). This is an instance of a more broad problem with current mechanistic interpretabiliy, lily work; we primarily locus on understanding neural nelwork activations, with little attention paid to how this structure in activations is computed via weights (Chughtai & Bushnaq 2025).
SDL latents may not contain the concepts needed for downstream use cases: When using SAEs for practical tasks, there sometimes exists single latent or small set of latents representing some concept of interest for task performance (and not representing much clse). For instance, KKntamneni et al. (2024) found a single latent whose activation pattern was more accurate than official dataset labels on the NLP task GLUE CoLA (Warstadt et al., 2019 More often than not though, a sparse set of latents that encode some useful concept of interest do not exist. It is unclear what causes this problem. One hypothesis is that the concept we want isn't how the model 'thinks' about the concept, and the SAE working as intended. Alternatively, the SAE training distribution could be too narrow, resulting in the SAE not being incentivised to learn the important latents. (Kissane et al. 2024c) found that SAEs trained on pretraining data generally do not have good latents for the concept of 'refusing' harmful user requests, while SAEs trained on chat formatted data do. Or, the SAE might not have large enough dictionary size to learn all concepts of interest. Many more hypotheses are plausible. A complicating factor in using SDL to identify the learned mechanisms of neural networks is that the latents identified by SDL depend heavily on the data set used to train them. This is an undesirable property for a decomposition method that was initially hoped to be capable of identifying the fundamental units of computation in neural networks (Kissane et, al. 2024c
Current decomposition methods lack solid thcorctical foundations. Given the practical and con- ceplual issues with SDL, there is broad agreement that the question of how to correctly decompose networks into atomic units remains& a central problem, evidenced by the large amount of effort focused on the direction in recent years. After investing considerable effort in SDL approaches, it is apparent that improved con- ceptual clarity beyond the idea of superposition (Elhage et al. 2021) is needed to advance neural network decomposition.
One of the most significant open questions is the absence of clarity around the nature of features, despite being the central focus of SDL's identification efforts. Satisfying formal definitions are elusive and con- ceptual foundations are not yet established. However, even without foundations, progress in mechanistic interpretability is possible - even confused concepts can be pragmatically useful (Henighan 2024).
Without solid conceptual foundations, it remains unclear whether the superposition hypothesis, which under- pins the SDL paradigm, is fundamentally valid or merely pragmatically useful (Henighan 2024 Templeton ct al. 2024). ). If it is the latter, there may be better methods than SDL for carving noural networks at their joints. Such methods may take into account feature geometry or take a more dynamic, developmenntll view of how mechanistic structure emerges in the training process (Hoogland et al., [2024] Wang al.] 2024b).
Moreover, although there is much emphasis comparatively less work examining how they might perform computation on them natively in superposition. Further conceptual work into this problem may suggest. new methodologies for decomposing networks, or provide bounds on the number of features we should expect models to be capable of learning (Hänni et al., 2024 Adler & Shavit. 2024: Bushnaq & Mendel, 2024). The field should explore very different approaches that show promise for decomposing neural networks into functional components and that do not exhibit many of the problem of the SDL paradigm, such as decomposing or analyzing network parameters (Braun) et al. 2025 Tsang et al., (018)) or methods that decompose networks according to analytical descriptions of the local functions they implement, such deep Taylor decomposition (Montavon et al. 2017
how models represent features in superposition, there is
Mcchanistic interpretability should idcally be built on more formal foundations. Some work attempts to ground mechanistic interpretability in the formalisms of causality |Geiger el al., (024a). However, this approach has not yet yielded canonical causal mediators (Mueller et al.) 2024) that may form a basis for dccomposition methods. How should we go about finding them? Given that our field's objective to understand the learned structures that underlie networks' generalization behaviors, exploring theories about why neural notworks generalize appears to be a promising avenue. But theories that attempt to characterize
13

---

Published in Transactions on Machine Learning Research (09/2025)
why neural networks generalize, such the spline theory of neural networks (Balestriero & richard baraniuk, 2018), theories of neural networks' simplicity bias (Valle-Perez et al. 2019), deep learning theory involving the neural tangent kernel (Jacot et al., 2018 Roberts et al. 2022), or singular learning theory (Watanabe 2009 Wei et al. 2023) have either not yet yielded mathematical objects that can be casily used for interpretability: or simply have not. been successfully linked to approaches for interpreting neural networks. Establishing these connections would make significant progress toward carving neural networks at their joints to facilitate mechanistie interpretability. And if we can carve trained networks at their joints, it may suggest ways to train networks such that they come 'pre-carved'' Thus, better theoretical foundations may also be important for developing models that are intrinsically decomposable by design, which we discuss next.
Intrinsic interpretability: Building more easily decomposable models The current strategy of training a model solely for performance and then interpreting it post hoc may not be optimal if our goal is a model that is both interpretable and performant. To this end, it may be beneficial to prioritize interpretability during model training, for which there are several plausible approaches.
Instead of post-hoc decomposing trained network activations into discrete codes (Section 2.1.2 network activations could be forced to use a discrete code from the outset, as in Tamkin et al. (2025) 5). MLPs could also be trained with sparser activation functions, such as Topk (Makhzani & Frey 2013 Bills et al. 2023) or
SoLU (Elhage et al., 2022j. Similar approaches could be potentially used to restrict attention superposition (Jermyn et al. 2023) by limiting the number of heads attending to any query-key pair. Several approaches, such as 'mixture of experts' (Shazeer al., 2017; Fedus et al., 2022; He, 2024), use sparsely activating components with large enough number of experts, and with sufficient activation sparsity, experts may become individually interpretable (Warstadt et al., 2019). Many attempts to incentivize interpretable activations directly so far have not been competitively performant, and have also allowed 'superposition to sneak through', mitigating benefits.
We can also target weight sparsity directly during training, including approaches such as Lo regularization (Louizos et al. 2018) or pruning (Mozer & Smolensky 1988 Frankle & Carbin 2019 Mocanu et al.) 2018)..
Han et al. (2015) achieve sparse weights by using magnitude pruning followed by finetuning. This highlights
general strategy of finetuning with interpretability in mind (also used by Tamkin et al. (2025)), avoiding the
potentially excessive cost of training from scratch. Another related strategy is largeting modularity (Kirsch
ct al_]1_113_ 2018 Andreas et al. 2016). For example, brain-inspired modular training (Liu et al. 2023) trains for
modularity by embedding neurons in a geometric space and encouraging geometrically local connections.
Existing research implements other approaches that can simplify linearity-reliant circuit analysis Elhage et al., 2021). For example, there work removing the layer norm operations (Heimersheim, 2024). using
input-switched affine transformations for recurrence (Foerster al. 2017). Other studies leverage archi- tectures that are mathematically analyzable in other ways than linearity, such bilinear activations in MLPs
(Sharkey 2023
Pearce et al. 2024).
After decomposing a network into components, whether through post hoc decomposition or by using intrinsi- cally decomposable models, our task remains unfinished. We must provide an interpretation of the functional role of each component (Step 2 Section 2.1.3 and validate that interpretation (Step 3 Section 2.1.4b. In the next section, we will discuss common mothods of interpretation and their shortcomings.
2.1.3 Reverse engineering step 2: Describing the functional role of components
After decomposing networks into parts, the next step of reverse engineering is to "describe" the functional role of these components. This step is best thought of as generating hypothesized "interpretations" or 'expla- nations'. These explanations form candidate descriptions of the functional role of a given component, and should not be taken to be definitive conclusions before they are thoroughly validated (see Section 2.1.4..
Descriptions of the functional role of neural network components can either indicate (1) the cause of a component's activation or (2) what occurs after that component has boen activated, or - prrferably both (Figure 6 In this section, we will discuss the existing set of tools available to mechanistic interpretability researchers to describe the functional role of network components.
14

---

Published in Transactions on Machine Learning Research (09/2025)
What is the functional role of component
output effects
intermediate effects
component of interest
intermediate causes
input causes
irrelevant components
Figure 6: To study the functional role of the blue component numerous approaches are possible. We could study its causes: the purple input components or the red intermediate components via e.g. feature synthesis (Olah et al. 2017a 2020b maximum activating examples (Olah et al.) 2017a Bricken 2023 or attributions (Sundararajan
al.J 2017). Or we could study its effects: the orange output components or the green intermediate components via e.g. the logit lens (Nostalgebraist, 2020), activation steering (Turner ct al. 2024 or attributions (Marks et al. 2024
Explanations for what causes components to activate Explanations of the causes of component activation can use three broad categories of methods, each with several problems: (1) Highly activating data set examples; (2) Attribution methods; and (3) Feature synthesis.
Highly activating data set examples. The simplest method is to use highly activating data set exam- ples (sometimes called 'exemplar representations' (Hernandez et al., 2022 ). These are inputs on which a particular component is strongly activated. For given component, analyzing apparent commonalities in the imputs suggests hypothoses for what causes that component to activate. This step may be carried out by humans AT systems (Section 2.4
Despite being widely used, this approach has several substantial issues. The first issue is that the method relies on human prior beliefs, which may lead interpreters to project their human understanding onto models that may, in fact, be using unfamiliar concepts. This bias could lead us to identify concepts in the model that do not truly explain the model's functioning (Freiesleben & König 2023 Donnelly & Roegiest 2019 Gale et al. 2020
Another issue with this approach is the potential for 'interpretability illusions. Bolukbasi et al. (0221 show that bias in data sets can create misleading explanations even when top activating examples are selected from real data sets, as opposed to synthetically created data. Depending on the data set from which the examples were drawn, human annotators identified dramatically different meanings for given directions in the activation space of BERT [Devlin 2018).
A third issue is that this approach often yields plausible explanations for arbitrarily chosen directions in the activation space (Szegedy et al. 2014 This means that plausible explanations based highly activating data set examples cannot be solely relied upon to identify the basic units of computations in neural networks other methods are needed to accurately identify them. Furthermore, it is possible to develop adversarial models that deliberately yield misleading feature visualizations (Geirhos et al. 2025).
Many of the issues with highly activating data set examples stem from the fact that they merely provide correlational explanations for the activation of a network component, rather than causal explanations. To identify causal explanations, other methods, such attribution methods, are necessary.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Attribution methods are necessary for causal explanations but are often difficult to interpret.
Attribution methods (Bachrens et. al. 2009; Simonyan et al.] 2014a Nguyen et al.
2016b Selvara et al.
2019; Sundararajan et al. 2017 Fong & Vedaldi 2017] Lundberg & Lee 2017
Ribeiro et al. 2016) are
intended to measure the causal importance of upstream variables (such as inputs) (such as a network component).
on downstream variables
Specific examples of attribution methods include Grad-CAA (Selvaraju al. 2019) and Integrated Gradients (Sundararajan et al. 2017), as well as perturbation model-agnostic approaches such as LIME (Ribeiro et al.
2016 and SHAP Lundbberg & Lee
2017
Attributions methods are often categorized as gradient-based (Mozer & Smolensky 1988
Baehrens et al.
2009.
Simonyan et al. 2014a Nguyen et al.) 2016b Selvaraju el al.] 2019 Sundarara aaaaaaaaaajaaa jan et al. 2017
Wang
iet al.
2024d) or sampling.. perturbation- or ablation- based (Fong & Vedaldi 2017 Vig et al. 2020 Geiger
ct al.
2020 Ghorbani & Zou 2020 Meng ct al.) 2022b Chan ct al. 2022a Nanda 2023a] However, on a
theoretical level, many gradient-based methods identily only a first-order approximation of the ideal attribu-
tion, which is sometimes poor approximation (Watson 2022). Most approaches only attribute important
single components, though some work, such as integrated Hessians, attributes important feature interac- tions (Janizek et al. 2020). Further extending the understanding of feature interactions, various methods have been proposed. For instance, KernelSHAP-IQ characterizes higher-order Shapley Interaction ion Index as a solution to a weighted least square problem, offering one way to quantify such interactions (Fumagalli iet al. 2024). Specific to Graph Neural Networks (GNNs), recent advancements such as GraphSHAP-IQ by Muschalik et al. (2025) enable the exact and efficient computation of any-order Shapley Interactions. Other specialized methods, like BiLRP, have been developed to identify feature interactions in specific contexts such text similarity models (Vasileiou & Eberle, 2024 Adebayo et al. (2018) revealed more practical im- plications, demonstrating that some gradient- gradient-based methods identify attributions that are independent both of the model and of the data generating process. Furthermore, an adversary can train a model or perturb
an imput to reveal any attribution map (Dombrowski et al. 2019 Ghorbani et al., 2019 Heo et al., 2019 Kindermans et al. 2019: Slack et al.) 2020 Zhang el al., 2020 For specific architect ures like Graph Neural Networks (GNNs), methods such as GNN-LRP have been developed to provide higher-order explanations by identifying relevant walks in the input graph using Layer-wise Relevance Propagation (Schnake et al., 2022p. Perturbation methods present certain issues, such as taking models off their training distribution and eliciting unusual behavior, among other theoretical complications (Feng et al. 2018 Molnar et al. 2021 Hooker et al.) 2021 Molnar et al. 2024 Freiesleben & König 2023 Slack et al. and accurate attribution methods thus remains an open problem.
2021 Developing efficient
Feature synthesis. Feature synthesis is a strategy that combines highly activating data set examples and gradient- attribution methods (Erhan al. 2009 SZzeedy al. 2014 Olah et al.] 2017a). This approach attempts to syuthesize inputs that maximize the activation of a component subject to some regular- ization, such as consistency with generative model (Nguyen et al. 2016a 2017) or total variation distance (Mahendran & Vedaldi 2014). However, criticisms of feature synthesis methods suggest that natural dataset examples may serve interpretation better (Zimmermann et al. 2021 Borowski et al. 2021 or show that current methods struggle to identify trojans (Casper et al. 2023a).
Explanations for the downstream effects of components Alternatively, the functional role of a component can be described through its downstream effects.
Studying the direct effect. The logit lens (Nostalgebraist, 2020) applies the models unembedding matrix to an intermediate residual stream representations, converting it into distribution over the model's output vocabulary. Direct logit attribution is generalization of this technique, applying to any appropriately sized
vector space in the model, c.g. the output of MLP layers (Geva et al. 2021 2022b a Dar et al. 2023). atlention blocks, gradients (Katz et al., 2024) of these, and SDL decoder weights (Bricken] 2023). Adding a trainable affine or linear transformation before unembedding, as also referred to as the tuned lens, improves decoding accuracy (Belrose et al.) 2023a Yom Din et al. (024), at the cost of less faithfully representing when the model has completed computation.
4 'owweerr, Freiesleben & König 2023 argue that adversarial examples do not truly undermine saliency maps as these are highly dissimilar to real interprelatiliiy challenges.
16

---

Published in Transactions on Machine Learning Research (09/2025)
In the language of causality PPaarl 2009). unembedding a residual stream vector measures the direct effect of that vector on the output (MeGrath et al. 2023). However, the logit lens cannot measure the indirect effect, the effects resulting from the influence that the embedding has on the hidden activations of subsequent layers. Other mothods, such as causal interventions (sce below), are necessary to mcasure the indirect effect. In the future, it may be possible to exlend logit-lens-like approaches to not only project the effects of network components on directions in output vocabulary space, but also on intermediate downstream components.
Causal Interventions. Causal interventions typically substitute ("patch") the value of some network component, usually activation vector, with different value during forward pass, observing the resulting effect on the model. There exist a related set of techniques: ablation (for the special case of zero-ing or
otherwise attempting to delete activations entirely), activation patching, causal mediation analysis, causal
tracing, and interchange intervention (Vig et al. 2020 Geiger et al. 2020 Meng et al. 2022a Chan et al.
2022a; Nanda 2023a
More surgical patches are also sometimes also made on edges between network components. This approach, known as "path" patching, allows us to isolate the effect of one particular component on only one other
component, instead of on the entire rest of the network (Goldowsky-Dill et al. 2023). Causal scrubbing (Chan et al. 2022a) is generalization of path patching that allows for testing hypotheses concerning any given connection between a set of network components.
Causal intervention methods can also generate supervision signal to identify subspaces of interest. For instance, distributed aligmment search (Geiger et al. 2024c) learns a linear subspace that represents a particular concept, using data with interventions that concept supervision (Geiger et al., 2024c Guerner
ct al. 2024 Wu ct al. 2023 Other work learns masks over network components remove irrelevant
components De Cao et al. 2020 Csordás et al. 2021 Davies et al. 2023).
A causal intervention typically requires a forward pass of the model. This may make performing one for every network component in large models, long contexts, or when using finer-grained components such as sparse auloencoder latents prohibitively expensive. Faster allernatives that work well in practice (Marks) et al. 2024 Templeton et al. 2024h include gradient- approximations to activation patching, such as
attribution patching (Nanda 2023a Syed et al. 2024). AtP* (Kramár et al. 2024). and integrated gradients
Sundararaaan al. 2017).
Observing the effects of components on sequential behavior. Another way to study the effects of model components is to patch in activated components and observe their effect on model behavior. Steering is one example of this (Rimsky et al.] 2024 Turner et al.; 2024). where components (activation vectors) are activated in order to influence the network's behavior, often interpretable ways. To determine the functional role of an activation vector, related method is have a language model itself decode the Ghandeharioun et al. 2024 Huang et al., 2024b Kharlapenko et al. 2024). These approaches, known "patchscopes", patch activations from one forward pass of model into a different forward pass (perhaps in different model). The context of the new forward pass is designed to elicit relevant information from the activations of the original forward pass.
activations (Chen et al. 2025 Watkins 2023
A related sequential behavior based technique is simply to read the cfain-of-thoughtt (Wei et al. 2024 Kojima ct al.] (024) produced by a language model's output. While this could be considered an 'interpretability technique" in the sense that aims to explain model decisions, it does not use model internals in those explanations, at least not directly. Recent research demonstrates that chains of thought are not entirely faithful to the model's underlying decision-making process AAgarall et al. 2024 Atanasova et al.] 2023 Turpin et al. 2023 Lanham et al. 2023 Ye & Durrett 2022). A promising future direction for interpretabil. ity research may be to incorporate model internals into chain-of-thougg training, which could incentivize faithfulness. Another possibility is building monitors based on model internals to improve transparency in thoinhtf faithfulness.
For instance, by patching activations from the corrupt prompt "the capital of Italy is" into the clean prompt "the capital of France is", we observe the effect the output ("Rome" vs." var'Pa"ss)). This tells us which componcnt values relevant for the differing output hetween the two prompts, but not the information that remains consistent (e.g. the fact that the answer is rity).
17

---

Published in Transactions on Machine Learning Research (09/2025)
2.1.4 Reverse engineering step 3: Validation of descriptions
Initial descriptions of network components' functional roles should be treated as hypotheses that first require validation to ensure that they are rcasonable.
Conflating hypotheses with conclusions has regrettably been commonplace in mechanistic interpretability
research, making validation an important area for the field to improve (Madsen et al. 2024 Stander et al,
2025 Unfortunatcly, it is often hard to distinguish faithful explanations of ncural network components
foom merely plausible ones. Numerous examples of model interpretations fail sanity checks (Adebayo el al.
2018 Leavitt & Morcos, 2020 Miller et al., 2024); "interpretability illusions" in which seemingly convincing
interpretations of a model later turned out to be false (Bolukbasi et al., 2021 Makelov al., 2023): ; or
instances where different approaches to explaining the same phenomenon yielded different interpretations
(e.g- Chan et al. (022bb) or Chughtai et al. (2023) vs. Stander et al. 2025 vs. Wu et al. (2024a) ) Several
other instances were previously cited in this review (Section 2.1.3). Hypotheses in interpretability require
extensive validation beyond what appearances might imply.
Validating a hypothesis involves posing a simple question: the ncural network's behavior? Testing the hypothesis often requires multiple approaches (Mueller et al., 2024). To validate descriptions, many approaches simply apply different description method than the one used to generate the initial hypothesis (Section 2.1.3). If one description method yields a different description from another, it invalidates the hypothesis, which necessitates returning to an carlier step in the reverse engineering cycle (Figure 3). However, methods for validating descriptions are not limited to other component description methods. Hypothesis validation may take many forms, including the following:
Does the hypothesis make good predictions about
Predicting activations and countorfactuals: By using natural language explanations of a given network component's function, is possible to predict the component's 's activation levels on different inputs. This analysis can be carried out by humans or by AT systems (Section 2.4 as in (Hernandez et al., 2022 Bills al., 2023 Shaham et al. 2025 Juang et al. 2024).. In essence, our interpretations should enable us to successfully predict counterfactual scenarios in neural networks. For instance, if we ablate or activate particular network components, we should be able to predict specific downsstream effects on other components.
Predicting and explaining unusual failures or adversarial examples: Good explanations of neural network bohavior should help us identify and explain cases where that behavior fails to produce expected outcomes. For instance, Hilton et al. (2020) validated their methods by explaining cases where a deep reinforcement learning agent's neural network failed to achieve maximum reward and also explained specific hallucinations exhibited by the network. Another approach is to use use an interpretability approach to
handcraft an input for network that functioned as an adversarial example (Carter et al.) 2019 Casper
et al. 2023c Mu & Andreas 2020: Hernandez et al. 2022).
Handcrafting a network that reconstructs a network behavior: If our explanations for network behavior are sufficient, we should be able to use them to build replacement parts for the original network. Cammarata al. (2020) validated their interpretation of a curve detector's function in a convolutional neural network by substituting its parts with simple handcrafted replacements.
If the weights of a toy neural network were handcrafted by humans, it is possible to obtain a ground truth explanation for how it works. This proves useful for testing explanations produced by interpretability methods. For example, Conmy et al. (2023) validated a tool's ability to attribute model behaviors to internal components by rumning it on simple model that implemented a known algorithm. (Soe also Section 2.1.4).
Testing on ground truth:
Using the hypothesis to achieve particular engineering goals: Another way to test explanations
is to assess their utility in downstream applications [Doshi-Veler & Kim 2017 Casper et al. 2023a). For
example, Templeton et al. (2024) discussed examples where manually editing a large language model based on an interprelation led to predictable high-level changes in its behavior. Meanwhile, Marks et al. (0024)
showed how an interpretability tool could assist humans with debugging a classifier in a toy task. Farrell ct al. (2024) use unlcarning (Section 3.2.2) demonstrate that learned SDL latents don't quite match human concepts, and might not be optimal for particular downstream use cases, highlighting potential issues with SDL.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Using the hypothesis to achieve specific engineering goals competitively: Achieving not only useful, but competitive methods sets an even higher standard. For interpretability tools, the highest evaluation criteria require fair comparisons against relevant baselines on real-world tasks instead of cherry-picking them. However, the practice of conducting evaluations using non-chery-pickodd tasks remains relatively uncommon. Although attempls have been made to use techniques in the current interpretability Loolkit, in such ovaluations, they have not proven to be consistently uscful (Adebayo et al., 2020 Denain & Stcinhardi, 2023 Casper et al.) 2022b Hase et al.; 2023; Durmus et al.) 2024). Some of the most promising research directions are to use interpretability methods to achieve things that would be hard or impossible to achieve without them (Schut et al. 2023). Unless interpretability methods demonstrate that they are competitive with alternative approaches to achieve engineering goals, then the act of demonstrating their usefulness may lead to a bias loward developing methods that only perform well in best-case scenarios and on simple tasks, rather than those that can handle worst-case scenarios and practical challenges.
Interpretabllity rescarchers have historically faced challenges in adoquately validating their hypotheses due to the high costs in terms of time and cognitive labor. In the following section, we explore two potential solutions that could simplify the validation process: model organisms and interpretability benchmarks.
'Model organisms' facilitate hypothesis validation. Although interpretability is sometimes motivated by achieving engineering goals, it is often also approached through the perspective of the natural sciences (Olah et al., 2020b). In certain natural sciences, such genomics and neuroscience, it is common for researchers to investigate a few extensively studied species known as 'model organisms" or 'model systems! By conducting in-depth studies on a select group of organisms, like E. coli, fruit flies, mice, and macaque monkeys, researchers can leverage the insights and tools gained from those organisms and apply them to other species. For example, imaging specific types of neural activity in mice is more tractable due to existing hypotheses about which proteins should be fluorescently labeled in order to identify specific types of neurons. The use of model organisms allows for cross-checking results with previous work, enabling stronger validation of hypotheses.
Currently, interpretability rescarchers lack consensus on which networks should serve as model organisms. Essentially, what should the Drosophila melanogaster of mechanistic interpretability be? In mechanistic interpretability, an ideal model organism should be open source, easy and cheap to use, representative of a broad range of systems and phenomena, have a replicable training process with open source training data, and have multiple instances with different random soeds, among other critcria (Sharkey et al.] 2022a) Thus far, researchers have mostly used model organisms that possess only some of these criteria, such as a transformer than can perform modular addition (Nanda et al., 2023a) or GPT-2 (Radford et al.) 2018
Model organisms not only support cross-validation of hypotheses, but also facilitate the progressive construc- tion of experimental infrastructure by providing a reliable foundation for experiment design. This simplifies the process of rigorous hypothesis testing, thus helping prevent oversimplification and 'interpretability illu- sions'.
Studying solely model organisms, instead of more directly pursuing engineering goals, risks merely making true statements about neural network structure, rather generating insights that are of immediate practical benefit. For mechanistic interpretability to make the fastest and most substantial progress toward engineering goals, both scientific and engineering wins should be pursued in parallel.
Furthermore, certain choices made while studying model organisms risk steering the field in suboptimal directions. For instance, interpretability research is often motivated by the engineering goal of understanding state-of-the-art models thoroughly enough to make assurances of their safety (Bereska & Gavves 2024; Tegmark & Omohundro 2023 Dalrymple et al. 2024). However, limiting its focus by studying small toy models (c.g. Nanda et al. (022aa) or how larger models accomplish select subtasks (Arditi et al.) 2024 risks incentivizing research and methods that fail to generalize to more safety-relevant real-world setlings.
Validating interpretability methods using benchmarks. Beyond validating individual hypotheses, we may wish to validate entire interpretability methods. Benchmarking is proven approach to making
19

---

Published in Transactions on Machine Learning Research (09/2025)
incremental improvements in other areas of machine learning, with several approaches to benchmarking interpretability methods being developed in recent years.
One desideratum for interpretability benchmarks is to evaluate interpretations against ground truth expla- nations (Freiesleben & König, 2023 Zhou et al. 2022.. Benchmarks can be established using models with known ground truth explanations. Such models can be created by compiling simple programs into weights of models that exactly implement the known program (Lindner et al. 2023 Weiss et al.) 2021: Thurnherr & Scheurer 2024 Gupta et al., 2024). Alternatively, predetermined explanations can be enforced at train- ing time in conventional models using Interchange Intervention Training (Gupta et al., 2024 Geiger et al, 2022). Other interpretability benchmarks that evaluate specific steps in the interpretability pipelines also exist, such as model decomposition (Huang et al. 2024a Makelov et al. 2024), generating descriptions of network component functions (Schwettmann et al. 2023), or testing natural language explanations (Huang ct al.] 2023). Other benchmarks offer tests for multiple facets of the interpretability pipeline in order to test explanation faithfulness, but may require further adaptation to the context of mechanistic explanations (Hedström et al. 2023
While progress toward the goals of reverse engineering continues, it must be recognized that it is an ambitious goal, especially for very large neural networks. It may be the case that the field fails ultimately to make progress toward complete reverse engineering because of the fundamental difficulty of understanding such large, complex systems. One manifestation of this might be that, even though we might make progress toward 'carving networks at their joints', the components that result are either too complex or numerous to make ready use of them. Similar concerns have motivated some researchers in computational neuroscience to aim to study the learning process of neural system, rather than the mechanisms that result (Lillicrap) & Kording 2019
2.2
Concept-based interpretability: Identifying components for given roles
2.2.1
Concept-based probes
When attempting to localize a human-interpretablee concept within the network, an intuitive approach is to 'probe' for it (Köhn 2015 Gupta et al.) 2015 Alain & Bengio 2017 Ettinger et al. 2016). A concept-based probe is a classifier trained to predict a concept from the hidden representation of another model (Hupkes & Zuidema 2018). These concepts are typically human concepts of special interest. Probing requires labeling lunction that assigns classification labels to input data, indicating the 'value" of the concept on that data (Note: A binary value indicating the presence absence of concept is a special case of this approach). Once the labels are assigned, probe, which is simple parameterized model, is trained to predict concept labels based on hidden activations. If the probe is a linear model, then we have localized the concept as a vector in latent space.
Probes were first introduced in NLP Köhn explored in the field (Conneau et al.
2015
Gupta et al. 2015) and have since been extensively
2018 Tenney et al.] 2019
Rogers et al.) 2020 Gurnee et al. 2023
Peters et al.) 2018; Burns et al.] 2023 Marks & Tegmark 2024
They have also been applied in vision
(Alain & Bengio 2017 Kim et al. 2018b) and deep reinforcement learning (McGrath et al. 2022 Forde
et al. 2023). Probing also includes concept activation vectors (Kim el al. 2018a), information--heerelle
probing (Voita & Titov 2020), and structural probing (Hewitt & Manning 2019).
Although relatively simple to implement, probing has two main challenges (Ravichander ct al. 2021 Belinkov
2022b):: (1) The need for carefully chosen data for well-defined concepts, and (2) probes detect correlations
instead of causal variables in hidden activations.
2.2.2 Probes need carefully chosen data for well-defined concepts
Concept-based probing requires a labeling function that assigns labels for the (human) concept of interest to input data, Obtaining a labeling function is not always trivial and may require substantial human effort to define data set for a single concept. Morcover, it is only possible to identify concepts that we have defined preciscly enough create high-quality data. This limitation implics that concept-based probing can only identify concepts that we were already looking for, rather than reveal unexpected features in the
20

---

Published in Transactions on Machine Learning Research (09/2025)
network. Another approach, known axis in activation space that corresponds to positive or negative propositions by enforcing probabilistic consistency conditions (Burns et al. 2023). Despite being more unsupervised than standard concept-based probing, even CCS requires the construction of data sets with clear positive and negative cases. A potential path forward for concept-basee decomposition is to develop methods that automatically develop data sets for probing and concept localization (Shaham et al. 2025) (Section 2.4
Search (CCS), probes not for single concepts but
2.2.3 Probes detect correlations, rather than causal variables, in hidden activations
Probes are tools for a correlational analysis, measuring if hidden activations serve as signals for given concept. Indeed, from an information-theoreticc point of view, an arbitrarily powerful probe measures the mutual information between hidden representation and a concept (Hewitt & Liang 2019 Pimentel et al. 2020
However, training a probe to associate concept with specific hidden activations does not necessarily imply that those activations causally mediate how that concept is used by the network, or even if the network uses the concept at all (Ravichander et al. 2021 Geiger et al. 2024b Flazar et al., 2021 Belinkov 2022b). Probes can be successfully trained on hidden activations that lack any causal connection the output, only localizing correlated hidden activation vectors. For this reason, probing should be used only to generate hypotheses about which network components might be causally linked to concept. Confirming such hypotheses requires further invostigation using causal interventions or other probing methods.
To improve the causal relevance of probe vectors, one approach is to use counterfact ual data, which involves intervening on the concept of interest (for instance, observing the resulting output if the dog in an image was changed to a cat) (Elazar et al., 2021 Mueller 2024 Geiger et al. 2024b). Methods include distributed alignment search (Geiger et al.) 2024c] Wu al. 2023 Huang et al. 2024a), causal probing (Guerner et al. 2024), using attribution methods to measure the effect of concept vectors on network predictions (Kim et al.
2018b1, and various concept erasure methods (Ravfogel et al. 2020 2022 Blazar et al. 2021 Belrose et al. 2023b bb. While these methods can identify cansal mediators of concepts in hidden representations, they require more specialized data than probes do.
At times, it might be acceptable for probes to identify merely correlated hidden activations if the correla- tions generalize to the test. distribution. However, probing approaches face considerable risk of not only discovering correlated features, but also spurious correlations due the high dimensionality of hidden ac- tivations. Validating probes is therefore essential to avoid overfitting. This includes evaluating them out-of-distribution lest data that varies along task-specific dimensions to ensure that general purpose lea-- tures have been found. One question that remains unanswered is how to use regularization to achieve good probe generalization.
2.2.4 Concept-based intrinsic interpretability
Although probes begin with a trained network and search for specific concepts within it, it is also feasible to leverage the concepts in the network training process itself, such as in the case of concept bottleneck models (Koh et al., 2020b. This beneficial as it is more likely that the components to which concepts are assigned are causally relevant. Instead of specific concepts, networks can also be trained to use particular causal structures (Geiger et al.) 2022). While it may not be possible to prespecify all relevant concepts structures, integrating this approach with methods for disentangling concepts could prove useful (Chen et al.] 2018). [Cloud et al. (2024) incentivize modularity via applying data-dependent, weighted masks to gradients during backpropagation.
2.3 Proceduralizing mechanistic interpretability into circuit discovery pipelines: case study
How should we codify the process of mechanistic interpretability to yield the deepest possible insights? To form a complete pipeline by combining various methods, several methodological choices regarding decompo- sition, description, and validation, must be made. Circuit discovery has emerged as a prominent pipeline in
recent. mechanistic interpretability research (Wang al. 2023: Hanna al. 2023 Heimersheim & Janiak
21

---

Published in Transactions on Machine Learning Research (09/2025)
Its objective is to describe how a neural network performs a task of interest while making specific choices for network decomposition, component description, and hypothesis validation. In this section, we look at the typical choices in each step in greater depth and discuss how this popular pipeline could be improved.
2023
The 'circuit discovery' pipeline takes the following steps:
Task Definition. For a given model we want to study, we select a task that the model can perform, and a dataset on which the network performs that task. This is a concept-based step, since the definition of the task was based how human researchers define task distribution.
2. Decomposition. During the decompasition step, it is common to think of the neural network as directed acyelic graph (DAG), where activations are represented by nodes and the "abstract weights" between them represented by edges. Most work thus far has selected architectural components (Section 2.1.2p, such as attention heads and MLP layers, to be the nodes. However, more recent work has also used SDL latents for nodes MMarks et al. 2024
3. An initial description step: Identify task-relevant vs. -irrelevant subgraphs. The circuit discovery procedure then identifies task-relevant nodes and edges. Typically, causal interventions are used, drawing samples from some "clean" and "counterfactual" data sets. Circuit discovery methods
are generally based iterative activation patching (Wang et al. 2023 Chan et al. 2022a
Lieberum
et al. 2023 or integrated gradients (Marks el al. 2024).
4. An iterative descriptionn-valiattoo loop. After obtaining a task-relevant subgraph, the next step involves describing the function of each node or edge individually. This step is less formulaic than previous steps. Researchers rely on their intuition, attempting to create testable hypotheses for the function of a component or edge of the circuit, and then design custom experiments to validate or invalidate their hypothesis. Only after several iterations of hypothesis testing through experimentation are researchers finally satisfied with their explanation. In research papers, this loop is rarely made explicit, only the final description is presented. However, Chan et al. (2022b) detail this process for understanding the induction task, and Nanda (2023b) provides another description of such a loop (building on work by Li ct al. 2023
5. Final Validation. Circuits are commonly evaluated based on three attributes (Wang et al. 2023
faithfulness, which refers to how closely the circuit approximates the entire net work'ss behavior, min- imality, which assesses nodes in the subgraph are unnecessary, and completeness, which determines whether any nodes not included in the subgraph are important for task behavior. Additional ad hoc validation methodologies also exist. For example, [Wang et al. (2023) generate adversarial examples for their task based on their mechanistic understanding, while Shi et al. (2024) devise suite of formal statistical hypothesis tests for circuit efficacy.
This circuit discovery procedure has yielded valuable insight, but falls short using current methods. The pipeline has several issues:
Task definition is concept-based. Defining circuits has thus far been with respect to tasks defined by humans. Milller et al.] (2024) demonstrate that. the within-task variance of model performance across the distribution of data points in a task is large, implying that the circuit provides a good approximation of the average case performance on the dataset, but a poor one for any individual data point. This suggests that the process of first selecting task and then studying how the model performs it may not be an effective approach to achieve "reverse engincering" -style interpretability. This approach may also create spurious correlations, an issue shared with concept-based probing (Section 2.2.1 Thus, it might be worth learning the task decomposition instead HHann et al. 2024).
Network decomposition methods are flawed. Perhaps most importantly, prior circuit discovery work has attempted to decompose models in either architectural bases (Wang et al. [2023 Conmy et al., (023) or sparse autoencoder latents (Marks et al. 2024: Huben et al. 2024). which are imperfect ways to decompose
22

---

Published in Transactions on Machine Learning Research (09/2025)
neural networks for mechanistic interpretability (Section 2.1.2
Future work could locate circuits in improved decompositions or simultaneously learn both network decompositions and circuits.
Circuit faithfulness is low. Simple early circuits were found to be unfaithful (Chan et al. 2022b 2023
Miller et al. (024) show that existing measures of faithfulness depend on the causal intervention imple- mentation used, and further demonstrate that such metrics are misleading when applied to several complex end-to-end circuits. Makelov et al.) (2023) argue that subspace activation patching via distributed alignment search may lead to interpretability illusion mechanisms, although these findings are contested by /Wu et al.) (002cc)
Scalable methods are only approximate. Identifying relevant components through individual interven- tions is costly when there are many components. Attribution patching Syed et. al. (2024) was designed to identify potential relevant candidates for further testing through intervention, which becomes more impor- tant as the number of components expands significantly through sparse dictionary learning (Marks et al., 2024). However, attribution patching uses gradients, which only yield a first-order approximation of the effect of ablating components (Wu et al. 2024cc Molchanov et al.) 2017), leaving it unclear whether this method and any improvements on it (Kramár et al. 2024 produce adequate approxinations.
Circuit discovery algorithms struggle with backup and negative behavior. Additional challenges for circuit analysis arise from the effects of "backup" and "negative" " behavior (Wang et al.) 2023
McGrath
itt al. 2023; McDougall et al., 2024), which actively suppress task performance and are thus not captured
by maximizing task performance metrics. Despite this, they remain important factors to consider; Muellrr
(2024) provides further discussion of these issues.
'Streetlight' Interpretability: Interpretability that focuses on models or behaviors that seem easier to interpret than average can be termed 'streetlight interpretability' after the |streetlight effect.. Much work in mechanistic interetability so far arguably belongs in this category, since the tasks studied so far have been deliberately selected to be simple to define and study mechanistically (Wang et al., 2023). This gives a misleading impression of the level of difficulty involved in implementing circuit discovery for any arbilrary task that a network implements. Indeed, attempts to study arbitrary circuits have proceeded less successfully Nanda et al. 2023b
Solving issues with current mechanistic interpretability pipelines remains an open challenge that promises sig- nificant benefit. Upon establishing reasonable procedures, automating the overall pipeline will become more feasible. However, some individual steps in mechanistic interpretability can already be fruitfully automated, as discussed in the next section (Section 2.4
2.4 Automating steps in mechanistic interpretability research
Historically, mechanistic interprelability research has required considerable manual researcher effort, though it typically studies models that are smaller than those at the frontier. To make interpretability useful for downstream use cases, scalable approaches are crucial. In this section, discuss automated interpretabilityy methods. We will explore previous cases where manual tasks in mechanistic interpretability have been successfully automated and address open problems in further automation.
Automating feature description and validation. A task that is amenable to automation is 'describing the functional role of model components' (Section 2.1.3.. With the increasing sophistication of language models, researchers have generated descriptions of the functional role of neurons in image models (Hernandez) let al. 2022), neurons in language models (Bills al. 2023), ; and sparse autoencoder latents in language models (Huben et al. 2024 Bricken 2023 Juang et al.) 2024 using highly activating data set examples. These interpretations are validated by assessing how effectively human or model can use them to predict the activation of a fcature in a given data set example, or predict where a feature is active within a single image or text excerpt. The success of these predictions can be used as a quantitative measurement of 'interpretabill This was previously used to measure progress toward a decomposition method that carves networks at the joints of their generalization structure, assuming that such a decomposition would be maximally interpretable (Section 2.1.2). While imperfect, these methods for interpretation hypothesis generation and validation might be improved by automating the generation of inputs to test. the interpretation hypotheses by ensuring that
23

---

Published in Transactions on Machine Learning Research (09/2025)
generations activate the interpreted feature (Huang et al. 2023]. defining more rigorous statistical tests (Bloom & Lin 2024b). Automatic component labeling could expand in the future to include descriptions of feature effects, relationships between features (Bussman et al. 2024), or how components interact during runtime produce behavior.
Automating circuit. discovery pipelines. An approach called Automated Circuit DisCovery' (ACDC) automates part of the pipeline discussed in Section 2.3 to identify computational subgraphs involved in particular tasks {Conmy al. 2023). Several works have since improved upon and accelerated this process (Syed et al. 2024] Kramár et al. 2024 Marks et al. 2024 Note that ACDC-like approaches in general assist in identifying relevant subgraphs for pre-defined task, but do not automate important subscquent steps, such as describing the functional role of subgraph components.
While significant progress has been made toward automating steps of mechanistic interpretability pipelines, fully automating current pipelines would not yield satisfactory explanations of model behavio Further methodological progress required for fully automated neural network interpretability to be capable of generating the quality of interpretations necessary to achieve our goals.
For attempt this using leading decomposition and description methods, Marks al. 2024
24

---

Published in Transactions on Machine Learning Research (09/2025)
3 Open problems in applications of mechanistic interpretability
Ultimately, we need mechanistic interpretability methods that enable to solve concrete scientific and engineering problems. While predicting the impact of fundamental science in advance is difficult, having concrete goals in mind during research is usually beneficial. We want mechanistic interpretability methods to help us achieve various outcomes, such as monitoring and anditing Al systems more effectively (Section 3.2 controlling of AI system behavior more precisely (Section 3.2.2p, predicting AI system outcomes more ac- curately (Section 3.3), enhancing AT system capabilities (Section 3.4). and extracting knowledge from AT systems (Section 3.5 We should also anticipate that mechanistic interpretability will uncover "unknown problems" present in systems, revealing that the true realm of challenges and possibilities is greater than what we currently perceive it to be.
As highlighted in the previous section, progress in mechanistic interpretability methods is multifaceted. Each axis of methodological advancement leads to varying degrees of progress toward different goals. Before we discuss open questions in its applications, we identify distinct axes of methodological progress that lead to different amounts of progress toward different goals.
3.1 Axes of mechanistic interpretability progress.
Decomposition vs. description of network components: Improvements in network decomposition versus component description methods offer varying benefits for different goals. Decomposition methods vary in their efficacy at carving networks at the joints of their generalization structure, while description methods can yield descriptions that vary in depth. Deeper descriptions of a component are typically more causal or mechanistic, whereas shallower descriptions may rely more on correlations and only connect to inputs or outputs without referencing intermediate causes or effects (Figure G). Deeper descriptions thus attempt to explain more about how the component interacts with other components within the network's algorithm. Certain goals can be achieved with minimal or no progress in decomposition or description, while others may demand substantial progress.
Extent of network decomposition or description: The extent of network decomposition or description needed may vary depending on the goal. Certain goals only require an understanding of specific network components (such as an individual features or a circuit), while others might require enumerating or under- standing of larger circuits or the entire model (as in 'enumerative safety").
Extent of task distribution analyzed: The scope of task distribution analysis also depends the intended goal. For instance, monitoring model for a single kind of behavior might only require decomposing or understanding the model only over a narrow task distribution, while others, such as formal verification, might demand understanding over the entire distribution of tasks.
Mcchanistic understanding post vs. during training: Understanding the mechanisms fixed model might suffice for some goals, but more ambitions goals might require an understanding of not only the models' mechanisms, but also how they change during the learning process.
In this section, we'll discuss how mechanistic interpretability has been used or could be leveraged to further the field's various goals. We'll assess the progress made thus far, and identify the advancements along different axes of methodological progress that will be most crucial to success.
3.2 Using mechanistic interpretability for better monitoring and auditing of Al systems for potentially unsafe cognition
3.2.1
Mechanistic interpretability-basse evaluations could help us detect unsafe or unethical AI cognition
Currently, we rely on "black box" evaluations to understand a model's capabilities, but studying input- output behavior alone may not reveal all dangerous behaviors. Such behaviors include deceiving users (Park)
et al. 2023b; Ward et al.) 2023 Scheurer et al. 20044] Meinke al. 2024); for instance, by intentionally underperforming on evaluations ("sandbagging" van der Weij et al. [2024) leveraging situational awareness
25

---

Published in Transactions on Machine Learning Research (09/2025)
Laine et al. 2024; giving dishonest responses tailored to match the user's beliefs ("sycophancy"; Sharma
et al. (0244b)) ). Interpretability techniques could be used to uncover the mechanisms underlying these potentially harmful behaviors and thus help to detect and characterize them. This becomes increasingly important as the capabilities of models increases, especially when using training methods that incentivize models to feign particular properties for the purpose of passing evaluation.
Using interpretability methods to identify internal signs of concern (also known as "white-box" evaluations (Capper et al., 2024) or aonderstanding-baeedd evaluations (Hubinger 2023 is therefore an important
problem. White-box evaluation methods could serve as tools to detect potential biases that arise when models learn to use spurious correlations (Gandelsman et al.) 2024a Casper ct al. 2022a Abid et al. 2022) However, human judgment might be required to determine which features are "supposed" to be relevant to the task (Marks et al., 2024 Kim et al. 2018a Goyal et al. 2022
Despite current shortcomings in decomposition and description, white-box evaluations are likely feasible to- day. Even shallow, correlation-based descriptions could signal potentially concerning cognition. For example, developing new methods that reliably distinguish between features that merely recognize deceptive behavior vs. mechanisms that cause deceptive behavior may be challenging. However, a correlation that flags both can facilitate catching the latter. To be useful, it may not even be necessary to decompose or describe the entire network; having descriptions for components that are used on concerning subdistribu- tions of model behavior might suffice. For instance, imperfect interpretability methods may help evaluators develop hypotheses about how models will behave, thus guiding further inquiry. Meanwhile, recent work proposes incorporating SDL (Section 2.1.2) into safety cases for advanced AI systems. By monitoring in- ternal representations, it could aid in detecting potential sabotage deceptive behavior before deployment Grosse 2024 While such approaches show promise, they have difficulty in validating whether learned features capture all concerning patterns of reasoning reliably.
method
Evaluations for unsafe cognition may be a particularly important use case as it plays well to the comparative advantages of mechanistic interpretability relative to the other areas of machine learning. The majority of other areas of machine learning already focus on controlling or steering the behavior of AT systems to alter input-output behavior. It is therefore unclear that this is to mechanistic interpretabilityys comparative advantage. On the other hand, interpretability is perhaps the only research area that attempts to understand the mechanisms of model cognition. This implies that it might be particularly fruitful for interpretability researchers to tackle problems that become easier to solve through improving such understanding: auditing for unsafe cognition, debugging unexpected behavior, and monitoring systems in deployment.
Enabling real time monitoring of AI systems for potentially unsafe cognition Beyond white-bon evaluations, interpretability has further applications in monitoring. For instance, internals could be used to passively monitor the system during deployment, much like content moderation systems currently in use today. Alternatively, internals could be used to flag when a model takes an action for abnormal reasons even in absence of satisfactory descriptions (Section 2.1.3 known as "mechanistic anomaly detection" " (Christiano) 2022 Johnston et al.) 2024), which may be a sign of suspicious behavior. Mechanistic anomaly detection primarily requires progress decomposition methods (Section 2.1.2p as it is necessary to be confident about what constitutes individual mechanism within the network. Current SDL methods identify active latents, but not active mechanisms, which are implemented by network parameters. It may not be necessary to have deep descriptions of the function of individual mechanisms as long as anomalies can be detected.
Improving our ability to red-team AI systems and elicit unsafe outputs Beyond white-box eval- uations, leveraging interpretability could improve our ability to conduct adversarial attacks, red-team, jailbreak Al systems (Casper et al. 2024). This process is beneficial as it exhibits failure modes models may display in the wild, when facing adversarial pressure from wide deployment or malicious actors, thereby enabling developers to elfectively preempl and address them. Furthermore, il may form a significant ele- ment of safety cases (Clymer et al. 2024 Balesni et al. 2024a Grosse 2024 Goemans et al.] 2024) for AI systems, by providing assurances of form: We tried hard to red-team the system, yet failed to exhibit con- cerning behavior despite having more affordances than users may have. One reasonable assumption is that developers may have white-box access to models, while users may not. Although many existing red-teaming
26

---

Published in Transactions on Machine Learning Research (09/2025)
methods (Perez et al. 2022 Zou et al. 2023b already require gradient access, we could additionally leverage interpretability insights to accelerate red-teaming. For instance, [Arditi et al. (2024) discovered universal "refusal direction" in that-finetune the behavior of refusing harmful requests. Lin et al. (2024) used this to red-team models by optimizing for inpuls that minimize the projection of the residual stream onto this direction during the forward pass. This approach may be more efficient than optimizing over the whole model, as previous methods did (Zou) et al. 2023b). Mechanistic interpretability techniques also promise improve our ability to attribute model outputs to their corresponding inputs (Section 2.1.3). This could enhance human red-teamers' ability to find key input features responsible for bad behavior in models, thus speeding up iteration cycles. Though such approaches are possible today with only feature-based understanding, they might improve with more crisp mechanism-basse understanding.
language models, which is causally important for models engaging in
3.2.2 Using mechanistic interpretability for better control of Al system behavior
Ensuring the safe deployment of AI first requires effective control over their behavior. Currently, the tech- niques used for this purpose are mostly unrelated to interpretability (e.g. Christiano et al. (2017; Rafailov
ct al. 2023); ; inter alia), but sometimes inspired by it (Rimsky et al. (2024), Zou ct al. (2024). Kirch
jet al. 2024;; inter alia). Mechanistic interpretability could assist in interpreting (Lee et al.) 2025) and improving (Conmyy & Nanda 2024 these control methods, in developing new ones. In this section, we outline interpretabiliy-inspiree control methods, and envision future possibilities with further progress in interpretability methods.
One new control method derived from mechanistic interpretability insights is activation steering (a.k.a. activation addition). A fixed activation vector, hypothesized to linearly represent model concept, added to an intermediate activation of a model at inference time (Li et al., 2024a] Turner et al.] 2024 Zou et all, 2023a: Rimsky et al.] (022).. [Turner et al. (2024) introduced activation steering, directly inspired by the Linear Representation Hypothesis (discussed in Section 2.1.2). This technique results from the hypothesis, and its success can be thought of as evidence for the hypothesis. Moderate success can be achieved in steering using basic decomposition and description methods. Mechanistic interpretability decompasition methods enable the steering of models toward a narrower range of behaviors with fewer side effects (Chalnev et al.] 2024). Advancements in mechanistic interpretability methods are likely to result in improved steering capabilities, such as activating entire mechanisms instead of individual features.
Machine unlcarning was originally defined as the problem of scrubbing the influence of particular data points on a trained machine learning model (Cao & Yang 2015). In the context of modern generative models, machine unlearning is more broadly defined as removing particular undesirable knowledge or capabilities ('unlearning targets') from models, while preserving model performance on tasks involving non-targets LLuu ct al. 2024a). Targets for unlcarning that are of particular interest include sensitive private or copyrighted data, model biases (Liu et al.] 2024b), and hazardous knowledge that could be misused by malicious actors (Li et al.] 2024b); for instance, information regarding the creation of bioweapons. A better understanding of how knowledge or capabilities are implemented within model internals can help in the development of
new techniques for machine unlearning [Belrose et al.) 2023b Zou et al. 2024 Guo et al., 2024 Pochinkov & Schoots 2024 Ashuach et al.] 2024), as well as to better evaluate unlearning efficacy through white-box, non-behavioral, techniques (Lynch et al. 2024 Deeb & Roger 2024 Hong et al.) 2024). Thus far, mechanistic interpretability methods that modify intermediate activations (but not weights) for unlearning have yet to yield competitive results (Farrell et al. 2024).
Unlearning falls under the broader aim of model (knowledge) editing, which seeks to make precise modifications to a machine learning model that incorporates specific knowledge with desirable generalization properties, while minimizing the impact other knowledge (Wang et al., 2024c). By attempting to carve neural networks at their joints, mechanistic interpretability could improve our ability to make interventions on knowledge with few side effects. Meng et al. (2022a) make initial progress toward interprelabbility-beesee model editing with their ROME technique. However, Thibodeau (2022) and Hase et al. (0023) highlight flaws the technique, indicating that mochanistic interpretability has not yet found appropriate model components to intervene on (Section 2.1.2). With better comprehension of neural networks, we should anticipate more surgical model editing techniques in the future.
27

---

Published in Transactions on Machine Learning Research (09/2025)
Editing any given capability or piece of knowledge presents greater challenge than deleting them. Mcaning- ful progress in unlearning and editing methods may depend on improved network decomposition methods, as it would require isolating the individual mechanisms that correspond to specific knowledge or capabili- ties. Progress in mochanistic interpretability may clucidate the structure of knowledge and capabilities in AI models, leading lo a better understanding of what kinds of model edits possibilities are realistic in future. Knowledge and capabilities could, fact, be part of large mechanisms that overlap with each other, making it challenging to isolate them into discrete components. For mechanistic interpretability to effectively guide editing, strong description methods will be necessary to understand how to modify specific targets without affecting others.
Finally, mechanistic interpretability may provide tools to rigorously understand how finetuning alters models. This may assist in debugging instances in which finetuning leads to undesired and spurious effects Casper et al. 2023b). Recent work (Jain et al. 2024 Prakash et al. 2024 Lee et al. 2025) suggests that ex- isting finetuning methodologies primarily make shallow edits to existing model representations and circuitry. Importantly, this suggests that harmlessness training (which trains models to refuse to answer harmful re- quests) may be cheaply undone. Empirical evidence supports this claim, both with further finetuning (Gade ct al. 2024 Lermen et al. 2024), as well as with causal interventions on the forward pass (Arditi et al. 2024h. Separately, localizing knowledge and capabilities within models may improve the sample efficiency of finetuning, by selectively modifying only relevant parameters (as in, e.g. Wu et al. (2024b)). Further advancement tools for comparing feature-level differences between models (such as Lindsey et al. (2024)) may accelerate our ability to debug finetuning or other control methods (Bricken et al.) (024). Mechanis- tie interpretability work has thus yielded several insights into how finetuning changes models and how to improve it, and may provide further insights and improvements in the future.
3.3 Using mechanistic interpretability for better predictions about Al systems
Accurately predicting model behavior in new scenarios regimes is difficult (arguably impossible) without understanding model internals. Interpretability could hopefully facilitate two kinds of predictions:
Predicting model behavior in novel situations
Predicting capabilities that arise during training or finetuning
3.3.1 Predicting behavior in novel situations
In order to determine whether an Al system may potentially underperform poorly or pose a safety risk in new situations, the ability to predict its behavior in untested settings is imperative. A model's behavior, which may only become apparent in unforeseen circumstances, cannot be fully captured by its performance on a finite set of behavioral evaluations.
By understanding the mechanisms of jailbreaking, we anticipate the means through which user might bypass existing sa saleguards (Lee et al. 2025 Arditi et al. 2024 Similarly, if models have "trojans", backdoors (Hubinger et al. 2024). adversarial examples, or biases, comprehending a model's internal mechanisms could improve our ability to predict when models will display undesirable behavior, even if these scenarios were not encountered during standard training or behavioral evaluations. Casper et al. (2023a) benchmark feature synthesis tools through their ability to aid developers identifying trojans, while interpretability assisted in identifying cases of adversarial examples (Gandelsman al.] 2024b Mu & Andreas 2020; Waan et al., 2023 Kissane et al. 2024a). ; and SDL was used to uncover biases based spurious correlations in an LLM- based classifier (Marks et al. 2024.. Beyond specific failures, interpretability methods can also be used to gain a broader understanding of network behavior. For example, prior work identified signatures in model internals that predict. a model's likelihood of hallucinating (Yu el al. 2024] or its knowledge of particular facts [Gottesman & Geva 2024). Mechanistic interpretability might also be able to identify when networks have learned 'shortcuts' (Geirhos et al. (220)) that might lead to generalization failures under distributional shifts, which may result unexpected and undesirable behavior. Identifying these 'shortcuts' in advance may help predict these failures in advance (Lapuschkin et. al. 2019
28

---

Published in Transactions on Machine Learning Research (09/2025)
Generally, being able to predict AI system's behavior in advance is more challenging but also more desirable - than merely being able to monitor its behavior and cognition. Mechanistic interpretability could allow us to make a certain type of claim, namely, "there exists no mechanisms that would cause the model to deliberately behave undesirably" (Olah) 2023). For a strong version of this claim, substantial progress in both decomposition and description methods is necessary. However, weaker versions of the claim, addressing specific undesirable behaviors, might be more feasible with near-term methods. For instance, if it is possible to decompose networks and identify all components, even basic description methods might let recognize that there are mechanisms relating to bioweapons or illicit substances within the network, thus letting us predict that models are probably not capable of instructing users how to fabricate bioweapons or illicit substances (a possibility sometimes referred to as "enumerative safety" (Elhage et al.) 2021 Olah 2023) As Al systems become increasingly agentic, claims about even more general behavior may be possible. Understanding their values or goals (or, less authropomorphically, 'the internal mechanisms that determine their action plans and actions") should enable us to better prodict their behavior across a broad range of contexts (Colognese & Jose 2023
When deploying Al in high-stakes scenarios, rigorous and reliable predictions are necessary, much like those demanded of safety-criticll software applications. Sometimes, such software is formally verified, thereby ensuring certain safety-critical aspects of its behavior are guaranteed, since its compliance with specific properties is mathematically proven. In the context of mechanistic interpretability, the equivalent would be
formal verification of Al systems (Dalrymple et al.) 2024 Tegmark & Omohundro 2023 Critch & Krueger 2020): mathematically proving that an AI system's 's behavior will satisfy a desired property on any input given distribution. Formal verification of AT systems remains an unresolved issue at present. The level of undorstanding of AI necessary to enable formal verification of large, general AI systems for nontrivial properties is well beyond the current capabilities of mechanistic interpretability. However, some recent studies using toy models provide a glimpse into what solving formal verification of Al might look like. Approaches inspired by mechanistic interpretability have been used to prove accuracy bounds on a single- layer transformer trained on a synthetic task, albeit with great difficulty (Gross et al.) 2024). Program synthesis through mechanistic analysis offers an alternative approach by converting simple trained neural networks into more interpretable, controllable, and verifiable programs (Michaud et al.) 2024).
Several open questions remain about the tractability of scaling these approaches from toy models to frontier systems. For instance, for program synthesis, it is uncertain to what extent computations within real-world neural networks can be reduced to operations that can be cleanly represented in symbolic code, or what the total length of such code would be. Ensuring the safety of agentic systems with formal guarantees is further complicated by the need to model a system's interactions in an arbitrarily complex environment that might not be formalizable (Seshia et al. 2022 Wongpiromsarn et al. 2023 Dalrymple et al., 2024).
3.3.2 Predicting capabilities that arise during training or finetuning
The most competitive methods of AI development systems result in uninterpretable systems that often fail in ways that surprise their developers OOpenAl et al. 2024 Team et al. 2024 Anthropic 2024). Applying mechanistic interpretability to alleviate this issue is a key area for future research.
Improved mechanistic understanding of model training could enhance the ability to predict when certain capabilities will appear. For instance, it has been observed that new model capabilities can emerge as a function of scale (Wei et al. (2022), though also see |Schaeffer et al. (2023) ). Evidence suggests that new capabilities may be learned in a somewhat discrete MMchaudd et al. (2023) or stagewise (Wang et al., 2024bb fashion, and that in synthetic data settings, the emergence of new capabilities coincides with abrupt changes in the trajectory of model parameters (Park et al. 2024a
Other work shows a correlation between in-context learning capabilities and the emergence of induction heads, an attention- based circuit mechanism (Olsson et al. 2022). By connecting these threads of research, the long-term hope for mochanistic interpretability rescarch to link small-scale mechanistic structure to larger- scale structure, such as the evolving shape of the loss landscape during model scaling (Olah) 2023). To make progress toward this goal, research needs to move beyond simply improving 'decomposition' (Section 2.1.2
29

---

Published in Transactions on Machine Learning Research (09/2025)
quality and instead be capable of describing the dynamic changes in the mechanistic structure of networks throughout the learning process.
'description' (Section 2.1.3)
We may also want to link the emergence of capabilities to specific properties of the training data set. Through mechanistic interpretability, we can create data sets that facilitate training models to demonstrate desirable attributes and predict their behavior. By attributing model outputs to specific training examples, influence functions have been applied to LLMs (Koh & Liang 2017] Grosse et al. 2023) to predict limitations in their generalization abilitics, such a lack of robustness when the order of certain phrases flipped [Berglund et al. 2024). Other work examines how data set composition shapes the emergence of in-context and weights-basee learning (Reddy 2024).
A related problem of interest involves predicting which model capabilities, that may not be present in a given model, can be "elicited" with sufficienlly advanced prompting or finetuning strategies (Greenblatt ct al. 2024). Prakash et al. (2024) find evidence that finctuning improves capabilities primarily by enhancing existing circuils, rather than developing fundamentally new mechanisms. Relatedly, (Jain et al.) 2024 and Lee et al. (2025) show that finetuning can mask capabilities present in base model in way that easily be reversed via simple changes to the model. Improved mechanistic understanding of finetuning could help reveal capabilities obscured in this fashion. Since capabilities are behaviors that often span multiple sequential steps, it may be necessary to have mechanistic interpretability methods that examine mechanisms spanning multiple time steps. However, current mechanistic interpretability research is primarily focused on understanding mechanisms involved in predictions at a single time step.
3.4 Using mechanistic interpretability to improve our ability to perform inference, improve training, and make better use of learned mechanisms
A mechanistic understanding of Al models could be leveraged to improve their utility, from faster inference and better training, to enhancing and manipulating representations.
By understanding the internal generation process of AI models, we could accelerate their inference. For example, it could help identify which parts of the computation could be skipped without changing the model's final output (Voita et al. 2019 Din et al., 2024 Voita et al., 2024 Gromov et al.) 2024). The ability to inspect the information or functions implemented in a model could facilitate the development of more effective distillation methods by recognizing gaps that should be distilled (Gottesman & Geva 2024) and discovering novel ways to distill them (Zhang et al. 2024).
Another aspect that mechanistic interpretability could enhance is model training. Interpreting how the model processes specific examples and using this information to influence its predictions (Koh & Liang] 2017 Grosse ct al. 2023) may inform the selection of better training data to improve the model's capabilities in desired ways. Moreover, better monitoring of the training process be achieved by correlating certain drops in training loss with capability gains (Olsson et al. 2022 Wang et al.
2024a) or identifying a general order
in which specific capabilities emerge during training (Michaud et al. 2023). In addition, identifying the contributing components of a given task could help to devise novel, parameter-efficien training methods. Finally, being able to decompose networks into their functional components presents possibilities to build components that lend themsclves to learning computational structures that we better understand (Crowson
et al. 2022 Fu al. 2023).
Mechanistic interpretability has the potential to not only accelerate Al inference and training, but also enhance its utility. Intervening in the model's computation has the potential to remove unwanted bugs in its reasoning abilities, and achieve ber balance between its knowledge recall process and latent reasoning (Yu) et al. 2023 Jin et al. 2024 Biran et al. 2024 Balesni et al. 2024b). More broadly, understanding the inner workings of different models could lead to better rocombination of what they have Iearned, such as combining model parameters (Wortsman et 2022 and transferring representations across models (Ghandeharioun
et al.
2025 Csiszárik et al. 2021
30

---

Published in Transactions on Machine Learning Research (09/2025)
3.5 Using mechanistic interpretability for 'microscope Al'
Current approaches for knowledge discovery from data involve statistical or causal analysis, dimensionality reduction, or using machine learning models that are inherently interpretable. These techniques can be valuable, but are influenced by human priors, typically assume linear relationships between variables, and cannot handle massive multimodal data. On the other hand, deep learning models are capable of encoding complex, non-linear relationships and extracting meaningful features from massive data sets without hu- man intervention. Historically, these abilities had limited scientific value, as without methods to interpret these models, we could not understand the patterns they found. However, with ongoing advancements in interpretability research, this is beginning to change.
Applying interpretability for knowledge discovery is sometimes called microscope AL This approach in- volves training a neural network to model a data set, then applying interpretability techniques to the model to gain insight into any (potentially novel) predictors it discovers. In this way, the superhuman pattern matching skills of deep neural networks can serve as a tool to parse complex data sets. By understanding these networks' detailed mochanisms, mechanistic interpretability may add now details to our understanding of the domains in which they have learned to perform so well.
Current methods allow for versions of microscope AI, depending on the kind of insights that we want to learn. Some examples of these applications include extracting novel chess concepts from AlphaZero and teaching them to top grandmasters (Schut et al. 2023) using CNN trained on defendant mugshots and judge decisions to reveal how facial features affect judgments (Ludwig & Mullainathan 2023), transforming psychology articles into a causal graph with an LLM to enable link prediction and produce expert-level hypotheses (Tong et al. 2024b), analyzing a CNN to learn previonsly unknown morphological features for predicting immune cell protein expression (Cooper et al., 2022). and applying methods like BiLRP assess similarity and extract insights from historical documents such as astronomical tables (Eberlc et al. 2022).
among several other studies (O'Brien et al. 2023b; Hicks et al., 2021 Narayanaswamy et al. 2020 et al. 2021). As interpretability methods improve along various axes, deeper insights in and across more domains will become possible.
Korol
Currently, the majority of scientists are unable to access microscope Al due to the need for specialized expertise in machine learning, interpretability, and domain knowledge to recognize significant new patterns, a combination of skills that is rare in many fields. This may change as interpretability research becomes more widely adopted in the sciences and as interpretability becomes increasingly automated and accessible.
3.6 Mechanistic interpretability on broader range of models and model families
The vast. majority of mechanistic interpretability research to date has focused on just three model families: CNN-based image models (e.g. Erhan et al. 2009 ; Nguyen et al. (2016c); Olah et al. (2020b)),
text models (c.g. Devlin (2018): Rogers et al. 20200) and GPT-based text models (c.g. Elhage et al. (2021)
Wang et al. (2023) Nanda et al. (2023a)). ). The degree of generalizability of these findings to other models
and contexts is currently a somewhat open question. Given that future frontier models may use architectures
that differ from the current state-of-the-art, and are expected to be multimodal by default, interpretability rescarchers may need to expand the range of models and modalities that they study and try to identify universal approaches that can be applied to all of them.
Assessing how well interpretability methods apply to architectures beyond those for which they were devel- oped, and whether we can develop techniques that generalize effectively across architectures remain open questions. This is especially important due to the recent success of other competitive architectures as alter- natives to CNNs and transformers. Notable alternatives include diffusion models (Sohl-Dickstein et al. 2015) Rombach et al. 2 and Vision Transformers (Dosovitskiy et al. 2021) for image generation wwalliiooonnllsliiii /classificationn and RWKV (Peng et al. 2023) and later state space models (SSMs) (Gu & Dao 2024) for language model- ing. Recent studies show that certain methods transfer from CNNs to SSMs (Paulo et al. 2024). and from
transformers to some SSMs e.g. (Meng et al. (202bb) vs. Sharma et al. (2024a) and (Wang al. (2023 vs.
Ensign & Marriga-Alonso
2024
31

---

Published in Transactions on Machine Learning Research (09/2025)
Beyond the transferability of interpretability methods, related open question concerns the transferability of conclusions across model families. The overwhelming majority of mechanistic interpretability research focuses on the transformer model family and therefore does not distinguish between observations that are modcl-specific and those that are not. Consequently, we may be overlooking valuable insights that could be obtained by comparing the results across multiple model families. These insights may, for example, evince or refute the 'universality hypothesis", which states that (Li et al., 2015 Olah et al. 2020b) different neural networks learn similar features and circuits to one another.
3.7 Human computer interaction with model internals
As we saw Section 3.3 the ability to control and understand neural networks tightly linked. Thus, mechanistic interpretability has great potential to facilitate new types of human-AI interaction. Systems amenable to human comprehension and control would allow diverse users to intuitively manipulate and interact with them based their preferences, greatly broadening their utility. As a starting point, Al engineers who build and test AI systems have an obvious interest in working with the internal workings of neural nets. If experts could visualize and interact with internal representations, it would unlock obvious benefits for scientific research.
However, interactive tooling has a much broader constituency, including policy-focused AI auditors, as well as end users. Consider an auditor looking for bias or safety issues in neural net. With a way probe the network directly instead of relying discovering potential low-probability but high-stakes errors. For end users, transparency into network facilitates appropriate calibration of trust. One recent idea proposes a dashboard that can show, in real time, the internal features that influence a chatbot's answers during a text chat (Chen et al. 2024 Zou et al., 2023a Viégas & Wattenberg/ 2023). Such a dashboard might help users spot AI errors, or display warnings safety-relevant. features activate. More generally, results from mechanistic interpretability could be used to blend direct-manipulation interfaces with text interfaces, providing users with richer palette of controls Carttrr & Nielsen 2017
testing behavior, the auditor can be much more successful in
32

---

Published in Transactions on Machine Learning Research (09/2025)
4 Open socio-technics problems in mechanistic interpretability
Effective practical application of mechanistic interpretability brings both technical challenges and complex social ramifications. It could enable us to act on AI policy and governance, presenting a valuable opportunity to implement regulatory slandards and social ideals through lechnical means (Section 4.1p. h. Such consequen-
tial impacts inevitably come with important social and philosophical considerations, which likewise require
rigorous inquiry if we are to fully realize the potential benefits of AI (Section 4.2
4.1 Translating technical progress in mechanistic interpretability into levers for Al policy and governance
Current frontier AI governance efforts rarely specify concrete ways in which a mechanistic understanding of AT models might be used to help implementation. For example, OpenAT's Preparedness Framework commits to mitigating biological risks posed by its Al systems (OpenAl 2023), but the framework lacks details on specific measures that might be taken. Progress in interpretability could potentially enable the removal of any knowledge from the model which could aid users in creating biological weapons (Li et al. 2024b). However, remains uncertain how technical progress will translate into better AI governance, largely due to the numerous open technical problems in mechanistic interpretability. However, there are several promising routes toward better levers for AI policy and governance.
These avenues include assisting companics and governments to identify risks through evaluations and enhane- ing forecasts about new AT developments; more thorough oversight of Al systems in deployment; simplifying how Al systems operate within existing liability law through clearer explanations of Al decisions; enabling governments to establish risk mitigation regulations and companics to commit to concrete mitigation com- mitments; and protecting copyright law.
An understanding of model internals can help Al labs assess the risks from frontier models CCaag et al. 2024 Shevlane et al.)
2023 Casper et al. 2024 and thus better fulfill their obligations under the EU Al
Act to "perform model evaluation with a view to identifying and mitigating systemic risk" (Council of the European Union 2024). More specifically, a mechanistic understanding of models could help evalua-
tors elicit dangerous capabilities via improved finetuning (United Kingdom Al Safety Institute 2024]. guide their adversarial red-tcam attempts (Tong et al.] 2024a), and ensure that Al systems are not intentionally underperforming evaluations (van der Weij et al., 2024). Advancements in mechanistic interpretability could also assist companies and govermments in anticipating when or if AI models will obtain specifie dangerous capabilities. Improved forecasting capabilities could enhance threat modeling by reducing the "reasonable disagreement amongst experts over which risks to prioritize" (Anthropic) 2024). For instance, it may help build consensus on whether language models are just stochastic parrots (Bender et al. [021] or if they have coherent world models (Li ct al., 2023). We also might be able to use interpretability to build ev- idence supporting or refuting different models of catastrophic threat, such as determining the validity of mesa-optimization (Hubinger et al.) (021)) or inner alignment concerns (Carlsmith) 2023). It would also
give additional time for companics to prepare adequate risk mitigation mcasures, (OpenA] 2023) and for governments to establish appropriate guidance or regulation [LK Government 2022
The EU AI Act mandates that developers of General Purpose Al models with systemic risk have obligations to report incidents involving their system to the AI office. Interpretability tools have the potential to continuously monitor AI inference and detect incidents that require reporting. Compared to incidents in other domains (for instance, nuclear socurity), AT systems allow us to log all inputs and system states, even those that may lead to catastrophic harm. Access to a small number of such data points may greatly improve our ability to mitigate similar future failures (Greenblatt & Shlegeris| 2024 For example, interpretability could be used to investigate the critical "features" of the input that led to the Al incident. This could improve our ability to further red-team the system (Section 3.3) and generate more similar data points that could result in similar incidents. This, in lurn, may help us reduce the likelihood of future incidents (Chan et al. 2024). The incident could be utilized more directly in the construction of test time monitors to detect similar future incidents (sec Roger (2023)). More speculatively, interpretability could be used to verify companies' compliance with domestic regulation (O'Brien et al.) (2023a), or to authenticate states' compliance with future international treaties regarding the use of AT (Aarne et. al. 2024
33

---

Published in Transactions on Machine Learning Research (09/2025)
Leveraging a mechanistic understanding of model internals could also make decision rationales for AI model outputs more easily obtainable. This could aid in enforcing citizens' rights under the EU General Data Protection Regulation "to obtain an explanation of the decision reached" by a system "based solely on automated processing" (GDP) 2016 Gilpin et al., 2019). ) Model editing tools could also resolve problems regarding the copyright status of existing generative models (Grynbaum & Mac 2023 According to the US Copyright Act (United States Congress), for any copyrighted work, an artifact "from which the work can be perceived, reproduced, or otherwise communicated.... ... with the aid of a machine or device" is considered copy of the work (Lee et al., 2024). Interpretability tools could help detect and remove memorized works that can be reproduced verbatim by generative models.
4.2 Social and philosophical problems in mechanistic interpretability
The ability to interpret advanced Al systems holds immense potential to advance the science of Al and increase our ability to control it |Critcc & Krueger 2020 Tegmark & Omohundro) 2023 Daltymple et al. 2024). In this paper, we provide an overview of various interpretability tools that offer novel insights. Nonetheless, interpretability research has thus far produced few tools that are used to make state-of-the-art systems safer in the real world (Rauker al. (023). Modern Al systems are still generally trained, evaluated, monitored, and debugged using techniques that do not rely on understanding their internal workings.
The absence of paradigmatic clarity is a major socio-technical factor for this. Questions such as which goals the field of interpretability should pursue, how success should be graded, and how we should define interpretability warrant more thoughtful answers than exist at present. In interpretable AI research, the motivations and methods employed are often described as "diverse and occasionally discordant" (Lipton 2018). At times, the objective of AI interpretability rescarch articulated as advancing a fundamental "understanding" or "uncovering the true essence" (Christensen & Cheney 2015) of what is happening inside black-box models. The intrinsic validity of this paradigm deserves philosophical inquiry.
However, from an engincer's perspective, pursuing "understanding" without practical downstream appli- cation misses an engineer's objective. Proponents of this view may contend that quantifiable benchmarks linked to concrete practical goals are accurate measures of the success of interpretability. This motivation is concrete and useful, but some have criticized interpretability research as artificially limiting the solution space to engineering problems. When interpretability tools are studied with motivations such as fairness or safety, Krishnan (2000) argues that, "Since addressing these problems need not involve something that looks like an 'interpretation' (etc.) of an algorithm, the focus interpretability artificially constrains the solution space by characterizing one possible solution as the problem itself." Thus, some argue that interpretability research has failed to produce competitive techniques (Rauker et al. 2023 Casper et al.) 20226). omitted non-interpretabiiiiy basolincs (Rudin) 2019 Krishnan 2020), and graded interpretability tools on their own curve (Doshi- Velez & Kim 2017 Miller 2019 Rauker et al.] 2023). In all salety-relevant applications of mechanistic interpretability, it important to assess the usefulness of interpretability against alternative methodologies. Failing to do so or misrepresenting these comparisons can lead to follow-up work that rests on false assumptions (Lipton & Steinhardt] 2019 Leech et al. 2024 This is particularly problematic when
it impacts the efforts of critical safety work.
Despite these concerns, it is not always imperative for mechanistic interpretability to strictly outperform uninterpretable baselines: It may still be helpful to develop methods that offer interpretabllity-baeee advan- tages, along with the benefits of more competitive uninterpretable methods (e.g. Rimsky et al. (2024)). While interprotablltt-baadd methods do not currently outperform black-box basclines, if they perform in a simi- lar ballpark, further progress in mechanistic interprelability could soon lead to better methods, especially in problems which we believe might disproportionately benefit from improved mechanistic understanding (Section 3
Another potential reason for lack of clarity is that models are often studied in a vacuum: Smart & Kasirzadeh (2024) emphasize that the usefulness or correctness of model interpretations can depend on the broader context of the modcl's development or deployment. For example, it may be hard to identify representations of fairness within models, since understanding this requires an understanding of broader contexts. The same data may lead to different conclusions under different definitions of fairness.
34

---

Published in Transactions on Machine Learning Research (09/2025)
Finally, the interpretability community must exercise caution in their communication to minimize potential abuses of the results of its work. Unfortunately, selective transparency can be used to actively mislead (Ananny & Crawford 2018). Furthermore, interpretability is at risk of being used for purposes that might serve corporate interests at the potential expense safety. The field of AI interpretability highly influ- enced by research leams the private sector. On one hand, industry resources and research contributions have significantly advanced interpretability research. On the other hand, compared to academia, corpora- tions publish selectively, often have financial conflicts of interest, and may provide limited transparency. Meanwhile, the recent definite but ultimately modest progress in mechanistic interpretability has been used to lobby against specific AT regulation by falsely claiming that, "Although advocates for AT safety guidelines often allude to the 'black box' nature of Al models, where the logic behind their conclusions is not transparent, recent advancmments in the Al sector have resolved this issue, thereby ensuring the integrity of open-source code models." (Andreessen Horowitz 2023
35

---

Published in Transactions on Machine Learning Research (09/2025)
5 Conclusion
While mechanistic interpretability has made mcaningful progress in both methods and applications, signifi- cant challenges remain before we can achieve many of the field's ambitious goals, if they can be achieved.
The path forward requires progress along multiple axes. We would benefit from stronger theoretical founda- tions for docomposing neural networks at the joints of their generalization structure. Current methods like sparse dictionary learning, while promising, face both practical limitations in scaling to larger models and deeper conceptual challenges regarding their underlying assumptions. We must also develop more robust methods for validating our interpretations of model behavior, moving beyond correlation-based descriptions to capture true causal mechanisms. Additionally, we need better techniques to understand how mechanisms evolve during training and how they interact to produce complex behaviors.
These methodological advances could unlock several promising applications. Improved interpretability meth- ods could enable more effective monitoring of potent ial risks, belter control over model behavior, and more accurate predictions of capabilities. For Al capabilities, mechanistic understanding could lead to more effi- cient architectures, better training procedures, and more targeted ways to enhance model performance. In various scientific domains, microscope AI approaches could help extract valuable insights from model inter- nals. To achieve these diverse goals, the field must ensure focus on generating insights that have real-world utility. This will involve establishing better benchmarks and comparing interpretability- based approaches to non-interpretability baselines. However, fastest progress will likely come from the field pursuing both scientific and engineering goals simultaneously, rather than one at the expense of the other.
The practical impact of progress in mechanistic interpretability extends beyond technical achievements. Interpretability tools could provide crucial mechanisms for governance and oversight. They could help verify compliance with safety standards, detect potential risks before deployment, and provide clearer attribution of model decisions. However, realizing these benefits will require careful attention to the risks of potential misuse and of giving false assurance about Al safety.
Looking toward the future, many expect current AT capabilities to be only a foretaste of what to come. As AI capabilities advance, the ncod for mechanistic understanding of their mariiinn-mmking processes becomes increasingly urgent. While the black-box nature of AT models remains unresolved, the untapped potential of mechanistic interpretability is what makes it such an exciting research area, and highlights the importance of solving its many open rescarch problems.
36

---

Published in Transactions on Machine Learning Research (09/2025)
LS managed the project and made major contributions to planning, writing, and editing content, as well as coordinating other contributors and synthesizing their perspectives where necessary. BC substantially contributed to the framing, writing and editing of the final manuscript, and made all of the figures. All authors (LS, BC, JBn, JL, JW, LB, NGD, SH, AO, JBm, SB, AGA, AC, NN, JR, MW, NS, JM, WS, EJM, SC, MT, DB, ET, AG, MG, JH, DM, TM) contributed to the initial writing and editing of various sections and giving foedback on versions of the manuscript.
We also greatly thank Fayth Tan for assistance with editing and Schmidt Sciences for their feedback various stages of the project and for funding the work.
References
Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repcaling directive 95/ 96/46/cc (general data protection regulation) (text with cea relevance), 2016. URL
Security, 2024. URL httpa//wwwww/./a//o/o//////////a
bitts://eeur-i wttrrreeeeeuurpp....o
Onni Aarne, Tim Fist, and Caleb Withers. Secure, governable chips. Report, Center for New American chips
Abubakar Abid, Mert Yuksekgonul, and James Zou. Meaningfully debugging model mistakes using concep- tual counterfactual explanations. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learn- ing, volume 162 of Proceedings of Machine Learning Research, pp. 66 86-88. PMLR, 17-23 Jul 2022. URL https://pproceedingg wwtiinggss........ www.ppressvvvee wtteess//vi///abdd22/ abid22a.htma..
Julius Adebayo, Justin Gilmer, Michael Muelly Ian Goodfellow, Moritz Hardt, and Been Kim. San- ity checks for saliency maps. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL.
new.rddinnnnnnnnnnss.eeee wwtppppprccpppp.rrrpppppp https/cccapp..eeeeeeeeeeaeaepeee fills//paper2218//tell pdf
Julius Adebayo, Michael Muelly, Πariaa Liccardi, and Been Kim. Debugging tests for model explanations. Tn H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information
Processing Systems, volume 33, pp. 700-712. Curran Associates, Inc., 2020. URL http:::/proceeddingg Paper Paaerr.-d..p.
Micah Adler and Nir Shavit. On the complexity of neural computation in superposition, 2024. URL https:
15889
Chirag Agarwal, Sree Harsha Tanneru, and Himabindu Lakkaraju. Faithfulness vs. plausibility: On the
(un)reliability of explanations from large language models, 2024. URL https://arxiv http:://arxx..orr///b//222. org/abs/2402. 04614
Rishabh Agarwal, Levi Mclnick, Nicholas Frosst, Xuczhou Zhang, Ben Lengerich, Rich Caruana, and Geolfrey Hinton. Neural additive models: Interpretable machine learning with neu- ral nets. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (ods.). Advances in Neural Information Processing Systems, volume 34, pp. 4699-4711. Curran Associates, Inc., 2021. URL http://wwwwg.gooooo...oooooopppp
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2017. URL
Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and
37

---

Published in Transactions on Machine Learning Research (09/2025)
Joshua Batson. Circuit tracing: Revealing computattinnal graphs in language models. Transformer Circuits Thread, 2025. URL
html
Mike Ananny and Kate Crawford. Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. New Media & Society, 20(3):973 989, 2018. doi: 10.1177/ 14614444816666445 URL
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pottern recognátion, pp. 39-48, 2016.
Andreessen Horowitz. Written evidence to the UK House of Lords Communications and Digital Select Committee inquiry: Large language models (LLM0114), 2023. URL https://commi cttp///sss.tttes.....mmmo// .arass.paallll parliament. Written evidence submitted to the UK Parliament.
uk/ htt/://tteeeerreee////77//////t
Anthropic. Giving claude role with system prompt, 2024. URL https://docs www.huiiitttrooccccocoooo com/en/
docs/buuld-withh paapud//prommttegnneeeereeee system-prompts
Anthropic.
Responsible
scaling policy. htt.tt/pppp.tcooo///////////000.
Policy
document,
Anthropic,
2024.
URL
Satrrppic-Reppaaaille-SSssa
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Fric J Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Chenyu Zhang, Ruiqi Zhong, Sean O hEigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Edwards, Aleksandar Petrov, Christian Schroeder de Witt, Sumeet Ramesh Motwani, Yoshua Bengio, Danqi Chen, Philip Torr, Samuel Albanie, Tegan Ma- haraj, Jakob Nicolaus Foerster, Florian Tramèr, He He, Atoosa Kasirzadch, Yejin Choi, and David Krueger. Foundational challenges in assuring aligmment and safety of large language models. Trunsactions on Ma- chine Learning Research, 2024. ISSN 2835-8856. URL hattps://openreee http://ppaareeeeeeeeeee/e/aaaaaa wow.rr/fooormmmdddoooooaaaa Survey Certification, Expert Certification.
Andy Arditi, Oscar Balcells Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnce, and Noel Nanda. Refusal in language models is mediated by single direction. In The Thirby-eighth Annual Conference Newral Information Processing Systems, 2024. URL
Tomer Ashuach, Martin Tutek, and Yonatan Belinkov. Revs: Unlearning sensitive information in language models via rank editing in the vocabulary space, 2024. URL http:://aaxxi..rr///b///00...000
Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasicwicz, Jakob Grue Simonsen, and Tsabelle Augenstein. Faithfulness tests for natural language explanations. In ACI, 2023. URL https:
18029
Kola Ayonrinde, Michael T. Pearce, and Lee Sharkey. Interpretability as compression: Reconsidering sae explanations of neural activations with mdl-saes, 2024. URL ftttpa://axxiv.or//ab///// 11179
David Bachrens, Timon Schrocter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Mueller. How to explain individual classification decisions, 2009. URL.
1128.
Mikita Balesni, Marius Hobbhahn, David Lindner, Alexander Meinke, Tomek Korbak, Joshua Clymer, Buck Shlegeris, Jérémy Scheurer, Charlotte Stix, Rusheb Shah, Nicholas Goldowsky-Dill Dan Braun, Bilal Chughtai, Owain Evans, Daniel Kokotajlo, and Lucius Bushnaq. Towards evaluations based safety cases for ai scheming, 2024a. URL htttss org/ abs/ /alss///11...1 03336
Mikita Balesni, Tomek Korbak, and Owain Evans. The two-hop curse: Lims trained on a->b, b->c fail learn a a->c, 2024b. URL htttp::://axxivorr//bbaaa
38

---

Published in Transactions on Machine Learning Research (09/2025)
Randall Balestriero and richard baraniuk. A spline theory of deep learning. In Jemnifer Dy and An- dreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 374 383. PMLR, 10 15 Jul 2018. URL https: 7/proceedings. mlr. www..prrssvv//// pptpees/veoobaalesrrrrrootttooee www.sstrreroot..htaaa
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Under- standing the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 2020. ISSN 0027-8424. doi: 1.107//pna..19077777 URL hattps://wit pnas. .rg/ cog//ontett/ ear1y//2000/08/33/11077771
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48 (1):207 219, March 2022a. doi: 10.1162/coli a 00422. URL httpge///eelaatiooooooor////2222
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Compulational Lingusiicss, 48 (1):207-218) 219, March 2022b. doi: 10. 10.1162/coll a 00422. URL http:///cciattlooog...og///222
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023a. URL
https://arx: http:///aariv..oaaaaaa/aeae argg/bb//300/ 20000002301 08112
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances Newral Information Processing Systems, volume 36, pp. 66044 66063. Curran Associates, Inc., 2023b. URL
mewpeeedddinnnnnnniippeee newripu wwtppccccppppaaaeee files/ pdf
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transpareney, FAccT '21, pp. 610 623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 4781430383 doi: 10. 10.1145/3442188.4445222 URL https:
//doi org/ 110 1141/144411188 3445922
Leonard Bereska and Stratis Gavves. Mechanistic interpretability for Al safety a review. Transac- tions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreviov. id=aPUeetPKu6 Survey Certification, Expert Certification.
Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: LLMs trained on "a is b" fail to learn "b is a". In The Twelfth International Conference on Learning Representations, 2024. URL https: new/ffoummiiiii
GPKTIktAOK
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https: 2023.
//openaipublic. core. core. windows wwwp:w/reee/eerorneeeeeee http://eurooe-eeelllaeeeeeeeeeee peppaaaaaaa/eppppp/r/eeeeeee...
Eden Biran, Danicla Gottosman, Sohce Yang, Mor Geva, and Amir Globcrson. Hopping too late: Exploring the limitations of large language models on multi-hop queries, 2024. URL http:////aawiiwooo/ab//////
12775
Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, and Connor Leahy. Interpreting neural networks through the polytope lens, 2022. URL attpps//arrxvv.or/aaaa
12312.
Joseph Bloom and Johnny Lin. Understanding sae features with the logit lens ai align- ment forum, Mar 2024a. URL https: http://wwww.ligmmnfffrrro wwwp:/rgggppp/ooooee hwtp://www..pps////////o.o...... understanding-sat wwtts-sae-fattts--eee -ightreee--itt-------g--------- logit-
39

---

Published in Transactions on Machine Learning Research (09/2025)
Joseph Isaac Bloom and Johnny Lin. Understanding SAE features with the logit lens. Align- ment Forum, March 2024b. URL https://www..o wwtp://wwwwggmm.torrrr.o wttp://wwwww....p/oo.////ooo/taa with-
sagesttaniig---a-featte -iteee--ttttttttot------- logit-lens
Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wat- tenberg. An interpretability illusion for bert, 2021. URL pittps://arxti wttp://wwwwggga///.....o..
Judy Borowski, Roland Simon Zimmermann, Judith Schepers, Robert Geirhos, Thomas S. A. Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain {cnn} activations better than stale-of-the-art feature visualization. In International Conference Learning Representalions, 2021. URL
Dan Braun, Jordan Taylor, Nicholas Goldowsky-Dill and Lee Sharkey. Identifying functionally important features with end-to-end sparse dictionary learning. In The Thirty-cightt Annual Conference on Neural Information Processing Systems, 2024. URL httt:p//pppaaooreeeaac//rrrrrrrp htt..a/tee.oooorraa/a/aaaaaaaaaa
Dan Braun, Lucius Bushmaq, Stefan Heimersheim, Jake Mendel, and Lee Sharkey. Interpretability in param- eter space: Minimizing mechanistic description length with attribution- parameter decomposition, 2025. URL httpps://rxxx.. wwwperiivv..rv//aaee httgg/ab//25011. 14926]
Leo Breiman. Classification and regression frees. Routledge, 1984.
Wieland Brendel and Matthias Bethge. Approximating CNNs with bag-of-local-featree models works surprisingly well imagenet. In International Conference Learning Representations, 2019. URL
httpss://openreview
Trenton Bricken, Oct 2023. URL http::///maaaattrrcr.cco///////r
Index.html
Trenton Bricken, Siddharth Mishra-Sharma, Jonathan Marcus, Adam Jermyn, Christopher Olah, Kelley Rivoire, and Thomas Henighan. Stage-wise model diffing. Transformer Circuils, 2024. URL https:
//transformer-cinnnm
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Carbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Tlya Sutskever, and Dario Amodci. Language models are few-shot learners, 2020. URL https: wrg/ www//bo////a/aaooooooooooe ab2// 14165
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 2023. URL
httpps:/openrrvview wtt.:/tottoooottdd-taauuuyoobb/
Lucius Bushnaq and Jake Mendel. Circuits in superpositiin:: Compressing many small neural networks into one. Aligmment Forum, oct 2024. URL http:///www..attaooooooooooootot
maaningg-aan-mmaaa -nerrll
Lucius Bushnaq, Stefan Heimersheim, Nicholas Goldowsky-Dill Dan Braun, Jake Mendel, Kaarel Hänni, Avery Griffin, Jörn Stöhler, Magdalena Wache, and Marius Hobbhahn. The local interaction basis: Identifying computationnallyyreevaaan and sparsely interacting features in neural networks, 2024. URL
2205 10928
Bart Bussman, Michael Pearce, Patrick Leask, Joseph Bloom, Lee Sharkey, and Neel Nanda. Show-
ing sae latents are not atomic using meta-saes, Aug 2024. URL https://www. lesswrong. wwt.:oonggcc///oo/e/////oo are -ot--oo---o awt--aaamiiissnnn meta-sae5
S.R. Cajal. Estructura de los centros nerviosos de las aves (1888). Jiménez y Molina, 1924. URL https:
7/books. //books.google.... www.:ggoooo.ooooooooooooo http........bboooooc.seeeeaa.a..
40

---

Published in Transactions on Machine Learning Research (09/2025)
Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schubert, Michael Petrov, and Chris Olah. Curve detectors. Distill, 2020. doi: 10.2391 223391//iisiill.00002..000
Yinzhi Cao and Junleng Yang. Towards making systems lorget with machine unlearning. In 2015 IEEE
Symposium on Security and Privacy, pp. 463 480, 2015. doi: 10.1109/SP.2015.000
Joe Carlsmith. Scheming ais: Will ais fake alignment during training in order to get power?, 2023. URL
https://arxiv.org: wwt.://.ggg/bbb//ae///ooee at1.ab////..1/7.....
Shan Carter and Michael Nielsen. Using artificial intelligence to augment human intelligence. Distill, 2017.
doi: 10.239115/disttll.000000 https://distill..ubb///00
Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas Distill,
2019. doi: 10.23915 10.23915/dlsttil1000111 distill.0000.. https://distill.ppu 2019/ http///////titaaioaataaaaa activation-atlas.
Stephen Casper, Max Nadeau, Dylan Hadfeeld-Menell and Gabriel Kreiman. Robust feature-level adver- saries are interpretability tools. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and
A. Oh (ods.), Advances in Neural Information Processing Systems, volume 35, pp. 33093-33106. Cur- ran Associales, Inc., 2022a. URL
neurips wwtpses...ppp...pppprrrrr..eee filess/paper/2222/fill/
Stephen Casper, Max Nadeau, Dylan Hadfield-Menell, and Gabriel Kreiman. Robust feature-level adver- saries are interpretability tools. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and
A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 33093-331000 Cur- ran Associates, Inc., 2022b. URL pestps://proceedinnee
neurips.ec.mm wew..pscccppppprrrrrrrr httprr/pp.cccppppppeeaaaappppppp fflles/paper/222//ieee Papee Paper-aapaeeeeeeee Conf coanferene..pde pdf
Stephen Casper, Tong Bu, Yuxiao Li, Jiawei Li, Kevin Zhang, Kaivalya. Hariharan, and Dylan Hadfield- Menell. Red teaming deep neural networks with feature synthesis tools. In Thirty-seventh Conference Newral Information Processing Systems, 2023a. URL https://openreviee wwtp://ooooomdddd0000mm.m..
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémn Scheurer, Javier Rando, Rachel Freedman, Tomek Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel- Raphael Segerie, Micah Carroll, Andi Peng, Phillip J.K. Christoffersen, Mchul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michaud, Jacob Pfau, Dmitrii Krashenin- nikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadffeld-Menell. Open problems and fundamental limitations of reinforcement learn- ing from human feedback. Transactions on Machine Learning Research, 2023b. ISSN 2835-8856. URL https: //tp:////peeeieeeeee tet/Eforum?iddexx24PP41P3 Survey Certification, Featured Certification.
Stephen Casper, Kaivalya Hariharan, and Dylan Hadfield-Menell. Diagnostics for deep neural networks with automated vopy/paste attacks, 2023c. URL http:p/://aaxivoorr//////./ 2211
Stephen Casper, Carson Fzell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, Marvin Von Hagen, Silas Alberti, Alan Chan, Qinyi Sun, Michael Gerovitch, David Bau, Max Tegmark, David Krueger, and Dylan Hadfield-Menell Black-box access is insufficient for rigorous ai audits. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, volume 35 of FAcoT 24, pp. 2254-2272. ACM, June 2024. doi:
0444///630106.3559000 URL http://dpxx...///// http:://dxxxi...... org/10. 1145//33/10666..111 0066300066666663777777
Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy, Improving steering vectors by targeting sparse au- toencoder features, 2024. URL https://arxiv wttpe/ivvorggab////eee httpg/ab//41110211999
Alan Chan, Carson Ezell, Max Kaufmann, Kevin Wei, Lewis Hammond, Herbie Bradley, Emma Bluemke, Nitarshan Rajkumar, David Krueger, Noam Kolt, Lennart Heim, and Markus Anderljung. Visibility into ai agents. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAceT FAceT "24, pp. 958 993,,73, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704505,. doi: 10. 2.1114536300 (6630006.3658944 URL
wtt.://o......//33011111 3658948
41

---

Published in Transactions on Machine Learning Research (09/2025)
Lawrence Chan, Adrià Garriga-alonso, and Nicholas Goldowsky-Dill method for rigorously testing interpretability hypotheses [redwood research] ai alignment forum, Oct 2022a. URL http:://ww...iigmeeettoooo wwwgamentforum...m..mmm... http://wwwwo.rooopoot//////////a causal
Causal scrubbing: A
Lawrence Chan, Adrià Garriga-alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, Tao Lin, Jenny Ni- Lishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: Results on induction heads - ai alignment forum, Dec 2022b. URL https: wttp:///wwwwwiggnnnfeo.......... wrligneentoruumoorg/// htg.raaeeeeetttaaaaaeaeerrrereer -en-
Lawrence Chan, Adri'a Garriga-Alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, Tao Lin, Jenny Nitishin- skaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: results on a paren bal- ance checker. Alignment Forum, 2023. URL https:///wwt //www. aliggmentforum org/s htsssaayaaeeeeeeasss p/EjuuffaQaz2mn74500
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie, A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol, 15(3), mar 2024.
ISSN 2157-6904. doi: 10.1145/3641280 URL http:: org/oo..///../..
10. 1145/3641289
David Chanin, James Wilken-Smith, Tomás Dulka, Hardik Bhatnagar, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders, 2024. URL https:
/aaxxiv http:///rr/abb//2099// ab0//409///2./. 14507
Haozhe Chen, Carl Vondrick, and Chengzhi Mao. Selfie: selfiinterprttationn of large language model embed- dings. In Proceedings of the s1 International Conference on Machine Learning, ICMT. JMLR.org, 2025.
Ricky T. Q. Chen, Xuechen Li, Roger Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL https://proceeding pdf.
Lee3df ke33fffbbb666223357777332------3
Yida Chen, Aoyu Wu, Trevor DePodesta, Catherine Yeh, Kenneth Li, Nicholas Castillo Marin, Oam Patel, Jan Riecke, Shivam Raval, Olivia Seow, Martin Wattenberg, and Fernanda Viégas. Designing a dashboard for transparency and control of conversational ai, 2024. URL hattps://arziii httppp////ari...or//b///////// 2406. 07882
Lars Thøger Christensen and George Cheney. Pecring into transparency: Challenging idcals, proxies, and organizational practices. Communicatton Theory, 25(1):70 90, February 2015. doi: 10.1111 http:/cmm..1000
URL https://doi. org/10. 1ot.///oo..//mm 12052.
Paul
F
Christiano.
Can
different mechanisms? https: //ww. al aliaaligggaala wlligmeentfors orum www..orgg/ossooe orgp/posts/JI//WP291AArr
we
efficienlly distinguish
Dec
2022.
URL
can hen-we-efficae eiccefffiiceellllyyyiiiiiiuuul
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fer- gus; S. Vishwanathha, and R. Garnett (eds.), Advances in Newral Information Processing Systems, vol- ume 30. Curran Associates, Inc., 2017. URL httpps:/proceeding.cee mewreiiinnnneuuui.... www.newwww........e..eeee filles/paper/
pdf
Bilal Chughtai and Lucius Bushnaq- 2025.
interpretability may be doomed. URL https://www.eessvrre
Activation space
Less Wrong,
be-ooodd
Bilal Chughtai and Yeu-Tong Tong Lau.
Understanding
positional
features
layer
SAEs. Less Wrong, 2024.
URL
42

---

Published in Transactions on Machine Learning Research (09/2025)
Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara En- gelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 6243-6267. PMLR, 23 Jul 2023. URL
mll wttppppsessssppppppspspee
.tml
Mark M. Churchland and Krishna V. Shenoy. Temporal complexity and heterogeneity of single-neuron activity in premotor and motor cortex. Journal of Neurophysiology, 6744)24335 4257, 2007. doi: 10.1152/ jn.00095.2007 URL
PubMed
ID: 17376854
Alex Cloud, Jacob Mldmann-Wezleer Evžen Wybitul, Joseph Miller, and Alexander Matt Turner. Gradient routing: Masking gradients to localize computation in neural nelworks, 2024. URL https://arxiv
Abs/2440.0433
Joshua Clymer, Nick Gabrieli, David Krueger, and Thomas Larsen. Safety cases: How to justify the safety of advanced ai systems, 2024. URL https::/arxxvvorg// wttp://r//abbs/....0/0/ 10462
Paul Colognese and Arun Jose. High-level interpretability: detecting an Al's objectives. Align-
ment Forum, September 2023. URL https::/www.lllgmeettttee wwtp.//ppaaa///////.........//..
Arthur
Conmy and Neel Nanda. rum, 2024.
Activation
steering with SAFs. Aligmment Fo-
URL
full post-progresss upeeee--ddattteeeeeeeeee the -he- hnt.-ppcddd-tttteeettttttttttaia Steeooinnnteiinnneeeee with
SAEs
Arthur Commy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (ods.), Advances in Neural Information Processing Systems, volume 36, pp. 16318 16352. Curran Associates, Inc., 2023. URL, /ttp/ www.eeeeurpp...ee. wwtpcc//pppp//e..eeeeeece files/ pdf
Alexis Conneau, German Kruszewski, Guillaume Lample, Loie Barrault, and Marco Baroni. What you can cram into a single $&!## vector: Probing sentence embeddings for linguistic properties. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Arnuul Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 2126-2136, Mclbourne, Australia, July 2018. Association for URL
Computational Linguistics. doi: 10. 10. 11155//11/11111111
http://ccaarbboooooooo.////-111 org/P18-1198/
Jack Cooper, In Hwa Um, Ogujen Arandjelovié, and David J Harrison. Lymphocyte classification from hoechst stained slides with deep learning. Cancers, 14(23):5957, Dec 2022. doi: 10.3390 10.3390/cancent http://aancccsseee 12t.uee9re1233955
URL https://doi org/10. 10. 3390/ cancers14235957 PMID: 36497439; PMCID: PMC9735034.
Council of the European Union Proposal for a Regulation of the European Parliament and of the Council lay- ing down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union
legislative acts, 2024. URL https://data.co http//iataccoosnll/ consi consilium lium. europa eu/ddc//doccmann//11-56222 202-66622222221177 en/pdf
Andrew Critch and David Krueger. Ai research considerations for human existential safety (arches), 2020.
URL https://arxiv. http:///wwxvvvogaaaaaa/ hatg/abs/2006 04948
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (ods.), Computer Vision ECCV 2022, pp. 88-105, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031- 19836-6.
43

---

Published in Transactions on Machine Learning Research (09/2025)
Adrián Csiszárik, Péter Körösi-Szabó, Åkos K. Matszangosz, Gergely Papp, and Dánicl Varga. Similarity and matching of neural network representations. In A. Beygebzime, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openrevier het/fforum?iddaeeIIIRR..
Róbert Csordás, Sjoord van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. In International Conference on Learning Rep- resenlations, 2021. URL btttps://openreviex.. http://ppaaaeeeeeeeeeeorrrraaaaa wttp://orrrumiid/////euuggu
Róbert Csordás, Christopher Potts, Christopher D. Manning, and Atticus Geiger. Recurrent neural networks learn store and generate sequences using non-linear representations, 2024. URL hnttps://arxt org/
abs/2408 10920
David Dalrymple, Joar Skalse, Yoshua Bengio, Stuart Russell, Max Tegmark, Sanjit Seshia, Sleve Omohun- dro, Christian Szegedy, Ben Goldhaber, Nora Ammann, Alessandro Abate, Joe Halpern, Clark Barrett, Ding Zhao, Tan Zhi-Xuan, Jeannette Wing, and Joshua Tenenbaum. Towards guaranteed safe ai: A framework for ensuring robust and reliable ai systems, 2024. URL https: www.aaiwww.....aaaaeeoooeeeoo 2405. 066244
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What is one grain of sand in the desert? analyzing individual neurons in deep ulp models. In Proceedings of the Thirty- Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAT Symposium on Educational Advances in Artificial Intelligence, MAAT1//AAT119EAAAA v33i01 33016309. URL htttps://bii.orggg/ 10t.o..99aaaaaa
AAAI Press, 2019. ISBN 978-1-57735-809-1 doi: 10.1609 /aaai.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In Anna Rogers, Jordan GoadeGraber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Popers), pp. 16124-16170, 70, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/w1/2022 acl- acl- 1933. URL https://acllnthho..og org 2023.222...o .aee3a.ccl1a0 acl-long. 893
Adam Davies and Ashkan Khakzar. The cognitive revolution in interpretability: From explaining behavior to interpreting representations and algorithms, 2024. URL
org/ abs/a///..... 05859
Xander Davies, Max Nadean, Nikhil Prakash, Tamar Rott Shaham, and David Bau. Discovering variable binding circuitry with desiderata, 2023. URL https://arxiv... www.aaawwiioooogbbaaaaa httggabb/23011 03637
Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNLP), pp- 3243 3255, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v 20tm2/0020.mmmpmmmii... 262. URL https://acllntthlllgg org/2020. emnlp-main m22
Aghyad Deeb and Fabien Roger. Do unlearning methods remove information from language model weights?, 2024. URL https://arxiv..rrg/ abs//410...22...///// 2410. 088277
Jean-Stanislas Denain and Jacob Steinhardt. Auditing visualizations: Transparency methods struggle to detect anomalous behavior, 2023. URL
2aabb./2206.... 13498.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXic:1810.04800 2018
Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to conclusions: Short-cutting transformers with linear transformations, 2024. URL https:://aaxivoorrg/bbbbbeeeeeee
094355
Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alchh-BBu, E. Fox, and R. Garnett (cds.), Advances in Neural In- formation Processing Systems, volume 32, Curran Associates, Inc., 2019. URL httpp::/pproceddinn htt httpa//seaaapaaoo///aooaaaaaaaaa Paper pdf
44

---

Published in Transactions on Machine Learning Research (09/2025)
Jonathan Donnelly and Adam Roegiest. On interpretability and feature representations: An analysis of the sentiment neuron. In Leif Azzopardi, Benno Stein, Norbert Fuhr, Philipp Mayr, Claudia Hauff, and Djoerd Hiemstra (eds.), Advances in Information Retrieval, pp. 795 802, Cham, 2019. Springer International Publishing. ISBN 978-3-030-15712-8.
Finale Doshi- and Been Kim. Towards a rigorous science of interpretable machine learning, 2017. URL
08608
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gclly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreever: wttp://wwww...etooooooooooooooo.
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits. In The Tiigtt-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:
/openrevies.net/ee wwt.:e/ooormmmmmddlllecccooooo
Esin Durmus, Alex Tamkin, Jack Clark, Jerry Wei, Jonathan Marcus, Joshua Batson, Kunal Handa, Liane Lovitt, Meg Tong, Miles McCain, Oliver Rausch, Saffron Huang, Sam Bowman, Stuart Ritchie, Tom Henighan, and Deep Ganguli. Evaluating feature steering: A case study in mitigating social biases, 2024. URL
Oliver Eberle, Jochen Buttner, Florian Krautli,
Muller, Matteo Valleriani, and Gregoire Mon- tavon. Building and interpreting deep similarity models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3):1149 1161, March 2022. ISSN 1939-3539. doi: 10.1109/tpamil http://pmm..2200000073333
URL http://ēx. http::///xxxdiioooogooo.ooo wrg.oo........oooooo.ooooooo..oo www.:////......//......... 11MIITPAMIIII 2020. 3020738
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg, Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175, 03 2021. ISSN 2307-387X. doi: 10.1162/tac) a 00359. URL https::/doi.i... wttg//.////. 11tp:/...a1ct/.aaaee.//ae _00559
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathemat- ical framework for transformer circuits. Transformer Circuits Thread, 2021. attps://transfont sutpiits.uubb21//ffaaae htt/////weoo//ndee..ttttoe
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield- Dodds, Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacob- son; Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah, Softmax linear units. Transformer Cireuits Thread, 2022. https:/ https://transformmrr strcuits.uub/2022, /owl///dex..ttm.......
Joshua Engels, Isaac Liao, Eric J. Michaud, Wes Gurnee, and Max Tegmark. Not all language model features 14860
are linear, 2024a. URL
Joshua Engels, Logan Riggs, and Max Tegmark. Decomposing the dark matter of sparse autoencoders, 2024b. URL
14670
Danielle Ensign and Adrià Garriga-Alonso, Investigating the indirect object identification circuit in mamba, 2024. URL Sullhttp::: httpt:://////tttii
Dumitru Erhan, Y. Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. Technical Report, Universily of Montreal, 01 2009.
45

---

Published in Transactions on Machine Learning Research (09/2025)
Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. Probing for semantic evidence of composition by means of simple classification tasks. In Proceedings of the 1st Workshop Evaluating Vector-Space Representations for NLP, pp. 134 139, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1 /1111111112222 URL http://aclanthoooy..orr/////// 2524
Eoin Farrell, Yeu-Tong Lau, and Arthur Conmy. Applying sparse autoencoders to unlearn knowledge in language models, 2024. URL
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, Journal of Machine Learning Research, 23(120):1-39, 2022. URL .tmml
http://jmlr.ors httpjjmwlr.ooooppeeem wwrg:/aperrs//3/211111
Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Scrre. Craft: Concept recursive activation factorization for explainability, 2023. URL https:
7/arxiv. attp/ 10154
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Tyyer, Pedro Rodriguez, and Jordan Boyd-Graber. Patholo- gies of neural models make interpretations difficult. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi ichi Tsujii (cds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3719-3728, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 1111186/31//11--//
URL htttp::/aaclatthooooo..oog/
Javier Ferrando, Gabricle Sarti, Arianna Bisazza, and Marta R. Costa-jussà, A primer on the inner workings of transformer-based language models, 2024. URL https://arxiv.orrg/ httpg/g/ab//24000
Jakob N. Foerster, Justin Gilmer, Jascha Sohl-Diekstein, Jan Chorowski, and David Sussillo. Input switched affine networks: an rmn architecture designed for interpretability. In Proceedings of the 34th International Conference Machine Learning Volume 70, 1CML'17, pp. 1136 1145. JMLR.org, 2017.
Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, October 2017. doi: 10.1109/ iccv.2017.371. URL http: //dx doi wtt..././..////..........
2017.377
Jessica Zosa Forde, Charles Lovering, George Konidaris, Ellie Pavlick, and Michael L. Littman. Where, when & which concepts does alphazero learn? lessons from the game of hex. Unpublished manuscript, 2023.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https::/openrevvev.
net//forum?iddrJJ--DBBCCC
Timo Freiesleben and Gunnar König. Dear xai community, we need talk! In Luca Longo (ed.), Explainable Artificial Intelligence, pp. 48 65, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-44064-9,
Dan Friedman, Andrew Kyle Lampinen, Lucas Dixon, Danqi Chen, and Asma Chandeharioun. Interpretabil- ity illusions in the generalization of simplified models, 2024. URL htttp:///pperrevvieeeeeef/uuuuu
v675Iyu0ta
Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference on Learning Representations, 2023. URL httpp://
Fabian Fumagalli, Maximilian Muschalik, Patrick Kolpaczki, Eyke Hüllermeier, and Barbara Hammer. Kernelshap-iq: Weighted least-square optimization for shapley interactions, 2024. URL https:://// pettps///arxis org/abs/2405.10853
Pranav Gade, Simon Lermen, Charlie Bogers-Smithh and Jeffrey Ladish Badllama: cheaply removing safety fine-tuning from laaa 2-chat 13b, 2024. URL atttpps//aarxv..or//a//////
46

---

/tt////112222//////// Published in Transactions on Machine Learning Research (09/2025)
Ella M. Gale, Nicholas Martin, Ryan Blything, Anh Nguyen, and Jeffrey S. Bowers. Are there any 'object detectors' in the hidden layers of cmns trained to identify objects or scenes? Vision Research, 176: 60 71, 2020. ISSN 0042-6989. doi: https://doi.org/11..0000 sciencedirect.
www.ess..20..6..... URL https://wwe
Yossi Gandelsman, Alexei Efros, and Jacob Steinhardt. Interpreting CLIP's image representation via text-based decomposition. In The Twelfth International Conference on Learning Representations, 2024a. URL http://ppareeeeeee..aaarrrrrraas
Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting the second-order effects of neurons in clip, 2024b. URL
04341
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024. URL
2406.04099
Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Afra Alishahi, Yonatan Belinkov, Grzegorz Chrupala, Dieuwke Hupkes, Yuval Pinter, and Hassan Sajjad (eds.), Proceedings of the Third BlockbozNT.P Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 163 173, Online, November 2020. Association for Computational Linguistics. doi:
1.16. URL https: hattps://acaane /aaa://hhoan
org/2020. blackboxnlp-111155
Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 7324-7338. PMLR, 17-23 Jul 2022. URL https:://pocceedinnnee wttpeoceeednngnm...... wtagggg.mmr...rrr..... ppress/v1622 www.////egegeeaaaa.. html
Atticus Geiger, Duligur lbeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. Causal abstraction: A theoretical foundation for mechanistic interpretability. In Machine Learning Research, 2024a. URT. https:
wwwpe/i..rr/aab/////// 04709
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2024b. Curran Associates Inc. ISBN 9781713845393.
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Teard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Francesco Locatello and Vanessa. Didelez (eds.), Proccedings of the Third Conference on Causal Learning and Reasoning, volume 236 of Proceedings of Machine Learning Research, pp. 160-187. PMLR, 01-03 Apr 2024c. URL https://pproceedingg mlr wttpppssssse//e/eppppppp geiger24a. htmml
Robert Gcirhos, Patricia Rubisch, Clandio Michaclis, Matthias Bethge, Felix A. Wichmann, and Wicland Brendel. Imagenet-trained CNNs are biased Lowards texture; increasing shape bias improves accuracy and nosustness. International Conference on Learning Representations, 2019. URL httpp::/openrevvee net/foruuridd-Byyhgj0000
Robert Geirhos, Jörn-Henrit Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine In- telligence, 2(11):665-67 663, November 2020. ISSN 2522-5839. doi: 01008//22225602000025555 URL httpp:/dxx..o doi http://dx..oo..oo/11..... 100.111842255-0000000000
Robert Geirhos, Roland S. Zimmermann, Blair Bilodeau, Wicland Brendel, and Boen Kim. Don't trust your eyes: on the (un)reliability of feature visualizations. In Proceedings of the 41st International Conference on Machine Learning, ICML, 24. JMLR.org, 2025.
47

---

Published in Transactions on Machine Learning Research (09/2025)
Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key- value memories. Tn Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott. Wen-tau Yih (eds.); Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484 5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistios. doi: 10. (0.18653/v1 (1653/v1/20// 1tt.22222..enmmmmaiee emmllp-main.446, URL hnttps://aclanthollan org/ 2021/22211..2 emnlp-main 446/
Mor Geva, Avi Caciularu, Guy Dar, Paul Roit, Shoval Sadde, Micah Shlain, Bar Tamir, and Yoav Gold- berg. LM-debugger: An interactive tool for inspection and intervention in transformer-based language models. In Wanxiang Che and Ekaterina Shutova (eds.), Proceedings of the 2022 Conference on Em- pirical Methods in Nutural Language Processing: System Demonstrations, pp. 12 21, Abu Dhabi, UAE, December 2022a. Association for Computational Linguistics. doi: 10.18653/v1 w00....mmp--eeooss22222 URL https://aclanthology org/2022 emnlp-demos demos. 2/
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zornitsa Kozarova, and Yue Zhang (eds.); Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 30-45, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. URL
doi: 10.18653/v /ttp//22222mmmmm.
hwwpooloooyyogg//22220 org/2022. emnlp-main.3
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika. Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12216-12233 Singapore, December 2023. Association for Computational Linguisties. doi: 111538811/20233 http:///0000..mmmmmmmmmm.....m URL https://acclanthoolggg www.:/lloogooooooogooooooo 2023. emnlp-main. 751n
Asma Ghandeharioum Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. In ICML, 2024. URL https://
arzzv.org//bas/4011
Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: a unifying framework for inspecting hidden representations of language models. In Proceedings of the 41st Interna- tional Conference on Machine Learning, ICML '24. JMLR.org, 2025.
Amirata Ghorbani and James Y Zou. Neuron shapley: Discovering the responsible neurons. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5922 5932. Curran Associates, Inc., 2020. URL /tKphttps//pproeeedngg www.oeeeeedinnnnn.nnn..... http:///ea..eeeoooo///////////ee
pdf
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. Proceedings of the AAAI Conference on Artificial Intelligence, (31)))33681 3688, Jul. 2019. doi: 10. 22.1609/aaii.vaa v33101.
33013681. URL httpps:/ojs maaii.rrg/indxx phppppppppaa.. /AAI/ httaa///rrtcceeeeeee/eeee
Leilani H. Gilpin, Cecilia Testart, Nathaniel Fruchter, and Julius Adebayo. Explaining explanations to society. In NewrIPS Workshop on Ethicul, Social and Governance Issues in AI, 2019. URL https:
//arxiv. wwweeorg/aaae htrg/abs/1901.000 06560
Arthur Goemans, Marie Davidsen Buhl, Jonas Schuett, Tomek Korbak, Jessica Wang, Benjamin Hilton,
and Geoffrey Irving. Safety case template for frontier ai: A cyber inability argument, 2024. URL https:
Nicholas Goldowsky-DD Dill, Chris MacLood, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching, 2023. CRL httts:///rxxiv https://arxvv..rgg//aa http:///aaiii..oora///////////// 05969
Daniela Goltesman and Mor Geva. Estimating knowledge in large language models without generating a single token. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 3994-4019, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ve /v// w1/2224..eeemeeee 22024.cmmlp-main 232. URL http:://acllatthooooy.or///
amnlp-maii main. 232/
48

---

Published in Transactions on Machine Learning Research (09/2025)
Priya Goyal, Adriana Romero Soriano, Caner Hazirbas, Levent Sagun, and Nicolas Usunier. Fairness indi- cators for systematic, assessments of visual feature extractors. In Proceedings of the 2022 ACM Con- ference on Fairness, Accountability, and Transparency, FAceT '22, pp. 70 88, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393522. doi: 011114//3331146.3533333 URL
https://doi..rr//10 wwe.e11/////11...1 3533074
Mara Graziani, Laura O' Mahony, An-Phi Nguyen, Henning Müller, and Vincent Andrearzzyk. Uncovering unique concept vectors through latent space decomposition, 2023. URL fttps://arxiv.org/bbbb 06913
Ryan Greenblatt and Buck Shlegeris. Catching AIs red-handed. Alignment Forum, January 2024. URL
httpe///ww..liineetffrrrre http:////ws///o///////////eonanr
Ryan Greenblatt, Fabien Roger, Dmitrii Krasheninnikov, and David Krueger. Stress-testing capability clic- itation with password-locked models. In The Tiihty-eighth Annuol Conference on Neural Information Processing Systems, 2024. URL http://ppeaaaeeeeeeee.eeeerrrras
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The unrea- sonable ineffectiveness of the deeper layers, 2024. URL httpp:://rxii...rg//////////// wtt..or///abb...///sb///// 17887
Jason Gross, Rajashree Agrawal, Thomas Kwa, Fuan Ong, Chun Hei Yip, Alex Gibson, Soufiane Noubir, and Lawrence Chan. Compact proofs of model performance via mechanistic interpretability In 1CML Workshop on Mechanistic Interpretability, 2024. URL https://arxiv, orggabbs/20..1117.//
Roger Grosse. Three sketches of ASL-4 safety case components, November 2024. URL https://alignnest anthropic wtwp/o///////////e//////////eeee s024/safet:-ccssss
Roger Grosse, Juhan Bac, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilé Lukošiuté, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023. URL https://arxiv org/abs////eeooooo /208 032966
Michael M. Grynbaum and Ryan Mac. The Times sucs OpenAI and Microsoft over A.I. use of copyrightod work. The New York Times, December 2023. URL https://wwn http:///wwwwtttttt..t.. wytiintttmss...... http://////////////////sseeeee media media/new-yote opemem---pe--eee mnoorrotttt--i-iiioooooooott-tte wtwpstttwaawsitttttata.... html
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL
https://arxiv. org/abs////o.oo hb////b///// 0ttp//311.00000
Clément Guerner, Anej Svete, Tianyu Liu, Alexander Warstadt, and Ryan Cotterell. A geometric notion of causal probing, 2024. URL
wwwp:/wwwwwwwpww/w////////////// 15054
Phillip Huang Guo, Aaquib Syed, Abhay Sheshadri, Aidan Ewart, and Gintare Karolina Dziugaite. Robust unlearning via mechanistic localizations. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. URL
Abhiject Gupta, Gemma Boleda, Marco Baroni, and Sebastian Padó. Distributional vectors encode ref- erential attributes. In Lluis Mårquez, Chris Callison-Buree and Jian Su (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 12-21, Lisbon, Portu- gal, September 2015. Association for Computational Linguistics. doi: 10.18653/ 10.18653/v1/D15-11 w0/0://1v/111100/0/ URL http://cclannooooog.orrr////ss
Rohan Gupta, Iván Arcuschin, Thomas Kwa, and Adrià Garriga-Alonso, Interpbench: Semi-synthetic trans- formers for evaluating mechanistic interpretability techniques. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL httpps://openreviex... net/
Tormm?id=
Wes Gurnce and Max Tegmark. Language models represent space and time. In The Twelfth International Conference on Learning Represendations, 2024. URL htttp::/pppaarevia..aat//omomm/d E8xbmvFiin
49

---

Published in Transactions on Machine Learning Research (09/2025)
Wes Gurnce, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
Kaarel Haani, Rio Popper, and Jake Mendel. A starting point for making sense of task structure (in machine learning). Less Wrong, February 2024. URL htttps:/wwww.eesvvrrs
of
Frank R. Hampel. The influence curve and its role in robust estimation. Journal of the Americon Statistical Association, 69(346):383 393, 1974. doi: /0.1080/01621455.. 01w00010211199994.11111010..1010 1044..042.......
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS11, pp. 1145- Cambridge, MA, USA, 2015. MIT Press.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compule greater-than?: Interpreting mathematical abilities in pre-trained language model. In Thirty-seventh Conference Neural Informa- tion Processing Systems, 2023. URL https::/ wttp:oormm..dddPPooaa.....
Kaarel Hänni, Jake Mendel, Dmitry Vaintrob, and Lawrence Chan. Mathematical models of computation in superposition. In ICML 2024 Workshop Mechanistic Interpretability, 2024. URL httpe://openreevie: net:foorum??d::ccJPPP...
Zellig S. Harris. Distributional structure. WORD, 10(2-)):146 162, 1954. doi: 10.10800/00433956.1994..
11659520. URL https://doi.org// http:///////11110000000333330 1080/00437956 1954. 11659520
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandcharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. Tn Thirty-seventh Conference Neural Information Processing Systems, 2023. URL
EldbU1Ztbd]
Trevor Hastic and Robert Tibshirani. Generalized Additiee Models. Statistical Science, 1(3) - 3310, 1986. doi: 10.1214/ss/ //14////1/17011041111
http::://ol..rrg/00...////////
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pallern Recognition (CVPR), pp. 770 778, 2016. doi: 10.1109/CVPP /CV...00....00
Xu Owen He. Mixture of a million experts, 2024. URL
wwtp:///ggbbb./..ooooo 04153
Anna Hedström, Leander Weber, Daniel Krakowezyk, Dilyara Bareeva, Franz Motzkus, Wojcicch Samek, Sebastian Lapuschkin, and Marina M.-C. Höhne. Quantus: An explainable toolkit for responsible evaluation of neural network explanations and beyond. Journal of Machine Learning Research, 24(34):
1-11, 2023. URL http://jmlr. orggpapprssv244/220000 0142. html
Stefan Heimersheim.
You can remove gpt2's layernorm by fine-tuning, 2024. URL https://arxiv.org
0tt//2099.13100
Stefan Heimersheim and Jett Janiak. A circuit for Python docstrings in a 4-layer attention-only transformer. Alignment Forum, February 2023. URL http:///wwwwaligneetffrruu. org post/psts/
u6KXXakFoxfWeoAkm/
Tom Henighan. Caloric and the utility of incorrect theories. Transformer Circuits, 2024. URL https: //transformer--iirette.. putp2202//pp111-ppppte///ieeee http:////neee.ataaaaaallloeeeeee
Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversar- ial model manipulation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://prooeeddnngs http.appppppp.cppppp..eeeeee.ee. 7
50

---

Published in Transactions on Machine Learning Research (09/2025)
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep features. In International Conference on Learning Represenlations, 2022. URL
John Hewitt and Percy Tiang. Designing and interpreting probes with control tasks. In FMNLP, 2019. URL
https://arxiv: org/abs/1909 https://aaravvorrr/bbb///0 /000b/9
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Compatational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4129-4138 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653 1115555/11/NIS/ v1/119-11/ URL
John Hewitt, John Thickstun, Christopher Manning, and Percy Liang. Backpack language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9103-90 (928, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: (0.18653/v1 /12033//11222cclll//// acl URL org/ acl-long.506
Steven A. Hicks, Jonas L. Isaksen, Vajira Thambawita, et al. Explaining deep neural networks for knowledge discovery in electrocardiogram analysis. Scientific Reports, 11:10949, 2021. doi: 10.1038/ $#155660021-902855.. URL
Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, and Chris Olah. Understanding rl vision. Distill, 2020. doi: 10.23915, 23315/distill.000000 /00029.. https://distill.pbbb/ 20200 www//e/sttaaiinng-liiiiiooo https:///sssllllaaa///mmm/mmmmss
Geoffrey Hinton. Shape representation in parallel systems. Proceedings of Leh Seventh International Joint Conference on Artificial Intelligence, 1981. URL
absps/
shape81. pdf
Geoffrey E. Hinton, Simon Osindero, and Yee whye Teh. A Fast Learning Algorithm for Deep Belief Nets. Newral Computation, 18(7):1527-1554, 07 2006. ISSN 0899-7667. doi: (t111////ccc.220.188...... URL
https://doi..o
1162/ 2006 18.7 7. 1527
Oskar John Hollinsworth, Curt Tigges, Atticus Geiger, and Neel Nanda. Language models lincarly represent sentiment. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackbozNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 58 87, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10.18653 /v000533///// wla.kboolackbbolppp... 1.5. URL ptttp:://aclaathoooog..og// http://ooo0..0...aaaaaooola/pl1p
Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, and Mor Geva. Intrinsic evaluation of unlearning using parametric knowledge traces, 2024. URL
Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet. The developmental landscape of in-context learning, 2024. URL https://arrxv..rrg/ httrgg/bs/2442...0mm 02364
Giles Hooker, Lucas Mentch, and Siyu Zhou. Unrestricted permutation forces extrapolation: Variable importance requires at least one more model, or there no free variable importance, 2021. URL https:
//arxiv.org/abb/11000 03151
Xiyang Hu, Cynthia Rudin, and Margo I. Seltzer. Optimal sparse decision trees. In Newral Information Processing Systems, 2019. URL
org. arg/CorpusIDD: 13910464999999
Jing Huang, Atticus Geiger, Karel D'Oosterlinck, Zhengxuan Wu, and Christopher Potts. Rigorously assessing natural language explanations of neurons. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi (eds.), Proceedings of the 6th BlackborNLP Work- shop: Analyzing and Interpreting Neural Networks for NLP, pp. 317-331, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1. /220222./. blackboxnlp-122 1.24. seselaanthology.or/22223 obapaaa.og....r/0000....eeeooaa. blackboznlp--1.22
URL https:
51

---

Published in Transactions on Machine Learning Research (09/2025)
Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. RAVEL: Evaluat- ing interpretability methods on disentangling language model representations. In Lun-Wei Ku, An- dre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8669-8687, Bangkok, Thailand, August 2024a. Association for Computaational Linguistics. doi: 10. 10.18653/v 12122/22240220 aal-longg440) URL https: : //aclanthology.oo orgg acl-long.-onn.......... 470/
Xinting Huang, Madhur Panwar, Navin Goyal, and Michael Hahn. Inversionview: A general-purpose method for reading information from neural activations. In ICML Interpretabilityy Workshop, 2024b. URL https:
7/arxiv http:/abb2245...5555
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference Learning Represenlations, 2024. URL https:///pennreview http:/ppaaeeeeeeeeeeeeoorrrraaaa
Evan
Hubinger. rum, March 2023.
Towards understaanding-besed
safety evaluations.
Alignment Fo-
URL
towards- ungersainddrs baweorssanddng--bdddd-aaaaaa based-ssaeety---vllaaiii
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems, 2021. URL https://arxii: httpps///ariivooor//////00.....1 2ttg///as/190..11200
Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, and Ethan Perez. Sleeper agents: Training deceptive llms that persist through safety training, 2024. URL wrw.orggg//b..oo.ooooeoooo.o ab/////aa/.2....////////////1/1/ 2401. 005666
Dieuwke Hupkes and Willem Zuidema. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hicrarchical structure (extended abstract). In Proceedings of the Twenty- Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pp. 5617 5621. International Joint Conferences on Artificial Intelligence Organization, 2018. doi: 10.24963 /ij////ijcci... /tt.:///i.aa.i////////.1/ 276./796. URL https://doi.org: orggrogg//11 10. 24963/1jcai. 2018/796
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31, Cur- ran Associates, Inc., 2018. URL https::/pproeedding www.oeeeiinnnn.uuuss........ wwwrrps.cc/papppeeeeee fa2bbafff44402236666666111776663
pdf.
Alon Jacovi. Trends in explainable ai (xai) literature, 2023. URL
Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Tim Rocktäschel, Edward Grefenstette, and David Krueger. Mechanistically analyzing the effects of fine-tuning proce- durally defined lasks. In The Twelfth International Conference on Leurning Representations, 2024. URL (tte//orrmmm11-AAKKKKKK11
https://openrews
Sarthak Jain and Byron C. Wallace. Attention is not Explanation. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapler of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), wwweeiiinoooddiigtttttta-------e
pp. 35446,356, Minneapolis,
title = 'Backward Lens: Pro- jecting Language Model Gradients into the Vocabulary Space', author 'Katz, Shahar and Belinkov, Yonatan and Geva, Mor and Wolf, Lior", editor = "Al-Onaizan, Yaser and Bansal, Mohit and Chen,
52

---

Published in Transactions on Machine Learning Research (09/2025)
Yun-Nung', booktitle 'Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing', month = nov, year "2024', address "Miami, Florida, USA', publisher = "As- sociation for Computational Linguisties', wrl = 'https:/ /atp///atooogg.orgg////o htwp:/ogg..rrg/22...mmmm (1118658/1122224...
142/", doi = pages = 22890-2422) abstract = "Understanding how Transformer- based Language Models (LMs) learn and recall information a key goal of the deep learning community. Recent interpretability methods project weights and hidden states obtained from the forward pass to the models' vocabularies, helping to uncover how information flows within LMs. In this work, we ex- tend this methodology to LMs' backward pass and gradients. We first prove that a gradient matrix can be cast as a. low-rank linear combination of its forward and backward passes' inputs. We then develop methods to project these gradients into vocabulary items and explore the mechanics of how new infor- mation is stored in the LMs' neurons. sola, June 2019. Association for Computational Linguisties. doi: 10.18653/v1/Ne 1357. URL httpp://aclaanthoooog... http///llaaaatolooooooo/////---0 1357/
Jelt Janiak, Chris Mathwin, and Stefan Heimersheim. Polysemantic attention head in a 4-layer transformer. LessWrong Blog, 2023. URL L.l.samattccattttttoooooaaaaaaaa transffoorrr
Joseph D. Janizek, Pascal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks, 2020. URL
04138
Adamn Jermyn, Chris Olah, and Tom Henighan. Atlention head superposition.
Trans- /ineee/i/iee..eeee html#
former Circuits, 2023. URL
attention--uperpostttie
Zhuoran Jin, Pengfei Cao, Hongbang Yuan, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Associalion for Compulational Linguisties ACL 2024, pp. 1193 1215,, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL https: https://aclaathoology org/ 2024. www..f.innnnnsaac....
David Johnston, Arkajyoti Chakraborty, and Nora Belrose, Mechanistic anomaly detection research update, Aug 2024. URL htttlh/tt::/bbbbt eleuther hewearrcerereeeeeeeeeeeee hutptce/epppde////eee
Caden Juang, Gonccedilalo Paulo, Jacob Drori, and Nora Belrose. Open source automated interpretability for sparse autoencoder features, Jul 2024. URL
eleuther aiapettteer.a//uttoneeee.r
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakol,, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Andrew J Ballard, Andrew Cowie, Bernardino Romera-Paredess, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W Semior, Koray Kavukeuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, 596:583-555 -89,, 2021. doi: 10.1038 00/5//1111---0100811999 URL https://doi..org 10. 11t.00000000001111000000--000000
Subhash Kantammeni, Josh Engels, Senthooran Rajamanoharan, and Ncol Nanda. SAE probing: What is il good for? absolutely something! Al Aligument Forum, nov 2024. URL httpp:///ww// www.wwwesswoonggom com/ sobod---oorbbbbouuuulsassttttttt
Doots//WilllyyyTeeedd////ssaappp whaperrroooooottt--------------- int---is-----ood--ooo gord-
Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf. Backward lens: Projecting language model
gradients into the vocabulary space. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 2390 2422, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/vv ha/////vv///// /v21//21122222 emnly-main. 142. URL
emnlp-mainnnn....e 142/
Dmitrii Kharlapenko, neverix, Neel Nanda, and Arthur Commy. Self-xxplanning SAE features. Align- ment Forum, August 2024. URL http:///www...pooooooooooooooots
self-explaining- wwwednnn-sse-fftturee
53

---

Published in Transactions on Machine Learning Research (09/2025)
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2668-2677 PMLR, 10-15 Jul 2018a. html
URL httpps///proceedings wwr.ggg..m...... wwt.e/rrressv/eiiieeeeeee
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2668 2677. PMLR, 10-15 Jul 2018b. URL
mlr Batss//vv0//immmdd.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (Un)reliabillis of Saliency Methods, pp. 267-280. Springer Inter- national Publishing, Cham, 2019. ISBN 978-3-030-280544 doi: (0.1007/998---0000-88000 28954-66- 14. URL
Nathalic Maria Kirch, Severin Field, and Stephen Casper. What features in prompts jailbreak 1lms? inves- ligating the mechanisms behind allacks, 2024. URL htttps://arxiv.org/aaeet
03343
Louis Kirsch, Julius Kunze, and David Barber. Modular networks: learning to decompose neural compu- tation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, pp. 2414-2423, Red Hook, NY, USA, 2018. Curran Associates Inc.
Connor Kissane, Arthur Conmy, and Neel Nanda. Attention output saes improve circuit analysis ai alignment forum, Jun 2024a. URL https://ww. wwwp://wwwwama..anii..a. wwtp:///.wo/oo////////////////// attention-output-saan satptttssss-nnooeeeeeeeeeeeeee-e
Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Commy, and Neel Nanda. Interpreting atlention layer outpuls with sparse autoencoders, 2024b. URL http://aawxw...oaab//////// 17759
Connor Kissane, Robert Krzyzanowski, Neel Nanda, and Arthur Conmy. SAEs are highly dataset dependentt a case study on the refusal direction. Alignment Forum, nov 2024c. URL http:::::www.wwlggmnn...eee potpostt/rrponn722upppopop odd aa/oo////saaeeeeee waaes-are-hiee http://raattatteeeeeeeeee---aaae
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 94th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1885-1894 PMLR, 06-11 Aug 2017. URL https://proceeddings mww.iinnnmm..r....mmmmmm.m wwt.:peee/////o////aaaaam
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5338 5348. PMLR, 13 18 Jul 2020. URL https::/ppooceedingg mlr. wttppp/eeeevev/eeeeeeea htmm
Arne Köhn. What's in an embedding? analyzing word embeddings through multilingual evaluation. In Lluís Mårquez, Chris Callison-Burch, and Jian Su (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2067-2073, Lisbon, Portugal, September 2015. Association http://hooooooooooooo-e-ee org/D15-1246
for Computational Linguistics. doi: 10. N0.18633vv1/D15 1146.. 1246. URL
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9781713871088.
Edward Korot, Nikolas Pontikos, Xiaoxuan Liu, et al. Predicting sex from retinal fundus photographs using automated deep learning. Scientific Reports, 11:10286, 2021. doi: 10.1038/s41598 021-89743-x URL
54

---

Published in Transactions on Machine Learning Research (09/2025)
Eliza Kosoy, Emily Rose Reagan, Leslie Lai, Alison Gopnik, and Danielle Krettek Cobb. Comparing machines and children: Using developmental psychology experiments to assess the strengths and weaknesses of lamda responses, 2023. URL
11243
János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. Atp*: An efficient and scalable method for localizing Ilm behaviour to components, 2024, URL https://arxivvorg//aae wttp://.wrrg//bb///0000044
Mukund Krishnan. Against interpretability: a critical examination of the interpretabiliity problem in machine learning. Phioosphy & Technology, 33(3)4887-502, 2020. doi: 10.0007/1133-FF-01400037299 URL https:
//doi.org//i 10101100073333 000.0000000033337000--007732222
Alex Krizhevsky, Tly Sutskever, and Geoffrey E. Hinton. Imagenct classification with deep convolutional neural networks. In NewrIPS, pp. 84 90, may 2012. doi: 10.1 145/3065356.
Rudolf Laine, Bilal Chughtai, Jan Betley, Kaivalya Hariharan, Mikita Balesni, Jérémy Scheurer, Marius Hobbhahn, Alexander Meinke, and Owain Evans. Me, myself, and AI: The situational awareness dataset (SAD) for LLMs. In The Thirty-eight Conference on Newral Information Processing Systems Datasets and Benchmarks Track, 2024. URL httpps://openrevier wttpriiwwweeeeeeeeeeeeeeeeeeeeee http///rrrnnnnnnun/nnrrrreee
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Demison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kermion, Kamilé Lukošiutē, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schicfer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez. Measuring faithfulness in chain-offthought reasoning, 2023. URL h org/abs/2307, 13702.
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature Communicationn,, 10(1):1096, Mar 2019. ISSN 2041-1723. doi: 10. 100111/////14477---------------1 7/doi. 10...oo//1.....000..
URL https:
Quoc V. Le, Marc Aurelio Ranzato, Rajal Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jell Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML'12, pp. 507 514, Madison, WI, USA, 2012. Omnipress. ISBN 978145031281..
Matthew L. Leavitt and Ari Morcos. Towards falsifiable interpretability rescarch, 2020. URL https:
12016
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning, May 2015. URL https://www.nst nature. com/ http/aaricoo//auueee55555
Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: a case study on dpo and toxicity. In Proceedings of the 41t International Conference on Machine Learning, TCMT'24. J.r..oorg 2025.
Katherine Lee, A. Feder Cooper, and James Grimmelmann. Talkin' 'bout ai generation: Copyright and the generative-ai supply chain. Journal of the Copyright Sociely of the USA, 2024. URL https://arxiv. org/abs/2309.00 08133
Gavin Leech, Juan J. Vazquez, Misha Yagudin, Niclas Kupper, and Laurence Aitchison. Questionable practices in machine learning, 2024. URL https:///axxv..orr abs////7. 12222222772222
Simon Lermen, Charlie Rogers-Smith, Smith, and Jeffrey Ladish Lora inn-tuningg efficiently undoes safety training in llama 2-chat 70b, 2024. URL
20624
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on synthotic task. In The Eleventh International Conference on Learning Representations, 2023. URL /tt.:/ts///peaeevv....e wtt.://www.e.oooononnom DeG07 Tc2vT
55

---

Published in Transactions on Machine Learning Research (09/2025)
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. In NewrIPS, 2024a. URL https:
//arxiv. wwwxiii.orgaaeee http/abb/2200..033
Nathaniel Lá, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann- Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burgee Rassin Lababidi, Lennarl Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Adam Khoja, Zhenqi Zhao, Ariel Herbert- Voss, Cort B. Breuer, Samuel Marks, Oam Patel, Andy Zou, Mantas Mazcika, Zifan Wang, Palash Oswal, Weiran Liu, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Russell Kaplan, Ian Steneker, David Campbell, Brad Jokubaitis, Alex Levinson, Jean Wang, William Qian, Kallol Krishna Karmakar, Steven Basart, Stephen Fitz, Mindy Levine, Ponnurangam Kumaraguru, Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The wmdp benchmark: Measuring and reducing malicious use with unlearning, 2024b.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Dmitry Storcheus, Afshin Rostamizadeh, and Sanjiv Kumar (cds.), Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges al NIPS 2015, volume 44 of Proceedlings of Machine Learning Reseurch, pp. 196 212, Montreal, Canada, 11 Dec 2015. PMI.R URL htttps://proceeddngg mlr. wwwss///4/eiieeaeee #rress/v441ii5Sooeerggntt html
Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chin- chilla, 2023. URL htttps://arxiiiii htttp:::/aaxxiv.rg/abba///////// 094588
Tom Lieberum, Senthooran Rajamanoharan, Arthur Commy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024. URL http:://tss/stt
wttpab/../2200// 05147.
Timothy P. Lillicrap and Konrad P. Kording, What does it mean to understand a neural network?, 2019. URL https://arxiv, htt//bbs/1900 063774
Yuping Lin, Pengfei He, Han Xu, Yue Xing, Makoto Yamada, Hui Liu, and Jiliang Tang. Towards under- standing jailbreak attacks in LLMs: A reprosentation space analysis. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 7067-7085, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10. 10.18653/ v1 www.eemaillmmeeeeeeeemm.... URL httph:ttt www.aanntooooooooeeoooooo http:/222222m2mm22mm/mm manlp-main... 401/
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Alexander Rives, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123 112311100,,,, 2023. doi: 10.1126 siienc..dee..
David Lindner, Janos Kramar, Sebastian Farquhar, Matthew Rahtz, Thomas McGrath, and Vladimir Miku- lik. Tracr: Compiled transformers as a laboratory for interpretability. In Thirly-seventh Conference Newral Information Processing Systems, 2023. URL https::/openrevies htt.eeennee/eoooriiiittttbbbaaae
Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah, Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits, 2024. URL https:
//transformer-ciiccutee http://cwccccapp//////////////ae html Equal contribution.
Zachary C. Lipton. The mythos of model interpecaabbbii:yyy: In machine learning, the concopt of interprotability is both important and slippery. Queue, 16(3):31-55) June 2018, ISSN 1542-7730. doi: 111114/3236336. 3241340. URL httpp:://ao..org//// wtt..////.001113333333333333333 100.11115/3366855 3241340
Zachary C. Lipton and Jacob Steinhardt, Troubling trends in machine learning scholarship: Some ml papers suffer from flaws that could mislcad the public and stymic future rescarch. Queue, 17(1):45-77 77, Febru- ary 2019. ISSN 1542-7730. doi: 11114//3317787.3255553 URL
1145333171.44
3328534
56

---

Published in Transactions on Machine Learning Research (09/2025)
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalic Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. Rethinking machine unlearning for large language models, 2024a. URL https://arxiv. www.:/rrg/abbaaaaeee 08787
Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Machine unlearning in generative ai: A survey, 2024b. URL http::////axii..oaaaaa/////s//// 20516
Ziming Liu, Eric Gan, and Max Tegmark. S is belioving: Brain-inspirod modular training for mechanistic interpretability, 2023. URL httppp///aaxxvvor/abbba///////// 087446
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačió, Thomas Y. Hou, and Max Tegmark. Kan: Kolmogorov-arnold networks, 2024c. URL https://arxiv.. wrw. /bs// 2404.19755
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through regu- larization. In International Conference Learning Representations, 2018. URL https://openreevis.
Jens Ludwig and Sendhil Mullainathan. Machine learning as a tool for hypothesis generation. Working Paper 31017, National Bureau of Economic Research, March 2023. URL
nber. ptbbrr.org/papprr//3100077
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Newral Information Processing Systems, NIPS117, pp. 4768 4777, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menel. Eight methods to evaluate robust unlearning in llms, 2024. URL https://arxiv. hwtgg/abs/2400
wwwp://///.bbbb......oog 2682.
Andreas Madsen, Himabindu Lakkaraju, Siva Reddy, and Sarath Chandar. Interprotability needs a new paradigm, 2024. URL http:::///aaiivoor///a////////// 05386
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them, 2014. URL
1412.0035
Aleksandar Makelov, Georg Lange, and Neel Nanda. Is this the subspace you are looking for? an inter- pretability illusion for subspace activation patching. In NewrIPS Workshop on Attributing Model Behavior at Scale, 2023. URL httpp:s://arri...orraaaaa//////a
Aleksandar Makelov, Georg Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024. URL https://opeerevieee wtt.:///e/ooooo/ddd/tooooo.
Alireza Makhzani and Brendan Frey. k-sparse autoencoders, 2013. URL http:://arx.v.ong/bbb///// 5663
Valerio Mante, David Sussillo, Krishna V. Shenoy, and William T. Newsome, Context-dependent computa- tion by recurrent dynamics in prefrontal cortex. Nature, 503:78-88 84, 2013. doi: 12..0e/2aaau URL
https: http://.....g.....0//////eel////
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/ false datasets. In First Conference on Language Modeling, 2024. URL
Samuel Marks, Can Rager, Eric Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models, 2024. URL
57

---

Published in Transactions on Machine Learning Research (09/2025)
Chris Mathwin, Dennis Akar, and Lee Sharkey. Gated attention blocks: Preliminary progress toward re- moving attention head superposition. Less Wrong, apr 2024. URL wttp:/:///wwww.sssrrooggggg wwwpe/rooooooooeeooooomeoeeeeeee cttps///wwooooccooooos/s//////// posts/ otgpa///wwooootoooo.....oo.ooooo
Callum Stuart McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppres- sion: Comprchensively understanding motif in language model attention heads. In The 7th BlackborNLP Workshop, 2024. URL ltttps://openrevieer http://ww...eeommmmmdddddsdedd..
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašov, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings doi: 10.1073 ppo.pp20.222111133 URL
of the National Academy of Sciences, 119(47))c22006625111 2022.
attps://www... pnas.
2ttp://p22aa222221111122 2206625119
Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shanc Legg. The hydra effect: Emergent self-repair in language model computations, 2023. URL https://arxxi.cco wtwp:/wwwwwrg/b///0///////// 15771
Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. Frontier models are capable of in-context scheming, 2024. URL https://arxiv owg.ooggg/a.o.oooo.o abs////////// 24022 04984
Jake Mendel. SAE feature geometry is outside the superposition hypothesis. ment Forum, 2024.
Align-
URL. https: //www. awwgnmentfooonmmmmmmmmm..m wwwooorrr.ororooooeeeeeee org/ wttg/ootss//FBB.b22ff333 3 Wmzz6/ sae- Seature-geomettin 18 outside- the-thheeeeeeee superposittion--rpotheeee
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022a. arXiv:2202.05262.
Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT, In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advonces in Neural Information Processing Systems, 2022b. URL https://oppenrevier http://paarreeeeeeeeeeerrrrrrrde htth:/forunni==-116WW53 h6WAS6EEE
Eric J Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. In Shirty-seventh Conference on Newral Information Processing Systems, 2023. URL https://openrevies met/fforun?ddssttTWWaaa.
Eric J. Michaud, Isaac Liao, Vedang Lad, Ziming Liu, Anish Mudide, Chloc Loughridge, Zifan Carl Guo, Tara Rezaei Kheirkhah, Mateja Vukelić, and Max Tegmark. Opening the ai black box: program synthesis via mechanistic interpretability, 2024. URL https:// httrs:://axxiv../r/bbb/////// 22wgg/abs/442...o 05110
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word repre- sentations. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff (eds.), Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pp. 1516-751,44 Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL htttps://aclaathholoog. wwt.:ey-org///1//1/011
Joseph Miller, Bilal Chughtai, and William Saunders. Transformer circuit evaluation metrics are not robust. In First Conference on Language Modeling, 2024. URL
Tim Miller, Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267:1-38, 2019. ISSN 0004-3702. doi: https://toi.org//0.11/////// a0t. URL http::htp/////t sciencedirect.cmm http://...........cc...../aaa//a
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications,, 9:2383, 2018. doi: 10.0088/441667-005444463333 URL httpe://bas.orr//00000////111110
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruming convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, 2017.
URL http://///.ppppeeeeeeeeeee///tta
58

---

Published in Transactions on Machine Learning Research (09/2025)
Christoph Molnar, Gunnar König, Julia Herbinger, Timo Freiesleben, Susanne Dandl, Christian A. Schol- beck, Giuseppe Casalicchio, Moritz Grosse Grose-Wentrupp Wentrup, and Bernd Bischl. General pitfalls of model-agnostic interpretation methods for machine learning models, 2021. URL
04131
Christoph Molnar, Gunnar König, Bernd Bischl, and Giuseppe Casalicchio. Model-agnostic feature impor- tance and effects with dependent features: a conditional subgroup approach. Dala Mining and Knowledge Discovery, 38:2903 2941, 2024. doi: 610618-022-00901-9
URL https://doi http///oi..or/1111100010
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65: 211- 222, May 2017. ISSN 0031-3203. doi: 10.1016/j.patcoog 2016.11.100......111111111111110 URL http: htt./xxxxooocco /ttp:////.x.ii.or/////... org/10.
1016/o. 2016. 11.
Mordvintsev. Inceptionism: Going deeper into neural networks, 2015. URL https://research.gonne http://cpppoooiiimnonnnnnn beennngddeee-iing nanto-neural-netwoon
Marius Mosbach, Vagrant Gantam, Tomás Borgara-Browns Dictrich Klakow, and Mor Geva. From insights actions: The impact of interpretability and analysis research on nlp, 2024. URL https://arxiv.org org/
abs/2406. 12618
Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In D. Touretzky (ed.), Advances in Neural Information Processing Systems, vol- une 1. Morgan-Kauffnann, 1988. URL https::/ htwp:/rrreeeiinn....errrr.ppppee ccppppp http:///pppppeeeeeeppeeeeeeeee files/paper/1988/ 710000740007cccea310002227777773
.dff
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 17153 17163. Curran Associates, Inc., 2020. URL http::t/ppcccceediiinneeo// neurips ct/ cttpaper_aaiiiieeeeeeee 2ppp:/22000111101666661166666662
pdf
Aaron Mueller. Missed causes and ambiguous effects: Counterfactuals pose challenges for interpreting neural networks, 2024. URL https://arxiv. httgg/bb/2400.... 04690
Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability, 2024. URL https://arxiv.. http::/://aaaav.o//aa///////////
Maximilian Muschalik, Fabian Fumagalli, Paolo Frazzetto, Janine Strotherm, Luca Hermes, Alessandro Sperduti, Eyke Hüllermeier, and Barbara Hammer. Exact computation of any-order shapley interactions for graph neural networks, 2025. URL
16944
Neel Nanda. Attribution patching: Activation patching at industrial scale, Mar 2023a. URL https: /wwww oee.lmmcc...oeeetammmtttteoeeeea
Neel Nanda. Othello-GPT: March 2023b.
Reflections on the research process. Alignment Forum, http://w...ppt.///aa/aaaaaaaaaaa
URL https://www.llggmm/tt othello-gpt pept-reflections--o on- tehe-reeearchh-roceee
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference Learning Repre- sentations, 2023a. URL https://oppereviiee het:/forum2dd-9XSSDDDr
Neel Nanda, Senthooran Rajamanoharan, J'anos Kram'an ar, and Rohin Shah. Fact finding: At- tempting to reverse-engincer factual recall on the neuron level (Post 1). Alignment Fo-
rum, December 2023b. URL
59

---

http:://ddiiilllppp/////////// Published in Transactions on Machine Learning Research (09/2025)
Arunachalam Narayanaswany, Subhashini Venugopalan, Dale R. Webster, Lily Peng, Greg S. Corrado, Paisan Ruanviboonsuk, Pinal Bavishi, Michael Bremner, Philip C. Nelson, and Avinash V. Varadarajan. Scientific discovery by generating counterfactuals using image translation. In Anne L. Martel, Purang Abolmacsumi, Danail Stoyanov, Diana Mateus, Maria A. Zuluaga, S. Kevin Zhou, Daniel Racoccamu, and Leo Joskowicz (eds.), Medical Image Compuling and Compuler Assisled Intervention MICCAT 2020, pp- 273,283, Cham, 2020. Springer International Publishing. ISBN 978-3-030-59710-8.
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, 16, pp. 2016a. Curran Associates Inc. ISBN 9781510838819.
Red Hook, NY, USA,
Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks, 2016b. URL https://arxiv. org/ /bb/////////
1602.0380 03616
Anh Nguyen, Jason Yosinski, and Jeff Clune, Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks, 2016c. UR..https://aatt htttps://axxi.. org/ abs/
1602.03613
Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug and play generative networks: Conditional iterative generation of images in latent space, 2017. URL https://arxiv..or org/
Abs/1612.00005
Nostalgebraist.
Interpreting gpt: The logit lens ai alignment forum, Aug 2020. URL https://www http:///e..aa..t..a/////a/////an
-ens
Joe O'Brien, Shaun Ec, and Zoe Williams. Deployment corrections: An incident response framework for frontier ai models, 2023a. URL htttps://axxivvorg//bb/////////
Thomas O'Brien, James Stremmel, Lola Pio-Lopez, Paul McMillen, Cody Rasmussen-lvey, and Michael Levin. Machine learning for hypothesis generation in biology and medicine: Exploring the latent space of neuroscience and developmental bioclectricity. Preprint, September 2023b. URL attpps://ol org/10.
31219/osf 1o/269
Chris Olah. Interpretability dreams. Transformer Circuils, May 2023.
URL https://
Lransformer-ciraute http:///ww.pppppooo////////////a html An informal note on future
goals for mechanistic interpretability.
Chris Olah, 2024. URL
html
Chris Olah and Adam Jermyn. What is a linear representation? what is multidmmennsionnnnll fcature? Trans- former Cireuils, 2024. URL http://:/.aaaaaaeaceerc..r/ei//a html#
linear-repressnatiinns
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11), Nov 2017a. doi: 1223115disstill0000
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017b. doi: http://silll.lll///////eeeeeeeee
10011/diisill.00000
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexan-
der Mordvintsev. The building blocks of interpretability. Distill, 2018.
doi: 10.23915 10.23915/distill.oom distill.0000
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
An overview of carly vision in inceptionvl. Distill, 2020a. doi: 10.23915 00.23915/distill.oo (distill.00024..00 http://iilllluuu/////2//rrriii/
60

---

Published in Transactions on Machine Learning Research (09/2025)
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An intruductoon to circuits. Distill, 2020b. doi: 11.23915/disiil.00000 https://disttill.pubb/ http:lppu//2///orriiiii
James Oldfield, Christos Thelepis, Yannis Panagakis, Mihalis A. Nicolaou, and Toannis Patras. Panda: Unsupervised learning of parts and appearances in the feature maps of gans, 2023. URL https://arxiv. org/
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022. URL https://arxiitii http:///axvvvv.vv www./rgg/////aeeeeee//oeeeeeee/e www.b//.......... 2219/9 11895
OpenAL. OpenAI preparedness framevcor Framework, OpenAT, 2023. URL https://cdn..opent /om/ openai prapppprrreeesss-trrsee---------
Beta version.
OpenAT, Josh Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapire Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Audrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Docarcaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Fleti, Tyna Floundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha CootisooLopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hosse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanic Lin, Matcusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adamn Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddic Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Ben- jamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworck, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Cheleea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner,
61

---

Published in Transactions on Machine Learning Research (09/2025)
Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL htttps://arxae
Core Francisco Park, Maya Okawa, Andrew Lee, Ekdeep Singh Lubana, and Hidenori Tanaka. Emergence of hidden capabilities: Exploring learning dynamics in concept space. In The Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2024a. URL https:://openrevvee..ntttt ооиEcт6втI
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Cansal Representation Learning Workshop al NeurIPS 2023, 2023a. URL https://openreviiee net/ peee/foorummtc--Poo..g.ee.
Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hicrarchical concepts in large language models. In ICML 2024 Workshop on Mechanistic Interprelabilily, 2024b. URL
Peter S. Park, Simon Goldstein, Aidan O'Gara, Michael Chen, and Dan Hendrycks. Ai deception: A survey of examples, risks, and potential solutions, 2023b. URL httppp///aaxxi..og//bb////////// 14752
Gonçalo Paulo, Thomas Marshall, and Nora Belrose. Does transformer interpretability transfer to rnns?, 2024. URL httpp://arxxi.. org/ http:/bb/2440..000 05971
Michael T. Pearce, Thomas Dooms, and Alice Rigg. Weight-based decomposition: A case for bilinear mlps, 2024. URL http:::///xxxiiorr/bbbbbe 039447
Judea Pearl. Causality. Cambridge University Press, 2 edition, 2009.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derezyuski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanislaw Wožniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14048-14077 14077, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/vbl htt////..innnnnss--ee.. wwwpanthooogyoorg///00 httg//22...ndinne......... Sandingg--ennp..95 a9nlp.9367/
URL https:
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Trving. Red teaming language models with language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empiri- cal Melhods in Natural Language Processing, pp. 3419 3448, Abu Dhabi, United Arab Emirates, De- cember 2022. Association for Computational Linguistics. doi: 10.18653 20/553/11/2/ ewnlp-manaiiii main 225. URL org/2022.22222 emnlp-man main. 225/
Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Fampérical Methods in Natural Language Processing, pp. 1499 1509, Brussels, Belgium, October- 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/18-- 1179. URL httpp://aclaatthologg wtt.://twtoooooooo////---999999
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. information-theoreee probing for linguistic structure. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (ods.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4609-4622 Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ v1 acl2020.all-main. 420. URL sttps://acllathoolgg..oogg
acl-main. 420
62

---

Published in Transactions on Machine Learning Research (09/2025)
Nicholas Pochinkov and Nandi Schoots. Dissecting language models: Machine unlearning via selective pruning, 2024. URL https://arxiv www.:/orgggggo. org/abs/2400 0ttp//2233..220
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In Proceedings of the 2024 International Conference on Learning Representations, 2024, arXiv:2402.14811
Danish Pruthi, Mansi Gupla, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. Learning to deceive with attention- based explanations. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.); Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4782 4793, Online, July 2020. Association for Computational Linguistics. doi: 10. 00011633/11/2020.a--mmaa https://aclantee .ww.loog..oogggggoeee org 20200 acl main. main.432/
URL
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Πlya Sutskever. Language models are unsupervised multitask learners, 2018. URL https://cdd
language_modee are umsuperv.seedmmliiaa.....e.
muluimultaat m_liiiaat___eereerrrresss learners. pdd
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea. Finn. Direct preference optimization: Your language model is secretly a reward model. In Thärty-seventh Con- ference on Neural Information Processing Systems, 2023. URL https://openrevier: http:ppaareeieeeeeeeortrrriaaiaa net/forum?id
HPuSIXJaa9
Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mechanistic interpretability for transformer-base based language models, 2024.
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, Janos Kramar, Rohin Shah, and Neel Nanda. Improving sparse decomposition of language model activations with gated sparse autoencoders. In The Thirty- Annual Conference on Neural Information Processing Systems, 2024. URL htttp::/ wwwpeeeewwweeeeeefffeeeeeeeeeeee wwtp//orumm.dd/1l111.......
David Raposo, Matthew T. Kaufman, and Anne K. Churchland. A category-free neural population supports evolving demands during decision-making. Nature Neuroscience, 17:1784 1792, 2014. doi: 10.1038/mn 3865. URL bittps://doi: org/ / 10w.0.138//..... 3865
Tilman Rauker, Anson Ho, Stephen Casper, and Dylan Hadfield- Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 464 483, Los Alamitos, CA, USA, February 2023. IEEE Computer Society. doi: 10.1109 10.1109/SaTMLLS5 T555/ 00009.. URL https://doi. wtw.oooooeeccccoccoosoooeeeee org/10 org/10.11091 1tt11110//Saaa S00000//AAMM555555... 00223000333
Shauli Ravfogel, Yanai Flazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by itcrative nullspace projection. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Anreual Meeling of the Association for Compulational Linguistics, pp. 7237-7256 Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ wrw.//20g00000 acl- 647
vl/2020).cclmminn..6. 647.. URL
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan Cotterell. Linear adversarial concept erasure. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18400 18421. PMLR, 17 23 Jul 2022. URL https://ppoocedinng. mlr preas/vv122/raffoee2222 html
Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. Probing the probing paradigm: Does probing accuracy entail task relevance? In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pp. 3363-3317 Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021. Leac-mmain 295. URL A httppe///cllaaaooooooooro/0/0000 each: 2955
63

---

Published in Transactions on Machine Learning Research (09/2025)
Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context clas- sification task. In The Twelfth International Conference on Learning Represenlations, 2024. URL
https://openreviers net/forunn1d-aaMJffGG. http://ppaaaiiii.e.eeeooooaaaaaa
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 'why should i trust you?": Explaining the prodictions of any classifier. In Proccedings of the 22nd ACM SIGKDD International Conference Knowledgee Discovery and Dala Mining, KDD '16, pp. 1135 1144, New York, NY, USA, 2016. As-
sociation for Computing Machinery. ISBN 9781450342322. doi: 10. 10.1155/2836622.2937777
URL
https://doi.org 10. 1ee11111229997777 2939778
Logan Riggs, Sam Mitchell, and Adam Kaufman. Finding sparse linear connections between features in LLMs. AI Alignment Forum, dec 2023. URL http:://www...eeeeeoooooooo/oo/s
f finding- sparse-Linear Iinear- comeetions--beteenn features--- inmssminnmmmmm
Mattia Rigotti, Omri Barak, Melissa Warden, Xiao-Jing Wang, Nathaniel D. Daw, Earl K. Miller, and
Stefano Fusi. The importance of mixed selectivity in complex cognitive tasks. Nature, 497:585-590 2013.
doi: 10.1038/ 10.1038/naturellee n210/ URL https://ddi..org// 1ttp://0..000//aaeee111106
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering Ilama 2 via contrastive activation addition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (cds.), Proceedings of the 62nd Annuul Meeling of the Associalion for Compulational Linguislics (Volume 1: Long Papers), pp. 15504 15522, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/v /002//024422l2a/ 828. URL https:
wrgg//22..ccooo org/ 2024 a01-4.acl-ooor 8288/
Daniel A. Roberts, Sho Yaida, and Boris Hanin. The Principles of Deep Learning Theory: An Effective Theory Approuch to Underslanding Neural Networks. Cambridge University Press, May 2022. ISBN
9781316519332. doi: 10.1017 0001799711009023405, URL
Fabien Roger. Coup probes: Catching catastrophes with probes trained off-policy. Align- ment Forum, November 2023. URL. https://www.allgnneet wwtp://wwww..t.eeeea///////aaaaa
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how
BERT works. Transactions of the Association for Computational Linguistics, 8:842-866, 2020. doi: 10.
1162/tacl 00349. URL atttps://aclanttoooog.oorr
tacl-1.54/
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the TEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Sam Roweis and Zoubin Ghahramani. A unifying review of linear gaussian models. Neural Computation, 11 (2):305-345, 1999. doi: 10.1162 01116/2/0889766990000666777
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use inter- pretable models instead. Nature Machine Inteligence, 1:206 215, 2019. doi: 10.1038/642256--19-000
Naomi Saphra and Sarah Wiegreffe. Mechanistic?, 2024. URL
Rylan Schacffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilitics of large language models
mirage? In shirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
//openreview wwwriiiwwwee/fooree nett/oorumidd--Tooaale
Jérémy Scheurer, Mikita Balesni, and Marius Hobbhahn. Large language models can strategically deceive their users when put under pressure, 2024. URL http:::///ardd..r//bb//////
Thomas Schnake, Oliver Eberle, Jonas Lederer, Shinichi Nakajima, Kristof T. Schutt,
Muller,
and Gregoire Montavon. Higher-order explanations of graph neural networks via. relevant walks. JEEE
Transactions on Pattern Analysis and Machine Intelligence, 44(11):7581-7596, November 2022. ISSN 1939-3539. doi: 10. 10. 1000///ammm2021.333
URL URL httl:ttpp///dddd..ii wtw.///.........oo 1109/TPAMI. 20.. 3115452
64

---

Published in Transactions on Machine Learning Research (09/2025)
Lisa Schut, Nenad Tomasev, Tom McGrath, Demis Hassabis, Ulrich Paquet, and Been Kim. Bridging the human-ai knowledge gap: Concept discovery and transfer in alphazero, 2023.
abs/2310.1644
Sarah Schwettmann, Tamar Rott Shaham, Joanna Materzynska, Neil Chowdhury, Shuang Li, Jacob Andreas, David Bau, and Antonio Torralba. FIND: A function description benchmark for evaluating interpretabilityy methods. In Thirtg-seventh Conference on Newral Information Processing Systems Dalasels and Bench- marks Track, 2023. URL https:// http://www.......eeoooooomattaaa
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basee localization. Inter- national Journal of Computer Vision, 128(2)3336-377 359,, Oct 2019.
doi: (0000//11163-01900
Sanjit A. Seshia, Dorsa Sadigh, and S. Shankar Sastry. Toward verified artificial intelligence. Commun. ACM, 65(7):46 55, June 2022. ISSN 0001-0782. doi: 10.1145/3503914, URL httpht/tp///d http::://doi.or//10..o 1145/ 3503914
Tamar Rott Shaham, Sarah Schwettmann, Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob An- drcas, and Antonio Torralba. A multimodal automated interpretability agent. In Proceedings of the 4 1st International Conference on Machine Learning, ICML'24. JMLR.org, 2025.
L. Shapley. 7. A Vulue for n-Person Games. Contribulions to the Theory of Games II (1953) 307- 317., pp. 69-79. Princeton University Press, Princeton, 1997. ISBN 9781400829156.
doi: doi:
URL
Lee Sharkey. A technical note on bilinear layers for interpretability. 2023. URL https: https://arxivv http:///xxvvvvv///aa/// abs/ 3305.03452.
Lee Sharkey, Sid Black, and Beren Millidge. Current themes in mechanistic interpretabil- ity research, Nov 2022a. URL http://www..eeeerreercc/ccosss/s current-themee
http://momeeeessese_eeeee dnde depth,
Lee Sharkey, Dan Braun, and Beren. [interim research report] taking features out of superposition with sparse autoencoders ai alignment forum, Dec 2022b. URL /ttp////t/////aalllaaaaattttoor. www.orrwwmm..opt..aeaeee
Arnab Sen Sharma, David Atkinson, and David Bau. Locating and editing factual associations in mamba. In First Conference on Language Modeling, 2024a. URL
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin DURMUS, Zac HatfieldDoodds, Scott It Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda. Zhang, and Ethan Perez. Towards understanding sycophancy language models. In The Twelfth International Conference on Learning Representations, 2024b. URL http::///ppprrrreeeeeeeetoorrrrr
Noam Shazeer, Azalia Mirhoseini, "Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. URL
C. S. Sherrington. Observations on the scratch-reflex in the spinal dog. The Journal of Physiology, 34(1-2): 1 50, 3 1906. doi: 10. doi/10. 1t1111//jjphhoooo/ 1906.999660000 sp001139 185 citations as of iisert current date).
URL httpp://ppyyoce oolinelilrarry woabbbrrrrlllrre..........r..eee wwwey. com/
Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul Christiano, and Allan Dafoe. Model evaluation for extreme risks, 2023. URL htttss://aaxti: / /ttp////ww..r///////......
65

---

Published in Transactions on Machine Learning Research (09/2025)
Claudia Shi, Nicolas Beltran-Veles Achille Nazaret, Carolina Zheng, Adrià Garriga-Alonso, Andrew Jesson,
Maggie Makar, and David M. Blei. Hypothesis testing the circuit hypothesis in Ilss, 2024. URL https:
//arxiv. wwt./os///aaaaooooooaoo/// 13032
Buck Shlegeris, Fabien Roger, Lawrence Chan, and Fuan McLean. Language models are better than humans at next-token prediction, 2024. URL fttp::://axxiv.org/bbbsee
11281
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising
image classification models and saliency maps, 2014a. URL http:ppps//aaax...o////////22222 6034
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutiooo networs:: Visualising image classification models and saliency maps, 2014b. URL http:://aaxxii..rg//a////222..
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference AI, Ethics, and Society, AIES '20, pp. 180-186, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450371100. doi: 10. 10.1155/3375627.3335330
URL https://doi. httt::://oi.orr/11.... 1145/
3375627.3375830
Dylan Slack, Anna Hilgard, Himabindu Lakkaraju, and Sameer Singh. Counterfactual explanations can be manipulated. In M. Ranzato, A. Beygelzimer Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, volume 34, pp. 62 75. Cur- ran Associates, Inc., 2021. URL
neurips. httpr//ssec/pppp......ae////////
pdf
Andrew Smart and Atoosa Kasirzadeh. Beyond model interpretability: socio-structural explanations in machine learning. Al and SOCIETY, September 2024. ISSN 1435-5655. doi: 10.1007/400146:024200066
URL http://dt.......// doi http:////d......rg///...oo 00000/00//0...00//880660600---00 024-0006611
Lewis Smith. The "strong" feature hypothesss could be wrong, Aug 2024a. URL https:///w:/ http://wwww.es..o.......
Lewis Smith. Fcature overloaded terminology. Losssrong, 2024b. Available at: http:/aa...atttttg./////eeeessss
P. Smolensky. Neural and conceplual interprelation of PDP models, pp. 390 431. MIT Press, Cambridge,
MA, USA, 1986. ISBN 0262631105.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learn- ing using nonequilibrium thermodynamics. In Francis Bach and David Blei (ods.), Proceedings of the 32nd International Conference on Machine Learning, volune 37 of Proceedings of Machine Learning Re- search, pp. 2256-22622 2266,, Lille, France, 07-09 Jul 2015. PMLR URL https::/pproceednngs mlr press/ mlt.mress//
w37/sohl-dicksteinl/ html
Nitish Srivastava, GeofTrey Hinton, Alex Krizhevsky, Tlya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56): 1929-1958, 2014. URL http::///al..org/pppppppppe http:/pppprs//i1./iiiaaataaaae
html
Dashiell Stander, Qinan Yu, Honglu Fan, and Stella Biderman. Grokking group multiplication with cosets. In Proceedings of the 41st International Conference on Machine Learning, ICML '24. JMLR.org, 2025.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, ICML'17, pp. 3319 3328. JMLR.org, 2017.
Aaquib Syed, Can Rager, and Arthur Conmy. Altribution patching outperforms automated circuit discovery. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (ods.). Proceedings of the 7th BlackbozNLPP Workshop: Analyzing and Interpreting Neural Networks for NLP. pp. 407-416, Miami, Florida, US November 2024. Association for Computational Linguistics. doi: 10. 1016885/112000 http:////////22bbbooooo//epe blwckboxnlp- URL https: htttss//accladae http:///aaaattlolooooo////////// http://000...aacccoom--111.2..
66

---

Published in Transactions on Machine Learning Research (09/2025)
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, 2014. URL https://arxiv: http://aaxxvv..r//bbbbasa// org/abs/ htsp//bb/1311..... 1399.6199
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015. doi: 10.1109 (1L109/CVV..20...2222 72985944.
Alex Tamkin, Mohammad Taufeeque, and Noah D. Goodman. Codebook features: sparse and discrete interpretability for neural networks. In Proceedings of the 4ist International Conference on Machine Learning, ICM JMLR.org, 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Hean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Toannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Ruther- ford, Erica Moreira, Kareem Ayoub, Megha Goel, Jack Krawczyk, Cosmo Du, Ed Chi, Heng-Tro Tze Cheng, Eric Ni, Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Severyn, Hanzhao Lin, YaGuang Li, Yong Cheng, Abe Ittycheriah, Mahdis Mahdich, Mia Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan, Jeremiah Liu, Andras Orban, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapathy, Steven Zheng, HyunJcong Choc, Agoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jelf Pitman, Rushin Shah, Emanuel Taropa, Majd Al Merey, Martin Baeuml, Zhifeng Chen, Laurent El Shafey, Yujing Zhang, Olcan Sercinoglu, George Tucker, Enrique Pi- queras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roclofs, Anaïs White, Anders Andreassen, Tamara von Glchn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone, Tomas Ko- cisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rac, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Gaurav Singh Tomar, Evan Senter, Martin Chadwick, Ilya Kornakov, Nithya Attaluri, lñaki Iturrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grim- stad, Ale Jakse Hartman, Xavier Garcia, Thamumalayan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech Badia, David Reitter, Mianna Chen, Jenny Bremnan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravi Addanki, Antoine Micch, Annic Louis, Denis Teplyashin, Geoff Brown, Elliot Catt, Jan Balaguer, Jackic Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wei Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahm, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoe, Alex Kaskasoli, Sébastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanbur,, Srivatsan Srinivasan, Hyeontaek Lim, Sarah Hodkinson, Pranay Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipanjan Das, Dominika Rogozińska, Vitaliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Al- bin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessic Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang,
67

---

Published in Transactions on Machine Learning Research (09/2025)
Shuo yiin Chang, Paul Komarek, Ross Mcllroy, Mario Lučié, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung, Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lene, Timothy Chung, Acdan Pope, Loren Maggiore, Jackic Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchelti, Maja Trebacz, Kevin Robinson; Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora. Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko Tinuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufinan, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih- Kuan Yeh, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang, Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami,, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Vic- tor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thornton, Fangxiaoyu Feng, Wojciech Stokowiec, Ce Zheng, Phoebe Thacker, Çağlar Unlü, Zhishuai Zhang, Mo- hammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoc, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon-Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Pagannii, Jcan-Baptiste Lespiau, Alexandre Moufarek, Samer Hassan, Kaushik Shiv- akumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Ti, Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri, Aaron Cohen, Venus Wang, Kristie Seymore, Sergey Zubkov, Rahul Goel, Summer Yue, Sai Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad Mohananey, Jonah Joughin, Egor Filonov, Tomasz Kepa, Yomna Eldawy, Jiawern Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Subha Puttagunta, Kalpesh Krishna, Leslie Baker, Norbert Kalb, Vamsi Bedapudi, Adam Kurzrok, Shuntong Tei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragagnolo, Tej Toor, Tina Chen, Valentin Anklin, Feiran Wang, Richie Feng, Milad Gholami, Kevin Ling, Lijuan Liu, Jules Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Sid- dhinita Wandckar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Clemens Lombriser, Maksim Mukha, Botu Sun, Hafeezul Rahman Mohammad, Josip Matak, Yadi Qian, Vikas Peswani, Pawel Janus, Quan Yuan, Leif Schelin, Oana David, Ankur Garg, Yifan He, Oleksii Duzhyi, Anton Älgmyr, Timothée Tot-- taz, Qi Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr Chuklin, Josie Li, Carrie Spadine, Travis Wolfe, Kareem Mohamed, Subhabrata Das, Zihang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay, A Akanksha Maurya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovitch, Pavan Kumar Reddy M, Aarush Selvan, Mikhail Dektiarev, Golnaz Ghiasi, Erdem Guven, Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akerlund, Srançois-Xavier Aubet, Terry Huang, Chen Zhu, Eric Zhu, Elico Teixeira, Matthew Fritze, Francesco Bertolini, Liana- Elconora. Marinescu, Martin Bolle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thiet, Tulsee Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejczuk, Dominik Ra- biej, Vipul Ranjan, Krzysztof Styre, Pengcheng Yin, Jon Simon, Malcolm Rose Harriott, Mudit Bansal,
68

---

Published in Transactions on Machine Learning Research (09/2025)
Alexei Robsky, Geoff Bacon, David Greene, Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler, Ben Horn, Assaf Tsrael, Francesco Pongetti, Chih-Wei "Louis" Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah Ó Donnaile, Sébasticn Percira, Linda Friso, Adam Stambler, Adam Kurzrok, Chenkai Kuang, Yan Romanikhin, Mark Geller, 7J Yan, Kane Jang, Cheng- Lee, Wojeiech Fica, Eric Malmi, Qijun Tan, Dan Banica, Daniel Balle, Ryan Pham, Yan- ping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh, Chris Hidey, Niharika Ahuja, Pranab Saxena, Dan Dooley, Srividya Pranavi Potharaju, Eileen O'Neill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safranek-Shrader, Andrew Goodman, Joshua Kessinger, Eran Globen, Prateek Kolhar, Chris Gorgolewski, Ali Ibrahim, Yang Song, Ali Eichen- baum, Thomas Brovelli, Sahitya Polluri, Preethi Lahoti, Cip Baelu, Ali Ghorbani, Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre-Louis Cedoz, Chenmei Li, Shiyuan Chen, Niccolò Dal Santo, Siddharth Goyal, Jitesh Punjabi, Karthik Kappaganthu, Chester Kwak, Pallavi LV, Sarmishta Velury, Himadri Choudhury, Jamie Hall, Premal Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chikkerur, Yenai Ma, Adams Yu, Soo Kwak, Victor Åhdel, Sujeevan Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoorn, Alex Bailey, Taylan Bilal, Huanjic Zhou, Mchrdad Khatir, Charles Sutton, Wojeiech Readkowski, Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung, Kai Yang, Nihal Balani, Arthur Bražinskas, Andrei Sozanschi, Matthew Hayes, Héctor Fernández Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders, Michael Mandl, Ante Kärrman, Pawel Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jessica Mallet, Mitch Rudominer, Eric Johnston, Sushil Mittal, Akhil Udathu, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci, Gamaleldin Elsayed, Elnaz Davoodi, Marin Georgiev, lan Tenney, Nan Hua, Geoffrey Cideron, Edouard Leurent, Mahmoud Al- nahlawi, Iomut Georgeseu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snock, Mukund Sundararajan, Xuezhi Wang, Zack Ontiveros, Itay Karo, Jeremy Cole, Vinu Rajashekhar, Lara Tumeh, Eyal Ben-David, Rishub Jain, Jonathan Uesato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung- Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, An- drew Lee, Nino Vieillard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Geoffrey Irving, Ed- ward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Castaño, Trene Giannoumis, Wooyeol Kim, Mikolaj Rybiński, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeck- emeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Twuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishck Chakladar, Ginger Perng, Elena Allica Abellan, Mingyang Zhang, Ishita Dasgupla, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Doussc, Fan Yang, Jeff Piper, Nathan Ic, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnie Lui, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Due Dung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Tili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulanova, Rémi Leblond, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Prascem Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, MohamnadHHossenn Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthieu Geist, Ser tan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee- Thorp, Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nico-
69

---

Published in Transactions on Machine Learning Research (09/2025)
las Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Neje Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papa- makarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloc Thornton, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri, Benigno Uria, Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang, Yeqing Li, Nir Levine, Ariel Stolovich, Rebeca Santamaria-Fernandez, Fernandez, Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Charlie Deck, Hyo Lee, Zonglin Ti, Kyle Levin, Raphael Hoffmann,, Dan Holtmann-Riee Rice, Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganeh, Siim Pöder, Mukarram Tariq, Yanhua Sun, Lucian Ionila, Mojtaba Seyedhosseini, Pouya Tafti, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszez, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christopher A. Choquette- Choo, Yunjie Li, TJ Lu, Abe Ityycheriah, Prakash ShrofT, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya. Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Fri Latorre-Chimoto, Hanna Klimezak- Pluciíska, David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominík Grewe, Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yad- lowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Dua, Dong Li, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Amuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hardic Cate, James Manyika, Key- van Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, War- ren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang, Julia Wiesinger, Fammanouil Koukoumidis, Yuan Tian, Anand Tyer, Madhu Gurumurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Authony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Ken Durden, Harsh Mchta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Mikulik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang, Ke Ye, Jean Michel Sarr, Melanie Moranski Preston, Madeleine Flish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Tejvi M., Xinyun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Amruta Muthal, Senaka Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush, Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taitelbaum, Hila Noga, Zhuyun Dai, James Wang, Chen Liang, Jenny Hamer, Chun-Sung Ferng, Chencl Elkind, Aviel Atias, Paulina Lee, Vit Listík, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krunoslav Zaher, Paul Müller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph Hirnschall, Ashwin Sethi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Dhandhania, Manish Katyal, Akshay Gupta, Atharva Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatia, Yashodha Bhavnani, Omar Alhadlaq, Xiaolin Li, Peter Danenberg, Dennis Tu, Alex Pine, Vera Filippova, Abhipso Ghosh, Ben Limonchik, Bhargava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Ianna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael Kucharski, Mohak Patel, Mudit Jain, Maksim Zabelin, Paolo Pelagatti, Rohan Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi Ramachandruni, Xiangkai Zeng, Ben Bariach, Laura Weidinger, Tu Vu, Alek Andreev, Antoine He, Kevin Hui, Sheleem Kashem, Amar Subramanya, Sissie Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoe Le, Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. Gemini: A family of highly capable multimodal models,
70

---

Published in Transactions on Machine Learning Research (09/2025)
2024. URL http:://arriv..r///////22222121
Max Tegmark and Steve Omohundro. Provably safe systems: the only path to controllable agi, 2023. URL
https://arxiv.orr/bbbss. wwwporr////a///// 01933
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Cal- lum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Ex- tracting interpretable features from claude 3 sonnet. Trunsformer Circuils Thread, 2024. URL https: //bransformer--irruutt
html
lan Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korho- nen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Lánguistics, pp. 4593 1593-4601, Florence, Italy, July 2019. Association for Computational URL http:::://claaatiooo....ooo////
Linguistics. doi: 10.18653/v1/PD5) 1452..
1452/
Jacques Thibodeau. But is it really in rome? an investigation of the rome model editing technique ai alignment forum, Dec 2022. URL
http://mmmnnnooooooooroooro.paas bet-is-tt-rellll-inn- htt.::/www......t.....t..o.oooea ottpppssse//1//99//S5//tttop
Hannes Thurnherr and Jérémy Scheurer. Tracrbench: Generating interpretability testbods with large lan- guage models. In ICML 2024 Workshop on Mechanislic Interprelabilill, 2024. URL https://openreerier
ket/forum?id=vwuu22255
Demian Till. Do sparse autoencoders find "true features"? Less Wrong, 2024. URL httpss:/www..esseet/
Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Func- tion vectors in large language models. In Proceedings of the 2024 International Conference on Learning Representations, 2024. URL https::/ wtt..ttt//ooou//idd//tt/tt//////
arXiv:2310.15213,
Shengbang Tong, Erik Jones, and Jacob Steinhardt. Mass-producing failures of multimodal systems with language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024a. Curran Associates Inc.
Siyuan Tong, Kaiwen Mao, Zhe Huang, et al. Automating psychological hypothesis generation with ai: when large language models meet causal graph. Humanilies and Social Sciences Communiiattions, 11:896, 2024b. doi: 10.1057/s41599-022 000-90024-0--0---5 URL
Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights, 2018. URL htttp:://axxxi..coom
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2024. URL. httpps//arxiv..rr Abs/2338..022
Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. In A. Oh, T. Naumann, A. Globerson, K. Saeuko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, neurips wttpccprppacaeeeeeeeeeeee _iles// pp. 74952-74965. Curran Associates, Inc., 2023. URL http:::/ pdf
UK Government. National Al strategy. Government strategy, Government of the United Kingdom, 2022. URL https://www.ce gov. w//porrmar//oolllleeetorrratraaa -atnonalll--strttege--aaeee htm.tteeggytmml-eeeeeee
United Kingdom AT Safety Institute. Early lessons from evaluating frontier AI systems. Tech-
nical report, United Kingdom AI Safety Institute, 2024. URL https:://wwi aisi wer.a.gvv.uuuuuueee uk/work/
haaryyelessss-fooorrrraaaaaaaiop sisssttmms
71

---

Published in Transactions on Machine Learning Research (09/2025)
United States Congress. Copyright law of the united states (title 17) and related laws contained in title 17 of the united states code. URL httpps//ww..oorrrighttttttt copyright /tvtrgghhggo//tttel/// This publication contains
the text of Title 17 of the United States Code, including all amendments enacted by Congress through December 23, 2022. It includes the Copyright Act of 1976 and all subsequent amendments to copyright law; the Semiconductor Chip Protection Act of 1984, as amended; and the Vessel Hull Design Protection Act, as amended.
Guillermo Valle-Perez, Chico Q. Camargo, and Ard A. Louis Deep learning generalizes because the parameteerffunttion map is biased towards simple functions. In International Conference Learning Represenlations, 2019. URL htt.:t/pp///ppeeerree..e///////a
Teun van der Weij, Felix Hofstätter, Ollie Jaffe, Samuel F. Brown, and Francis Rhys Ward. Ai Ai sandbagging: Language models can strategically underperform on evaluations, 2024. URL https://arxiv.org// //s//////// 2406.07358
Alexandros Vasileiou and Oliver Eberle. Explaining text similarity in transformer models, 2024. URL
https://arxiv. www.oogg///oooeooooooo a20/////a//20.../////////// 06604
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Sys-
lems, volume 33, pp. 12388 12401. Curran Associates, Inc., 2020. URL https:://proceedings mwwiiiinnnnnnunns.......... neurips. pdf
Fernanda Viégas and Martin Wattenberg. The system model and the user model: Exploring ai dashboard design, 2023. URL http:///....a///////////////2222
Elena Voita and Ivan Titov. Information-theoreta probing with minimum description length. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 183 196, Online, November 2020. Association for Computational Linguistics. doi: 10066/8//112022mmmmmp// main..4. URL https://aclanthology .rgg
2020. wwn.l-main.n......
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self- attention: Specialized heads do the heavy lifting, the rest can be pruned. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annuul Meeling of the Association for Computational Linguistics, pp. 5897,5808, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/PI11 1580. URL sttpss:/acclamttooogg.orrg////// org/P19-1580
Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Asso- ciation for Computational Linguistics ACL 2024, pp. 1288-1301, Bangkok, Thailand and virtual meet- ing, August 2024. Association for Computational Linguistics. URL https: https:://alaaahhooooooo 2024. findings-acl.75
Chelsea Voss, Nick Cammarata, Gabriel Goh, Michael Petrov, Ludwig Schubert, Ben Egan, Swee Kiat Tim, and Chris Olah. Visualizing weights. Diställ 2021. doi: 10.23915/distti1.0000 https://disttillpubt
httplllp//pp///////// /2000/circuiss htnp://ircuitt///isaliiingggn wiws//aiiiigggeiieeeeeeeee
Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization. In ICML 2024 Workshop on Mechanistic Interpretabillly, 2024a.
URL https://openrevier: wtt.://r.nnet/oooooiimmm/eeeeii
George Wang, Matthew Saarugia-Roberte Jesse Hoogland, Liam Carroll, Susan Wei, and Daniel Murfet. Loss landscape geometry reveals stagewisc development of transformers. In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024b. URL https:://operrevien.o net/
forum?id=2Jabby2JM51
72

---

Published in Transactions on Machine Learning Research (09/2025)
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretabil- ity in the wild: circuit for indirect object identification in GPT-2 small. In The Fleventh International Conference on Learning Representations, 2023. URL https://opeereeiies http://wwww.....ooooooo/d///o/ee
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. Knowledge editing for large language models: A survey. ACM Comput. Surv., 57(3), November 2024c. ISSN 0360-0300. doi:
10.1145/3695900. URL https:://toii.... htttpp::/aoi..org11..11111 1111///990..44/66999999
Yongjie Wang, Tong Zhang, Xu Guo, and Zhiqi Shen. Gradient based feature attribution in explainable al: A technical review, 2024d. URL http::httt///sttaae http://ww..rg//////0003333/ 10411
Francis Ward, Francesca Toni, Francesco Belardinelli, and Tom Everitt. Honesty is the best policy: Defin- ing and mitigating ai deception. In A. Oh, T. Naumann, A. Globcrson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Newrul Information Processing Systems, volume 36, pp. 2313 2341. Cur- ran Associates, Inc., 2023. URL https: https:://proceeddngg nwwsineerrppp.. neurips.cc/paper files//paper/223//Eiee/ 0ffc//eehal1aree52200001aa66666 Prppreereeeeeeppp---oooreeeeeeee
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Thans- aclions of the Association for Compulational Linguislics, 7:625 641, 2019. doi: 10.1162/tacl 00290.
URL https://aclaanthool.g.. http:/aawaaooooooooo1o11/1-11111 orrp//9-00//Q111000009
Sumio Watanabe. Algebraic Geometry and Statisticol Learning Theory. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2009.
Matthew Watkins. ding spaces, Dec 2023.
Mapping the semantic void: Strange goings-on in gpt embed-
URL https://www.eesswroug hwtp:///a///////oo//o/rrrrrrr/rr
David S. Watson. Conceptual challenges for interpretable machine learning. Synthese, 200:65, 2022. doi:
2//0///11222202-34-883333 URL
Donglai Wei, Bolei Zhou, Antonio Torrabla, and William Freeman. Understanding intra-class knowledge
inside cmn, 2015. URL
02379
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgcaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Ma- chine Learning Research, 2022. ISSN 2835-8856. URL htttp:://operreviex http://pprrriieeeee.eeerrrrrreep rett://orrum?dddyyysssssuu Survey Certification.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. ofain-of-thhugghe prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc ISBN 9781713871088.
Susan Wei, Daniel Murfet, Mingming Gong, Hui Li, Jesse Gell-Redman, and Thomas Quella. Deep learning is singular, and that's good. IEEE Transactions on Neural Networks and Learning Systems, 34(12):
10486, 2023. doi: 10.1109/TNNI, 2022.3167409.
Gail Weiss, Yoar Goldberg, and Eran Yahav. Thinking like transformers. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11080-11090. PMLR, 18-24 Jul 2021. URL https://proceedngs. html
mlr. httpass/v33//weeeec
Tichakorn Wongpiromsarn, Mahsa Ghasemi, Murat Cubuktepe, Georgios Bakirtzis, Steven Carr, Mustafa O. Karabag, Cyrus Neary, Parham Gohari, and Ufuk Topeu. Formal methods for autonomous systems. Foundations and Trends@ in Systems and Control, 10(3 1880-44770 2023. ISSN 2325-6826. doi: 10.1561/ 2600000029. URL htdp://dx... doi org/ 1611100001611/8000000099
73

---

Published in Transactions on Machine Learning Research (09/2025)
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Contijo-Lopes, Ari S Mor- COS, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infer- ence time. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23998. PMLR, 17-23 Jul 2022. URL https:///proeeednngg mlr. ppwesa/v12//oottaareeeee
html
Wilson Wu, Louis Jaburi, Jacob Drori, and Jason Gross. Unifying and verifying mochanistic interpretations: A case study with group operations, 2024a. URL ftttps://arxiv.orgg/
07476
Zhengxuan Wu, Atticus Geiger, Thomas Ieard, Christopher Potts, and Noah Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL hattps://openrevier. httpps//gaaaaaateee..eeer/rrrddd wett/forun?.draRfChaai..
Zhengxuan WU, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky; Christopher D Manning, and Christopher Potts. ReFTT: Representation finetuning for language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https://openreviiet http://ppeerrvio...teffforrr net/forum?
id=fykjpIMeOV,
Zhengxuan Wu, Atticus Geiger, Jing Huang, Aryaman Arora, Thomas Teard, Christopher Potts, and Noah D. Goodman. A reply to makelov et al. (2023)'s "interpretability illusion' arguments, 2024c. URL https:
7/arxiv. wwrxiv.org/ooo
Keith Wynroc and Lee Sharkey. Decomposing the QK circuit with bilincar sparse dictionary learn- ing. Al Aligument Forum, jul 2024. URL https:///ww..eeswrron http:///g...o///////////aaaaaann recompoiig-tthe-gk-rircue www.::rwwwwttt..maaaaoaaaeeeeeee
Rui Xin, Chudi Zhong, Zhi Chen, Takuya Takagi, Margo Seltzer, and Cynthia Rudin. Exploring the whole rashomon set of sparse decision trees. Tn Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:///ppnrrviie..ee
forum?id
Xi
Ye and Greg Durrett. The unreliability of explanations in few-shot prompting for tex- tual reasoning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30378 30392. Curran Associates, Inc., 2022. URL http::///pooeedingg.nor.s....o http:////pppprrieeee//////////// 2205502880F005022000bbr0eee111p0
Adam Yedidia. Gpt-2's positional embedding matrix a helix, July 2023. URL http::t/tp//// //wp::///wwwwwww http:/wwwwsswwo........m/
Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to conclusions: Short-cutting transformers with linear transformations. In Nicoletta Calzolari, Min-Yea Kan, Veronique Hoste, Alessan- dro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC- 2024), pp. 9615- 9625, Torino, Italia, May 2024. ELRA and ICCL. URL http:t:://alaaattooogg..ggggggg wrg/2024.....0000... 1w:.:044..eeemmmmmamea 840
Jason Yosinski, Jeff Clune, Anh Mai Nguyen, Thomas J. Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. CoRR, abs/1506.0657 2015. URL http:://axi:: http:///arxd..rr//aa//5566 org/abs//506..
06579
Lei Yu, Meng Cao, Jackie CK Cheung, and Yue Dong. Mechanistic understanding and mitigation of language model fon-factual hallucinations. In Yaser Al- Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. November 2024. Association for Computational Linguistics. doi: 10.18653 /v1///11222222 findings- finding-eemnlp-46 URL httpp::://acllaaoooooooo///////p findirfssdddiiiniiine
Miami, Florida, USA,
74

---

Published in Transactions on Machine Learning Research (09/2025)
Qinan Yu, Jack Merullo, and Ellic Pavlick. Characterizing mechanisms for factual recall in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pp. 9924 9959, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 118633/v1222222 wwwweemmmimaaaaaaeeem URL https:://cclatttolllg/// .org org/
2023. em2ppmmnm-m..n.c 615.
Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: contextualized embedding as linear superposition of transformer factors. In Eneko Agirre, Marianna Apidianaki, and Ivan Vulié (ods.), Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Archilectures, pp. 1 10, On- line, June 2021. Association for Computational Linguistics. doi: 10.18653/v 221/2221..eeee ddelloo URL attps://aclanthologg) wtg/2021 wttp:/rr///...eeooooo11....
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision Cham, 2014. Springer International Publishing. ISBN 978-3-319-105900
ECCV 2014, pp. 818 833,
Enyan Zhang. Michael A. Lepori, and Ellie Pavlick. Instilling inductive biases with subnetworks, 2024. URL
Setps://arxiin http:/arxvvv...gaaaa. 2310. 10899
Xinyang Zhang, Ningfei Wang, Hua Shen, Shouling Ji, Xiapu Luo, and Ting Wang. Interpretable deep learning under fire. In Proceedings of the USENIX Security Symposium (Security), 2020.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene enns. In ICLR, 2015. URL http://arxiv. httgg/abs/412... 6856
Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. Do feature attribution methods correctly attribute fcatures? Proceedings of the AAAI Conference on Artificial Intelligence, 36(9):9623-960) 9633, Jun. 2022. doi: 10.1609 http://aii.v33.... 21196
CRL https://ojs.. aaaii org wwtgg/nddx.x.r hbtp/AA////aarttcce//iie/////
Roland Simon Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas S. A. Wallis, and Wieland Brendel. How well do feature visualizations support cansal understanding of CNN activations? A. Beygelzimer Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, 2021. URL httpps://openrevior http://ppwnrrrlooo..teooooooooaa wet.t/orrun.ddd1.aaaaaaa...
Andy Zon, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency, 2023a. URL
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023b. URL https://arxiv.
15043
Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving aligument and robustness with circuit breakers, 2024. URL https: http://aaxvvvv.. www.vvv.org/bb///40000 04313
75

---

Published in Transactions on Machine Learning Research (09/2025)
Appendix
Summary of open questions
Open problems mechanistic interpretability methods and foundations
Reverse engineering: Identifying the roles of network components
Reverse engineering step 1: Neural network decomposition
1. How should we decompose networks into more interprotable constituent parts?
(a) What isomorphism or what approximation of a neural network (or parts of it) is the best way to express it. for the purposes of interpreting it?
(b) How should coarse grain neural networks?
How should we build higher level abstractions on top of low-level network components!
How true is the linear representation hypothesis?
(a) To what extent do models encode concepts linearly in their representations?
(b) How should we characterize representations that are not lincarly represented in neural networks?
(c) What properties of concept, or of the training distribution, result in particular concept becoming encoded linearly (or not)?
the combination of the linear representation hypothesis and superposition the right frame for thinking about computation neural networks?
(a) Can fully determine the causes of feature superposition and polysemanticity within neural networks?
(b) How should we understand superposition in allention blocks?
(c) How should we understand cross-layer superposition?
(d) What new theoretical insights can be gleaned from considering how networks perform com- putation natively superposition, rather than treating superposition purely a compression strategy?
Can the problems with SDL be overcome?
(a) What lies in SDL reconstruction errors? Will the errors converge to zero with methodological progress?
(b) Is sparsily the correct proxy for interpretabillty?
(c) Can the approach be scaled to the largest models?
(d) Does SDL make sense if we don't believe in the lincar representation hypothesis?
(e) Is sparsity the best possible proxy for interpretability?
Is correct to think of SDL features as 'bags of features', there important information
contained within the geometry of representation space?
(g) If SDL finds compositions of the "truc" features, this a problom?
(h) Can SDL be applied to all architectural components to fully decompose networks?
How can we better measure the success of SDL techniques?
How should connect sparsely activating features into circuits? Will this be practically pos-
sible, and the best possible description of network mechanisms?
(k) Can develop new methods that address conceptual and practical issues with SDL?
How important is the geometry of activation space for explaining neural network behavior?
(a) How can we identify the underlying functional structure of networks (which defines why acti- vations are located in particular geometric arrangements in activation space)?
76

---

Published in Transactions on Machine Learning Research (09/2025)
(b) Must we understand global feature geometry only local feature geometry in order to under- stand computation in neural networks?
6. Can we connect theories for how neural networks generalize to interpretability?
(a) Can we distinguish parts of networks that underlie generalization from parts that underlie memorization?
(b) What mechanisms underlie the relationship between interpretability and generalization?
(c) Are there connections between adversarial robustness and superposition?
(d) Can connect interpretability to theories of deep learning like SLT?
Can we build intrinsically more interpretable models at low performance cost? How helpful is this?
(a) Interpretability training: Can we train networks that are interpretable by default at low per- formance cost?
(b) Interpretable inference:
Can we convert already-trained models into forms that much easier to completely interpret at little performance cost?
(c) How can we train large-scale models such that the concepts they use are naturally understand- able to humans?
(d) How can we design training objectives so that the model is incentivized to use known specific abstroctions?
(c) How we localize concepts we want to control (c.g., long-term plans) during training?
Reverse engineering step 2: Identifying the functional role of components
Can we improve max-activating input data set examples for understanding the causes of network component activations?
(a) How can we avoid imposing human bias to explanations?
(b) Can progress toward deeper descriptions based internal mechanisms?
(c) How might we develop interpretation methods that can recognize and work with unfamiliar concepts computational patterns that don't map cleanly to human intuitions?
How can we develop attribution methods that faithfully and efficiently compute which network components are important for some downstream metric?
(a) How can develop altribution methods that capture higher-order effects beyond first-order approximations of model behavior?
(b) Is possible to create perturbation-buse methods that don't force models to operate outside their training distribution?
(c) Can we develop hybrid approaches that combine the strengths of different attribution methods while mitigating their individual weaknesses?
3. How can belter measure the downstream effects of model components?
(a) How can we reliably distinguish between true causal pathways and compensatory effects like the "Hydra effect" when performing interventions?
Reverse engincering step 3: Validation of descriptions
Can we improve our ability to validate mechanistic explanations for model behavior ways that do not depend on rescarcher intuition and are computationally tractable use?
(a) Can improve on methodologies for evaluating hypotheses through their predictive power activations of network components?
(b) Can we develop methodologies for evaluating hypotheses through their predictive power for model behavior (e.g. unusual failures adv examples)?
77

---

Published in Transactions on Machine Learning Research (09/2025)
(c) Can we develop methodologies for handcoding weights that faithfully represent some hypotheses, as drop in replacement for subnetworks we claim to understand?
(d) Can we develop wider suite of networks with known ground truth explanations to validate techniques against?
(e) Can we use mechanistic explanations to achieve engineering goals?
(f) Can we use mechanistic explanations to achieve enginocring goals in a way that improves upon black box baselines?
2. Can develop "model organisms" as communily, which are understood deeply, and seen test-bed for new unproven interpretability methodologies to be tested?
Can we establish standardized baselines and benchmarks for comparing different interpretability approaches on real-world, con-cherry-picker tasks, where the ground truth is known?
4. What would constitute comprehensive set of "stress tests" for interpretability hypotheses that could reliably detect interpretability illusions?
How might we design ovaluation frameworks that assess interprotability methods on their average case and worst-case performance rather than just. best-case scenarios?
6. How can we ensure that our understanding of internals generalizes to disttiititii inputs?
Concept-based interpretability: Identifying network components for given roles
How can we reliably distinguish causal from merely correlated features when probing neural net- works?
2. Can we develop automated systems to generate high-quality probing data sets, reducing the current heavy reliance on human effort?
What regularization and validation techniques can be used prevent spurious correlations while ensuring probes find generalizable fcatures?
4. How can we improve probing for concepts that may not have clear positive/negation examples?
Proceduralizing mechanistic interpretability into circuit discovery pipelines
Can we develop techniques that build on lower level methods that provide deeper or more complete insights about neural networks?
2. How much can we learn from further work in the existing circuit discovery paradigm?
(a) Should we expect circuit discovery to benefit from further methodological progress in decom- posing neural networks? Will faithfulness go and explanation description length go down?
(b) Can we remove the constraint that task definition for circuit discovery inherently concept- based, which may be complicating mechanistic analysis?
(c) Can we get around the practical issues of negative and backup behavior?
(d) Will circuit discovery provide insights into arbitrary tasks, or will it only be helpful in cases where we are able to crisply define tasks?
Automating steps in mechanistic interpretability research
Can we improve on AT automated feature description and validation methods?
(a) Through automating the generation and testing of arbitrary hypotheses?
(b) Through describing differences between fcatures?
(c) Through descriptions of how components interact?
78

---

Published in Transactions on Machine Learning Research (09/2025)
2. Can we improve ACDC-like circuit discovery methods?
3. Can we automate other parts of the mechanistic interpretability pipeline?
(a) Conceptual interpretability research?
(b) Decomposition method discovery?
More ad hoc validation of hypotheses?
4. Should we take steps to mitigate potentially misaligned AI systems sabotaging AI automated inter- pretability?
Open problems in applications of mechanistic interpretability
Using mechanistic interpretability for better monitoring and auditing of Al systems for potentially unsafe cognition
Can we effectively use interpretability for safety evaluations?
(a) Can develop robust "white box" evaluations that detect concerning internal patterns without needing to understand the entire network?
(b) Can we reliably distinguish between features that merely recognize deceptive behavior versus mechanisms that generate deceptive bchavior?
(c) How can we validate that learned features capture all concerning patterns reasoning?
(d) Can we reliably identify which features are appropriately versus spuriously relevant to given task?
2. Can we leverage interpretability to enhance red-teaming and system testing?
(a) Can we use interpretability insights to make red-teaming more efficient than current methods?
(b) How we best use feature altribution to help human red-teamers identify problematic input patterns?
3. Can we develop effective test-time monitoring systems based interpretability?
(a) Can we get mechanistic anomaly detection to work?
(b) Can create passive monitoring systems based model internals that effectively flag con- cerning internal patterns during deployment?
(c) Can we develop monitoring systems that work with only feature-level understanding rather than requiring deep mechanical insights?
Using mechanistic interpretability for better control of Al system behavior
1. Can we improve steering methods through interpretability?
(a) How can we make activation steering more precise and reduce its side effects?
(b) Can we develop methods to steer entire mechanisms rather than just single features?
2. Can we achieve reliable model unlearning and editing?
(a) Will carving the network at its true joints help us improve on model unlearning and editing?
(b) Can mechanistic interpretability help us develop better methods for evaluating unlearning effi- cacy?
(c) Can mechanistic interpretability help us determine which classes of model edit are even possible, without damaging generalization in undesirable ways?
Can we better understand and improve finetuning through interpretability?
(a) Can make finetuning more sample-efficient by targeting specific parameters?
(b) Can we develop better tools for analyzing feature-level or mechanism-level differences bet weew model versions?
79

---

Published in Transactions on Machine Learning Research (09/2025)
Using mechanistic interpretability for better predictions about Al systems
1. Can predict model behavior in novel situations outside of the distribution of inputs we have access to with mechanistic understanding?
(a) Can we reliably predict when and how jailbreaking or safety bypasses might. occur?
(b) How we identify internal signatures that predict specific failure modes like hallucination?
(c) Can we develop methods to predict model behavior without requiring behavioral evaluations?
(d) Can we find "values" or "goals" in systems that might be indicative of behavior in more gener- ality?
(e) Is possible prove the absence of specific dangerous capabilities through mechanistic anal- ysis?
Can we develop formal verification methods for Al systems?
(a) Can current toy model verification approaches scale frontier systems?
(b) How much of neural computation can be reduced to verifiable symbolic operations?
(c) Can we create formal guarantees about system behavior in complex, non-formalizable environ- ments?
(d) What level of mechanistic understanding is necessary for meaningful formal verification?
3. Can we make rigorous claims about model safety?
(a) Can we definitively prove the absence of specific dangerous mechanisms?
(b) How can we verify claims about model values and goals in a rigorous way?
(c) What types of safety claims are possible with current interpretability methods?
(d) Can we develop "enumerative safety" approaches that reliably identify all relevant mechanisms?
4. Can we better predict AT capability development. through interpretability?
(a) Can we identify early signatures that predict emergent capabilities?
(b) How do model mechanisms evolve dynamically during training?
(c) Can we map the connection between small-scale circuits and large-scale capabilities?
(d) How does the loss landscape's structure relate to capability emergencc?
Can we understand the relationship between training data and capabilities?
(a) How do specific training examples influence the development of model mechanisms?
(b) Can we predict model limitations based on training data composition?
(c) Can design training data sets to reliably produce specific desired capabilities?
(d) How does data set structure affect the balance between in-context and weights- based learning?
6. Can we predict latent maskable capabilities?
(a) How can we identify capabilities that could be 'unlocked' through prompting or finetuning?
(b) Can detect when finetuning has masked rather than removed capabilities?
(c) How do we analyze mechanisms that span multiple timesteps or sequential behaviors?
(d) Can predict which model capabilities are fundamental versus superficially trained?
Using mechanistic interpretability to improve our ability to perform inference, improve training and make use of learned representations
Can we use interpretability to make inference more efficient?
(a) How can we identify skippable computations without affecting outputs?
(b) Can we create more effective distillation methods through mechanistic understanding?
(c) How we optimize model architecture based component. function analysis?
80

---

Published in Transactions on Machine Learning Research (09/2025)
(d) Can identify and optimize critical computational pathways?
Can we improve training through mechanistic insights?
(a) Can we better select training data by understanding example influence?
(b) How we monitor and optimize capability emergence during training?
(c) Can we develop more parameter-efficent training methods through component analysis?
(d) Can create better architectures through component understanding?
(e) Can we identify and enhance components with specific functionalities?
3. Can we instill capabilities directly into networks?
(a) Can design better inductive biases based on mechanistic insights?
(b) Is possible to create modular architectures with swappable components?
(c) Can develop reliable methods for combining model parameters?
(d) Is possible to transfer specific capabilities between models?
Using mechanistic interpretability for 'microscope Al'
Can we leverage Al models for scientific discovery?
(a) How we extract novel patterns and predictors that models have found?
(b) Can we make microscope Al techniques accessible to domain experts?
(c) How do validate scientific insights derived from model interpretability?
(d) Can we extend microscope AI beyond current simple correlational discoveries?
Can we develop better knowledge extraction methods?
(a) How
detect when models have found genuinely novel patterns?
(b) Can we automate the process of finding scientific insights in model weights?
(c) How do bridge the gap between model features and scientific concepts?
(d) Can we make these techniques usable without deep machine learning expertise?
Mechanistic interpretability on a broader range of models and model families
Can interpretability methods generalize across architectures?
(a) Do current interpretability methods (SDL, circuit analysis) transfer to SSMs? Or, like the transition from CNNs to transformers,
approaches necessary?
(b) Which insights are model-specific versus universal?
(c) How can we adapt methods for multimodal models?
2. How do different models trained on similar data compare mechanistically?
(a) Is the "universality hypothesis" true across models? To what extent do neural networks learn similar features and circuits to each other (and to humans?)
(b) Do different architectures learn fundamentally different features?
(c) How do mechanisms of particular tasks differ between transformers, CNNs, and SSMs?
(d) Are there insights can gain from comparing architectures?
3. Can we future-prodd interpretability research?
(a) How can we prepare for interpreting novel architectures?
(b) Should focus architecture-specific general methods?
(c) Can we identify truly fundamental interpretability principles?
(d) Will current methods work future frontier models?
81

---

Published in Transactions on Machine Learning Research (09/2025)
Human computer interaction with model internals
1. Can create interfaces that use mechanistic understanding to enhance human-neural network interaction?
(a) How can we visualize model internals in an intuitive way?
(b) Can we develop real-time interpretability dashboards?
(c) What's the right balance between simplicity and depth in these interfaces?
(d) How do we make complex model mechanisms understandable to non-experts?
2. Can we develop interpretability tools to help auditors?
(a) How we help auditors find potential failure modes directly?
(b) Can we develop tools to detect bias at the mechanism level?
(c) What interfaces would make auditing more efficient and thorough?
(d) How can we present technical findings to policy makers?
3. Can we improve end-user interaction with AT?
(a) How transparency features help users calibrate trust?
(b) Can we create intuitive controls based on model mechanisms?
(c) Can create intuitive ways to steer model behavior?
Governance
1. Can mechanistic analysis help identify and prevent failures?
(a) Can identify specific mechanisms that caused AI failures?
(b) How do we map the causal chain of mechanisms leading to incidents?
(c) Can detect when similar mechanisms are about to activate?
(d) Is it possible to isolate and modify failure-causing mechanisms?
Can we study mochanism patterns related governance?
(a) Can identify mechanisms responsible for specific dangerous capabilities?
(b) How do we detect deceptive evasive mechanisms?
(c) Can we map the mechanisms involved in model decision-making"?
(d) Is it possible to verify the absence of specific harmful mechanisms?
Can mechanistic insights verify compliance?
(a) How can we trace decision mechanisms to explain model outputs?
(b) Can identify mechanisms that process copyrighted content?
(c) Is possible to detect mechanisms that encode specific knowledge?
(d) How do verify modifications to problematic mechanisms?
Open socio-technical problems in mechanistic interpretability
Translating technical progress mechanistic interpretability into levers for Al policy and governance
1. Can we use mechanistic understanding to better evaluate AI capabilities?
(a) How can we use interpretability to improve capability elicitation?
(b) Can we use interpretability reliably detect when models are strategically underperforming capabilities evaluations?
82

---

Published in Transactions on Machine Learning Research (09/2025)
2. Can mechanistic understanding improve our ability to forecast when whether new capabilities will arise ahead of time?
How can we use interpretability better estimate the likelihood of different threat models?
4. Can we use interpretability to prevent Al incidents?
(b) Can interpretability to construct reliable test-time monitors to detect Al incidents?
(a) Can we use
reliably prevent similar incidents in the future, by using interpretability to design new evaluation tasks on incident scenarios?
Can interpretability help verify which workloads GPUs are being used for?
6. How should interpretability inform copyright law?
How can mechanistic understanding help resolve copyright challenges in generative AI, particularly regarding the detection and removal of memorized copyrighted works?
Social and philosophical challenges in mechanistic interpretability
1. What is interpretability?
(a) What are the goals of the field?
(b) How should success be graded?
(c)
Should we treat interpretability as a science or an engineering discipline? What implications does this have on what research should be done?
2. How can mitigate downside risks of interpretability research?
(a) How can we communicate the results of our research such that the risk of their misuse is minimized?
83

---

Table 1: Summary of principal methodologics in mechanistic interpretability (Section 2)
Published in Transaction on Machine Learning Research (09 (2025)
Methodology
Principal Method /
Description / Goal
Key Techniques
Examples
Major Open Problems
Limita-
Category
Approach
tions (Sec. 2)
Reverse
Engi-
1. Neural Network
Breaking down the network into simpler, analyzable parts to understand how general- izes (Section 2.1.2
neering
Decomposition
Architectural components (ncurons.
Architectural components often polyscmantic (Sec- SDL: High reconstruction er- ror, expensive, assumes lin- representation, sparsity proxy for interpretability is questionable, feature geometry unexplained, doesn't decom- pose mechanisms themselves (Section 2.1.2). lack of solid thcoretical foundations (Sec-
layers) (Sec-
tion 2.1.2
tion 2.1.2
Dimensionality reduction (PCA, SVD) (Sec-
Dim. reduction can't find more features than dimensions (Section 2.1.2).
tion 2.1.2
Sparse Dictionary Learn- ing (SDL) (SAEs, Transcoders) tion 2.1.2
(Sec-
Intrinsically decompos- able models (e.g., sparse activations, modular training) (Section 2.1.2
tion 2.1.2
Intrinsic methods: Perfor- mance costs, superposition
can "sneak through' (Sec-
tion 2.1.2
Continued on next page

---

Table Summary of principal methodologies in mechanistic interpretability (Soction 2) (Continued)
Published in Transactions on Machine Learning Research (09 (2025)
Methodology Category
Principal Method / Approach
Description
Goal
Key Techniques
Examples
Major
Open Problems
Limita-
tions (Sec. 2)
2. Describing Com- ponent Functional Roles
Formulating and hypoth- esizing the functional role of decomposed components and their interactions (Soc-
Causes of activation: Highly activating exam- ples, attribution methods (Grad-CAM, Integrated Gradients, SHAP, LIME), feature synthesis (Sec-
Highly activating examples: Human bias, interpretability illusions, correlational (Sec- tion 2.1.3
Lion 2.1.3
Attribution: Often first- order approximations, can be mislcading manipu- lated, perturbation issues
tion 2.1.3
Downslream effects: Logit lens, causal interventions (patching, ablation), ob- serving sequential behav- ior (steering, patchscopes) (Section 2.1.3
(Section 2.1.3)
Feature synthesis: May not reffect natural data, struggle with trojans (Section 2.1.3
Logit lens: Measures direct ef-
fect only (Section 2.1.3
Causal interventions: Can be expensive (Section 2.1.3
Chain-of-thought, Not always faithful to internal process
(Section 2.1.3
Continued on next page

---

Table Summary of principal methodologies in mechanistic interpretability (Soction 2) (Continued)
Published in Transactions on Machine Learning Research (09 (202))
Methodology Category
Principal Method / Approach
Description
Goal
Key Techniques
Examples
Major
Open Problems
Limita-
tions (Sec. 2)
3. Validation of De- scriptions
Testing if hypothescs about component functions are cor- rect and distinguishing faith- ful from merely plausible ex- planations (Section 2.1.4).
Predicting activa-
Difficulty distinguishing faith- ful from plausible explanations (Section 2.1.4).
tions fion/couune counteerrars (Section 2.1.4
Explaining unusual fail-
Interpretability illusions (Sec-
ures adversarial examples (Section 2.1.4
tion 2.1.4)
High cost (time, cognitive validation (Sec-
Handcrafting network
labor) for
replacements
(Sec-
Lion 2.1.4
tion 2.1.4)
Lack of consensus on model or-
Testing on ground truth
ganisms (Section 2.1.4).
models (Section 2.1.4.
Current benchmarks may need adaptation for mechanistic ex-
Achieving engineering goals
(competitively)
planations (Section 2.1.4).
88
(Section 2.1.4
Risk that complete reverse engineering is too plex/ plex/numerous components (see end of Section 2.1.4
Using 'model organisms'
(Section 2.1.4
Interpretabill bench-
marks (Section 2.1.4
Continued on next page

---

Table Summary of principal methodologies in mechanistic interprotability (Soction 2) (Continued)
Published in Transactionn on Machine Learning Research (09 (2025)
Methodology Category
Principal Method / Approach
Description
Goal
Key Techniques
Examples
Major
Open Problems tions (Sec. 2)
Limita-
Based Interpretability
Concept-based
Identifying network compo- nents that correspond to pre- defined, human-interpretable concopts (Section 2.2.1
Probes
Linear
probes
(Sec-
Requires carefully chosen data for well-defined concepts (hu- man effort) (Section (.2.2)
tion 2.2.1
Concept Activation Vec-
tors (Section 2.2.1
Detects correlations, not nec- essarily causal variables (Sec- tion 2.2.3
information-theoreti
probing (Section 2.2.1
Structural probing (Soc-
Risk of spurious correlations due to high dimensionality
lion 2.2.1
Coiisass--Cossstentt
(Section 2.2.3 Causal variants require more specialized data (Sec-
Scarch (CCS) (Soc-
lion 2.2.2
Causal probing variants using counterfactual data, distributed alignment search (Section 2.2.3
tion 2.2.3
Concept-base based Intrinsic pretability
Building models that are in- herently interpretable with re- spect to specific human con- cepts from the training pro- cess itself (Section 2.2.4
Inter-
Concept Bottleneck Mod- els (Section 2.2.4
May not be possible to prespecify all relevant concepts/strune /structurussss (Soc- tion 2.2.4)
Training for particu- lar causal structures (Section 2.2.4
Gradient routing (Sec- tion 2.2.4
Continued on next page

---

Table Summary of principal methodologies in mechanistic interpretability (Soction 2) (Continued)
Published in Transactions on Machine Learning Research (09 (2025)
Methodology Category
Principal Method / Description
Goal
Key Techniques
Examples
Major Open Problems Limita- tions (Sec. 2)
Approach
Proccduralizing & Automating Interpretability
Circuit Pipelines
Discovery
A structured procedure to identify and describe how a neural network performs a specific task of interest (Soc- lion 2.3
Steps: Task Definition, Decomposition (DAG), Identify task-relevant subgraphs (e.g. itera- tive activation patching, integrated gradients), Iterative doscription- validation, Final Val- idation (faithfulness, minimality, completeness)
Task definition is concept- based, may not be true 'reverse engineering' (Sec- tion 2.3
Relies on flawed network decomposition methods (Sec- tion 2.3
Low circuil faithfulness in practice (Section 2.3
Scalable methods for identi- fying components (e.g., attri- bution patching) are approxi- mate (Section 2.3
(Section 2.3 Automated variants like
ACDC (Section 2.4
Struggles with 'backup' and 'negative" behavior (Sec- tion 2.3 "Streetlight interpretability' focus on simpler, easier- to-interpret (Section 2.3
tasks/models
Continued on next page

---

Table Summary of principal methodologies in mechanistic interpretability (Soction 2) (Continued)
Published in Transactions on Machine Learning Research (09 22025)
Methodology Category
Principal Method /
Description
Goal
Key Techniques
Examples
Major Open Problems tions (Sec. 2)
Limita-
Approach
Automating Inter- pretability Steps
Using AI/LLMs to automate parts of the mechanistic inter- pretability research process to handle scale and reduce man- ual elfort (Section 2.4
Automating feature de- scription and validation (e.g. LLMs explaining neurons neurons/latento /latents based on activating examples) (Sec-
Current automated methods are imperfect and may re- quire further methodological progress to yield satisfactory explanations (see end of Sec- tion 2.4 Automated circuit discovery doesn't automate describing functional roles of components (Section 2.4
tion 2.4
Automating parts of circuil discovery (e-g.,
ACDC identifying sub- graphs) (Section 2.4
88


Reviews:
[{"review_id": "m47rJUPuJ0", "reviewer": "Reviewer_1PFg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The formalisation of the topics in interpretability research is a relevant need in the community, hence the paper fills an important gap\n- The literature review covers a broad range of existing work\n- The formalisation of sparse dictionary learning is interesting and relevant, and so are the concepts of \"imperfect interpretability\" and \"model organisms\".\n\nWeaknesses:\n- The structure of the paper can be vastly improved as right now it reminds of a collection of summaries. As of now, the authors mention the main publications and research directions with endless lists of bullet points and mini paragraphs. While I see the struggle of handling such a large amount of literature, they could have done a better job at summarising, grouping and collecting works in collective views, literature maps and summary tables. For example, point 1 on page 18 actually generalises to many of the other methods that have been described in the previous pages, but here it is merely presented as a component of only one of them. What is the point of compiling a literature review if no such level of abstraction is introduced to simplify the understanding of the readers on a given topic? \n- The figures are very poor quality. Figure 2 is just white. Figure 6 is broken. Apart from these two, I find the other ones hard to interpret as there is no legend of the symbolism being used, and a lot is left to intuition. This can be improved. \n-  The discussion lacks depth if there is no argument against the need for interpretability. There may be a part of the model that will always remain impossible to interpret, as there is a misalignment between how humans and machine works, and there will only be. I fear this paper is missing the opportunity to shift the discussion on whether there is the need for \"perfect interpretability\", which may as much be unachievable. As in most engineering systems, one mainly cares about the safe boundary or margin, which ensure smooth and safe functioning of the system, rather than a perfect functioning. A similar concept could have been expanded in the context of \"imperfect interpretability\", argumenting on what and how such safe margin could be defined and tested by future research. The authors also touch on it in figure 5, highlighting \"irrelevant components\", but fail to describe this concept further and how does it tie to assessing their relevance in the model (e.g. to pruning) and on whether they could, at times (if pruning is followed by a degradation in  performance) they may represent what is left \"uninterpretable\" in the model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is an extensive review of mechanistic interpretability. It covers a large range of publication and it contributes by creating a solid foundation of literature that describes the approaches, limitations and existing challenges in the field. -", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Following the weaknesses, I suggest:\n- revising the paper structure, enhancing the level of abstraction and finding connections between the different lines of works, or putting them in comparison side by side with the aid of tables and other visualizations\n- improving all the figures by clarifying the legend, ensuring that they are visible and detailed enough to provide high quality information to the reader\n- Enhance the discussion on the need of interpretability or the possibility that we will have to live with an uninterpretable boundary\n\nThe authors could consider extending the references to the discovery of concepts, e.g.:\n- how it ties to superposition and polysemanticity: \nGraziani, M., O'Mahony, L., Nguyen, A. P., Müller, H., & Andrearczyk, V. Uncovering Unique Concept Vectors through Latent Space Decomposition. Transactions on Machine Learning Research.\n- how it can be used for model editing: \nOldfield, J., Tzelepis, C., Panagakis, Y., Nicolaou, M., & Patras, I. PandA: Unsupervised Learning of Parts and Appearances in the Feature Maps of GANs. In The Eleventh International Conference on Learning Representations.\n- how it can be obtained and iteratively refined with NMF:\nFel, T., Picard, A., Bethune, L., Boissin, T., Vigouroux, D., Colin, J., ... & Serre, T. (2023). Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2711-2721).\n\nAdditionally, it would also be worth mentioning the work on meta-evaluations:\n- Hedström, Anna, et al. \"Quantus: An explainable ai toolkit for responsible evaluation of neural network explanations and beyond.\" Journal of Machine Learning Research 24.34 (2023): 1-11.", "broader_impact_concerns": "No main concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UFPeYK2l0g", "reviewer": "Reviewer_ZeP2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nWriting is clear\n\nCoverage is thorough\n\nSeveral important though usually under-discussed ideas are highlighted including: the need for validation, the use of 'model organisms', and the need for more diverse model types\n\nWeaknesses:\n\nIt is long. Though on the whole it is not terribly repetitive or long-winded, so this may not be a weakness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This piece is a thorough review of the current state and potential future of mechanistic interpretability, including its implications across other disciplines. It cites a wealth of relevant literature.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think it is crucial that the authors give some space to the possibility that some of the goals of mech-interp may simply be impossible to achieve. Especially when describing future potential uses of mech-interp, we need to accept that current challenges might actual reflect inherent difficulties in understanding a large distributed system that will not be resolved by better methodology. Perhaps put another way, even if we can \"carve the model at its joints\" it may still not be in a very usable shape for us. More on this can be found here: https://arxiv.org/abs/1907.06374 \n\nI think the authors are also missing an additional interdisciplinary benefit of advances in mech-interp which is that it can help neuroscientists decide how to analyze real neural networks. Methods won't be able to be directly ported, but certainly concepts and approaches can be shared. More on this here:  https://www.sciencedirect.com/science/article/pii/S1389041723000906 I think this would be a strong addition to the piece.\n\nThe authors may want to add this recent work to their references. I don't think it changes any of the core claims, but does represent a lot of them: https://transformer-circuits.pub/2025/attribution-graphs/methods.html \n\nMaybe explain or give more examples of attribution methods after this: \"Attribution methods are necessary for causal explanations but are often difficult to interpret\"\n\nword missing: \"networks is that the latents identified by depend on the data set used to train them.\"", "broader_impact_concerns": "Implications are appropriately addressed, especially when warning against over stating what mech-interp is capable of.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yI1KtEUb3l", "reviewer": "Reviewer_eDJL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nWhile there have been many recent reviews/surveys of MI, this one offers a sharp assessment of the current issues and shortcomings on the methodological front.  I found Sec 2, and particularly the frank discussion of issues with SDL, to be insightful and a pleasure to read.  \n\n## Weaknesses\nIn contrast to the sharp and targeted assessment of Sec 2, Secs 3 and 4 were often about implications of understanding models generally instead of MI specifically.  I would recommend zeroing back in on the specific role of MI in these applications and societal implications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission is a forward-looking assessment of mechanistic interpretability (MI) research.  At a high level, the first half discusses open problems in MI methodology (Sec 2) while the second half focuses on future applications and societal implications (Secs 3,4).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The weakness above should be address or rebutted, as well as the following more minor points:\n- Please explain the term “streetlight interpretability”, which is likely not standard terminology to a broader audience.\n- The review defines/frames MI in terms of generalization.  Page 2: “In this review, we use the term ‘mechanistic interpretability’ in a technical sense, referring specifically to work that investigates the mechanisms underlying neural network generalization”, and then multiple times in Sec 2 generalization is positioned as central to MI.  While MI can certainly inform our understanding of generalization, it is a broader pursuit to understand the internal mechanisms of networks.  Please correct me if I’m misreading.\n- Nit: page 26, “Current approaches for knowledge discovery from data involve statistical or causal analysis, dimensionality reduction, or using machine learning models that are inherently interpretable. These techniques can be valuable, but are influenced by human priors, typically assume linear relationships between variables, and cannot handle massive multimodal data. **On the other hand, neural networks can do these things.**”  The bolded sentence is awkward if you mean it as neural networks ameliorate all three issues, as “do these things” does not fit with being “influenced by human priors” or “assume linear relationships”.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2iolYjButX", "reviewer": "Reviewer_uHkd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n  - The paper is overall well-written, easy to follow and discusses the status-quo of mechanistic interpretability.\n  - The discussion on sparse dictionary learning (SDL) is detailed and a useful contribution\n  - The survey addresses relevant broader scientific, engineering, and societal goals, summarizing the importance of interpretability for the use of AI.\n\n**Weaknesses**\n  - The survey mostly focuses on a review of existing problems and approaches that have not worked, but remains quite vague with regard to what methodological approaches could provide a way out. This has been widely recognized as a key issue of mech interp, and the paper does not sufficiently address this challenge.\n  - While I appreciate the AI for the Sciences perspective in the introduction, the remainder of the survey does not follow-up on this motivating direction, which makes it appear unconnected. What concrete scientific problems do the authors think can be addressed effectively with mechanistic interpretability (as compared to other interpretability approache and the “Explainable AI for the Sciences” community)\n  - The paper lacks a clear methodological summary of the status-quo beyond the vague distinction into  ‘reverse engineering’ and ‘concept-based interpretability’ (Section 2). The focus on the human perspective for ‘concept-based interpretability’ is not clear. Should an interpretability approach not focus on providing any concept, even if not aligned with human perspectives?\n  - The survey lacks a dedicated discussion and connection to frameworks developed for evaluating explanations in the context of attributions, e.g. how do commonly used evaluation metrics like “faithfulness” and “plausibility” transfer to mech interp? Section 2.3 touches upon these points (see “5. Final Validation”), but only refers to the limited recent works, see, e.g [Hed23, Vil21] for broader reviews.\n\n    - [Hed23] Hedström, A., Weber, L., Krakowczyk, D., Bareeva, D., Motzkus, F., Samek, W., ... & Höhne, M. M. C. (2023). Quantus: An explainable ai toolkit for responsible evaluation of neural network explanations and beyond. Journal of Machine Learning Research, 24(34), 1-11. \n    - [Vil21] Vilone, G., & Longo, L. (2021). Notions of explainability and evaluation approaches for explainable artificial intelligence. Information Fusion, 76, 89-106.\n\n  - Figure 6 does not convey any relevant visual information as it summarizes the headings of Section 3.\n  - What’s the use of discussing and dissecting a fairly recent direction of “sparse dictionary learning” and SAEs that has been advocated for by the mech interpretability community from 2021 onwards, to realize that it comes with many flaws that remain difficult to resolve in the context of deep learning models (see Figure 4). One conclusion from this survey may be that this research direction has not been able to fulfil the promises, what are alternatives that may be able to better reconcile these methodological issues. Sparse coding and compressed sensing have a long and theoretically well-grounded history which should be discussed in this context as many of the encountered issues have been identified previously in other contexts, e.g. source separation.\n3.2.1 \n  - Short-sighted survey with respect to undesired black-box behaviors, see e.g. [Lap19, Gei20]\n\n    - [Lap19] Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K. R. (2019). Unmasking Clever Hans predictors and assessing what machines really learn. Nature communications, 10(1), 1096.\n\n    - [Gei20] Geirhos, R., Jacobsen, J. H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11), 665-673.\n\n  -  The paper lacks a clear prioritization, in my view the main priority should be to ground mech interp in better methodology, yet on this point the paper remains vague and focuses on a summary of all things that have not worked.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper outlines key open challenges in mechanistic interpretability, the study of how neural networks implement computations internally. It highlights two main methodological approaches (reverse engineering and concept-based interpretability) and discusses their current limitations, such as the shortcomings of sparse dictionary learning. The authors emphasize the need for better theoretical foundations, scalable tools, and validation techniques to reliably understand and control model behavior, especially in the context of safety-critical applications.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I think in its current state, the paper is of limited interest to the ML/TMLR community as it remains fairly superficial (see Weanesses above) with regards to methodological machine learning research and instead focuses on providing a summary of mech interp in 2025.\nI have summarized below the key changes that, in my opinion, must be addressed carefully. Collectively, these issues would require a **major revision** of the current draft:\n\n- Provide concrete methodological proposals for advancing mechanistic interpretability.\n\n- Strengthen the connection between the scientific motivation and the rest of the paper, Explainable AI for the Sciences and what does mech interp really contribute to deepening our understanding of AI from first principles?\n\n- Expand the methodological summary beyond the reverse engineering vs. concept-based distinction.\n\n- Clarify the role of human vs. non-human concepts in concept-based interpretability.\n\n-  Include a more extensive discussion on evaluation frameworks and metrics like faithfulness and plausibility, especially in reference to more than a decade of research in developing faithful attribution approaches.\n\n- Remove/revise Figure 6 to convey more meaningful visual information.\n\n- Discuss alternative methods to sparse dictionary learning and relate to prior work in sparse coding and source separation, should this direction be pursued?\n\n- Broaden the discussion on undesired black-box behavior.\n\n- Prioritize methodological grounding as a central challenge for the field.\n\n\nSome additional related works that were missed\n\nSection 2.1.2\n  - Gradient-based Explanations\n    - [Bae10] Baehrens, D., Schroeter, T., Harmeling, S., Kawanabe, M., Hansen, K., & Müller, K. R. (2010). How to explain individual classification decisions. The Journal of Machine Learning Research, 11, 1803-1831.\n\n  - Theoretical view on decomposition of neural networks\n    - Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K. R. (2017). Explaining nonlinear classification decisions with deep taylor decomposition. Pattern recognition, 65, 211-222.\n\nSection 2.1.2\n  - Observing the effects of components from a graph-perspective, decomposition into Higher-order structure in Graphs and Transformers\n    - [Tsa18] M. Tsang, D. Cheng and Y. Liu, \"Detecting statistical interactions from neural network weights\", Proc. Int. Conf. Learn. Representations, 2018.\n    - [Ebe20] Eberle, O., Büttner, J., Kräutli, F., Müller, K. R., Valleriani, M., & Montavon, G. (2020). “Building and interpreting deep similarity models”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3), 1149-1161.\n    - [Jan20] J. D. Janizek, P. Sturmfels and S. Lee. \"Explaining explanations: Axiomatic feature interactions for deep networks\", CoRR, 2020.\n    - [Sch21] Schnake, T., Eberle, O., Lederer, J., Nakajima, S., Schütt, K. T., Müller, K. R., & Montavon, G. (2021). Higher-order explanations of graph neural networks via relevant walks. IEEE transactions on pattern analysis and machine intelligence, 44(11), 7581-7596.\n    - [Vas24] A Vasileiou and O Eberle. “Explaining Text Similarity in Transformer Models”, NAACL 2024.\n    - [Fum24] Fumagalli, F., Muschalik, M., Kolpaczki, P., Hüllermeier, E., & Hammer, B. (2024). KernelSHAP-IQ: Weighted Least-Square Optimization for Shapley Interactions. arXiv preprint arXiv:2405.10852.\n\n  - Section 2.2.3. The notion of “spurious correlations” seems relevant here.\n\n- Section 2.3. References to works on theoretically grounded subgraph detection in the context of interactions of features/components for interpretability\n\n    - [Sch21] Schnake, T., Eberle, O., Lederer, J., Nakajima, S., Schütt, K. T., Müller, K. R., & Montavon, G. (2021). Higher-order explanations of graph neural networks via relevant walks. IEEE transactions on pattern analysis and machine intelligence, 44(11), 7581-7596.\n\n    - [Fum25] Fumagalli, F., Muschalik, M., Frazzetto, P., Strotherm, J., Hermes, L., Sperduti, A., ... & Hammer, B. (2025). Exact Computation of Any-Order Shapley Interactions for Graph Neural Networks. ICLR 2025.", "broader_impact_concerns": "I have no serious concerns regarding broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)