Mechanistic Interpretability for AI Safety - A Review
Authors: Leonard Bereska
Stratis Gavves
Venue: Accepted by TMLR ()

Abstract: Understanding AI systems' inner workings is critical for ensuring value alignment and safety. This review explores mechanistic interpretability: reverse engineering the computational mechanisms and representations learned by neural networks into human-understandable algorithms and concepts to provide a granular, causal understanding. We establish foundational concepts such as features encoding knowledge within neural activations and hypotheses about their representation and computation. We survey methodologies for causally dissecting model behaviors and assess the relevance of mechanistic interpretability to AI safety. We examine benefits in understanding, control, alignment, and risks such as capability gains and dual-use concerns. We investigate challenges surrounding scalability, automation, and comprehensive interpretation. We advocate for clarifying concepts, setting standards, and scaling techniques to handle complex models and behaviors and expand to domains such as vision and reinforcement learning. Mechanistic interpretability could help prevent catastrophic outcomes as AI systems become more powerful and inscrutable. 
For an HTML version of the paper, visit https://leonardbereska.github.io/blog/2024/mechinterpreview/.

Published in Transactions on Machine Learning Research (08/2024)
Mechanistic Interpretability for AI Safety A Review
Leonard Bereska (leonard, Jeeesan egavves) @raa..l University of Amsterdam
Efstratios Gavves
Reviewed on OpenReview: https: // openrevee.. net/ forun? id= ePUVetPRu6
//
Abstract
Understanding AT systems' inner workings is critical for ensuring value alignment and safety. This review explores mechanistic interpretability: reverse engineering the computational mechanisms and representations learned by neural networks into human-undersaandan algorithms and concepts to provide a granular, causal understanding. We establish founda- tional concepts such as features encoding knowledge within neural activations and hypothe- ses about their representation and computation. We survey methodologies for causally dissecting model behaviors and assess the relevance of mechanistic interpretability to Al safety. We examine benefits in understanding, control, alignment, and risks such pability gains and dual-use concerns. We investigate challenges surrounding scalability, automation, and comprehensive interpretation. We advocate for clarifying concepts, setting standards, and scaling techniques to handle complex models and behaviors and expand to domains such as vision and reinforcement lcarning. Mcchanistic interpretability could help prevent calastrophic outcomes as AT systems become more powerful and inscrutable. For an HTML version of the paper, visit https: attps://leenardderees.. wiw.eess.a...tt....ee.eeeee 1o/blog//02// mechinterpreview/
1 Introduction
As AI systems rapidly become more sophisticated and general (Bubeck et al., 2023; Bengio et al., 2023), advancing our understanding of these systems is crucial to ensure their alignment (Ji et al., 2024) with human values and avoid catastrophic outcomes (Hendrycks et al., 2023; Hendrycks & Mazeika, 2022). The field of interpretability aims to demystify the internal processes of AI models, moving beyond evaluating performance alone. This review focuses on mechanistic interpretability, an emerging approach within the broader interpretability landscape that strives comprehensively specify the computations underlying deep neural networks. We emphasize that understanding and interpreting these complex systems is not merely academic endeavor il 's a societal imperalive to ensure AT remains Iruslworthy and beneficial.
The interpretability landscape is undergoing a paradigm shift akin to the evolution from behaviorism to cognitive neuroscience in psychology. Historically, lacking tools for introspection, psychology treated the mind as a black box, focusing solely on observable behaviors. Similarly, interpretability has predominantly relied on black-box techniques (Casper et al., 2024), analyzing modcls based on input-output relationships or using autribution methods that, while probing deeper, still neglect the model's internal architecture. However, just advancements in neuroscience allowed for deeper understanding of internal cognitive processes, the field of interpretability is now moving towards a more granular approach. This shift from surface-level analysis to a focus on the internal mechanics of deep neural networks characterizes the transition towards inner interpretability (Räuker et al., 2023).
Mcchanistic interpretability, as an approach inner interprotability, aims to completely specify a neural network's computation, potentially in a format as explicit as pseudocode (also called reverse engineering). striving for a granular and precise understanding of model behavior. It distinguishes itself primarily through

---

Phibishiee in Commsactioo on Machine Learning Ressearci (08/2024)
-input
relations
relations
gradients
probes
mechanism
+
f(
-+ *
2
output
Behaan Rehavioral
Attributinn
Attributional
Concept-based
Mechanistic
Figure 1: Interpretability paradiet offer distinct lenses for understanding neural networks: inhaviorad an- alyzes cont-ouuieee relations; Attributional quantifics individual input feature iefunendee Con b me
Concept-based
identifies high-level sepresentations governing behavior; Mechanistic uncovers precise causal mechanisms from inputs to outputs.
its ambition for comprehensive reverse engineering and its strong motivation towards AT safety. Our review serves as the first comprehensive exploration of mechanistic interpretability research, with the most accessible introductions currently scattered in a blog or list format (Olah, 2022; Nanda, 2022d; Olah et al., 2020; Sharkey et al., 2022a; Olah et al., 2018; Nanda, 2023f; 2024). Concurrently, Ferrando et al. (2024) and Rai et al. (2024) have also contributed valuable reviews giving concise, tochnical introductions to mechanistic m
interpretability in transformer- language models. Our work complements, these efforts by synthesizing the research (addressing the 'research debt" (Olah & Carter, 2017)) and providing a structuredi accessible, and comprehensive introduction for AI rescarchers and practitioners. "
The structure of this paper provides cohesive overview of mechanistic interpretability, situating the mecha- nistic approach in the broader interpretability landscape (Section 2), presenting core concepue and hypotheses (Section 3), explaining methods and techniques (Section 4), presenting a taxonomy and survey of the cur- rent field (Section 5), exploring relevance to AI safety (Section 6), and addressing challenges (Section 7) and and future directions (Section:8) 8).
2
Interpretability Paradigms from the Outside In
Interprctain
We encounter a spectrum of interpretability paradigms for decoding AI systems' decision-making, ranging from external black-box techniques to internal analyses. We contrast these paradigms with mechanistic interpretability highlighting its distinct causal bottom-up perspective within the broader interpretability landscape
landscape (see Figure 1).
Behavioral interpretability bytey treats the model as a black box, analyzing input-outpur relations. Techniques such as minimal pair analysis (Warstadt et al., 2020), sensitivity and perturbation nntturation analysis (Casalicchio et al., 2018) examine maput-ouue relations to assess the model's robustness and variable dependencies (Shapley, 1988; Ribeiro et al., 2016; Covert et al., 2021). Its model-agnostic nature is practical for complex or seppriitatt models but lacks insight into internal decision processos and causal depth (Jumelet, 2023).
Iitributional interpretability aims to explain outputs by tracing predictions to individual input contri- butions using gradients. Raw gradients can be discontinuous sensitive to slight perturbations. Therefore, iiscontimmnn
tediiigue such as GraothGrad (Smilkov et al., 2017) and Integrated Gradients (Sundararajan et al., 2017) average across gradients. Other popular cechniiqunt are layer-wise relevance propagation (Bach et al., 2015), DeepLIFT (Shrikumar et al., 2017), or GradCAM (Selvaraju et al., 2016). Attribution enhances transparehn by showing input feature influence without requiring an understanding of the internal structure, enabling ei
techniques
enabling
decision validation, compliance, and trust while serving as a bias detection tool, but also has eeeamnntal limitations (Bilodeau et al., 2024). --
2

---

Published in Transactions on Machine Learning Research (08/2024)
representation
emergence
defining features
linearity
superposition
features
simulation
(alternative)
feature
computation
circuit
universality
motif
prediction orthogonality
Figure 2: Overview key concepts and hypotheses mechanistic interpretability, organized into four sub- section (pink boxes): defining features (Section 3.1), representation (Section 3.2), computation (Section 3.3), and emergence (Section 3.4). In turquoise, it highlights definitions like features, circuits, and motifs, and in orange, it highlights hypotheses like linear representalion, superposition, universalily, simulation, and predic- tion orthogonality. Arrows show relationships, e.g., superposition enabling an alternative feature definition or universality connecting circuits and motifs.
Concept-based interpretability adopts a top-down approach to unraveling a model's decision decision-making processes by probing its learned representations for high-level concepts and patterns governing behavior. Techniques include training supervised auxiliary classifiers (Belinkov, 2021), employing unsupervised con- trastive and structured probes (see Section 4.2) explore latent knowledge (Burns et al., 2023), and using neural representation analysis quantify the representational similarities between the internal representa- tions learned by different neural networks (Kornblith et al., 2019; Bansal et al., 2021). Beyond observational analysis, concept-based interpretability resentation engineering (Zou et al., 2023) - potentially enhancing safety by upregulating concepts such as honesty, harmlessness, and morality.
enable manipulation of these representations also called rep-
Mechanistic interpretability bottom-up approach that studies the fundamental components of models through granular analysis of features, neurons, layers, and connections, offering intimate view of opera- tional mechanics. Unlike concept-based interpretability, it aims to uncover causal relationships and precise computations transforming inputs into outputs, often identifying specific neural circuits driving behavior. This reverse engineering approach draws from interdisciplinary fields like physics, neuroscience, and systems biology to guide the development of transparent, value-aligned Al systems. Mechanistic interpretability is the primary focus of this review.
3 Core Concepts and Assumptions
This section introduces the key concepts and hypotheses of mechanistic interpretability, summarized in Figure 2. We start by defining features the basic units of representation (Section 3.1). We then examine the nature of these features, including the challenges posed by polysemantic neurons and the implications of the superposition and linear representation hypotheses (Section 3.2). Next, we explore computation through circuits and motifs, considering the universality hypothesis (Section 3.3). Finally, we discuss the implications for understanding emergent properties, such as internal world models and simulated agents with potentially misaligned objectives (Section 3.4).
3

---

Published in Transactions on Machine Learning Research (08/2024)
3.1 Defining Features Representational Primitives
Features as fundamental units of representation. The notion of feature in neural networks is cen- tral yet elusive, reflecting the pre-paradigmatic state of mechanistic interpretability. We adopt the notion of features as the fundamental units of newral network representations, such that features cannot be fur- ther disentangled into simpler, distinct factors. These features are core components of neural network's representation, analogous to how cells form the fundamental unit of biological organisms (Olah et al., 2020).
Definition 1: Feature
Features are the fundamental units of neural network representations that cannot further decom- posed into simpler independent factors.
Concepts as natural abstractions. The world consists of various entities that can be grouped into categories or concepts based on shared properties. These concepts form high-level summaries like "tree" 'velocity,' allowing compact world representations by discarding many irrelevant low-level details. Neural networks can caplure and represent such natural abslractions (Chan et al., 2023) through their learned features, which serve as building blocks of their internal representations, aiming to capture the concepts underlying the data.
Features encoding input patterns. In traditional machine learning, features are understood as charac- teristics or attributes derived directly from the input data stream (Bishop, 2006). This view particularly relevant for systems focused on perception, where features map closely to the input data. However, in more advanced systems capable of reasoming with abstractions, features may emerge internally within the model as representational patterns, even when processing information unrelated to the input. In this context, fea- tures are better conceplualized as any measurable property or characteristic of a phenomenon (Olah, 2022), encoding abstract concepts rather than strictly reflecting input attributes.
Features representational atoms. A key property of features their irreducibility, meaning they cannot be decomposed into expressed as combination of simpler, independent factors. In the context of input-related features, Engels et al. (2024) define feature as irreducible if it cannot be decomposed into or expressed as a combination of statistically independent patterns or factors in the original input data. Specifically, a feature is reducible transformations reveal its underlying pattern, which can be separated into independent co-occurring patterns or is a mixture of patterns that never co-occur. We propose generalizing this notion of irroducibility features encoding abstract concepts not directly tied to input patterns, such that features cannot be reduced to combinations or mixtures of other independent components within the model's representations.
Features beyond human interpretability. Features could be defined from a Iumman-centric perspective as semantically meaningful, articulable input patterns encoded in the network's activation space (Olah, 2022). However, while cognitive systems may converge on similar natural abstractions (Chan al., 2023), these need not necessarily align with human-interpretable concepts. Adversarial examples have been interpreted non-interpretable features meaningful to models but not humans. Imperceptible perturbations fool networks, suggesting reliance alien representational patterns (Ilyas et al., 2019). As models surpass human capabili- ties, their learned fcatures may become increasingly abstract, encoding information in ways incongruent with human intuition (Hubinger, 2019a). Mechanistic interpretability aims to uncover the actual representations learned, even if diverging from human concepts. While human-interpretable concepts provide guidance, a noo-humann-centree perspective that defines features as independent model components, whether aligned with human concepts or not, is a more comprehensive and anture-proof approach.
3.2 Nature of Features: From Monosemantic Neurons Non- Linear Representations
Neurons as Computational Units? In the architocture of neural networks, neurons are the natural computational units, potentially representing individual features. Within neural network representation
4

---

Published in Transactions on Machine Learning Research (08/2024)
Privileged
Privileged
Non-Privileged
Non-privileged
monosemantic neurons:
polysemantic neurons:
Figure Contrasting privileged and non-privileged bases. In a non-privileged basis, there is no reason to expect features to be basis-aligned calling basis dimensions neurons has meaning. In privileged basis, the architecture treats basis directions differently features can but need not align with neurons (Bricken et al., 2023). Leftmost: Privileged basis; individual features (arrows) align with basis directions, resulting in monosemantic neurons (colored circles). Middle left: Privileged basis, where despite having more features than neurons, some neurons are monosemantic, representing individual features, while others are polysemantic (overlapping gradients), encoding superposition of multiple features. Middle right: Non- privileged basis where, even when the number of features equals the number of ncurons, the lack of alignment between the feature directions and basis directions results in polysemantic neurons encoding combinations of features. Rightmost: Non-privee Non-privileged, polysemantic neurons as feature directions do not align with neuron basis.
h E R", the basis directions are called neurons. For neuron to be meaningful, the basis directions must functionally differ from other directions in the representation, forming a privileged basis where the basis vectors are architecturally distinguished within the neural network layer from arbitrary directions in activation space, as shown in Figure 3. Typical non-linear activation functions privilege the basis directions formed by the neurons, making it meaningful to analyze individual neurons (Elhage et al., 2022b). Analyzing neurons can give insights into a network's 's functionality (Sajjad et al., 2022; Mu & Andreas, 2020; Dai et al., 2022; Ghorbani & Zou, 2020; Voita et al., 2023; Durrani et al., 2020; Goh et al., 2021; Bills et al., 2023; Huang et al., 2023).
Monosemantic and Polysemantic Neurons. A neuron corresponding to single semantic concept is called monosemantic. The intuition behind this term comes from analyzing what inputs activate given neuron, revealing its associated semantic meaning or concept. If neurons were the representational primitives of neural networks, all neurons would be monosemantic, implying a one-to-one relationship between neurons and features. Comprehensive interpretability would be as tractable as characterizing all neurons and their connections. However, empirically, especially for transformer models (Elhage et al., 2022b), neurons are often observed to be polysemanlic, i.e., associated with multiple, unrelated concepts (Arora et al., 2018; Mu & Andreas, 2020; Elhage et al., 2022a; Olah et al., 2020). For example, single neuron may be activated by both images of cats and images of cars, suggesting it encodes multiple unrelated concepts. Polysemanticity contradicts the interpretation of neurons as representational primilives and, in practice, makes il challenging to understand the information processing of neural networks.
To understand the widespread occur- rence of polysemanticity neural networks, several hypotheses have been proposed:
Exploring Polysemanticity: Hypotheses and Implications.
i.) One trivial sccnario would be that feature directions are orthogonal but not aligned with the basis directions (neurons). There is no inherent reason to assune that features would align with neurons in a poniiievieeged basis, where the basis vectors are not architecturally distinguished. However, even in a privileged basis formed by the neurons, the network could represent features not in the standard basis but as linear combinations of neurons (see Figure 3, middle right).
(.) An alternative hypothesis posits that redundancy due to noise introduced during training, such random dropout (Srivastava al., 2014), can lead to redundant representations and, consequently,
5

---

Published in Transactions on Machine Learning Research (08/2024)
to polysemantic neurons (Marshall & Kirchner, 2024). This process involves distributing single feature across several neurons rather than isolating it into individual ones, thereby encouraging polysemanticity.
Wi.
Finally, the superposition hypothesis addresses the limitations in the network's representative capac- ity the number of neurons versus the number of crucial concepts. This hypothesis argues that the limited nunber of neurons compared to the vast array of important concepis necessitales a form of compression. As a result, an n-dimensional representation may encode features not with the n basis directions (ncurons) but with the x exp(n) possible almost orthogonal directions (Elhage et al., 2022b), leading to polysemanticity.
Hypothesis 1: Superposition
Neural networks represent more features than they have neurons by encoding features in overlapping combinations of neurons.
Superposition Hypothesis. The superposition hypothesis suggests that neural networks can leverage hign-dimensional spaces to represent more features than their actual neuron count by encoding features almost orthogonal directions. Non-orthogonality means that features interfere with another. However, the benefit of representing many more features than neurons may outweigh the interference cost, mainly when concepts are sparse and non-linear activation functions can error-correct noise (Elhage et al., 2022b).
Observed model
Hypothetical disentangled model
Figure 4: Observed neural networks (left) can be viewed as compressed simulations of larger, sparser networks (right) where neurons represent distinct features. An "almost orthogonal' projection compresses the high- dimensional sparse representation, manifesting as polysemantic neurons involved with multiple features in the lover-dimensions observed model, reflecting the compressed encoding. Figure adapted from (Bricken et al., 2023).
Toy models can demonstrate under which conditions superposition occurs (Elhage et al., 2022b; Scherlis et al., 2023). Neural networks, via superposition, may effectively simulate computation with more neurons than they possess by allocating each feature to linear combination of neurons, creating what is known overcomplete linear basis in the representation space. This perspective superposition suggests that polysemantic models could be seen compressed versions of hypothetically larger neural networks where each neuron represents single concept (see Figure 4), Consequently, an alternative definition of features could be:
Definition 2: Feature (Alternative)
Features are elements that a network would ideally assign to individual neurons neuron count were not limiting factor (Bricken al., 2023). In other words, features correspond the disentangled concepts that larger, sparser network with sufficient capacity would learn represent with individual neurons.

---

Published in Transactions on Machine Learning Research (08/2024)
Toy Model of Superposition
A toy model (Elhage et al., 2022b) investigates the hypothesis that neural networks can represent more features than the number of neurons by encoding real-world concepts in a compressed manner. The model considers high-dimensional vector 2, where each element I; corresponds a feature capturing real-world concept, represented by weight dg. These features are assumedd to have the following properties:
random vector with varying importance determined
2.) Concept sparsity: Real-world concepts occur sparsely.
ú.) More concepts than neurons: The number of potential concepts vastly exceeds the avail- able neurons.
wi. Varying concept importance: Some concepts more important than others for the task at hand.
The imput vector a represents features capturing these concepts, defined by a sparsity level and an importance level a, for each feature 24, reflecting the sparsity and varying importance of the underlying concepts. The model dynamics involve transforming xr into hidden representation h of lower dimension, and then reconstructing it as x':
h Wx. x' = ReLU(( h b).
The network's performance is evaluated using a loss function & weighted by the feature importances 0;; reflecting the importance of the underlying concepts:
L Σ- a.(x; - x1)2
This toy model highlights neural networks' ability encode numerous features representing real-world concepts into a compressed representation, providing insights into the superposition phenomenon observed in neural networks trained on real data.
X
X'
Increasing sparsity
W
W
h
Increasing
h
h
Toy model architecture
Effect of sparsity
Figure 5: Illustration of the toy model architecture and the effects of sparsity. (left) Transformation of a five-feature input vector æ into a two-dimensional hidden representation h, and its reconstruction as x' using the weight matrix W and its transpose, with feature importance indicated by a color gradient from yellow green. (right) The effect of increasing fcature sparsity S the encoding capacity of the netw highlighting the network's enhanced ability to represent features in superposition sparsity increases from 0 to 0.9, illustrated by arrows in the activation space h, which correspond to the columns of the matrix W.
Rescarch on superposition, including works by (Elhage et al., 2022b; Scherlis al., 2023; Henighan et al., 2023), often investigates simplified models. However, understanding superposition in practical, transformer- based scenarios crucial for real-world applications, pioneered by Gurnee et al. (2023).

---

Published in Transactions on Machine Learning Research (08/2024)
The need for understanding networks despite polysemanticity has led to various approaches: One involves training models without superposition (Jermyn et al., 2022), for example, using softmax linear unit (Elhage et al., 2022a) as an activation function to empirically increase the number of monosemantic neurons, but at the cost of making other neurons less interpretable. From a capabilities standpoint, polysemanticity may be desirable as it allows models to represent more concepts with limited compute, making training cheaper. Overall, engincering monosemanticity has proven challenging (Bricken et al., 2023) and may be impractical until we have orders of magnitude more compute available.
Another approach is to train networks in standard way (creating polysemanticity) and use post-hoe analysis to find the feature directions in activation space, for example, with Sparse Autocncoders (SAEs). SAEs aim to find the true, disentangled features in an uncompressed representation by learning a sparse overcomplete basis that describes the activation space of the trained model (Bricken et al., 2023; Sharkey et al., 2022b; Cunningham et al., 2024) (also see Section 4.2).
If not neurons, what are features then? We want identify the fundamental units of neural networks, which we call features. Initially, neurons seemed likely candidates. However, this view fell short, particularly in transformer models where neurons oflen represent multiple concepts, a phenomenon known as polyseman- ticity, The superposition hypothesis addresses this, proposing that due to limited representational capacity, neural networks compress mumerous features into the confined space of neurons, complicating interpretation.
This raises the question: How are features encoded if not in discrele neuron wnils? While a priori features could be encoded in an arbitrarily complex, non-linear structure, a growing body of theoretical arguments and empirical evidence supports the hypothesis that features are commonly represented linearly, i.c., as linear combinations of neurons hence, as directions in representation space. This perspective promises enhance our comprehension of neural networks by providing more interpretable and manipulable framework for their internal representations.
Hypothesis 2:
Linear Representation
Features are directions in activation space, i.e., linear combinations of neurons.
The linear representation hypothesis suggests that neural networks frequently represent high-level features linear directions in activation space. This hypothesis can simplify the undorstanding and manipulation of neural network representations (Nanda et al., 2023b). The prevalence of linear layers in neural network architectures favors linear representations. Matrix multiplication in these layers most readily processes linear features, while more complex non-linear encodings would require multiple layers to decode.
However, recent work by Engels et al. (2024) provides evidence against a strict formulation of the linear representation hypothesis by identifying circular features representing days of the week and months of the year. These multi-dimensional, non-linear representations were shown be used for solving modular arith- metic problems in days and months. Intervention experiments confirmed that these circular features are the fundamental unit of computation in these tasks, and the authors developed methods to decompase the hidden states, revealing the circular representations.
Establishing non-linearity can be challenging. For example, Li et al. (2023a) initially found that in a GPT model trained on Othello, the board state could only be decoded with a non-linear probe when represented in terms of white and black picces, scomingly violating the lincarity assumption. However, Nanda (2023c); Nanda et al. (2023b) later showed that linear probe sufficed when the board state was decoded in terms of "one's own" and 'the opponent's pieces, reaffirming' the linear representation hypothesis in this case. In contrast, the work by Engels et al. (2024) provides clear and convincing existence proof for non-linear, multi-dimennsionll representations in language models.
While the linear representation hypothesis remains a useful simplification, it is important to recognize its limitations and the potential role of non-lincar representations (Sharkey et al., 2022a). As neural networks continue to evolve, ongoing recvaluation of the hypothesis crucial, particularly considering the possible emergence of non-linear features under optimization pressure for interpretability (Hubinger, 2022). Alter-
8

---

Published in Transactions on Machine Learning Research (08/2024)
native perspectives, such the polytope lens proposed by Black et al. (2022), emphasize the impact of non-linear activation functions and discrete polytopes formed by piecewise linear activations potential primitives of neural network representations.
Despite these exceptions, empirical evidence largely supports the linear representation hypothesis in many contexts, especially for feedforward networks with ReLU activations. Semantic vector calculus in word embeddings (Mikolov et al., 2013), successful linear probing (Alain & Bengio, 2016; Belinkov, 2021), sparse dictionary learning (Bricken al., 2023; Cunningham et al., 2024; Deng et al., 2023), and linear decoding of concepts (O (O'Mahony et al., 2023), tasks (Hendel et al., 2023), functions (Todd et al., 2023), sentiment (Tigges et al., 2024), refusal (Arditi et al., 2024), and relations (Hernandez et al., 2023; Chanin et al., 2023) in large language models all point to the prevalence of linear representations. Moreover, linear addition techniques for model steering (Turner et al., 2023; Sakarvadia et al., 2023a; et al., 2023b) and representation engineering (Zou et al., 2023) highlight the practical implications of linear feature representations.
Building upon the linear representation hypothesis, recent work investigated the structural organization of these linear features within activation space. Park et al. (2024) reveal a geometric framework for categorical and hierarchical concepts in large language models. Their findings demonstrate that simple categorical concepts (e.g., mammal, bird) are represented as simplices in the activation space, while hierarchically related concepts are orthogonal. This geometric analysis aligns with earlier observations on feature clustering and splitting in neural networks (Elhage et al., 2022b). It suggests that the linear features are not merely scattered directions but are organized reflect semantic relationships and hierarchies.
3.3 Circuits as Computational Primitives and Motifs as Universal Circuit Patterns
Having defined features as directions in activation space as the fundamental unils of neural network rep- resentation, we now explore their computation. Neural networks can be conceptualized as computational graphs, within which circuits are sub-graphs consisting of linked features and the weights connecting them. Similar how features are the representational primitive, circuits funetion as the computational primitive (Michaud al., 2023) and the primary building block of these networks (Olah et al., 2020).
Observed models trained on similar task and data
Hypothetical disentangled models
Features
Neurons
Motif
Figure 6: Comparing observed models (left) and corresponding hypothetical disentangled models (right) trained on similar tasks and data. The observed models show different neuronal activation patterns, while the dissection into feature-level circuits reveals motif shared circuit pattern emerging across models, hinting at universality models converging on similar solutions based on common underlying principles.
Definition 3: Circuit.
Circuits are sub-graphs of the network, consisting of features and the weights connecting them.

---

Published in Transactions on Machine Learning Research (08/2024)
The decomposition of neural networks into circuits for interpretability has shown significant promise, partic- ularly in small models trained for specific tasks such as addition, seen in the work of Nanda et al. (2023a) and Quirke & Barez (2023). Scaling such comprehensive circuit analysis to broader behaviors in large language models remains challenging. However, there has boen notable progress in scaling circuit analysis of narrow behawiors to larger circuits and models, such as indirect object ident ification (Wang al., 2023) and greater-than computations (Hanna al., 2023) in GPT-2 and multiple-choice question answering (Lieberum et al., 2023).
In search of general and universal circuits, researchers focus particularly more general and transferable behaviors. McDougall et al. (2023)'s work on copy suppression in GPT-2's attention heads sheds light on model calibration and self-repair mechanisms. Davies et al. (2023) and Feng & Steinhardt (2023) focus on how large language models represent symbolic knowledge through variable binding and entity-attribute binding, respectively. Yu et al. (2023); Nanda et al. (2023c); Lv et al. (2024); Chughtai et al. (2024); Ortu et al. (2024) explore mechanisms for factual recall, revealing how circuits dynamically balance pre- trained knowledge with new context ual information. Lan & Barez (2023) extend circuit analysis sequence continuation tasks, identifying shared computational structures across semantically related sequences.
More promisingly, some repeating patterns have shown universality across models and tasks. These universal patterns are called motifs (Olah et al., 2020) and can manifest not just as specific circuits or features but also as higher-level behaviors emerging from the interaction of multiple components. Examples include the curve detectors found across vision models (Cammarata al., 2021; 2020), induction circuits enabling in-context learning (Olsson et al., 2022), and the phenomenon of branch specialization in neural networks (Voss et al., 2021). Motifs may also capture how models leverage tokens for working memory parallelize computations in a divide-and-conquee fashion across representations. The significance of motifs lies in revealing the common structures, mechanisms, and strategies that naturally emerge across neural architectures, shedding light on the fundamental building blocks underlying their intelligence. Figure contrasts observed neural network models with hypothetical disentanglod models, illustrating how a shared circuit pattern can emerge across different models trained on similar tasks and data, hinting at an underlying universalily.
Definition 4: Motif
Motifs are repeated patterns within network, encompassing either features circuits that emerge across different models and lasks.
Universality Hypothesis. Following the evidence for motifs, we can propose two versions for a univer- sality hypothesis regarding the convergence of features and circuits across neural network models:
Hypot 3: Weak Universality
There underlying principles governing how neural networks learn to solve certain tasks. Models will generally converge analogous solutions that adhere to the common underlying principles. However, the specific features and circuits that implement these principles can vary across different models based on factors like hyperparameters, random seeds, and architectural choices.
Hypothesis 4: Strong Universality
The same core features and circuits will universally and consistently arise across all neural network models trained on similar tasks and data distributions and using similar techniques, reflecting a set of fundamental computational motifs that neural networks inherently gravitate towards when learning.
The universality hypothesis posits a convergence in forming features and circuits across various models and fasks, which could significantly ease interpretability efforts in AT. Tt proposes that artificial and biological
10

---

Published in Transactions on Machine Learning Research (08/2024)
neural networks share similar features and circuits, suggesting standard underlying structure (Chan et al., 2023; Sucholutsky et al., 2023; Kornblith et al., 2019). This idea posits that there is fundamental basis in how neural networks, irrespective of their specific configurations, process and comprehend information. This could be due to inbuilt inductive biases in neural networks or natural abstractions (Chan et al., 2023) - concepts lavored by the natural world that any cognitive system would naturally gravitale lowards.
Evidence for this hypothesis comes from cross-species neural structures in neuroscience, where similar neural found in different species (Kirchner, 2023). Additionally, machine learning models, including neural networks, tend to converge on similar features, representations, and classifications across different tasks and architectures (Chen et al., 2023a; Hacohen et al., 2020; Li et al., 2015; Bricken et al., 2023). Marchetti et al. (2023) provide mathematical support for emerging universal features.
structures and functions
While various studies support the universality hypothesis, questions remain about the extent of feature and circuit similarity across different models and tasks. In the context of mechanistic interpretability, this hypothesis has been investigated for neurons (Gurnee al., 2024), group composition circuits (Chughtai et al., 2023), and modular task processing (Variengien & Winsor, 2023), with evidence for the weak but not the strong formulation (Chughtai et al., 2023).
3.4 Emergence of World Models and Simulated Agents
Internal World Models. World models are internal causal models of an environment formed within neural networks. Traditionally linked with reinforcement learning, these models are explicitly trained to develop compressed spatial and temporal representation of the training environment, enhancing downstream task performance and sample efficiency through training on internal hallucinations (Ha & Schmidhuber, 2018). However, in the context of our survey, our focus shifts to internal world models that potentially form ömplicilly as a by-product of the training process, especially in LLMs trained on next-token prediction also called GPT.
LLMs are sometimes characterized as stochaslic porrols (Bender et al., 2021). This label stems from their fundamental operational mechanism of predicting the next word in sequence, which is seen relying heavily memorization. From this viewpoint, LLMs are thought to form complex correlations based on observational data but cannot develop causal models of the world due to their lack of access to interventional data (Pearl, 2009)..
An alternative perspective on LLMs comes from the active inference framework (Salvatori et al., 2023), a theory rooted in cognitive science and neuroscience. Active inference postulates that the objective of minimizing prediction error, given enough representative capacity, is adequate for a learning system develop complex world representations, behaviors, and abstractions. Since language inherently mirrors the world, these models could implicitly construct linguistic and broader world models (Kulveit et al., 2023).
The simulation hypothesis suggests that models designed for prediction, such as LLMs, will eventually simulate the causal processes underlying data creation. Seen as an extension of their drive for efficient compression, this hypothesis implies that adequately trained models like GPT could develop internal world models as a natural outcome of their predictive training (jamus, 2022; Shanahan et al., 2023).
Hypothesis 5: Simulation
A model whose objective is text prediction will simulate the causal processes underlying the text creation if optimized sufficiently strongly (janus, 2022).
In addition to thcoretical considerations for emergent causal world models (Richens & Everitt, 2024; Nichani et al., 2024), mechanistic interpretability is starting to provide empirical evidence on the types of internal world models that may emerge in LLMs. The ability to internally represent the board state in games like chess (Karvonen, 2024) or Othello (Li et al., 2023a; Nanda et al., 2023b), create lincar abstractions of spatial and temporal data (Gurnee & Tegmark, 2024), and structure complex representations of mazes, demonstrating understanding of maze topology and pathways (Tvanitskiy et al., 2023) highlight the growing rbstioation
11

---

Published in Transactions on Machine Learning Research (08/2024)
capabilities of LLMs. Li et al. (2021) identified contextual word representations that function models of entities and situations evolving throughout a discourse, akin to linguistic models of dynamic semantics. Patel & Pavlick (2022) demonstrated that LLMs can map conceptual domains (e.g, direction, color) to grounded world representations given a few examples, suggesting they learn rich conceptual spaces (Gardenfors, 2004) reflective of the non-linguistic world.
The prediction orthogonality hypothesis further expands on this idea: It posits that cendiction-focusee models like GPT may simulate agents with various objectives and levels of optimality. In this context, GPT are simulators, simulating entities known as simulacra that be either agentic non-agentic, with different objectives from the simulator itself (jamus, 2022; Shanahan et al., 2023). The implications of the simulation and prediction orthogonality hypotheses for Al safety and alignment are discussed in Section 6.
Hypothesis 6: Prediction Orthogonality
A model whose objective is prediction can simulate agents who optimize toward any objectives with any degree of optimality (jamus, 2022).
In conclusion, the evolution of LLMs from simple predictive models to entities potentially possessing complex internal world models, as suggested by the simulation hypothesis and supported by mechanistic interpretabil- ity studies, represents significant shift in our understanding of these systems. This evolution challenges to reconsider LLMs' capabilities and future trajectories in the broader landscape of Al development.
4 Core Methods
Mechanistic interpretability (MI) employs various tools, from observational analysis to causal interventions. This section provides comprehensive overview of these methods, beginning with a taxonomy that categorizes approaches based on their key characteristics (Section 4.1). We then survey observational (Section 4.2), followed by interventional techniques (Section 4.3). Finally, we study their synergistic interplay (Section 4.4).
Figure offers a visual summary of the methods and techniques unique to mechanistic interpretability.
Observation
Intervention
Structured Probes
Sparse Autoencoder
Activation Patching
Attribution Patching
Causal Scrubbing
Logit Lens
Figure 7: Overview of key methods and techniques in mechanistic interpretabilityy research. Observational approaches include structured probes, logit lens variants, and sparse autoencoders (SAEs). Interventional methods, focusing on causal understanding, encompass activation patching variants for uncovering causal mechanisms and causal scrubbing for hypothesis evaluation.
4.1 Taxonomy of Mechanistic Interpretability Methods
We propose a taxonomy based on four key dimensions: causal nature, learning phase, locality, and compre- hensiveness (Table 1).
The causal nature of methods ranges from purely observational, which analyze existing representations with- out direct manipulation, to interventional approaches that actively perturb model components to establish causal relationships. The learning phase dimension distinguishes between post-hoc techniques applied trained models and intrinsic methods that enhance interpretability during the training process itself.
12

---

Published in Transactions on Machine Learning Research (08/2024)
Locality refers to the scope of analysis, spanning from individual neurons (e.g., feature visualization) to entire model architectures (e.g., causal abstraction). Comprehensiveness varies from partial insights into specific components to holistic explanations of model behavior.
Table 1: Taxonomy of Mechanistic Interpretability Methods
Method
Causal Nature
Phase
Locality
Comprehensiveness
Key Examples
Feature Visualization
Observation
Post-hoc
Local
Partial
Zeiler & Fergus (2014)
Zimmermann et al. (2021)
Exemplar methods
Obscrvation
Post-hoc
Local
Partial
Grosse et al. (2023)
Garde et al. (2023)
Probing Techniques
Observation
Post-hoc
Both
Both
McGrath al. (2022)
Gurnee et al. (2023)
Structured Probes Logit Lens Variants
Observation
Post-hoc
Both Global
Both
Burns et al. (2023)
Observation
Post-hoc
Partial
nostalgebraist (2020)
Belrose et al. (2023)
Sparse Autoencoders
Observation
Post-hoc
Both
Comprehensive
Cunningham et al. (2024)
Bricken el al. (2023)
Activation Patching
Intervention
Post-hoc
Local
Partial
Meng al. (2022a)
Wang et al. (2023)
Path Patching Causal Abstraction
Intervention Intervention
Post-hoc Post-hoc
Both Global
Both Comprehensive
Goldowsky-Dill al. (2023)
Geiger et al. (2023a)
Geiger et al. (2023b)
Wu et al. (2023a)
Hypothesis Testing
Intervention
Post-hoc
Global
Comprehensive
Chan al. (2022)
Jenner et al. (2023)
Intrinsic Methods
Pre/During
Global
Comprehensive
Elhage et al. (2022a)
Liu et al. (2023a)
The categorization is based on the methods' general tendencies. Some methods offer local and global partial and comprehensive interpretability depending the scope of the analysis and application. Probing techniques can range from local to global and partial to comprehensive; simple linear probes might offer local insights into individual features, while more sophisticated structured probes can uncover global patterns. Sparse autoencoders decompose individual ncuron activations (local) but aim to disentangle features across the entire model (global). Path patching exlends local interventions global model understanding by tracing information flow across layers, demonstrating how local perturbations can yield broader insights.
In practice, mechanistic interpretability research involves both method development and their application. When applying methods to understand model, combining techniques from multiple categories is often necessary and beneficial to build more comprchensive understanding (Section 4.4).
4.2 Observation
Mechanistic interpretability draws from observational methods that analyze the inner workings of neural networks, with many of these methods preceding the field itself. For detailed exploration of inner inter- pretability methods, refer to (Räuker et al., 2023). Two prominent categories are example-based methods and beasure-based methods:
i.) Exemplar methods identify real input examples that highly activate specific neurons or layers. This helps pinpoint inftuential data points that maximize neuron activation within the neural network (Grosse et al., 2023; Garde et al., 2023; Nanfack et al., 2024).
i.) Feature visualization encompasses techniques that generate synthetic inputs to optimize neuron activation. These visualizations reveal how neurons respond to stimuli and which features are sen- sitive to (Zeiler & Fergus, 2014: Zimmermann et al., 2021). By inspecting the synthetic inputs that drive neuron behavior, hypothesize about the features encoded by those neurons.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Probing for Features. Probing (Alain & Bengio, 2016; Hewitt & Manning, 2019) involves training classifier using the activations of a model, with the classifier's performance subsequently observed to deduce insights about the model's behavior and internal representations. However, the probe's performance may often reflect its own learning capacities more than the actual characteristics of the model's representations (Belinkov, 2021). This dilermma has led researchers to investigale the ideal balance between the complexity of probe and its capacity to accurately represent the model's features (Cao et al., 2021; Voita & Titov, 2020).
The lineor representation hypothesis offers resolution to this issue. Under this hypothesis, the failure of simple linear probe to detect certain features suggests their absence in the model's reprosentations. Conversely, suppose a more complex probe succeeds where a simpler one fails. In that case, it implies that the model contains features that a complex function can combine into the target feature but that the target feature itself is not explicitly represented. Thus, the hypothesis implies that using linear probes could suffice in most cases, circumventing the complexity considcrations generally associated with probing (Belinkov, 2021).
McGrath et al. (2022) analyzed chess knowledge acquisition in AlphaZero, revealing the emergence of strate- gic concepts during training. In language models, Gurnee et al. (2023) introduced sparse probing to decode internal neuron activations to understand feature representation and sparsity. They show that early layers use sparse combinations of neurons to represent many features in superposition, while middle layers seem to have dedicated monosemantic neurons for higher-level uantextual features.
Probing is limited in drawing causal behavioral conclusions. Tts primarily observational nature focuses how information is encoded rather than how it is used (sce Figure 1), necessitating careful analysis and integration with interventional techniques (Section 4.3), or alternative approaches (Elazar et al., 2021). While in explainable AI (Linardatos et al., 2020), probing has primarily analyzed high-level concepts like linguistic representations (Tenney et al., 2019; Dalvi et al., 2019), MI aims to probe towards uncovering underlying computational processes and functionality. This shift in goals towards uncovering mechanistic computation is a nuanced distinction rather than a clear-cut line between probing in MI and the broader explainability field.
Structured Probes. While focusing on bottom-up, mechanistic interpretabilitty approaches, we can also consider integrating top-down, concept-based structured probes with mechanistic interpretability.
Structured probes aid conceptual interpretability, probing language models for complex features like truth representations. Notably, Burns et al. (2023)'s contrust-consistend search identifies linear projections exhibit- ing logical consistency in hidden states, contrasting truth values for statements and negations.
However, structured probes face significant challenges in unsupervised probing scenarios. As Farquhar et al. (2023) showed, arbitrary features, not just knowledge-rllated ones, can satisfy contrast consistency equally well, raising doubts about scalability. For example, the loss may capture simulation of knowledge from hypothesized simulacra within sufficiently powerful language models rather than the models' true knowledge. Furthermore, Farquhar et al. (2023) demonstrates self-supervised probing methods (like (Burns al., 2023)) often detect prominent but unintended distractor feat uras in the data. The discovered features are also highly sensitive to prompt choice, and there is no principled way to select prompts that would reliably surface a model's true knowledge.
While structured probes primarily focus on high-level concept ual representations (Zou et al., 2023), their findings could potentially inform or complement mechanistic interpretability efforts. For instance, identifying truth directions through structured probes could help guide targeted interventions or analyze the underlying circuits responsible for truthful behavior using mechanistic techniques such as activation patching or circuit tracing (Section 4.3). Conversely, mechanistic methods could provide insights into how truth representa- tions emerge and are computed within the model, addressing some of the challenges faced by unsupervised structured probes.
The logit lens (nostalgebraist, 2020) provides a window into the model's predictive process by applying the final classification layer (which projects the residual stream activation into logits/ vocabulary
Logit Lens.
14

---

Published in Transactions on Machine Learning Research (08/2024)
space) to intermediate activations of the residual stream, revealing how prediction confidence evolves across computational stages. This is possible because transformers tend to build their predictions across layers iteratively (Geva et al., 2022). Extensions of this approach include the tuned lens (Belrose et al., 2023), which trains affine probes to decode hiddon states into probability distributions over the vocabulary, and the Future Lens (Pal et al., 2023), which explores the extent to which individual hidden states encode information about subsequent tokens.
Researchers have also investigated techniques that bypass intermediate computations to probe representa- tions directly. Din et al. (2023) propose using linear transformations to approximate hidden states from different layers, revealing that language models often predict final outputs in carly layers. Dar et al. (2022) present a theoretical framework for interpreting transformer parameters by projecting them into the embed- ding space, enabling model alignment and parameter transfer across architectures.
Other techniques focus on interpreting specific model components or submodules. The DecoderLens (Langedijk et al., 2023) allows analyzing encoder-decoder transformers by cross-attending intermediale en- coder representations in the decoder, shedding light on the information flow within the encoder. The Atten- tion Lens (Sakarvadia et al., 2023b) aims to elucidate the specialized roles of attention heads by translating their outputs into vocabulary tokens via learned transformations.
Feature Discntanglement via Sparse Dictionary Learning. Recent work suggests that the essential elements in neural networks are linear combinations of neurons representing features in superposition (EI- hage et al., 2022b). To disentangle these features, researchers have developed sparse autoencoders (SAEs), which decompose neural network activations into individual component features (Sharkey et al., 2022b; Cun- ningham et al., 2024). This process, known as sparse dictionary learning, reconstructs activation vectors as sparse linear combinations of directional vectors within the activation space (Olshausen & Field, 1997).
The theoretical foundations of SAEs are rooted in work on disentangled representations. Whittington et al. (2022) demonstrate that autoencoders can recover ground truth features under conditions of feature sparsity and non-negativity. Furthermore, Garfinkle & Hillar (2019) provides guarantees for the uniqueness and sta- bility of dictionaries for sparse representation, even in the presence of noise. These theoretical underpinnings support SAEs' ability to uncover true, disentangled features underlying the data distribution.
In practice, SAFs stand out for their simplicity and scalability (Sharkey et al., 2022b). They incorporate sparsity regularization to encourage learning sparse yet meaningful data representations, with the prccisc tuning of the sparsily penalty on hidden activations critical in diclating the autoencoder's 's sparsity level. We provide an overview of the SAE architecture in Figure 8.
SAEs' dictionary features exhibit higher scores on autointerpretabiiity metrics and increased monosemantic- ity (Bricken et al., 2023; Cunningham et al., 2024; Sharkey et al., 2022b). They are scalable lo Malle-of-theaar models and can detect safety-relevant features (Templeton al., 2024), measure feature sparsity (Deng al., 2023), and interpret reward models in reinforcement learning-based language models (Marks et al., 2023a).
Evaluating SAE quality remains challenging due to the lack of ground-truth interpretable features. Re- searchers have addressed this through various approaches: Karvonen et al. (2024) proposed using language models trained on chess and Othello transcripts as testbeds, providing natural collections of interpretable features. Sharkey et al. (2022b) constructed a toy model with traceable features, while Makelov et al. (2024); Makelov (2024) compared SAE results with supervised features in large language models to demonstrate their viability.
The versatility of SAEs extends to various neural network architectures. They have been successfully applied to transformer attention layers (Kissane et al., 2024) and convolutional neural networks (Gorton, 2024). Notably, Gorton (2024) applied SAEs to the early vision layers of InceptionV1, uncovering new interpretable features, including additional curve detectors not apparent. from examining individual neurons (Cammarata et al., 2020).
In circuit discovery, SAEs have shown particular promise (see also Section 4.4). He et al. (2024) proposed circuit discovery framework alternative to activation patching (discussed in Section 4.3.1), leveraging
15

---

Published in Transactions on Machine Learning Research (08/2024)
dictionary features decomposed from all modules writing to the residual stream. Similarly, O'Neill & Bui (2024) employed discrete sparse autoencoders for discovering interpretable circuits in large language models.
Recent advancements have focused on improving SAE performance and addressing limitations. Rajamanoha- ran et al. (2024) introduced a gating mechanism to separate the functionalities of determining which direc- tions use and estimating their magnitudes, mitigating shrinkage the systematic underestimation of feature activations. An alternative approach Dunefsky al. (2024) uses transcoders to faithfully approx- imate densely activating MLP layer with wider, sparsely-activating MLP layer, offering another path to interpretable feature discovery, type of sparse distillation (slavachalnev, 2024).
Sparse Dictionary Learning
Sparse autoencoders (Cunningham al., 2024) are proposed solution to polysemantic neurons. The problem of superposition is mathematically formalized as sparse dictionary learning (Olshausen & Field, 1997) problem to decompose neural network activations into disentangled component features. The goal to learn a dictionary vectors [f C. Rd that can represent the unknown, ground truth network features as sparse lincar combinations. If successful, the learnod dictionary contains monosemantic neurons corresponding to features (Sharkey al., 2022b). The autoencoder architec- ture consists of encoder and ReLU activation function, expanding the input dimensionality to dhid din- The encoder's output is given by:
h ReLU(W ene b),
(1)
Ania-1
Waech hifo
(2)
where Wene: WT Rduaxdd and E Ruia The parameter matrix W forms the feature dictio- nary, with rows fi as dictionary features. The autoencoder is trained to minimize the loss, where the I penalty on encourages sparse reconstructions using the dictionary features,
[(x) = | x'1 aa/h
(3)
tokens
Transformer
Sparse autoencoder
embed
X
attention
W
enc
residual stream
encode
h
MLP
"eatures" "
unembed
decode
W
dec
logits
X
Figure 8: Illustration of a sparse autoencoder applied to the MLP layer activations, consisting of an encoder that increases dimensionality while emphasizing sparse representations and a decoder that reconstructs the original activations using the learned feature dictionary.
16

---

Published in Transactions on Machine Learning Research (08/2024)
4.3 Intervention
Causality Theoretical Foundation. The theory of causality (Pearl, 2009) provides mathemati- cally precise framework for mechanistic interpretability, offering a rigorous approach to understanding high- level semantics in neural representations (Geiger et al., 2023a). By treating neural networks causal models, with their compute graphs serving as causel graphs, researchers can perform precise interventions and ex- amine the roles of individual parameters (Mueller et al., 2024). This causal perspective on interpretability has led to the dovelopment of various intervention techniques, including activation patching (Section 4.3.1), causal rbstioation (Section 4.3.2), and hypothesis testing methods (Section 4.3.3).
4.3.1 Activation Patching
Patch corrupted activations into clean circuits.
AND
OR
Embedding
Unembedding
Attention layer
MLP layer
Activations
Patch clean activations into corrupted circuits.
OR
AND
Rome
Parie- Rome
(a)
(b)
Figure 9: (a) Activation patching in a transformer model. Left: The model processes the clean input 'Colosseum in Rome," caching the latent activations (step i.). Right: The model runs with the corrupted input 'Fiffel Tower in Paris" (step si.). The pink arrow shows MLP layer activation (green diamond) patched from the clean into the corrupted run (step mi.). This causes the prediction to change from 'Paris' to 'Rome,' demonstrating how the significance of the patched component is determined (step iu.). By comparing these carefully selected inputs, researchers can control for confounding circuitry and isolate the specific circuit responsible for the location prediction behavior. (b) Activation patching directions: Top: Patching corrupted activations (orange) into clean circuits (turquoise) reveals sufficient components for identifying OR logic scenarios. Bottom: Patching clean activations (green) into corrupted circuits (orange) reveals necessory components that are useful for identifying AND logic scenarios. The AND and OR gates demonstrate how these patching directions uncover different logical relationships between model components.
Activation patching is collective term for set of causal intervention techniques that manipulate neural network activations to shed light on the decision-making processes within the model. These techniques, including causal tracing (Meng et al., 2022a), interchange intervention (Geiger et al., 2021b), causal mediation analysis (Vig et al., 2020), and causal ablation (Wang et al., 2023), share the common goal of modifying a neural model's internal state by replacing specific activations with alternative values, such as zeros, mean activations across samples, random noise, or activations from different forward pass (Figure 9a).
The primary objective of activation patching is to isolate and understand the role of specific components circuits within the model by observing how changes in activations affect the model's output. This enables rescarchers to infer the function and importance of those components. Key applications include localizing behavior by identifying critical activations, such as understanding the storage and processing of factual infor- mation (Meng et al., 2022a; Geva et al., 2023; Goldowsky- et al., 2023; Stolfo et al., 2023), and analyzing component interactions through circuit analysis to identify sub-networks within model's 's computation graph that implement specified behaviors (Wang et al., 2023; Hanna al., 2023; Lieberum al., 2023; Hendel et al., 2023; Geva et al., 2023)-
17

---

Published in Transactions on Machine Learning Research (08/2024)
The standard protocol for activation patching (Figure 9a) involves:
step i. Running the model with a clean input and caching the lalent activalions;
step i. Executing the model with a corrupted input;
step tt.. Re-running the model with the corrupted input but substituting specific activations with those from the clean cache; and
step W. Determining significance by observing the variations in the modcl's output during the third step, thereby highlighting the importance of the replaced components.
This process relies on comparing pairs of inputs: a clean input, which triggers the desired behavior, and a corrupted input, which is identical to the clean one except for critical differences that prevent the behavior. By carefully selecting these inputs, researchers can control for confounding circuilry and isolate the specific circuit responsible for the behavior.
Differences in patching direction clean to corrupted (causal tracing) versus corrupted to clean (resample ablation) - provide insights into the sufficiency necessity of model components for a given behavior. Clean to corrupted patching identifies activations sufficient for restoring clean performance, even if they are unnecessary due to redundancy, which is particularly informative in OR logic scenarios (Figure 9b, OR gate). Conversely, corrupted to clean patching determines the necessary activations for clean performance, which is useful AND logic scenarios (Figure 9b, AND gate).
Activation patching can employ corruption methods, including zero-, mean-, random-, or resample ablation, each modulating the model's internal state in distinct ways. Resample ablation stands out for its effectiveness in maintaining consistent model behavior by not changing the data distribution too much (Zhang & Nanda, 2023). However, it is essential to be careful when interpreting the patching results, as brcaking behavior by laking the model off-distribution is uninteresting for finding the relevant circuit (Nanda, 2023e).
Path Patching and Subspace Activation Patching. Path patching extends the activation patching approach to multiple edges in the computational graph (Wang et al., 2023; Goldowsky-Dill et al., 2023), allowing for a more fine-grained analysis of component interactions. For example, path patching can be used to estimate the direct and indirect effects of attention heads on the output logits. Subspace activation patching, also known as distributed interchange interventions (Geiger et al., 2023b), aims to intervene only linear subspaces of the representation space where features are hypothesized to be encoded, providing tool for more targeted interventions.
Recently, Ghandeharioun et al. (2024) introduced petchscopes, a framework that unifies and extends activa- tion patching techniques: using the model's text generation to explain internal representations, enables more flexible interventions across various interpretability tasks, improving early layer inspection and allowing for model analysis.
Limitations and Advancements. Activation patching has several limitations, including the effort re- quired to design input templates and counterfactual datasets, the need for human inspection to isolate important subgraphs, and potential second-order effects that can complicate the interpretation of results (Lange et al., 2023) and the hydra effect (MeGrath et al., 2023; Rushing & Nanda, 2024) (see discussion in Section 7.2). Recent advancements aim to address these limitations, such as automated circuit discovery algorithms (Conmy et al., 2023), gradient-based methods for scalable component importance estimation like attribution patching (Nanda, 2023d; Syed et al., 2023), and techniques to mitigate self-repair interference during analysis (Ferrando & Voita, 2024).
4.3.2 Causal Abstraction
Causal abstraction (Geiger et al., 2021a; 2023a) provides a mathematical framework for mechanistic in- terpretability, treating neural networks and their explanations causal models. This approach validates
18

---

Published in Transactions on Machine Learning Research (08/2024)
explanations through interchange interventions network activations (Jenner al., 2023), unifying var- lous interpretability methods such as LIME (Ribeiro et al., 2016), causal effect estimation (Feder et al., 2021), causal mediation amalysis (Vig et al., 2020), iterated mullspace projection (Ravfogel et al., 2020), and dircuit-based explanations (Geiger et al., 2023a).
To overcome computational limitations, distributed aligument search (Geiger et al., 2023b) introduced gradient-based distributed interchange interventions, extending causal abstraction to larger models (Wu et al., 2023b). Further advancements inchude causal prozy models (Wu et al., 2023a), which address the challenge of counterfactual observations.
Applications of causal abstraction span from linguistic phenomena analysis (Arora et al., 2024; Wu et al., 2022b), and evaluation of interpretability methods (Huang el al., 2024), to improving performance through representation finetuning (Wu et al., 2024), and improving efficiency via model distillation (Wu et al., 2022b).
4.3.3 Hypothesis Testing
In addition to the causal abstraction framework, several methods have been developed for rigorous hypothesis testing about neural network behavior. These methods aim to formalize and empirically validate explanations of how neural networks implement specific behaviors.
Causal scrubbing (Chan et al., 2022) formalizes hypotheses as a tuple (G,I, where G is the model's computational graph, I is an interpretable computational graph hypothesized to explain the behavior, and maps nodes of I to nodes of g. This method replaces activations in G with others that should be equivalent according to the hypothesis, measuring performance on the scrubbed model to validate the hypothesis.
Locally consistent abstractions (Jenner et al., 2023) offer a more permissive approach, checking the consis- tency between the neural network and the explanation only one step away from the intervention node. This method forms a middle ground between the strictness of full causal abstraction and the flexibility of causal scrubbing.
These methods form a hierarchy of strictness, with full causal abstractions being the most stringent, followed by locally consistent abstractions and cansal scrubbing being the most permissive. This hierarchy highlights trade-offs in choosing stricter or more permissive notions, affecting the ability to find acceptable explanations, generalization, and mechanistic anomaly detection.
4.4 Integrating Observation and Intervention.
To comprehensively understand internal neural network mechanisms, combining observational and interven- tional methods is crucial. For instance, sparse autoencoders can be used to disentangle superposed features (Cunningham al., 2024), followed by targeted activation patching to test the causal importance of these features (Wang et al., 2023). Similarly, the logit lens can track prediction formation across layers (nostalge- braist, 2020), with subsequent interventions confirming causal relationships at key points. Probing techniques can identify encoded information (Belinkov, 2021), which can then be subjected to causal abstraction (Geiger et al., 2023a) to understand how this information is utilized. This iterative refinement process, where broad observational methods guide targeted interventions and intervention results inform further observations, en- ables a multi-level analysis that builds a holistic understanding across different levels of abst raction. Recent work (Marks al., 2024; Bushnaq et al., 2024; Braun et al., 2024; O'Neill & Bui, 2024; Ge et al., 2024) demonstrates the potential of integrating sparse autoencoders with automated circuits discovery (Conmy et al., 2023; Syed et al., 2023), combining feature-level analysis with cireuit-level interventions to uncover the interplay between representation and mechanism.
5 Current Research
This section surveys current research in mechanistic interpretability across three approaches based on when and how the model is interpreted during training: Intrinsic interpretability methods are applied before train- ing to enhance the model's inherent interpretability (Section 5.1). Developmental interpretability involves studying the model's learning dynamics and the emergence of internal structures during training (Section 5.2).
19

---

Published in Transactions on Machine Learning Research (08/2024)
After training, post-hoc interpretability techniques applied to gain insights into the model's behavior and decision-making processes (Section 5.3), including efforts towards uncovering general, transferable principles across models and tasks, as well as automating the discovery and interpretation of critical circuits in trained models (Section 5.4).
intrinsic
developmental during training
post-hoc after training
before training
sparsity
predictive
universal
modularity
tractable
high-level
disentanglement
unifying theory
comprehensive
Figure 10: Key desiderata for interpretability approaches across training and analysis stages: (1) Intrinsic: Architectural biases for sparsily, modularily, and disentangled representations. (2) Developmental: Predictive capability for phase transitions, manageable number of critical transitions, and unifying theory connecting obscrvations to singularity geometry. (3) Post-hoc: Global, comprchensive, automated discovery of critical circuits, uncovering transferable principles across models/tasee and extracting high-level causal mechanisms.
5.1 Intrinsic Interpretability
Intrinsic methods for mechanistic interpretability offer a promising approach to designing neural networks that are more amenable to reverse engineering without sacrificing performance. By encouraging sparsity, modularily,, and monosemanticity through architectural choices and training procedures, these methods aim make the reverse engineering process more tractable.
Intrinsic interpretability methods aim to constrain the training process to make learned programs interpretable (Friedman et al., 2023b). This approach is closely related to neurosymbolic learning (Riegel et al., 2020) and can involve techniques like regularization with spatial structure, akin to the organization of information in the human brain (Liu et al., 2023a;b).
Recent work has explored various architectural choices and training procedures improve the interpretability of neural networks. Jermyn et al. (2022) and Elhage al. (2022a) demonstrate that architectural choices can affect monosemantieity, suggesting that models could be engineered to be more monosementic. Sharkey (2023) propose using bilinear layer instead of linear layer to encourage monosemanticity in language models.
Liu et al. (2023a) and Liu et al. (2023b) introduce biologically inspired spatial regularization regime called brain-inspired modular training for forming modules in networks during training. They showcase how this can help RNNs exhibit brain-like anatomical modularity without degrading performance, contrast. naive attempts to use sparsity reduce the cost of having more neurons per layer (Jermyn et al., 2022; Bricken et al., 2023).
Preceding the mechanistic interpretability literature, various works have explored techniques to improve interpretability, such as sparse attention (Zhang et al., 2021), adding L' penalties to neuron activations (Kasioumis el al., 2021; Georgiadis, 2019), and pruning neurons (Frankle & Carbin, 2019). These techniques have been shown to encourage sparsity, modularity, and disentanglement, which are essential aspects of intrinsic interprotability.
5.2 Developmental Interpretability
Developmental interpretability examines the learning dynamics and emergence internal structures in neural networks over time, focusing on the formation of features and circuits. This approach complements static analyses by investigating critical phase transitions corresponding to significant. changes in model behavior
20

---

Published in Transactions on Machine Learning Research (08/2024)
capabilities (Steinhardt, 2023; Schaeffer et al., 2023; Wei al., 2022; Simon et al., 2023). While primarily distinct field, developmental interpretability often intersects with mechanistic interpretability, as exemplified by Olsson et al. (2022)'s work. Their research, rooted in mechanistic interpretability, demonstrated how the emergence of in-context learning relates to specific training phase transitions, connecting microscopic changes (induction heads) with macroscopic observables (training loss).
A key motivation for developmental interpretability is investigating the universality of safety-rritical patterns, aiming to understand how deeply ingrained and thereby resistant to safety fine-tuning capabilities like deception are. In addition, researchers hypothesize that emergent capabilities correspond to sudden circuit formation during training (Michaud et al., 2023), potentially allowing for prediction or control of their development.
Singular Learning Theory (SLT), developed by Watanabe (Watanabe, 2009; 2018), provides a rigorous frame- work for understanding overparameterized models' behavior and generalization. By quantifying model com- plexity through the local learning coefficient, SLT offers insights into learning phase transitions and the emergence of structure in the model (Lau et al., 2023). Recent work by Hoogland et al. (2024) applied this coefficient to identify developmental stages in transformer models, while Furman & Lau (2024) and Chen et al. (2023b) advanced SLT's scalability and application to the toy model of superposition (Figure 5), respectively.
While direct applications to phenomena such as generalization (Zhang et al., 2017), learning functions with increasing complexity (Nakkiran et al., 2019), and the transition from memorization to gencralization (grokking) (Tiu et al., 2022a; Power et al., 2022; Liu et al., 2022b; Nanda et al., 2023a; Varma et. al., 2023; Thilak et al., 2022; Merrill et al., 2023; Liu et al., 2023c; Stander et al., 2023; Wang et al., 2024) are limited, these areas, along with neural scaling laws (Caballero et al., 2022; Liu & Tegmark, 2023; Michaud et al., 2023) (which can be connected to mechanistic insights (Hernandez et al., 2022)), represent promising future research directions.
In conclusion, developmental interprelability serves as an evolutionary theory lens for neural net worksrks, of- fering insights into the emergence of structures and behaviors over time (Saphra, 2023). Drawing parallels from systems biology (Alon, 2019), this approach can apply concepts like network motifs, robustness, and modularity to neural network development, explaining how functional capabilities arise. Sometimes, under- standing how structures came about is easier than analyzing the final product, similar to how biologists find certain features in organisms easier to explain in light of their evolutionary history. By studying the temporal aspects of neural network training, researchers can potentially uncover fundamental principles of learning and representation that may not be apparent from examining static, trained models alone.
5.3 Poc Interpretability
In applicd mochanistic interpretability, rescarchers explore various facets and methodologics to uncover the inner workings of AT models. Some key distinctions are drawn between global versus local interpretability and comprehensive versus partiol interpretability. Global interpretability aims to uncover general patterns and behaviors of model, providing insights that apply broadly across many instances (Doshi-Velez & Kim, 2017; Nanda, 2023e). In contrast, local interpretability explains the reasons behind a model's decisions for particular instances, offering insights into individual predictions or behaviors. Comprehensive interpretability involves achieving a deep and exhaustive understanding of a model's behavior, providing a holistic view of its inner workings (Nanda, 2023c). In contrast, partial interpretability often applied to larger and more complex models, concentrates on interpreting specific aspects or subsets of the model's behavior, focusing on the application's most relevant or critical arcas.
Large Models - Narrow Behavior. Circuit-style mechanistic interpretability aims to explain neural networks by reverse engineering the underlying mechanisms at the level of individual neurons or subgraphs. This approach assumes that neural vector representations encode high-level concepts and circuits defined by model weights encode meaningful algorithms (Olah et al., 2020; Cammarata et al., 2020). Studies on deep networks support these claims, identifying circuits responsible for detecting curved lines or object orientation (Cammarata et al., 2020; 2021; Voss et al., 2021).
21

---

Published in Transactions on Machine Learning Research (08/2024)
This paradigm has been applied to language models to discover subnetworks (circuits) responsible for specific capabilities. Circuit analysis localizes and understands subgraphs within a model's computational graph responsible for specific behaviors. For large language models, this often involves narrow investigations into behaviors like multiple choice reasoning (Lieberum et al., 2023), indirect object identification (Wang et al., 2023), or computing operations (Hanna et al., 2023). Other examples include analyzing circuits for Python docstrings (Heimersheim & Jett, 2023), "an" vs "a" usage (Miller & Neo, 2023), and price tagging (Wu et al., 2023b). Case studies often construct datasets using templates filled by placeholder values to enable precise control for causal interventions (Wang et al., 2023; Hanna et al,, 2023; Wu et al., 2023b).
Toy Models- Comprehensive Analysis. Small models trained specialized mathematical algo- rithmic tasks enable more comprehensive reverse engineering of learned algorithms (Nanda et al., 2023a; Zhong et al., 2023; Chughtai et al., 2023). Even simple arithmetic operations can involve complex strategies and multiple algorithmic solutions (Nanda et al., 2023a; Zhong et al., 2023). Characterizing these algo- rithms helps test hypotheses around generalizable mechanisms like variable binding (Feng & Steinhardt, 2023; Davies et al., 2023) and arithmetic reasoning (Stolfo et al., 2023). The work by Varma et al. (2023) builds on the work that analyzes transformers trained on modular addition (Nanda et al., 2023a) and ex- plains grokking in terms of circuit elficiency, illustrating how comprehensive understanding of a toy model can enable interesting analyses on top of that understanding.
Towards Universality. The ultimate goal is to uncover general principles that transfer across models and tasks, such induction heads for in-context learning (Olsson et al., 2022), variable binding mechanisms (Feng & Steinhardt, 2023; Davies et al., 2023), arithmetic reasoning (Stolfo et al., 2023; Brinkmann et al., 2024), or retrieval lasks (Variengien & Winsor, 2023). Despite promising results, debates surround the universality hypothesis - the idea that different models learn similar features and circuits when trained on similar tasks. (Chughtai et al., 2023) finds mixed evidence for universality in group composition, suggesting that while families of circuits and features can be characterized, precise circuits and development order may be arbitrary.
Towards High-level Mechanisms. Causal interventions can extract high-level understanding of com- putations and representations learned by large language models (Variengien & Winsor, 2023; Hendel et al., 2023; Feng & Steinhardt, 2023; Zou et al., 2023). Recent work focuses on intervening in internal represen- tations to study high-level concepts and computations encoded. For example, Hendel et al. (2023) patched residual stream vectors to transfer task representations, while Feng & Steinhardt (2023) intervened on resid- ual streams to argue that models generate IDs to bind entities to attributes. Techniques for representation engineering (Zou el. al., 2023) extract reading vectors from model activations to stimulate or inhibit specific concepts. Although these interventions don't operate via specific mechanisms, they offer a promising ap- proach for extracting high-level causal understanding and bridging bottom-up and top-down interpretability approaches.
5.4 Automation: Scaling Post-Hoc Interpretability
As models become more complex, automating key aspects of the interpretability workflow becomes in- creasingly crucial. Tracing a model's computational pathways is highly labor-intensive, quickly becoming infeasible the model size increases. Automating the discovery of relevant circuits and their functional interpretation represents pivotal step towards scalable and comprehensive model understanding (Nainani, 2024).
Dissecting Models into Interpretable Circuits. The first major automation challenge is identifying the critical computational sub-circuits or components underpinning a model's behavior for a given task. A pioneering line of work aims to achieve this via efficient masking or patching procedures. Methods like automated circuit discovery (Conmy et al., 2023) and attribution patching (Syed et al., 2023; Kramár et al., 2024) iteratively knock out model activations, pinpointing components whose removal has the most significant impact on performance. This masking approach has proven scalable even to large models (Lieberum et al., 2023).
22

---

Published in Transactions on Machine Learning Research (08/2024)
Other techniques take more top-down approach. Davies et al. (2023) specify high-level causal properties (desiderata) that components solving target subtask should satisly and then learn binary masks to expose those component subsets. Ferrando & Voita (2024) construct information flow graphs highlighting key nodes and operations by tracing attribution flows, enabling extraction of general information routing patterns across prediction domains.
Explicit architectural biases like modularity can further boost automation efficiency. Nainani (2024) find that models trained with brain-inspired modular training (Liu et al., 2023a) produce more readily identifiable circuits compared to standard training. Such bomain-insspired inductive biases may prove increasingly vital as models grow more massive and monolithic.
Interpreting Extracted Circuits. Once critical circuit. components have been isolated, the key remain- ing step is interpreting what computation those components perform. Sparse autoencoders prominent approach for interpreting extracted circuits by decomposing neural network activations into individual com- ponent features, as discussed in Section 4.2.
A novel paradigm uses large language models themselves as an interpretive tool. Bills et al. (2023) demon- strate generating natural language descriptions of individual neuron functions by prompting language models like GPT-4 to explain of inputs that activate a neuron. Mousi et al. (2023) similarly employ language models annotate unsupervised neuron clusters identified via. hierarchical clustering. Bai et al. (2024) describe the roles of neurons in vision networks with multimodal models. These methods easily leverage more capable general-purpost models in the future. Foote el al. (2023) take complementary graph-based approach in their neuron-to-grapy tool: automatically extracting individual neurons' behavior patterns from training data as structured graphs amenable to visualization, programmatic comparisons, and property searches. Such representations could synergize with language model-based annotation to provide descriptions of neuron roles.
However, robustly interpreting the largest pallimnnparrmeter models using automated techniques remains open challenge. Another novel approach, mechanistieinneeppttabblli interpretability-bane program synthesis (Michand et al., 2024), entirely sidesteps this complexity by auto-distilling the algorithm learned by trained model into human-readable Python code without relying on further interpretability analyses or model architectural knowledge. As models become increasingly vast and opaque, such synergistic combinations of methods uncovering circuits, annotating them, or altogether transcribing them into executable code will likely prove crucial for maintaining insight and oversight when scaling model size.
6 Relevance to Al Safety
Harmful 4
Helpful
Monitoring and evaluation
Substantiate threat models
Accelerate capabilities
Dual-use
Anticipate emergence
Accelerate safety research
Diverting resources
Causing overconfidence
Figure 11: Potential benefits and risks of mechanistic interpretability for Al safety.
How Could Interpretability Promote AI Safety? Gaining mechanistic insights into the inner work- ings of AT systems seems crucial for navigating AI safety as we develop more powerful models (Nanda, 2022c). Interpretability tools can provide an understanding of artificial cognition, the way AI systems process infor- mation and make decisions, which offers several potential benefits:
23

---

Published in Transactions on Machine Learning Research (08/2024)
Mechanistic interpretability could accelerate Al safety research by providing richer feedback loops and grounding for model evaluation (Casper, 2023). It may also help anticipate emergent capabilities, such as the emergence of new skills or behaviors in the model before they fully manifest (Wei et al., 2022; Stein- hardt, 2023; Nanda et al., 2023a; Barak et al., 2022). This relates to studying the incremental development of internal structures and representations as the model learns (Section 6.2). Additionally, interpretability could substantiate theoretical risk models with concrete evidence, such as demonstrating inner mésalignment (when model's behavior deviates from its intended goals) or mess-optimmzzatinn (the emergence of unin- tended subagents within the model) (Hubinger et al., 2019; von Oswald et al., 2023). It may also trigger normative shifts within the AT community toward rigorous safety protocols by revealing potential risks or concerning behaviors (Hubinger, 2019a).
Regarding specific AI risks (Hendrycks et al., 2023), interpretability may prevent malicious misuse by lo- cating and crasing sensitive information stored in the model (Meng et al., 2022a; Nguyen et al., 2022). It could reduce competitive pressures by substantiating potential threats, promoting organizational safety cul- tures, and supporting AT alignment (ensuring AT systems pursue intended goals) through better monitoring and evaluation (Hendrycks & Mazeika, 2022). Interpretability can provide safety filters for every stage of training: before training by deliberate design (Hubinger, 2019a), during training by detecting carly signs of misalignment and potentially shifting the distribution towards aligmment (Hubinger, 2022; Sharkey, 2022), and after training by rigorous evaluation of artificial cognition for honesty (Burns et al., 2023; Zou et al., 2023) and screening for deceptive behaviors (Park et al., 2023b).
The emergence of internal world models in LLMs, as posited by the simulation hypothesis, could have significant implications for AT alignment research. Finding an internal representation of human values and aiming the Al system's objective may be trivial way to achieve alignment (Wentworth, 2022), especially if the world model is internally separated from notions of goals and agency (Ruthenis, 2022). In such cases, world model interpretability alone may be sufficient for aligmment (Ruthenis, 2023).
Couditioning pre-trained models is considered a comparatively safe pathway towards general intelligence, as it avoids directly creating agents with inherent goals or agendas (Jozdien, 2022; Hubinger el al., 2023). However, prompting a model to simulate an actual agent, such as 'You are superintelligence in 2035 writing down an alignment solution,' could inadvertently lead to the formation of internal agents (Hubinger et al., 2023). In contrast, reinforcement. learning lends to create agents by default (Casper el al., 2023a; Ngo et al., 2022).
The prediction orthogonality hypothesis suggests that prediction-focused models like GPT can simulate agents with potentially misaligned objectives (janus, 2022). Although GPT may lack gemuine agency or intentionality, it may produce outputs that simulate these qualities (Bereska & Gavves, 2023; Shanahan al., 2023). This underscores the need for careful oversight and, better yet, using mechanistic interprelability to search for internal agents or their constituents, such as optimization search processes an endeavor known as searching for search (NicholasKees & janus, 2022; Jenner et al., 2024).
Mechanistic interpretability integrates well into various Al alignment agendas, such as understanding ex- isting models, controlling them, making Al systems solve alignment problems, and developing alignment theories (technicalities & Stag, 2023; Hubinger, 2020). It could enhance strategies like detecting deceptive alignment (hypothetical when a model ensures to appear aligned as to pursuc misaligned goals without raising suspicion) (Park et al., 2023b), eliciting latent. knowledge from models (Christiano et. al., 2021), and enabling better scalable oversight, such as in iterative distillation and amplification (Chan, 2023). A high degree of understanding may even allow for well-founded AT approaches (AI systems with provable guaran- tees) (Tegmark & Omohundro, 2023) or microscope AI (extract world knowledge from the model without letting the model take actions) (Hubinger, 2019a). Furthermore, comprehensive interpretability itself may be an aligument strategy if we can identify internal representations of human values and guide the model to pursue those values by retargeting an internal search process (Wentworth, 2022). Ultimately, understanding and control are intertwined, and deeper understanding can control AI systems more reliably.
However, there is spectrum of potential misalignment risks, ranging from acute, model-centric issues to gradual, systemic concerns (Kulveit, 2024). While mechanistic interpretability may address risks stemming directly from model internals such as deceptive alignment or sudden capability jumps it may be less helpful
24

---

Published in Transactions on Machine Learning Research (08/2024)
for tackling broader systemic risks like the emergence of misaligned economic structures novel evolutionary dynamics (Hendrycks, 2023b). The multi-scale risk landscape calls for balanced research portfolio to minimize risk, where research on governance, complex systems, and multi-agent simulations complements mechanistic insights and model evaluations. The perceived utility of mechanistic interpretability for AI safety largely depends on researchers' priors regarding the likelihood of these different risk scenarios.
How Could Mechanistic. Insight be Harmful? Mechanistic interpretability research could accelerate Al capabilities, potentially leading to the development of powerful Al systems that are misaligned with human values, posing significant risks (Soares, 2023; Kross, 2023; Hendrycks & Mazeika, 2022). While historically, interpretability research had little impact on Al capabilities, recent exceptions like discoveries about scaling laws (Hoffmann et al., 2022), architectural improvements inspired by studying induction heads (Olsson al., 2022; Fu et al., 2023a; Poli et al., 2023; Schuster et al., 2022), and efficiency gains inspired by the logit lens technique (Schuster et al., 2022) demonstrated its potential to enhance capabilities. Scaling interpretability research may necessitate automation (Conmy et al., 2023; Bills et al., 2023), potentially enabling rapid self- improvement of AI systems (RicG, 2023). Some rescarchers recommend selective publication and focusing on lower-risk areas to mitigale these risks (Hobbhahn & Chan, 2023; Shovelain & McKernon, 2023; Elhage et al., 2022b; Nanda et al., 2023a).
Mcchanistic interpretability also poses dual-use risks, where the same techniques could be used for both beneficial and harmful purposes. Fine-grained editing capabilities enabled by interpretability could be used for machine unlearning (removing private data dangerous knowledge from models) (Guo et al., 2024; Sun et al., 2024; Nguyen et al., 2022; Pochinkov & 2023) but could be misused for censorship. Similarly, while interpretability may help improve adversarial robustness (Räuker et al., 2023), it may also facilitate the development of stronger adversarial attacks (Mu & Andreas, 2020; Casper et al., 2023b).
Misunderstanding or overestimating the capabilities of interpretability techniques can divert resources from critical safety areas or lead to overconfidence and misplaced trust in AI systems (Charbel-Raphai) 2023; Casper, 2023). Robust evaluation and benchmarking (Section 8.2) are crucial to validate interpretability claims and reduce the risks of overinterpretation misinterpretation.
7 Challenges
7.1
Research Issues
Need for Comprchensive, Multi-Pronged Approaches. Current interpretability research often fo- cuses on individual techniques rather than combining complementary approaches. To achieve a holistic understanding of neural networks, we propose utilizing a diverse interpretability toolbox that integrates multiple methods (see also Section 4.4), such as: (t.) Coordinating observational (e.g., probing, logit lens) and interventional methods (c.g., activation patching) to establish causal relationships. (ti.) Combining feature-level analysis (e.g., sparse autoencoders) with circuit-level interventions (e.g., path patching) to interplay. (tti.) Integrating intrinsic interpretability approaches with
uncover pprreentatioonnmecanne post-hoc analysis for robust understanding.
For example, coordinated methods could be used for reverse engineering trojaned behaviors (Casper et al., 2023c), where observational techniques identify suspicious activations, interventional methods isolate the relevant circuits, and intrinsic approaches guide the design of more robust, architect
Cherry-Picking and Streetlight Interpretability. Another concerning pattern is the tendency to cherry-pice picky results, relying small number of convincing examples or visualizations as the basis for an argument without comprchensive evaluation (Räuker et al., 2023). This amounts to publication bias, show- casing an unrealistic highlight reel of best-case performance. Relatedly, many interpretability techniques are primarily evaluated on small toy models and tasks (Chughtai et al., 2023; Elhage et al., 2022b; Jermyn et al., 2022; Chen et al., 2023b), risking missing critical phenomena that only emerge in more realistic and diverse contexts. This focus on Mherry-picked results from toy models is a form of streetlight interpretability (Casper, 2023), examining AT systems under only ideal conditions of maximal interpretability.
25

---

Published in Transactions on Machine Learning Research (08/2024)
7.2 Technical Limitations
Scalability Challenges and Risks of Human Reliance. A critical hurdle is dernonstrating the scala- bility of mechanistic interpretability to real-world AT systems across model size, task complexity, behavioral coverage, and analysis efficiency (Elhage et al., 2022b; Scherlis et al., 2023). Achieving a truly comprehen- sive understanding of a model's capabilities in all contexts is daunting, and the time and compute required must scale tractably. Automating interpretability techniques is crucial, as manual analysis quickly becomes infeasible for large models. The high human involvement in current interpretability research raises concerns about the scalability and validity of human-generated model interpretations. Subjective, inconsistent human evaluations and lack of ground-truth benchmarks are known issues (Räuker et al., 2023). As models scale, it will become increasingly untenable to rely on humans to hypothesize about model mechanisms manually. More work is needed on automating the discovery of mechanistic explanations and translating model weights into human-readable computational graphs (Elhage et al., 2022b), but progress on that front may also come from outside the field (Lu al., 2024).
Obstacles to Bottom-Up Interpretability. There are fundamental questions about the tractability of fully reverse engineering neural networks from the bottom up, especially as models become more complex (Hendrycks, 2023a). Models may learn internal representations and algorithms that do not cleanly map to humano-understandsbbe concepts, making them difficult to interpret even with complete transparency (MeGrath et al., 2022). This gap between human and model ontologies may widen as architectures evolve, increasing opaqueness (Hendrycks et al., 2022). Conversely, model representations might naturally converge to more human-interrretablee forms as capability increases (Hubinger, 2019a; Feng & Steinhardt, 2023).
Analyzing Models Embedded in Environments. Real-world AT systems embedded in rich, interac- tive environments exhibit two forms of in-context behavior that pose significant interpretability challenges beyond understanding models in isolation. Externally, models may dynamically adapt to and reshape their environments through in-context learning from the interactions and feedback loops with their external en- vironment (Leahy, 2023). Internally, the hydra effect demonstrates in-context reorganization, where models flexibly reorganize their internal representations a context-dependent manner to maintain capabilities even after ablating key components (McGrath et al., 2023). These two instances of in-context behavior external adaptation to the environment and internal self=reorganization undermine interpreta approaches that assume fixed circuits. For models deeply embedded in rich real-world settings, their dynamic coupling with the external world via in-context environmental learning and their internal in-conlext representational re- organization make strong interpretability guarantees difficult to attain through analysis of the initial model alone.
Adversarial Pressure Against Interpretability. As models become more capable through increased training and optimization, there is a risk they may learn deceptive behaviors that actively obscure or mislead the interpretability techniques meant to understand them. Models could develop adversarial 'mind-reader" components that predict and counteract the specific analysis methods used to interpret their inner workings (Sharkey, 2022; Hubinger, 2022). Optimizing models through techniques like gradient descent could inadver- tently make their internal representations less interpretable to external obscrvers (Hubinger, 2019b; Fu al., 2023b; von Oswald et al., 2023). In extreme cases, a highly advanced AI system singularly focused on pre- serving its core objectives may directly undermine the fundamental assumptions that enable interpretability methods in the first place.
These adversarial dynamics, where the capabilities of the AI model are pitted against efforts to interpret it, underscore the need for interpretability research to prioritize worst-case robustness rather than just average- case scenarios. Current techniques often fail even when models are not adversarially optimized. Achieving high confidence in fully understanding extremely capable AI models may require fundamental advances make interpretability fraewwrrk resilient, against intelligent system's active deceptive efforts.
26

---

Published in Transactions on Machine Learning Research (08/2024)
8 Future Directions
Given the current limitations and challenges, several key research problems emerge as critical for advanc- ing mechanistic interpretability. These problems span four main arcas: emphasizing conceptual clarity (Section 8.1), establishing rigorous standards (Section 8.2), improving the scalability of interpretability tech- niques (Section 8.3), and expanding the research scope (Section 8.4). Each subsection presents specific roscarch questions and challenges that need to be addressed to move the field forward.
Prioritize robustness over capability advancement
Clarifying concepts
Corroborate refute core assumptions
Setting Standards
Integrate existing literature and terminology
Establish metrics, benchmarks, and algorithmic testbeds
Automation techniques
Expanding Scope
Vision, multimodal, and RL models
Scaling Up
Coverage and complexity
Top-down and Hybrid
Universality and overarching theories
During and before training
Figure 12: Roadmap for advancing mechanistic interpretability research, highlighting key strategic directions.
8.1 Clarifying Concepts
Integrating with Existing Literature. To mature, mechanistic interpretability should embrace exist- ing work, using established terminology rather than reinventing the wheel. Diverging terminology inhibits collaboration across disciplines. Presently, the terminology used for mechanistic interpretability partially diverges from mainstream AI research (Casper, 2023). For example, while the mainstream speaks of dis- tributed representations (Hinton, 1984; Olah, 2023) and the goal of disentangled representations (Higgins et al., 2018; Locatello et al., 2019), the mechanistic interpretability literature refers to the same phenomenon as polysemantáciity (Scherlis et al., 2023; Lecomte et al., 2023; Marshall & Kirchner, 2024) and superyosition (Elhage et al., 2022b; Henighan et al., 2023). Using common language invites 'accidental' contributions and prevents isolating mechanistic interpretability from broader research.
Mechanistic interpretability relates to many other fields in AI research, including compressed sensing (Elhage al., 2022b), modularity, adversarial robustness, continual learning, network compression (Ränker et al., 2023), neurosymbolic reasoning, trojan detection, and program synthesis (Casper, 2023; Michaud et al., 2024), and causal representation learning, These relationships can help develop new methods, metrics, benchmarks, and theoretical frameworks. For instance:
i.) Neurosymbolic Reasoning and Program Synthesis: Mechanistic interpretability aims for reverse engineering neural networks by converting their weights into human-readable algorithms. This endeavor draw inspiration from neurosymbolic reasoning (Riegel et al., 2020) and program synthesis. Techniques like creating programs in domain-specific languages (Verma et al., 2019b;a; Trivedi et al., 2021), extracting decision trees (Zhang et al., 2019) or symbolic causal graphs (Ren et al., 2023) from neural networks align well with the goals of mechanistic interpretability. Adopting those approaches can extend the toolkit for reverse engincering AI systems.
i.) Causal Representation Learning: Causal Representation Learning (CRL) aims to discover and disentangle underlying causal factors data (Schölkopf et al., 2021), complementing mechanistic interpretability's goal of understanding causal structures within neural networks. While mechanis- tic interpretability typically examines individual features and eircwils, CRL offers framework for
27

---

Published in Transactions on Machine Learning Research (08/2024)
understanding high-level causal structures. CRL techniques could enhance interpretability by identi- [ying causal relationships between neurons or layers (Bengio et al., 2019; Ke et al., 2021), potentially revealing model reasoning. Its focus on interventions and counterfactuals (Pearl & Mackenzie, 2018;
Peters et al., 2017) could inspire new methods for probing model internals (Goyal ct al., 2020; Besserve et al., 2019). CRL's 's emphasis on learning invariant representations (Peters et al., 2015;
von Kiigelgen et al., 2019) could guide the search for robust features, while its approach to trans- fer learning (Rojas-Carulle Carulla et al., 2018; Magliacane et al., 2018) could inform studies into model generalization.
i..) Trojan Detection: Detecting deceptive alignment models is a key motivation for inspecting model internals, as by definition deception is not salient. from observing behavior alone (Casper et al., 2024). However, quantifying progress is challenging due to the lack of evidence for deception as emergent capability in current models (Steinhardt, 2023), apart from sycophancy (Sharma et al., 2023; Denison et. al., 2024) and theoretical evidence for deceplice in behavior (Lang el al., 2024). Detecting trojans (or backdoors) (Hubinger et al., 2024) implanted via data poisoning could be a proxy goal and prooffof-conceet.. These trojans simulate outer misalignemnt (where the model's behavior is misaligned with the specified reward function or objectives due to poorly defined or incorrect reward signals) rather than inner misalignment such deceptive alignment (where the model appears aligned with the specified objectives but internally pursues different, misaligned goals). Moreover, activating a Irojan lypically results in immediate change of behavior, while deception can be subtle, gradual, and, at first, entirely internal. Nevertheless, trojan detection can still provide a practical testbed for benchmarking interpretability methods (Maloyan et al., 2024).
iv.) Adversarial Robustness: There is duality between interpretability and adversarial robustness (Elhage et al., 2022b; Räuker et al., 2023; Bereska, 2024). More interpretable models tend to be more robust against adversarial attacks (Jyoti et al., 2022), and vice versa, adversarially trained models are often more interpretable (Engstrom et al., 2019). For instance, techniques like input gradient regularization have been shown to simultaneously improve the interpretability of saliency maps and enhance adversarial robustness (Ross & Doshi-Velez, 2017; Du et al., 2021). Furthermore, interpretability tools can help create more sophisticated adversaries (Carter et al., 2019; Casper et al., 2021), improving our understanding of model internals. Viewing adversarial examples as inherent neural network features (Ilyas et al., 2019) rather than bugs also hints at alien features beyond human perception. Connecting mechanistic interpretability to adversarial robustness thus promises ways to gain theoretical insight, measure progress (Casper, 2023), design inherently more robust architectures (Fort & Lakshminarayanan, 2024), and create interprctability-gudee approaches for identifying (and mitigating) adversarial vulnerabilities (Carefa-Carrasco et al., 2024).
More details on the interplay between interpretabilityy robustness, modularity, continual learning, network compression, and the human visual system can be found in the review by Räuker et al. (2023).
Corroborate Refute Core Assumptions. Features are the fundamental units defining neural repre- sentations and enabling mechanistic interpretability's 's bottom-up approach (Chan, 2023), but defining them involves assumptions requiring scrutiny, as they shape interpretations and research directions. Questioning hypotheses by seeking additional evidence or counter-examples is crucial.
The linear representation hypothesis treats activation directions as features (Park et al., 2023a; Nanda et al., 2023b; Elhage et al., 2022b), but the emergence and necessity of lincarity is unclear - is it architectural bias or inherent? Stronger theory justifying linearity's necessity or counter-examples like autoencoders on uncorrelated data without intermediate linear layers (Elhage et al., 2022b) are needed. An alternative lens views features as polytopes from piecewise linear activations (Black et al., 2022), questioning if direction simplification suffices or added polytope complexity aids interpretability.
The superposition hypothesis suggests that polysemantic neurons arise from the network compressing and representing many fcatures within its limited set of neurons (Elhage et al., 2022b), but polysemanticity can also occur incidentally due to redundancy (Lecomte et al., 2023; Marshall & Kirchner, 2024; McGrath et al., 2023). Understanding superposition's role could inform mitigating polysemanticity via regularization
28

---

Published in Transactions on Machine Learning Research (08/2024)
(Lecomte et al., 2023). Superposition also raises open questions like operationalizing computation in su- perposition (Vaintrob et al., 2024; Hänni et al., 2024), allention head superyosition (Elhage et al., 2022b; Jermyn et al., 2023; Lieberum et al., 2023; Gould et al., 2023), representing feature clusters (Elhage et al., 2022b), connections to adversarial robustness (Elhage et al., 2022b; García Garcia-Carrasco et al., 2024; Bloom & Bailey, 2023), anti-correlated feature organization (Elhage et al., 2022b), and architectural effects (Nanda, 2023a).
8.2 Setting Standards
Prioritizing Robustness over Capability Advancement. As the mechanistic interpretability com- munity expands, it is essential to mainlain the norm of not advancing AT capabilities while simultaneously establishing metries necessary for the field's progress (Räuker et al., 2023). Researchers should prioritize developing comprehensive tools for analyzing the worst-case performance of Al systems, ensuring robustness and reliability in critical applications. This includes focusing adversarial tasks, such as backdoor detection and removal (Tamparth & Reuel, 2023; Hubinger et al., 2024; Wu et al., 2022a), and evaluating the accuracy of explanations in producing adversarial examples (Goldowsky-Din et al., 2023).
Establishing Metrics, Benchmarks, and Algorithmic Testbeds. A central challenge in mechanistic interpretability is the lack of rigorous evaluation methods. Relying solely on intuition can lead to conflating hypotheses with conclusions, resulting in cherry-picking and optimizing for best-case rather than average or worst-case performance (Rudin, 2019; Miller, 2019; Räuker et al., 2023; Casper, 2023). Current ad hoc practices and proxy measures (Doshi-Velez & Kim, 2017) risk over-optimization (Goodhart's law - When a mensure becomes a target, it ceases to be a good measure). Distinguishing correlation from causation is crucial, as interpretability illusions demonstrate that visualizations may be meaningless without causal linking (Bolukbasi et al., 2021; Friedman et al., 2023a; Olah et al., 2017).
To advance the field, rigorous evaluation methods are needed. These should include: (i) assessing out-of- distribution inputs, as most current methods are only valid for specific examples or datasets (Räuker et al., 2023; llyas et al., 2019; Mu & Andreas, 2020; Casper et al., 2023c; Burns et al., 2023); (it) controlling systems through edits, such as implanting or removing trojans (Mazeika et al., 2022) or targeted editing (Ghorbani & Zou, 2020; Dai et al., 2022; Meng et. al., 2022a;b; Bau el. al., 2018; Hase et al., 2023); (iii) replacing components with simpler everse-enginnered alternatives (Lindner et al., 2023); and (iv) comprehensive evaluation through replacing components with hypothesized circuits (Quirke et al., 2024).
Algorithmic testbeds are essential for evaluating faithfulness (Jacovi & Goldberg, 2020; Hanna et al., 2024) and falsifiability (Leavitt & Morcos, 2020). Tools like Tracr (Lindner et al., 2023) can provide ground truth labels for benchmarking search methods (Goldowsky-Dill et al., 2023), while toy models studying superposition in computation (Vaintrob et al., 2024) and transformers on algorithmic tasks can quantify sparsity and test intrinsic methods. Recently, Thurnherr & Scheurer (2024); Gupta et al. (2024) introduced datasets of transformer weights with known circuits for evaluating mechanistic interpretability techniques.
8.3 Scaling Techniques
Broader and Deeper Coverage of Complex Models and Behaviors. A primary goal in scaling mechanistic interpretability is pushing the Pareto frontier between model and task complexity and the coverage of interpretability techniques (Chan, 2023). While efforts have focused on larger models, it is equally erucial to scale to more complex tasks and provide comprehensive explanations essential for provable safety (Tegmark & Omohundro, 2023; Dalrymple et al., 2024; Gross et al,, 2024) and enumerative safety (Cunningham et al., 2024; Elhage et al., 2022b) by ensuring models won't engage in dangerous behaviors like deception. Future work should aim for thorough reverse engineering (Quirke & Barez, 2023), integrating proven modules into larger networks (Nanda et al., 2023a), and capturing sequences encoded in hidden states beyond immodiate predictions (Pal et al., 2023). Deepening analysis complexity is also key, validating the realism of toy models (Elhage et al., 2022b) and extending techniques like path patching (Goldowsky-Dill et al., 2023; Liu et al., 2023a) to larger language models. The field must move beyond small transformers
29

---

Published in Transactions on Machine Learning Research (08/2024)
algorithmic tasks (Nanda et al., 2023a) and limited scenarios (Friedman et al., 2023a) to tackle more complex, realistic cases.
Towards Universality. As mechanistic interpretability matures, the field must transition from isolated empirical findings to developing overarching theories and universal reasoning primitives beyond specific circuits, aiming for comprehensive understanding of Al capabilities. While collecting empirical data remains valuable (Nanda, 20231), establishing motifs, empirical laws, and theories capturing universal model behavior aspects is crucial. This may involve finding more circuits /feuttree (Nanda, 2022a;c), exploring circuits as lens for memorization cammoriiatiiongemnaaliiiatiooooi /generalization (Hanna et al., 2023), identifying primitive general reasoning skills (Feng & Steinhardt, 2023), generalizing specific findings to model-agnostic phenomena (Merullo et al., 2023), and investigating emergent model generality across neural network classes (Ivanitskiy et al., 2023). Identifying universal reasoning patterns and unifying theories is key to advancing interpretability.
Automation. Implementing automated methods is crucial for scaling interpretability of real-world state- of-the-art models across size, task complexity, behavior coverage, and analysis time (Hobbhahn, 2022). Manual circuit identification is labor-intensive (Lieberum et al., 2023), so automated techniques like circuit discovery and sparse autoencoders can enhance the process (Foote et al., 2023; Nanda, 2023b). Future work should automatically create varying datasets for understanding circuit. functionality (Conmy et al., 2023); develop automated hypothesis search (Goldowsky-Dill et al., 2023), and investigate attention head/MLi interplay (Monca ct al., 2023). Scaling sparse autoencoders to extract high-quality features automatically for frontier models is critical (Bricken el. al., 2023). Still, it requires caution regarding potential downsides like AI iteration outpacing training (RieG, 2023) and loss of human interpretability from tool complexity Velez & Kim, 2017).
8.4 Expanding Scope
Interpretability Across Training. While mechanistic interpretability of final trained models is prereq- uisite, the field should also advance interpretability before and during training by studying learning dynamics (Nanda, 2022b; Elhage et al., 2022b; Hubinger, 2022). This includes tracking neuron development (Liu et al., 2021), analyzing neuron set changes with scale (Michaud et al., 2023), and invostigating emergent computa- tions (Quirke & Barez, 2023). Studying phase transitions could yield safety insights for reward hacking risks (Olsson et al., 2022).
Multi-Level Analysis. Complementing the predominant bottom-up methods (Hanna et al., 2023), mech- anistic interpretability should explore top-down and hybrid approaches, a promising yet neglected avenue. The top-down analysis offers a tractable way to study large models and guide microscopic research with macroscopic observations (Variengien & Winsor, 2023). Its computational efficiency could enable extensive 'comparative anatomy' of diverse models, revealing high-level motifs underlying abilities. These motifs could serve as analysis units for understanding internal modifications from techniques like instruction fine-tuning (Ouyang al., 2022) and reinforcement learning from human feedback (Christiano al., 2017; Bai et al., 2022).
New Frontiers: Vision, Multimodal, and Reinforcement Learning Models. While some mecha- nistic interpretability has explored convolutional neural networks for vision (Cammarata et al., 2021; 2020), vision-language models (Palit et al., 2023; Salin et al., 2022; Hilton et al., 2020), and multimodal neurons (Goh ct al., 2021), little work has focused on vision transformers (Palit et al., 2023; Aflalo et al., 2022; Vilas et al., 2023; Pan et al., 2024). Future efforts could identify mechanisms within vision-language models, mirroring progress in unimodal language models (Nanda et al., 2023a; Wang et al., 2023).
Reinforecment learning (RL) is also crucial frontier given its role in advanced AI training via techniques like reinforcement learning from human leedback (RLHF) (Christiano el al., 2017; Bai et al., 2022), despite potentially posing significant safety risks (Bereska & Gavves, 2023; Casper et al., 2023a). Interpretability of RL should investigate reward/goo representations (Mini et al., 2023; Colognese & Jozdien, 2023; Colognese, 2023: Bloom& & Colognese, 2023; Bloom & Bailey, 2023), study circuitry changes from alignment algorithms (Prakash et al., 2024; Jain et al., 2023; Lee et al., 2024; Jain et al., 2024), and explore emergent subgoals
30

---

Published in Transactions on Machine Learning Research (08/2024)
proxies (Hubinger et al., 2019; Ivanitskiy et al., 2023) such internal reward models (Marks et al., 2023b). While current state-of-the-art AT systems prediction-trinned LLMs are considered relatively safe (Hubinger et al., 2023), progress on interpreting RL systems may prove critical for safeguarding the next paradigm (Aschenbrenner, 2024).
Acknowledgements
am grateful for the invaluable leedback and comments from Leon Lang, Tim Bakker, Jannik Brinkmamm, Can Rager, Louis van Harten, Jacqueline Bereska, Benjamin Shaffrey, Thijmen Nijdam, Alice Rigg, Arthur Conmy, and Tom Licberum. Their insights substantially improved this work.
31

---

Published in Transactions on Machine Learning Research (08/2024)
Glossary
circuits Sub-graphs within neural networks consisting of features and the weights connecting them. Circuits can be thought of as compulational primilives that perform understandable operations to produce (ideally interpretable) features from prior (ideally interpretable) features. Examples include circuits for detecting curves at specific orientations (Cammarata et al., 2020; 2021), continuing repeated patterns in text (Olsson et al., 2022), and resolving anaphoric references (Wang et al., 2023). While circuits can involve clearly interpretable features, the definition allows for intermediate representa- tions that are less easily interpretable.. 3, 9, 10, 20, 26, 27, 35
concepts An abstract idea or representation derived from observations of the world. Concepts refer to the natural abstractions that cognitive system, like a neural network, aims capture and represent through its learned features, which may may not align perfectly with human-defined concepts.. 4, 6, 7, 32, 34
deceptive alignment. When misaligned model aims appear aligned to gain more power to take control sufficiently powerful.. 24
deceptive inflation Theoretical result on deceptive behavior: policies produce trajectories that look better than they actually are from the human's perspective with limited observations to get higher reward signals during training. This deceptive behavior arises in reinforcement learning from human feed- back when the human provides feedback based only on partial observations of the trajectories, while the policy has full state information during training (Lang et al., 2024).. 28
disentangled In disentangled representations, individual dimensions components correspond to distinct, independent factors of variation in the data, rather than reprosenting tangled mixture of these factors.. 4, 9, 15, 16, 20, 27, 32, 33
eliciting latent knowledge Developing strategies make a machine learning model explicitly report la- tent facts knowledge embedded in its parameters, especially in cases where the model's output is untrusted (Christiano et al., 2021). This involves finding patterns in neural network activations that track the true state of the world (Mallen & Belrose, 2023).. 24
features
The fundamental units of how neural networks encode knowledge, which cannot be further de- composed into smaller, distinct concepts. Features are core components of a neural network's rep- resentation, analogous to how cells form the fundamental unit of biological organisms (Olah et al., 2020). The superposition hypothesis suggests an alternative definition: that features correspond to the disentangled concepts that a larger, sparser network with sufficient capucity would learn to represent with individual (monosemantic) neurons (Olah et al., 2020; Bricken et al., 2023).. 3, 4, 6-8, 10, 13, 16, 18, 20, 23, 27, 28, 32, 33, 35
grokking 'Grokking refers to the surprising phenomenon of delayed generalization where neural networks, on certain learning problems, generalize long after overfitting their training set.' (Liu et al., 2022a).
21, 22
hydra effect
The phenomenon where models can internally self-repair and maintain capabilities even when key components ablated, making it challenging to identify the relevant. components underlying particular behavior (McGrath et al., 2023).. 18, 26
inner misalignment Inner misalignment, or goal misgeneralization, occurs when an Al system develops goals or behaviors during training that misaligned with the intended objectives despite correctly specified reward signal.. 24, 28
internal world models Internal causal environment models formed within neural networks, implicitly emerging by-product of prediction (e.g., in large language models).. 11, 12, 24, 34
32

---

Published in Transactions on Machine Learning Research (08/2024)
irreducible We adopt the notion of features as the fundamental units of neural network representations, such that features cannot. be further decomposed into smaller, distinct factors. To make this more precise, we can formalize the definition of features as irreducible input patterns following Engels ct al. (2024): A feature f of sparsity probability - 8 > 0) into a higher-dinmensionll representation ational space. We say the feature is active on this subset. A feature is reducible into features a and b if there exists a transformation that decomposes f into a and such that the transformed distribution p(a, b) is either:
a function that maps a subset of the input space (with
1. Separable: p(a, b) p(a)p(b)
2. A mixture: p(a, b) wpi (a, b) + -1-ww w)pp(a, () where pi is lower-dimensionall
Features are defined as irreducible patterns that cannot be decomposed into separable or mixture dis- tributions via such transformations. This formalizes the notion that features form the fundamental atomic units underlying neural representations. Features that can be disentangled into statistically independent components (separable) or simpler lower-dimensional factors (mixtures) are not con- sidered the representational primitives. The key properties are that 1) features map from the input space to higher-dimensional representational spaces, 2) fealures are sparse and only acti- vated on subsets of the input, and crucially, 3) features are irreducible and cannot be expressed as transformations of other statistically independent components. 4
iterative distillation and amplification A technique for training AI systems by repcatedly distilling knowledge from & larger model into a smaller one while amplifying the smaller model's capabili- ties through feedback and interaction with humans.. 24
linear representation Features are directions in activation space, i.e., linear combinations of neurons.. 3, 8, 14, 28
machine unlearning Techniques for removing private data or dangerous knowledge from models..
mesa-optimization
The emergence of unintended subagents within model with their own objectives, potentially misaligned with the original training objective. 24
microscope AI Systems that extract and utilize knowledge from a model without allowing the model to take autonomous actions. This involves reverse engineering a trained model to understand its Icarned knowledge about the world, aiming leverage this understanding directly without deploying the model in operational capacity.. 24
modularity
The property of an Al system being composed of distinct, semi cemi-independent components or submodules that can be separately understood, modified, and recombined, rather than a monolithic, opaque structure.. 20, 21
monosemantic A neuron corresponding to single concept. The intuition is that analyzing what inputs activate given neuron reveals its associated semantic meaning concept. In contrast to polyse- mantic. 5, 8, 14, 16, 20, 32, 34
motifs Repeating patterns that emerge across models and tasks, manifesting as circuits, features, or higher- level behaviors from component interactions. Examples include curve detectors, induction circuits, and branch specialization. Motifs reveal common structures and mechanisms underlying neural network intelligence.. 3, 10, 21, 35
natural abstractions High-level summarics or descriptions of system or environment learned and used
by many cognilive systems. According to the natural abstraction hypothesis (Chan el al., 2023), a set of "natural' abstractions exist that represent redundantly encoded information in the world and tend to be learned by intelligent systems produced through local selection pressures. These natural abstractions form a relatively small, discrete set of concepts like 'tree,' "velocity, etc., that allow compact descriptions of the world while discarding many irrelevant low-level details.. 4, 11, 32
33

---

Published in Transactions on Machine Learning Research (08/2024)
outer misalignment Outer misalignment, reward hacking, occurs when the specified reward function or utility function fails to capture the desired objectives correctly. This leads the AT to optimize for behaviors that achieve high reward scores but are misaligned with the intended outcomes.. 28, 34
oversight (Scalable) oversight refers to the challenge of providing reliable supervision -hhrisi labels, reward signals, critiques to Al models, ensuring effectiveness even as models surpass human- level performance.. 23, 24
polysemantic Neurons that are associated with multiple, unrelated concepts, contradicting the interpre- tation of neurons as representational primitives and making it challenging to understand the infor- mation processing of neural networks. This term is derived from linguistic concepts of polysemy (Falkum & Vicente, 2015), and in the context of neural networks first introduced by Arora et al. (2018), who suggested that word embeddings of polysemous words may be stored as a superposition of vectors representing distinct meanings. Olah et al. (2020) first used the term polysemanticity, claborating on the concept of polysemantic neurons as a challenge for mochanistic interpretability..
5, 16, 28, 33
prediction orthogonality A model whose objective is prediction can simulate agents who optimize toward any objectives with any degree of optimality (janus, 2022).. 3, 12, 24
privileged basis In certain neural network representations, the basis directions formed by the individual neurons are architecturally distinguished from arbitrary directions in the activation space. This privileged basis makes mcaningful to analyze the propertics and roles of individual neurons, as the architecture encourages features to align with these basis directions. Hence, a privileged basis is necessary but not sufficient for the formation of monosemantic neurons. (Elhage et al., 2022b).. 5
representation engineering top-down approach to transparency research that treats representations as the fundamental unit of analysis, aiming to understand and control representations of high-level cognitive phenomena in neural networks like large language models. Representation engineering has two main areas: 1) Reading representations to probe and interpret. their contents, and 2) Controlling representations to manipulate high-level concepts like honesty morality (Zou et al., 2023).. 3, 9, 22
reverse engineering The process of deconstructing a neural network's computations to fully understand and specily its operations. This involves breaking down the network's functionality into explicit, interpretable components, potentially clear and detailed as pseudocode.. 1, 3, 20, 21, 25-27, 29
reward hacking See outer misulignment.. 30
simulacra The text oulputs generaled by predictive model simulating the causal processes underlying lext creation. These outputs simulate coherent and contextually relevant language, sometimes exhibiting agentic bchaviors or goals despite the predictive model itself lacking genuine agency or intentionality. Simulacra can be either agentic, mimicking intentional and persuasive language use, or non- con-agentic, merely generating descriptive text without simulated goals agency (janus, 2022; Bereska & Gavves, 2023).. 12, 14
simulation
The simulation hypothesis says that when scaled up sufficiently, predictive models will learn to simulate the real-world causal processes that generated their training data (janus, 2022). When these models are optimized for predictive accuracy on broad data distributions like natural language, they are incentivized to discover the underlying rules, physics, and semantics that govern the data to model and predict future observations effectively. This allows the models to go beyond just mem- orizing or pattern-matching their training sels, instead learning to simulate hypothetical scenarios, reason about counterfactuals, and exhibit behaviors characteristic general intelligence - all as a byproduct of the drive for efficient compression and accurate prediction. The simulation hypothesis suggests these models will develop rich internal world models capturing the causal dynamics of the training distribution.. 11, 12, 14, 24
34

---

Published in Transactions on Machine Learning Research (08/2024)
streetlight interpretability Examining Al systems under only ideal conditions of maximal interpretabil- ity, risking missing critical phenomena that only emerge in more realistic and diverse contexts.. 25
superposition
The superposition hypothesis suggests that neural networks leverage high-dimensional spaces to represent more features than the actual count of ncurons by encoding features in almost orthogonal directions (Elhage et al., 2022b).. 3, 5, 6, 16, 21, 28, 32, 34
sycophancy The tendeney of models to generate responses that align with user beliefs rather than providing truthful information. This behavior, encouraged by human feedback used in fine-tuning, is observed in state-of-the-art. assistanns across various tasks (Sharma et al., 2023). Sycophancy arises because human preference judgments often favor responses that match users' views, leading to a preference for convincingly written sycophantic responses over correct ones.. 28
universality The universality hypothesis proposes the emergence of common circwils across neural network models trained on similar tasks and data distributions. A stronger form posits that these common circuits represent a set of fundamental computational motifs that neural networks gravitate towards when learning. The weaker version suggests that for given task, dataset, and model architecture, an optimal way to solve the problem may exist, which different models will tend to converge towards, resulting in analogous circuits. The universality hypothesis implies that rather than each model learning arbitrary, unstructured representations, there an underlying universality the circuits that emerge, shaped by the learning task and inductive biases.. 3, 9, 10, 21, 22
well-founded AI Developing Al systems with provable safety guarantees about their behavior and align- ment with human values through rigorous mathematical modeling and verification. (TTgmmark & Omohundro, 2023; Dalrymple al., 2024).. 24
35

---

Published in Transactions on Machine Learning Research (08/2024)
References
Estelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, and Vasudev Lal. VI-
interpret: An interactive visualization tool for interpreting vision-language transformers. CVPR, June 2022. 30
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using lincar classifier probes. ICLR, 2016. 9, 14
Uri Alon. An introduction to systems biology: design principles of biological circuits. Chapman and Hal- 1/CRC, 2019. 21
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Palcka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. CoRR, 2024. 9
Aryaman Arora, Dan Juralsky, and Christopher Potts. Causalgym: Benchmarking causal interpretability
methods on linguistic tasks. CoRR, February 2024. 19
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. TACL, December 2018. 5, 34
Leopold Aschenbrenner. Situational awareness: The decade ahead. Series: Situational Awareness. June, 2024. 31
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, July 2015.
Nicholas Bai, Rahul Ajay Tyer, Tuomas Oikarinen, and Tsui- Weng. Describe-and-disoot:: Interpreting neurons in vision networks with language models. ICML MI Workshop, June 2024. 23
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Josoph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, April 2022. 30
Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural represen- talions. CoRR, June 2021. 3
Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. NeurIPS, 2022. 24
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. ICLR, December 2018. 29
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. CoRR, September 2021. 3, 9, 14, 19
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. CoRR, August 2023. 13, 15
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? ACM FAceT, March 2021. 11
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastion Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mecha- nisms. CoRR, February 2019. 28
36

---

Published in Transactions on Machine Learning Research (08/2024)
Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwarlz, Gillian Hadfield, Jeff Clune, Tegan Maharaj, Frank Hutter, Atilmm Günes Baydin, Sheila Mcllraith, Qiqi Gao, Ashwin Acharya, David Krueger, Anca Dragan, Philip Torr, Stuart Russell, Daniel Kahneman, Jan Brauner, and Sören Mindermann. Managing ai risks in an era of rapid progress. CoRR, November 2023.
Leonard Bereska. Mechanistic interpretability for adversarial robustness Blog, August 2024. 28
proposal. Leonard Bereska 's
Leonard Bereska and Eistratios Gavves. Taming simulators: Challenges, pathways and vision for the align- ment of large language models. AAAT-SS, October 2023. 24, 30, 34
Michel Besserve, Arash Mehrjou, Rémy Sun, and Bernhard Schölkopf. Counterfactuals uncover the modular structure of deep generative models. CoRR, December 2019.
28
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, llya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. OpenAI Blog. 2023. 5, 23, 25
Blair Bilodeau, Natasha Jaques, Pang Wei Koh, and Been Kim. Impossibility theorems for feature attribu- tion. Proc. Natl. Acad. Sci. U.S.A., Jamuary 2024." 2
Christopher M. Bishop. Pattern recognition and machine learning. Springer- New York Inc., 2006. 4
Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Mcrizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, and Comnor Leahy. Interpreting neural networks through the polytope lens. CoRR, November 2022. 9, 28
Joseph Bloom and Jay Bailey. Features and adversaries in memorydt. Less Wrong, October 2023. 29, 30
Joseph Bloom and Paul Colognese. Decision transformer interpretability. Al Alignment Forum, 2023. 30
Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Vi'egas, and M. Wattenberg. An interpretability illusion for bert. CoRR, April 2021. 29
Dan Braun, Jordan Taylor, Nicholas Goldowsky-Dill, and Lee Sharkey. Tdentifying functionally important features with end-to-end sparse dictionary learning. ICML MI Workshop, May 2024.
19
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah F. Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, October 2023. 5, 6, 8, 9, 11, 13, 15, 20, 30, 32
Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. A mechanistic analysis of transformer trained on symbolic multi-step reasoning task. CoRR, February 2024. 22
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lec, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribciro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpl-4. CoRR, April 2023.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. TCLR, 2023. 3, 13, 24,24, 29
Lucius Bushnaq, Stefan Heimersheim, Nicholas Goldowsky-Dill Dan Braun, Jake Mendel, Kaarel Hanni, Avery Griffin, Jorn Stohler, Magdalena Wache, and Marius Hobbhahn. The local interaction basis: Iden- tifying computationall--eleeaan and sparsely interacting features in neural networks. CoRR, May 2024.
19
37

---

Published in Transactions on Machine Learning Research (08/2024)
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. ICLR, October 2022. 21
Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schubert, Michael Petrov, and Chris Olah. Curve detectors. Distill, June 2020. 10, 15, 21, 30, 32
Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve circuits. Distill, 2021. 10, 21, 30, 32
Steven Cao, Victor Sanh, and Alexander M. Rush. Low-complexity probing via finding subnetworks. NAACL- HLT, April 2021. 14
Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas. Distill,
March 2019. 28
Giuseppe Casalicchio, Christoph Molnar, and Bernd Bischl. Visualizing the feature importance for black box models. ECML, PKDD, 2018. 2
Stephen Casper. The engineer's interpretability sequence. Al Alignment Forum, February 2023. 24, 25, 27, 28, 29
Stephen Casper, Max Nadeau, Dylan Hadfield-Menell, and Gabriel Kreiman. Robust feature-level adversaries are interpretability tools. NeurIPS, October 2021. 28
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel- Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem BIyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dy- lan Hadeleld-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback. CoRR, 2023a. 24, 30
Stephen Casper, Kaivalya Hariharan, and Dylan Hadlield-Menell. Diagnostics for deep neural networks with automated sopy/paste attacks. NeurIPS 2022 ML Safety Workshop (Best paper award), May 2023b. 25
Stephen Casper, Yuxiao Li, Jiawei Li, Tong Bu, Kevin Zhang, Kaivalya Hariharan, and Dylan Hadfield- Menell. Red tcaming deep neural networks with feature synthesis tools. NeurIPS, 2023c. 25, 29
Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, Mar- vin Von Hagen, Silas Alberti, Alan Chan, Qinyi Sun, Michael Gerovitch, David Bau, Max Tegmark, David Krueger, and Dylan Hadfield-Menell. Black-box access is insufficient for rigorous ai audits. ACM Conference on Fairness, Accountability, and Transparency. January 2024. 1, 28
Lawrence Chan. What would do if wasn't at are evals. AT Alignment Forum, May 2023. 24, 28, 29
Lawrence Chan, Adrià Garriga-alonso, Nicholas Goldowsky-Diin ryan greenblatt, jenny, Ansh Radhakr- ishnan, Buck, and Nate Thomas. Causal scrubbing: method for rigorously testing interpretability hypotheses [redwood research). AI Alignment Forum, December 2022. 13, 19
Lawrence Chan, Leon Lang, and Erik Jenner. Natural abstractions: Key claims, theorems, and critiques.
Al Aligument Forum, March 2023. 4, 11, 33
David Chanin, Anthony Hunter, and Oana-Maria Camburu. Identifying linear relational concepts in large language models. CoRR, 2023.
Charbel- Raphaäl. Against almost every theory of impact of interpretability. AI Alignment Forum, August 2023. 25
38

---

Published in Transactions on Machine Learning Research (08/2024)
Yiting Chen, Zhanpeng Zhou, and Junchi Yan. Going beyond neural network feature similarity: The network
feature complexity and its interpretation using category theory. CoRR, November 2023a. 11
Zhongtian Chen, Edmund Lau, Jake Mendel, Susan Wei, and Daniel Murfet. Dynamical versus bayesian
phase transitions in toy model of superposition. CoRR, October 2023b. 21, 25
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. NewrIPS, December 2017. 30
Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge, Jamuary 2021. 24, 32
Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality; Reverse engineering how networks learn group operations. ICML, 2023. 11, 22, 25
Bilal Chughtai, Alan Cooney, and Neel Nanda. Summing up the facts: Additive mechanisms behind factual
recall in llms. NewrIPS Workshop Attributing Model Behaviour at Scale, 2024. 10
Paul Colognese. Internal target information for ai oversight. Less BessWrong, 2023. 30
Paul Colognese and Jozdien. High-level interpretability; detecting an ai's objectives. Al Alignment Forum, 2023. 30
Arthur Commy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. NeurIPS, 2023. 18, 19, 22, 25, 30
Tan C. Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: a. unified framework for model explanation. J. Mach. Learn. Res., January 2021.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find
highly interpretable features in language models. ICLR, Jamuary 2024. 8, 9, 13, 15, 16, 19, 29
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. ACI, 2022. 5, 29
David 'davidad' Dalrymple, Joar Skalse, Yoshua Bengio, Stuart Russell, Max Tegmark, Sanjit Seshia, Steve Omohundro, Christian Szegedy, Ben Goldhaber, Nora Ammann, Alessandro Abate, Joe Halpern, Clark Barrelt, Ding Zhao, Tan Zhi-Xuan, Jeannelte Wing, and Joshua Tenenbaum. Towards guaranteed safe ai: A framework for ensuring robust and reliable ai systems. CoRR, May 2024. 29, 35
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Authony Bau, and James Glass, What is one grain of sand in the desort? analyzing individual neurons in deep nlp models. Proceedings of the AAAI Conference on Artificial Intelligence, July 2019. 14
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. ACL, December 2022. 15
Xander Davies, Max Nadeau, Nikhil Prakash, Tamar Rott Shaham, and David Bau. Discovering variable binding circuitry with desiderata. CoRR, July 2023. 10, 22, 23
Mingyang Deng, Lucas Tao, and Joc Benton. Mcasuring feature sparsity in language models. CoRR, 2023. 9, 15
Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, Ethan Perez, and Evan Hubinger. Sycophancy to subterfuge: Investigating reward-tampering in large language models. CoRR, 2024. 28
Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to conclusions: Short-cutting transformers with linear transformations. CRRR, March 2023. 15
39

---

Published in Transactions on Machine Learning Research (08/2024)
Finale Doshi- Velez and Been Kim. Towards rigorous science of interpretable machine learning. CoRR, March 2017. 21, 29, 30
Keke Du, Shan Chang, Huixiang Wen, and Hao Zhang. Fighting adversarial images with interpretable gradients. ACM TURC, October 2021. 28
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable Ilm feature circuits. ICML MI Workshop, June 2024, 16
Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. Analyzing individual neurons in pre- trained language models. EMNLP, October 2020.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. TACL, February 2021. 14
Nelson Elhage, Tristan Hume, Olsson Catherine, Nanda Neel, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jack- son Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacobson, Eli Tran- Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuils Thread, 2022a. 5, 8, 13, 20
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac -oddds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. Transformer Circuits Thread, 2022b. 5, 6, 7, 9, 15, 25, 26, 27, 28, 29, 30, 34, 35
Joshua Engels, Isaac Liao, Eric J. Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear. CoRR, May 2024. 4, 8, 33
Logan Engstrom, Andrew Nyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness as a prior for learned representations. CoRR, September 2019. 28
Ingrid Lossius Falkum and Agustin Vicente. Polysemy: Current perspectives and approaches. Lingua, April 2015. 34
Sebastian Farquhar, Vikrant Varma, Zachary Kenton, Johannes Gasteiger, Vladimir Mikulik, and Rohin Shah. Challenges with unsupervised Ilm knowledge discovery. CoRR, 2023. 14
Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. Causalm: Causal model explanation through counterfactual language models. Computational Linguistics, May 2021. 19
Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? CoRR, October 2023. 10, 22, 26, 30
Javier Ferrando and Elena Voita, Information flow routes: Automatically interpreting language models at scale. CoRR, February 2024. 18, 23
Javier Ferrando, Gabricle Sarti, Arianna Bisazza, and Marta R. Costa-jussà. A primer on the inner workings of transformer-based language models. CoRR, May 2024. 2
Alex Foote, Necl Nanda, Esben Kran, Ioannis Konstas, Shay Cohen, and Fazl Barez. Neuron to graph: Interpreting language model neurons at scale. CoRR, May 2023. 23, 30
Stanislav Fort and Balaji Lakshminarayanan. Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness. CoRR, August 2024. 28
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, March 2019. 20
Dan Friedman, Andrew Lampinen, Lucas Dixon, Danqi Chen, and Asma Ghandeharioun. Interpretability illusions in the generalization of simplified models. CoRR, 2023a. 29, 30
40

---

Published in Transactions on Machine Learning Research (08/2024)
Dan Friedman, Alexander Wettig, and Danqi Chen. Learning transformer programs. NewrIPS, June 2023b.
20
Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. ICLR, 2023a. 25
Deqing Fu, Tian-Qi Chen, Robin Jia, and Vatsal Sharan. Transformers learn higher-order oplimization methods for in-context learning: A study with linear models. CoRR, October 2023b. 26
Zach Furman and Edmund Lau. Estimating the local learning coefficient at scale. CoRR, February 2024, 21
Jorge Carcia-Caarassee Alejandro Maté, and Juan Trujillo. Detecting and understanding vulnerabilities in language models via mechanistic interpretability. IJCAI, August 2024. 28, 29
Albert Garde, Esben Kran, and Fazl Barez. Deepdecipher: Accessing and investigating neuron activation in
large language models. NeurIPS Workshop XAIA, October 2023. 13
Peter Gardenfors. Conceptual spaces: The geometry of thought. MIT press, 2004. 12
Charles J. Garfinkle and Christopher J. Hillar. On the uniqueness and stability of dictionaries for sparse
representation of noisy signals. IEEE Transactions on Signal Processing, December 2019. 15
Xuyang Ge, Fukang Zhu, Wentao Shu, Junxuan Wang, Zhengfu He, and Xipeng Qiu. Automatically identi- fying local and global circuits with linear computation graphs. CoRR, May 2024. 19
Atticus Geiger, Hanson Lu, Thomas Ieard, and Christopher Potts. Causal abstractions of neural networks. NewrIPS, 2021a. 18
Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah D. Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. 1CML, Jamuary 2021b.
17
Atticus Geiger, Chris Polts, and Thomas Teard. Causal abstraction for faithful model interpretation. CoRR, January 2023a. 13, 17, 18, 19
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. Finding align- ments between interpretable causal variables and distributed neural representations. CoRR, 2023b. 13,
18, 19
Georgios Georgiadis. Accelerating convolutional neural networks via activation map compression. CoRR, March 2019. 20
Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer food-forward layers build predictions by promoting concepts in the vocabulary space. EMNLP, October 2022. 15
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. EMNLP, October 2023. 17
Asma Ghandcharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. CoRR, January 2024. 18
Amirata Ghorbani and James Zou. Neuron shapley: Discovering the responsible neurons. NeurIPS, Novem-
ber 2020. 5, 29
Gabriel Goh, Nick Cammarata t, Chelsea Voss t, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, March 2021. 5, 30
Nicholas Goldowsky-Din Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. CoRR, 2023. 13, 17, 18, 29, 30
Liv Gorton. The missing curve detectors of inceptionvl: Applying sparse autoencoders inceptionvl early vision. ICML MT Workshop, June 2024. 15
41

---

Published in Transactions on Machine Learning Research (08/2024)
Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, interpretable attention heads in the wild. CoRR, 2023. 29
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bern- hard Schölkopf. Recurrent independent mechanisms. CoRR, November 2020. 28
Jason Gross, Rajashree Agrawal, Thomas Kwa, Euan Ong, Chun Hei Yip, Alex Gibson, Soufiane Noubir, and Lawrence Chan. Compact proofs of model performance via mechanistic interpretability. ICML MI Workshop, June 2024. 29
Roger Grosse, Juhan Bac, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhosscin Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilè Lukošiūté, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions. CoRR, August 2023. 13
Phillip Huang Guo, Aaquib Syed, Abhay Sheshadri, Aidan Ewart, and Gintare Karolina Dziugaite. Robust unlearning via mechanistic localizations. ICML MI Workshop, June 2024. 25
Rohan Gupta, Iván Arcuschin, Thomas Kwa, and Adrià Garriga-Alonso. Interpbench: Semi-synthetic trans- formers for evaluating mechanistic interpretability techniques. CoRR, July 2024. 29
Wes Gurnee and Max Tegmark. Language models represent space and time. ICLR, 2024. 11
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in haystack: Case studies with sparse probing. TMLR, 2023. 7, 13, 14
Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaci Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models. CoRR, January 2024. 11
David R. Ha and J. Schmidhuber. Recurrent world models facilitate policy evolution. NewrIPS, September 2018. 11
Guy Hacohen, Leshem Choshen, and Daphna Weinshall. Let's agrec to agree: Neural networks share classification order on real datasets. ICML, 2020. 11
Michael Hanna, Ollic Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. NewrIPS, 2023. 10, 17, 22, 30
Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. ICML MI Workshop, June 2024. 29
Kaarel Hänni, Jake Mendel, Dmitry Vaintrob, and Lawrence Chan. Mathematical models of computation in superposition. 1CML MI Workshop, August 2024. 29
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models, NewrIPS Spotlight, Jamuary 2023. 29
Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt. CoRR, 2024. 15
Stefan Heimersheim and Jett. A circuit for python doestrings in 4-layer attention-only transformer. AI Alignment Forum, February 2023. 22
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. EMNLP, October
2023. 9, 17, 22
Dan Hendrycks. Introduction to Al Safety, Ethics, and Society. Self-published, 2023a. 26
Dan Hendrycks. Natural selection favors ais over humans. CoRR, July 2023b. 25
42

---

Published in Transactions on Machine Learning Research (08/2024)
Dan Hendrycks and Mantas Mazeika. X-risk analysis for ai research. CoRR, June 2022. 1,24 25
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. CoRR, June 2022. 26
Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks. CoRR, October 2023. 1, 24
Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread, 2023. 2, 27
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, 7ac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling laws and inter- pretability of learning from repeated data. CoRR, 2022. 21
Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. CoRR, August 2023.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. NAACL HIT, June 2019. 14
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. CoRR, December 2018. 27
Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, and Chris Olah. Understanding rl vision. Distill, 2020. 30
Geoffrey E Hinton. Distributed representations. Carnegie Mellon University, 1984. 27
Marius Hobbhahn. Marius' aligument. agenda, 2022. 30
Marius Hobbhahn and Lawrence Chan. Should we publish mechanistic interpretability research? Al Aliga- ment Forum, April 2023. 25
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optinal large language models. CoRR, March 2022. 25
Jesse Hoogland, Liam Carroll, and Daniel Murfet. Stagewise development in neural networks. Al Alignment Forum, March 2024. 21
Jing Huang, Atticus Geiger, Karel D'Oosterlinck, Zhengxuan Wu, and Christopher Potts. Rigoronsly as- sessing natural language explanations of neurons. CoRR, September 2023. 5
Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. Ravel: Evaluating inter- pretability methods on disentangling language model representations. CoRR, 2024. 19
Evan Hubinger. Chris olah's views on agi safety. Al Alignment Forum, November 2019a. 4, 24, 26
Evan Hubinger. Gradient hacking. AT Alignment Forum, October 2019b. 26
Evan Hubinger. An overview of 11 proposals for building safe advanced ai. CoRR, December 2020. 24
Evan Hubinger. A transparency and interpretability tech tree. Al Alignment Forum, June 2022. 8, 24, 26,
30
43

---

Published in Transactions on Machine Learning Research (08/2024)
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. CoRR, May 2019. 24, 31
Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Conditioning predictive models: Risks and strategies. CoRR, February 2023. 24, 31
Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, and Ethan Perez. Sleeper agents: Training deceptive 1lms that persist through safety training. CoRR, 2024. 28, 29
Andrew Tlya,, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. NeurIPS, August 2019. 4, 28, 29
M. Ivanitskiy, Alexander F. Spies, Tilman Rauker, Guillaume Corlouer, Chris Mathwin, Lucia Quirke, Can Rager, Rusheb Shah, Dan Valentine, Cecilia Diniz Behn, Katsumi Inoue, and Samy Wu Fung. Structured world representations in maze-solving transformers, CoRR, December 2023. 11, 30, 31
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? CoRR, April 2020. 29
Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Edward Grefenstette,
Tim Rocktäschel, and David Scott Krueger. Mechanistically analyzing the elfects of fine-tuning on proce- durally defined tasks. CoRR, November 2023. 30
Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip Torr, Amartya Sanyal, and Puneet K. Dokania. What makes and breaks safety fine-tuning? mechanistic study. ICML MI Workshop, June 2024. 30
janus. Simulators. Less Wrong, September 2022. 11, 12, 24, 34
Erik Jenner, Adrià Garriga-alonso, and Egor Zverev. A comparison of causal scrubbing, causal abstractions, and related methods. AI Alignment Forum, June 2023. 13, 19
Erik Jenner, Shreyas Kapur, Vasil Georgiev, Cameron Allen, Scott Emmons, and Stuart Russell. Evidence of learned look-ahead in a chess-playing neural network. CoRR, June 2024. 24
Adam Jermyn, Chris Olah, and T Henighan. Circuits updates may 2023: Attention head superposition. Transformer Circuits Thread, 2023. 29
Adam S. Jermyn, Nicholas Schiefer, and Evan Hubinger. Engineering monosemanticity in toy models. CoRR, November 2022. 8, 20, 25
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O'Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo, and Wen Gao. Ai alignment: A comprehonsive survey. CoRR, Jamuary 2024.
Jozdien. Conditioning generative models for alignment. Al Alignment Forum, July 2022. 24
Jaap Jumelet. Evaluating and interpreting language models. NLP Lecture, November 2023. 2
Amlan Jyoti, Karthik Balaji Ganesh, Manoj Gayala, Nandita Lakshmi Tumuguntla, Sandesh Kamath, and Vinceth N. Balasubramanian. On the robustness of explanations of deep neural network models: A survey. CoRR, November 2022. 28
44

---

Published in Transactions on Machine Learning Research (08/2024)
Adam Karvonen. Emergent world models and latent variable estimation in chess-playing language models. COLM, July 2024. 11
Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Smith, C. M. Verdun, David Bau, and Samuel Marks. Measuring progress in dictionary learning for language model interpretabil- ity with board game models. ICML MI Workshop (Oral), July 2024. 15
Thcodoros Kasioumis, Joc Townsend, and Hiroya Inakoshi. Elite backprop: Training sparse interprotable neurons. International Workshop on Neuro-Symbolic Learning and Reasoning, 2021. 20
Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Christopher Pal. Systematic evaluation of causal discovery in visual model based reinforcement learning. CoRR, July 2021. 28
Jan Kirchner. Neuroscience and natural abstractions. Less Wrong, March 2023. 11
Connor Kissanc, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neol Nanda. Interpreting allention layer outputs with sparse autoencoders. ICML MI Workshop, June 2024. 15
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. ICML, July 2019. 3, 11
János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. Atp*: An efficient and scalable method for localizing Ilm behaviour to components. CoRR, March 2024. 22
Nicholas Kross. Why and when interpretability work is dangerous. Less Wrong, May 2023. 25
Jan Kulveit. Risks from ai misalignment at different scales, July 2024. 24
Jan Kulveit, Clem von Stengel, and Roman Leventov. Predictive minds: Lims atypical active inference agents. CoRR, November 2023. 11
Max Lamparth and Anka Reuel. Analyzing and editing inner mechanisms of backdoored language models. CoRR, 2023. 29
Michael Lan and Faul Barez. Localing cross-task sequence continuation circuits in transformers. CoRR, November 2023. 10
Leon Lang, Davis Foote, Stuart Russell, Anca Dragan, Erik Jenner, and Scott Emmons. When your ais deceive you: Challenges with partial observability of human evaluators in reward learning. CoRR, March 2024. 28, 32
Georg Lange, Alex Makelov, and Neel Nanda. An interpretability illusion for activation patching of arbitrary subspaces. AT Aligmment. Forum, August 2023. 18
Anna Langedijk, Hosein Mohebbi, Gabriele Sarti, Willem Zuidema, and Jaap Jumelet. Decoderlens: Layer- wise interpretation of encoder-decoder transformers. CoRR, 2023. 15
Edmund Lau, Daniel Murfet, and Susan Wei. Quantifying degeneracy in singular models via the learning cocfficient. CoRR, August 2023. 21
Connor Leahy. Barriers to mechanistic interpretability for agi safety. Al Alignment Forum, 2023. 26
Matthew L. Leavitt and Ari Morcos. Towards falsifiable interpretability rescarch. CoRR, October 2020. 29
Victor Lecomte, Kushal Thaman, Trevor Chow, Rylan Schaeffer, and Sanmi Koyejo. Incidental polyseman- licity. CoRR, 2023. 27, 28, 29
Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg. Jonathan K. Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. CoRR, 2024. 30
45

---

Published in Transactions on Machine Learning Research (08/2024)
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language
models. ACT-IICNLPP, August 2021. 12
Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on synthetic task. ICLR, 2023a.
8, 11
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspoter Pfister, and Martin Wattenberg. Inferenco-time inter- vention: Eliciting truthful answers from a language model. NeurIPS Spollight, July 2023b. 9
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? NTPS Workshop on Fealure Fatraction, December 2015.
11
Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chin- chilla. CoRR,, July 2023. 10, 17, 22, 29, 30
Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. Explainable ai: review of machine learning interpretabillty methods. Entropy, December 2020. 14
David Lindner, János Kramér, Sebastian Farquhar, Matthew Rahtz, Thomas McGrath, and Vladimir Miku- lik. Tracr: Compiled transformers as a laboratory for interpretability. CoRR, 2023. 29
Leo Z. Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. Probing across time: What does roberta know and when? EMNLP, September 2021. 30
Ziming Liu and Max Tegmark. A neural scaling law from lottery ticket ensembling. CoRR, 2023. 21
Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. NewrIPS, 2022a. 21, 32
Ziming Liu, Eric J. Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. 1CML, 2022b. 21
Ziming Liu, Eric Gan, and Max Tegmark. Seeing is believing: Brain-inspired modular training for mechanistic interpretability. Entropy, June 2023a. 13, 20, 23, 29
Ziming Liu, Mikail Khona, Ila R. Fiete, and Max Tegmark. Growing brains: Co-emergence of anatomical and functional modularity in recurrent neural networks. CoRR, 2023b. 20
Ziming Liu, Ziqian Zhong, and Max Tegmark. Grokking as compression: A nonlinear complexity perspective. CoRR, 2023c. 21
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled represen- tations. CoRR, June 2019. 27
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. CoRR, August 2024. 26
Ang Lv, Yuhan Chen, Kaiyi Zhang, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, and Rui Yan. Inter- preting key mechanisms of factual recall in transformer- based language models. CoRR, 2024. 10
Sara Magliacane, Thijs van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M. Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. NeurIPS, October 2018. 28
Alcksandar Makelov. Sparse autoencoders match supervised features for model steering on the ioi task. TCML MT Workshop, June 2024. 15
46

---

Published in Transactions on Machine Learning Research (08/2024)
Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. CoRR, May 2024. 15
Alex Mallen and Nora Belrose. Eliciting latent. knowledge from quirky language models. CoRR, December 2023. 32
Narek Maloyan, Ekansh Verma, Bulat Nutfullin, and Bislan Ashinov. Trojan detection in large language models: Insights from the trojan detection challenge. CoRR, April 2024. 28
Giovanni Luca Marchetti, Christopher Hillar, Danica Kragic, and Sophia Sanborn. Harmonics of learning:
Universal fourier features emerge in invariant networks. CoRR, December 2023. 11
Luke Marks, Amir Abdullah, Luna Mendez, Rauno Arike, Philip Torr, and Fazl Barez. Interproting reward models in rlliftuned language models using sparse autoencoders. CoRR, October 2023a. 15
Luke Marks, Amir Abdullah, Clement Neo, Rauno Arike, Philip Torr, and Fazl Barez. Beyond training objectives: Interpreting reward model divergence in large language models. CoRR, October 2023b. 31
Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. CoRR, March 2024. 19
Simon C. Marshall and Jan H. Kirchner. Understanding polysemanticity in neural networks through coding theory. CoRR, January 2024. 6, 28
Mantas Mazeika, Andy Zou, Akul Arora, Pavel Pleskov, Dawn Song, Dan Hendrycks, Bo Li, and David Forsyth. How hard is trojan detection in dnns? fooling detectors with evasive trojans. CoRR, September 2022. 29
Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppression: Comprehensively understanding an attention head. CoRR, October 2023. 10
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. PNAS, November 2022. 13, 14, 26
Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. CoRR, July 2023. 18, 26, 28, 32
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. NewrIPS, 2022a. 13, 17, 24, 29
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in transformer. ICLR, 2022b. 29
William Merrill, Nikolaos Tsilivis, and Aman Shukla. A tale of two circuits: Grokking competition of sparse and dense subnetworks. CoRR, 2023. 21
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. A mechanism for solving relational tasks in transformer language models. CoRR, May 2023. 30
Fric J. Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. CoRR, March 2023. 9, 21, 30
Eric J. Michaud, Isaac Liao, Vedang Lad, Ziming Liu, Anish Mudide, Chloe Loughridge, Zifan Carl Guo, Tara Rezaci Kheirkhah, Mateja Vukelić, and Max Tegmark. Opening the ai black box: program synthesis via mechanistic interpretability. CoRR, February 2024. 23, 27
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. NewrIPS, October 2013.
47

---

Published in Transactions on Machine Learning Research (08/2024)
Joseph Miller and Clement Neo. We found an neuron in gpt-2. AI Alignment Forum, February 2023. 22
Tim Miller, Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, February 2019. 29
Ulisse Mini, Peli Grietzer, Mrinank Sharma, Austin Moek, Monte MacDiarmid, and Alexander Matt Turner. Understanding and controlling a maze-solving policy network. CoRR, October 2023. 30
Giovanni Monea, Maxime Peyrard, Martin Josifoski, Vishrav Chaudhary, Jason Eisner, Emre Kiemmnn Hamid Palangi, Barun Patra, and Robert West. A glitch in the matrix? locating and detecting language model grounding with fakepedia. CoRR, 2023. 30
Basel Mousi, Nadir Durrani, and Fahim Dalvi. Can Ilms facilitate interpretation of pre-trained language models? FMNLP, 2023. 23
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. NewrIPS, June 2020. 5, 25, 29
Aaron Mueleer, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. CoRR, August 2024. 17
Jatin Nainani. Evaluating brain-inspired modular training in automated circuit. discovery for mechanistic interpretability. CoRR, January 2024. 22, 23
Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L. Edelman, Fred Zhang, and Boaz Barak. Sgd on noural networks learns functions of incrcasing complexity. NeurIPS, May 2019. 21
Neel Nanda. 200 cop in mi: Looking for circuits in the wild. Neel Nanda's 's Blog, 2022a. 30
Neel Nanda. 200 cop in mi: Analysing training dynamics. Neel Nonda's 's Blog, 2022b. 30
Neel Nanda. 200 cop in mi: Studying learned features in language models. Neel Nanda's 's Blog, 2022c. 30
Neel Nanda. A comprehensive mechanistic interpretability explainer & glossary. Neel Nonda's 's Blog, Decem- ber 2022d. 2
Neel Nanda. A longlist of theories of impact for interpretability. Al Alignmeett Forum, March 2022e. 23
Neel Nanda. 200 cop in mi: Exploring polysemanticity and superposition. Neel Nanda's 's Blog, January 2023a. 29
Neel Nanda. 200 cop in mi: Techniques, tooling and automation. Neel Nanda's 's Blog, 2023b. 30
Neel Nanda. Actually, othello-gpt has a linear emergent world representation. Neel Nanda's 's Blog, March
2023c. 8
Neel Nanda. Attribution patching: Activation patching at industrial scale. Neel Nanda's Blog, February 2023d. 18
Neel Nanda. How to think about activation patching. AT Alignament Forum, April 2023e. 18, 21
Neel Nanda. Mechanistic interpretability quickstart guide. Neel Nanda's Blog, January 2023f. 2, 30
Neel Nanda, An extremely opinionated annotated list of my favourite mechanistic interpretability papers v2. AI Alignment Forum, July 2024. 2
Neel Nanda, Lawrence Chan, Tom Licberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. ICLR, January 2023a. 10, 21, 22, 24, 25, 29, 30
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. BlackbozNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, September 2023b. 8, 11, 28
48

---

Published in Transactions on Machine Learning Research (08/2024)
Neel Nanda, S. Rajamanoharan, J. Kramár, and R. Shah. Fact finding: Attempting to reverse-engineer factual recall on the neuron level. AT Aligument Forum, 2023c. URL hllps://www.. alignmentforum. (333BB/ffck fon.dinggatemmiinn----reee-ee ennarr/---rreeeeee-enneeneereere factual-raaalll,, 2023c. 10
Geraldin Nanfack, Alexander Fulleringer, Jonathan Marty, Michael Eickenberg, and Eugene Belilovsky, Adversarial attacks on the interpretation of neuron activation maximization. Proceedings of the AAAT Conference on Artificial Intelligence, March 2024. 13
Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from deep learning per-
spective. CoRR, December 2022. 24
Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chume Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning. CoRR, October 2022. 24, 25
Eshaan Nichani, Alex Damian, and Jason D. Lee. How transformers learn causal structure with gradient descent. CoRR, February 2024. 11
NicholasKecs and janus. Searching for search. Al Alignment Forum, November 2022. 24
nostalgebraist. interpreting gpt: the logit lens. Al Aligrmmet Forum, August 2020. 13, 14, 19
Chris Olah. Distributed representations: Composition & superposition. Transformer Circuits Thread, 2023.
27
Chris Olah and Shan Carter. Research debt. Distill, March 2017.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, November 2017.
29
Chris Olah, Arvind Satyanarayan, lan Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, March 2018. 2
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, March 2020. 2, 4, 5, 9, 10, 21, 32, 34
Christopher Olah. Mechanistic interpretability, variables, and the importance of interpretable bases. Trans-
former Circuils Thread, 2022. 2, 4
B. A. Olshausen and D. Field. Sparse coding with an overcomplete basis set: strategy employed by v1? Vision Res, December 1997. 15, 16
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodde Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuils Thread, 2022. 10, 21, 22, 25, 30, 32
Laura O' O'Mahony, Vincent Andrearezyk, Henning Muller, and Mara Graziani. Disentangling neuron repre- sentations with concept vectors. CVPR Workshops, April 2023. 9
Charles O'Neill and Thang Bui. Sparse autoencoders enable scalable and reliable circuit identification in language models. CoRR, May 2024. 16, 19
Francesco Ortu, Zhijing Jin, Diego Doimo, Mrinmaya Sachan, Alberto Cazzaniga, and Bernhard Schölkopf. Competition of mechanisms: Tracing how language models handle facts and counterfactuals. ACL. 2024.
June 2024. 10
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishknnn Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. CoRR, 2022. 30
49

---

Published in Transactions on Machine Learning Research (08/2024)
Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C. Wallace, and David Bau. Future lens: Anticipating subsequent tokens from a single hidden state. CoNLI, 2023. 15, 29
Vedant Palit, Rohan Pandey, Aryaman Arora, and P. Liang- Towards vision-language mechanistic inter- pretability: A causal tracing tool for blip. ICCVW, 2023. 30
Xu Pan, Aaron Philip, Ziqian Xie, and Odelia Schwartz. Dissocting query-key interaction in vision trans- formers. ICML MI Workshop, June 2024. 30
Kiho Park, Yo Joong Choc, and Victor Veitch. The lincar representation hypothesis and the goometry of large language models. NeurIPS Workshop on Causal Representation Learning, November 2023a. 28
Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. ICML MI Workshop (Oral), June 2024. 9
Peter S. Park, Simon Goldstein, Aidan O'Gara, Michael Chen, and Dan Hendrycks. Ai deception: A survey of examples, risks, and potential solutions. CoRR, August 2023b. 24
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. ICLR, 2022. 12
Judea Pearl. Causality. Cambridge University Press, 2009. 11, 17
Judea Pearl and Dana Mackenzie. The Book of Why: The New Science of Cause and Effect. Penguin Books Limited, May 2018. 28
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference using invariant prediction: identi- fication and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, November 2015. 28
Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, October 2017. 28
Nicky Pochinkov and Nandi Machine unlearning evaluations as interpretability benchmarks. Al Alignment Forum, August 2023. 25
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. ICML, April 2023. 25
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. CoRR, Jamuary 2022. 21
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. ICLR, February 2024. 30
Philip Quirke and Fazl Barez. Understanding addition in transformers. CoRR, October 2023. 10, 29, 30
Philip Quirke, Clement Neo, and Fazl Barez, Increasing trust in language models through the reuse of verified circuits. CoRR, February 2024. 29
Daking Rai, Yilum Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mechanistic interpretability for transformer- based language models. CoRR, July 2024.
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders. CoRR, April 2024. 16
Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfiel--Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. TMLR, August 2023. 1, 13, 25, 26, 27, 28,
29
50

---

Published in Transactions on Machine Learning Research (08/2024)
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. ACI, July 2020. 19
Jie Ren, Mingjee Li, Qirui Chen, Huiqi Deng, and Quanshi Zhang. Defining and quantifying the emergence sparse concepts in dnns. CoRR, April 2023. 27
Marco Tulio Ribciro, Sameer Singh, and Carlos Guestrin. 'why should trust you?": Explaining the predic-
Lions of any classifier. NAACL, August 2016. 2, 19
RicG. Agi-automated interpretability is suicide. Less Wrong, May 2023. 25, 30
Jonathan Richens and Tom Everitt. Robust agents learn causal world models. ICLR Oral, February 2024.
11
Ryan Riegel, Alexander Gray, Francois Luus, Nawoed Khan, Ndivhuwo Makondo, Ismail Yunus Akhalwaya, Haifeng Qian, Ronald Fagin, Francisco Barahona, Udit Sharma, Shajith Ikbal, Hima Karanam, Sumit Neelam, Ankita Likhyani, and Santosh Srivastava. Logical neural networks. NeurIPS, June 2020. 20, 27
Mateo
Bernhard Scholkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. JMLR, 2018. 28
Andrew Slavin Ross and Finale Desei-Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. AAAI, November 2017. 28
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use inter- pretable models instead. Nat Mach Intell, May 2019. 29
Cody Rushing and Neel Nanda. Explorations of self-repair in language models. ICML, May 2024. 18
Thane Ruthenis. Internal interfaces are a high-priority interpretability target. AT Alignment Forum, De- cember 2022. 24
Thane Ruthenis. Workd-model interpretability is all we need. Al Alignment Forum, January 2023. 24
Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level interpretation of deep nlp models: A survey. TACL, November 2022. 5
Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and lan Foster. Memory injections: Correcting multi-hop reasoning failures during inference in transformer-basee language models. CoRR, September 2023a.
Mansi Sakarvadia, Arham Khan, Aswathy Ajith, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. Attention lens: A tool for mechanistically interpreting the attention head infor- mation retrieval mechanism. CoRR, October 2023b. 15
Emmanuelle Salin, Badreddine Farah, Stéphane Ayache, and Benoit Favre. Are dision-languag, transformers learning multimodal representations? probing perspective. AAAI, June 2022. 30
Tommaso Salvatori, Ankur Mali, Christopher L. Buckley, Thomas Lukasicwicz, Rajesh P. N. Rao, Karl Friston, and Alexander Ororbia. Brain-inspired computational intelligence via predictive coding. CoRR, 2023. 11
Naomi Saphra. Interpretability creationism. The Gradient, 2023. 21
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilitios of large language models a mirage? CoRR, May 2023. 21
Adam Scherlis, Kshitij Sachan, Adam S. Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks. CoRR, July 2023. 6, 7, 26, 27
51

---

Published in Transactions on Machine Learning Research (08/2024)
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning. Special Issue of Proceedings of the IEEE Advances in Machine Learning and Deep Newral Networks, February 2021. 27
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. NeurIPS Oral, October 2022. 25
Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? visual explanations from deep networks via gradient- based localization. ICCV, 2016. 2
Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role play with large language models. Nature, November 2023. 11, 12, 24
Lloyd S. Shapley. A value for -person games. Cambridge University Press, October 1988. 2
Lee Sharkey. Circumventing interpretability: How to defeat mind-readers. CoRR, December 2022. 24, 26
Lee Sharkey. A technical note on bilinear layers for interpretability. CoRR, May 2023. 20
Lee Sharkey, Sid Black, and beren. Current themes in mechanistic interpretability research. Al Alignment Forum, November 2022a. 2,
Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoencoders.
AI Alignment Forum, 2022b. 8, 15, 16
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, New- ton Cheng, Esin Durmus, Zac HatfieldDoodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndlusse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy language models. CoRR, October 2023. 28, 35
Justin Shovelain and Elliot McKernon. The risk-reward tradeoff of interpretability research. Less Wrong, July 2023. 25
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. ICML, 2017. 2
James B. Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J. Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. TCML, May 2023. 21
slavachalnev. Sparse mlp distillation. Less Wrong, 2024. 16
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. CoRR, June 2017. 2
Nate Soares. If interpretability research goes well, it may get dangerous. Less Wrong. April 2023. 25
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, 1lya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent noural networks from overfitting. JMLR, 2014.
Dashiell Stander, Qinan Yu, Honglu Fan, and Stella Biderman. Grokking group multiplication with cosets. CoRR, 2023. 21
Jacob Steinhardt. Emergent deception and emergent optimization. Bounded Regret, February 2023. 21, 24, 28
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic
reasoning in language models using causal mediation analysis. EMNLP, October 2023. 17, 22
52

---

Published in Transactions on Machine Learning Research (08/2024)
Ila Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C. Love, Erin Grant, Tris Groen, Jascha Achterberg, Joshua B. Tenenbaum, Katherine M. Collins, Katherine L. Hermann, Kerem Oktar, Klaus Greff, Martin N. Hebart, Nori Jacoby, Qiuyi Zhang, Raja Marjieh, Robert Geirhos, Sherol Chen, Simon Kornblith, Sunayana Rane, Talia Konkle, Thomas P. O'Connell, Thomas Un- terthiner, Andrew K. Lampinen, Klaus-Robert Müller, Mariya Toneva, and Thomas L. Griffiths. Getting aligned on representational alignment. CoRR, November 2023.
Chen Sun, Nolan Andrew Miller, Andrey Zhmoginov, Max Vladymyrov, and Mark Sandler. Learning and unlearning of fabricated knowledge in language models. 1CML MI Workshop, June 2024. 25
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. 1CML, June 2017. 2
Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. CoRR, October 2023. 18, 19, 22
technicalities and Stag. Shallow review of live agendas in alignment & safely. Less Wrong, 2023. 24
Max Tegmark and Steve Omohundro. Provably safe systems: the only path to controllable agi. CoRR, September 2023. 24, 29, 35
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, and Brian Chen. Scaling monoseeannicctt:: Extracting interpretable features from claude 3 sonnet. Transformer Cireuits Thread, 2024. 15
Ian Tenney, Dipanjan Das, and Ellic Pavlick. Bert rediscovers the classical nlp pipeline. ACL, August 2019.
14
Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon. CoRR, 2022. 21
Hannes Thurnherr and Jérémy Scheurer. Tracrbench: Generaling interpretability testbeds with large lan- guage models. 1CML MI Workshop, June 2024. 29
Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Language models linearly represent sentiment. ICML MI Workshop, June 2024. 9
Fric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. CoRR, 2023. 9
Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, and Joseph J. Lim. Learning to synthesize programs as inter- pretable and generalizable policies. NewrIPS, 2021. 27
Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Looch, Ulisse Mini, and Monte MacDiarmid.
Activation addition: Steering language models without optimization. CoRR, September 2023.
Dmitry Vaintrob, jake sete___ee,l,,,e and Kaarel. Toward mathematical framework for computation in super- position. AT Aligument Forum, 2024. 29
Alexandre Variengien and Eric Winsor. Took before you leap: A universal emergent. decomposition of retrieval tasks in language models. 1CML MI Workshop, December 2023. 11, 22, 30
Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficicncy. CoRR, September 2023. 21, 22
Abhinav Verma, Hoang M. Lc, Yisong Yuc, and Swarat Chaudhuri. Imitation-projecteed programmatic reinforcement learning. NewrIPS, 2019a. 27
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmect Kohli, and Swarat Chaudhuri. Program- matically interpretable reinforcement learning. CoRR, April 2019b. 27
53

---

Published in Transactions on Machine Learning Research (08/2024)
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. NewrIPS, 2020. 17, 19
M. Vilas, Timothy Schaumlöffel, and Gemma Roig. Analyzing vision transformers for image classification in class embedding space. CoRR, 2023. 30
Elena Voita and Ivan Titov. Information probing with minimum description length. EMNLP,
March 2020. 14
Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, gram, positional. CoRR, September 2023. 5
Julius von Kügelgen, M. Loog, A. Mey, and B. Scholkopf. Semi-supervised learning, causality, and the conditional cluster assumption. Conference on Uncertainty in Artificial Intelligence, May 2019. 28
Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera y Arcas, Max Vladymyrov, Razvan Pascamu, and João Sacramento. Uncovering mesa-oplimization algorithms in transformers. CoRR, September 2023. 24, 26
Chelsea Voss, Gabriel Goh, Nick Cammarata, Michael Petrov, Ludwig Schubert, and Chris Olah, Branch specialization. Distill, April 2021. 10, 21
Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasonors: A mechanistic journey to the edge of generalization. ICML MI Workshop, June 2024.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in gpt-2 small. ICLR, 2023. 10, 13, 17, 18, 19, 22, 30, 32
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohanancy, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 2020. 2
Sumio Watanabe. Algebraic Geometry and Statistical Learning Theory. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2009. 21
Sumio Watanabe. Matheraatical Theory of Bayesian Stalistics. Chapman and Hall, edition, April 2018. 21
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilitics of large language models. TMLR, October 2022. 21, 24
John Wentworth. How go from interpretability to alignment: Just retarget the search. Al Alignment Forum, August 2022. 24
James C. R. Whittington, Will Dorrell, Surya Ganguli, and Timothy E. J. Behrens. Disentangling with biological constraints: A theory of functional cell types. CoRR, September 2022. 15
Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Back- doorbench: A comprehensive benchmark of backdoor learning. NeurIPS Datasets and Benchmarks, Oc- tober 2022a. 29
Zhengxuan Wu, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, and Noah Goodman. Causal distillation for language models. NAACL-HITT, July 2022b. 19
54

---

Published in Transactions on Machine Learning Research (08/2024)
Zhengxuan Wu, Karel D'Oosterlinck, Atticus Geiger, Amir Zur, and Christopher Potts. Causal proxy models for concept-based model explanations. TCML, 2023a. 13, 19
Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D. Goodman. Interpretability at scale: Iden- tifying causal mechanisms in alpaca. CoRR, May 2023b. 19, 22
Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. Reft: Representation finetuning for language models. CoRR, May 2024. 19
Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models. CoRR, October 2023. 10
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks, ECCV, 2014. 13
Biao Zhang, Ivan Titov, and Rico Sennrich. Sparse attention with linear units. EMNLP, October 2021. 20
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep
learning requires rethinking generalization. ICLR, February 2017. 21
Fred Zhang and Neel Nanda. Towards best practices activation patching in language models: Metrics and methods. ICLR, 2023. 18
Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu. Interpreting cmns via decision trees. CVPR, 2019. 27
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. CoRR, 2023. 22
Roland S. Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas S. A. Wallis, and Wieland Brendel. How well do feature visualizations support causal understanding of enn activations?
NewrIPS, November 2021. 13
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazcika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathanicl Li, Michael Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency. CoRR, October 2023.
3, 9, 14, 22, 24,34 34
55


Reviews:
[{"review_id": "oBy1LL9DV3", "reviewer": "Reviewer_QHfE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**:\n\n1. The topic of survey, mechanistic interpretability, is important and may attract interests from many researchers. \n\n2. This paper does a extensive investigation for mechanistic interpretability. As far as I know, there are few survey papers on this topic. \n\n\n\n**Weakness**:\n\nMy main concerns come from the organization of the survey paper. I understand that mechanistic interpretability is more complex than attribution/concept interpretability. But at current vision, I find it difficult to provide a coherent overview of this field. To improve the readability, I have following comments:\n\n1. **There is short of overview figures/tables**, so as to (such as) clarify the connection between core concepts, the relationship between concepts and hypotheses, and so on. For example, how can we distinguish from features, semantics, neurons, and basis? What’s the connection between features, circuits, and motif? How does the Universality Hypothesis relate to these core concepts? I notice that the authors may give answers to some of the above questions in the texts, but this is not intuitive and impressive. It is crucial to help the readers form a clear overall picture.\n\n2. This survey only lists core concepts and hypothesis, but **lacks a summary and refinement of key research problems** in the field of mechanism interpretability. Key research problems are essential to investigating a research field. Based on my understanding of this survey, key research problems may be “How a neural network  encode semantics? Is it encoded in neurons?”, “Whether a single neuron corresponds to an explicit semantic?”, and so on. I suggest the authors to have a deep thinking on summarizing research problems. \n\n3. **There is a significant gap between the core concepts/research problems (Section 3) and methods/techniques (Section 4)**. The authors should give more details or examples on how to adopt these techniques in interpreting the mechanisms of neural networks. \n\n4. I find the survey a little long and difficult to read. I suggest to condense the paper and polish the language, to stress the key point and make the statement more precise and understandable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on investigating current methodologies in the field of mechanistic interpretability, which aims to uncover causal relationships and precise computations transforming inputs into outputs in a neural network. Towards this field, this paper presents its core concepts and hypotheses, explains methods and techniques, discusses evaluation, explores its relevance to AI safety, and points out challenges and future directions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weakness. The four points are important to my final decision.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FE7xDdEae6", "reviewer": "Reviewer_Jfw9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. I think this paper is a well-constructed and researched survey paper that goes over definitions, hypothesis, methods and relevant aspects of the field of mechanistic interpretability. I have  some thoughts on making it a more comprehensive repository of information on mechanistic interpretability, and some things to change below:\n2. It would be great to have a  more detailed anthology as in this paper. E.g., see this paper https://arxiv.org/abs/2311.04329 that links each definition to its place in the glossary allowing readers to go over and access all definitions easily.)\n3. If each definition could be numbered and then linked in the glossary that would also help retrieve them more easily.\n4. This paper interestingly falls in between two types of papers i.e., critique/position papers vs. a survey of all of the work in the field. Most of the paper is structured as a survey paper with definitions of important terms, hypotheses, algorithms and methods with the relevant citations needed for these. The latter part of the paper is more of a critique of aspects of the field. This division is great! However, in between some of the definitions/methods are critiques of certain methods that are not citable, or at least cited in this paper (they seem more subjective). For the sake of a comprehensive/unbiased survey paper, I would urge the authors to not provide subjective viewpoints in the descriptive/survey part. I’ve listed them all below.\n5. An overarching concern with this type of survey paper is that the field is emerging so fast that a lot of the things in this paper are already outdated (e.g., claims such as “so far no one has done so and so in a certain task”) are not largely true anymore even at the time of reviewing this paper. It’s clearly necessary to outline what has and hasn’t been done until this point in time but I would urge the authors to reduce such claims.\n6. For e.g., on page 8, “only relatively narrow behaviours like Python docstring formatting” have been explored so far”.\n7. On page 8:” Hypothesis: Universality” it might be nice to also have the hypotheses linked in a similar glossary?\n8. On page 9: on internal world models, there are several works on behavioural probing and mech interp that look into this that are not cited: https://openreview.net/forum?id=jE8xbmvFin, https://openreview.net/forum?id=gJcEM8sxHK, https://arxiv.org/abs/2106.00737 \n9. On page 12, re: “unsupervised methods can identify numerous features without a straightforward verification process”, can we add citations for this?\n10. On page 14, the overview of the different activation patching approaches is great, but if this could be elaborated in some detail and cited that would be helpful! \n11. Figure 7 is really great! \n12. The terms, inner misalignment and mesa optimisation could be helpful to define in the anthology.\n13, On page 22: “utilising a diverse interpretabiltiy toolbox”, how would the authors propose to do that? This sentence, without any explanation of what this implies doesn’t actually provide any insight or details into what could be done better.\n14. On page 24: this might be hard to do but it would be extremely helpful to have a table that maps the definitions to each other re: the point about how terms are redefined and new terms/definitions really mean the same underlying thing or concept.\n15. In section 2, RSA-like approaches could also be mentioned and cited here as a class of probing/interpretability techniques.\n16. Re: monosemantic and polysmenatic neurons: can we cite where these terms originate from? (e.g., used in the linguistics literature here, Vicente and Falkum (2017), but it would be nice to have the original paper that used this both in semantics, and also in the interpretability field).\n^maybe this paragraph should be phrased not in the context of neurons, but just talking about mono/polysemy, why it is important to make this distinction, and that it is about trying to find units that correspond to it. The next paragraph can then tie things together (as it does already).\n17. The definition of features could be made more concrete—is just saying “fundamental unit” sufficient? Is the implication that features are monosemantic (whether human interpretable or otherwise?) Also it is currently unclear to me if they are implying that features equate to neurons?\n18. When talking about world modeling and relating to AI safety: “emergent world models have significant implications … internal representation of human values” → it might be worth drawing the comparison between intrinsic values/human morals in the real world and the game/world representations that are typically studied e.g., in chess domains. These are usually smaller and less complex than a world that allows the incorporation of complex, nuances, human values and cultural information. \n19. Activation patching is defined but not path patching and more complex variants. It might also be worth diving into how the patching is done (and the links to causal interventions).\n20. “Causality as a theoretical foundation”: it would be great if this could/should be introduced a lot earlier? Especially when introducing patching techniques it is worth drawing the connection to causality and the theory behind a lot of this field of mechanistic interpretability methods. Even having some of the definitions of causal interventions from Pearl etc., would be helpful to readers new to this space!", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper serves as a survey and introduction to the field of mechanistic interpretability. It outlines the terminology, provides definitions, overviews recent approaches based on these, and also provides a critique of aspects of the field (e.g., fundamental limitations of some methods, weaknesses in the rigour of some evaluations, etc).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Simple change: numbering the definitions so it’s easy to refer to and go back to them.\n2. More involved change: Having an index / glossary at the end that lists out all the terms defined could be helpful to readers that want to skim through these terms and link back to the context of where they were defined.\n(e.g., format from several papers e.g., this paper: https://arxiv.org/abs/2311.04329)\n3. Addressing all comments above!", "broader_impact_concerns": "This paper is fairly well justified in its criticism of several aspects of the field and has a nuanced and balanced take on sensitive topics such as AI safety. I would urge the authors to take into account the above points though!", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9y4jaKCtQy", "reviewer": "Reviewer_ijrU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strenghts\n\n**Necessity of a common ground for Mechanistic interpretability research.** MI developed over the past years as a detached research area, with its own terminology and challenges, necessitating formal links to other better-known research branches in AI, especially with XAI. The reference to date introducing the notion of MI is [Olah 2020], and often cited as the primary contribution in the area, which is a blog post not followed by any peer-reviewed publication clarifying the terminology and the specific challenges of this research field. Following, other blog posts also explored aspects of MI, introducing novel ideas and terminologies, often conflating with existing literature in XAI. To this end, the effort of the authors to recollect several papers under the same umbrella and set the core concepts and challenges is valuable and can serve as a basis for future works in MI. Moreover, cross-fertilization with other AI research is promoted and could benefit both MI and XAI researchers.\n\n**Sound presentation of core ideas and useful taxonomy.** The authors contribute to pointing to the core concepts (features, circuits, motifs), and typical assumptions and hypotheses (superposition, linearity, universality, simulation). Core methods are divided into the observational vs interventional nature and into what aspects of training they target: pre/during/post- training. This taxonomy is helpful and does comprise many, if not all, works in MI. \n\n**Relevance to AI safety.** Authors point to applications of MI to safety/trustworthy AI, which could be integrated with current LLMs research. The discussion touches on both helpful and harmful aspects of MI applied to safe AI, making MI a potential new avenue of research also for trustworthy AI. \n\n**Future challenges.** Eventually, authors detect future challenges and research directions that can help improve current methodologies in MI. This is valuable when pointing to common standards that future research should share and to indicating problems and future avenues to the researchers in the field. \n\n# Weaknesses\n\n### Clarity\nSeveral parts require more scrutiny and better formalization. While previous papers and blog posts avoid being specific in terminology and formalization, the paper should avoid adopting a confusing conceptualization.\n\n**Definitions.** In section 3, the authors introduce the notions of features, circuits, and motifs. In the presentation, however, (human-understandable) concepts are often mentioned but not explained making somehow vague how should the various notions be interpreted. The reference to concepts keeps appearing in different parts of the text and appears to be rooted in intuition. However, this poses a degree of ambiguity in definitions. In section 3.1 the authors write *'' We adopt the notion of features as the smallest units of how neural networks encode knowledge, such that features cannot be further decomposed into smaller, distinct concepts.''* which becomes vague and cannot be parsed from the text. In this respect, it is not clear (Definition: Feature, Alternative) what are the disentangled concepts. This should be made clear, especially in light of the presupposed distinction between concept-based explanations and MI methods (Sec 2 - Mechanistic). Mentioning *patterns* in (Def: Motifs) is also ambiguous and not clear from the text. If a pattern is either a feature or a circuit, it should be specified. \n\nRegarding the hypotheses, universality is not also crystal clear what is the meaning that should be attributed to *analogous*: the same, or similar (up to what)? The definition of simulation is also a bit vague, regarding what it means to ''_simulate the underlying causal process of textual data_'' and what it means to ''_optimize strongly the model_''. As a stylistic note, it would be helpful to separate into a different subsection the notions of features and circuits and the discussion on the last three hypotheses (as mentioned at the beginning of section 3).\nI am also not convinced that violations of the linearity hypothesis have not been observed yet: it depends on the context and clear counter-examples of some linear notion of features are present in the literature [1].  If some sort of linearity of the features holds, it should be specified in what domains and contexts, e.g. [2].\n\n\n**Methods and Evaluations.** In the presentation of the core methods, at the attributional level methods based on (structured) probing have connections to XAI research. The review comprehends those that fall within MI, but it is not clear where the line should be drawn between XAI methods of this sort and MI methods. In light of the discussion in (Sec 4.1 - Structured Probes) regarding the ''_complementarity to MI efforts_''  it would be better to make clear the main differences between methods that use probes in MI and XAI. Feature Disentanglement is also mentioned only w.r.t. sparse autoencoders, but there is a rich literature in this respect, which should be mentioned in the potential future avenues of research, e.g. [1].\nThe presentation of Activation Patching can be improved. The explanation of the standard protocol can be made more precise by referring to Fig 7, and the example with A-OR-B and  A-AND-B cannot be understood entirely. \n\nAlso, it is not clear why methods based on causal abstractions and causal scrubbing are not discussed in Sec 4.2 and are related to rigorous hypothesis testing. This should be explained in the text. \n\n**Overall structure and taxonomy**\nWhile the overall structure is reasonable and offers a good distinction between methods, it would be helpful to resume what are the axes of distinctions between MI methods. The authors mention that methods can be distinguished based on their causal nature (observational/interventional) and the phase of learning they study (pre/during/post). Locality and globality are also mentioned for post-hoc methods and their comprehensiveness. These constitute other axes among which methods can be regrouped as customary in XAI research, see [3].\n\n### Minors\n\n**Connection to other research areas.** It is clear that MI has several intersections with other research on XAI. The main difference is the bottom-up approach of MI methods, although, it is not clear if all methods respect this general idea (e.g.  MI methods based on causal abstractions and high-level concepts, called top-down, ref. Sec 8.2 - Obstacles to bottom-up interpretability). Causality also seems to play a central role, due to disentangled representations and to interventions. Thus, the intersection with other works on XAI based on concepts [4] and Causal Representation Learning (CRL) [5]  can be fruitful for both research fields and MI. Recent works in CRL address the emergence of the Linear Hypothesis [6] and provably learning of (causal) concepts in foundation models [7]. Similarly, causality and CRL have been applied to defining causally interpretable representations [8].  Other works in this direction are also particularly relevant to Mechanistic Interpretability [9].\n\n**Listing key concepts and working assumptions.** It can be helpful to provide an enlarged illustration of how MI differs from other methods and list the key concepts and hypotheses that are peculiar to MI, compared to other methods in XAI. This can help facilitate researchers approaching the field from related areas.\n\n[1] Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations - ICML, Locatello et al. (2019) \\\n[2] Relative representations enable zero-shot latent space communication - ICLR, Moschella et al. (2023) \\\n[3] Explainable AI: A Review of Machine Learning Interpretability Methods - Entropy, Linardatos et al. (2021)  \\\n[4] Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors - ICML, Kim et al. (2018) \\\n[5] Towards Causal Representation Learning - IEEE, Scholkopf et al. (2021) \\\n[6] On the Origins of Linear Representations in Large Language Models - arxiv, Jiang et al. (2024) \\\n[7] Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Models - arxiv, Rajendran et al. (2024) \\\n[8] Interpretability is in the Mind of the Beholder: A Causal Framework for Human-interpretable Representation Learning - Entropy, Marconato et al. (2023) \\\n[9] Impossibility theorems for feature attribution, PNAS, Bilodeau et al. (2024)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Mechanistic Interpretability (MI) is a new avenue of research that targets reverse engineering the functioning of deep learning models. In this paper, the authors review works in this area by analyzing the fundamental concepts and hypotheses, collecting the core methods, presenting evaluations and tested scenarios, discussing applications to AI safety, and describing possible future avenues. MI so far stands as an independent branch of Explainable AI (XAI) for bottom-up explainability,\ncomprising the study of intrinsic interpretable methods,\n the analysis of neural networks' dynamics, \nand assessing post-hoc evaluations of trained models (especially LLMs). \nThe authors collected a series of works from these three branches to create a common ground for future research in MI.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Authors should address the main weaknesses of their work:\n\n1) Better formalization of the notions and concepts presented in their manuscript. It is crucial to fix the relevant parts, especially if this review appears to be the official reference for MI.\n\n2) Taxonomy listing how MI approaches are related to one another, w.r.t. to axes the authors highlighted. This could strengthen the presentation, providing a concise overview of different works in MI.\n\nI hope the authors will also take into account other suggestions if not all, which may improve connections to other fields of research.", "broader_impact_concerns": "There is no major negative ethical implications concerning this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper structure and content has been appreciated by all reviewers. The aim of structuring the literature around interpretability has been found as a worthy goal, and reviewers considered that an omni-comprehensive survey could be impossible. At the same time, the initial revision of the manuscript has been criticized for missing some literature connections and references and for presenting dimensions that appeared to be detached one from another in the presentation. Lastly, some statements were imprecised and needed to be rewritten.

Authors took all comments into account and provided a revised paper that substantially improved presentation and readability, as well as better covering the interpretability landscape. All reviewers are willing to accept the paper and I agree with them. Furthermore, two reviewers agreed on proposing the Survey certification, as the revised version now looks pretty polished.
Decision: Accept (poster)