Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition

## Abstract

Abstract Automated mechanistic interpretation research has attracted great interest due to its potential to scale explanations of neural network internals to large models. Existing automated circuit discovery work relies on activation patching or its approximations to identify subgraphs in models for specific tasks (circuits). They often suffer from slow runtime, approximation errors, and specific requirements of metrics, such as non-zero gradients.
In this work, we introduce contextual decomposition for transformers (CD-T) to build interpretable circuits in large language models. CD-T can produce circuits of arbitrary level of abstraction, and is the first able to produce circuits as fine-grained as attention heads at specific sequence positions efficiently .
CD-T is compatible to all transformer types, and requires no training or manually-crafted examples.
CD-T consists of a set of mathematical equations to isolate contribution of model features. Through recursively computing contribution of all nodes in a computational graph of a model using CD-T followed by pruning, we are able to reduce circuit discovery runtime from hours to seconds compared to state-of-the-art baselines.
On three standard circuit evaluation datasets (indirect object identification, greater-than comparisons, and docstring completion),
we demonstrate that CD-T outperforms ACDC and EAP by better recovering the manual circuits with an average of 97% ROC AUC under low runtimes.
In addition, we provide evidence that faithfulness of CD-T circuits is not due to random chance by showing our circuits are 80% more faithful than random circuits of up to 60% of the original model size.
Finally, we show CD-T circuits are able to perfectly replicate original models’ behavior (faithfulness = 1 absent 1 =1 = 1 ) using fewer nodes than the baselines for all tasks.
Our results underscore the great promise of CD-T for efficient automated mechanistic interpretability, paving the way for new insights into the workings of large language models. 1 1 1 CD-T will be released as an open-source python package to benefit the wide mechanistic interpretability research community.

## 1 Introduction

Transformers (Vaswani et al., ) have recently demonstrated impressive predictive capabilities (Brown et al., ) by learning intricate nonlinear relationships between features. However, the challenge of comprehending these relationships has resulted in transformers largely being thought of as black boxes. Despite this, transformers are increasingly utilized in high-stakes domains such as medicine (e.g. medical image analysis (He et al., )) and science (e.g. protein structure prediction aiding drug discovery (Jumper et al., )). This highlights the necessity of understanding model behaviors.
Mechanistic interpretability, a study to explain behaviors of neural networks in terms of their internal components, is at the frontier of interpretability research (Geiger et al., ; Geva et al., ; Räuker et al., ), as it uniquely provides an avenue for guiding manual modifications (Elhage et al., ; Vig et al., ) and reverse-engineering solutions (Elhage et al., ; Meng et al., ).

One main direction in mechanistic interpretability research is circuit discovery, where researchers aim to identify a computational subgraph (circuit) in neural networks responsible for solving a specific task (Olah et al., ). Prior circuit discovery work in language models has found subgraphs of attention heads and multi-layer perceptrons (MLPs) that partially or fully explain model behaviors on tasks such as indirect object identification, docstring completion, and greater-than comparisons (Wang et al., ; Hanna et al., ; Nanda et al., ; Heimersheim & Janiak, ). However, most of them require tedious manual inspection, and are thus limited to smaller models (Lindner et al., ).
Automated Circuit DisCovery (ACDC) (Conmy et al., ) has been proposed as a scalable end-to-end technique to identify circuits in models of arbitrary sizes by automating the pruning of unimportant edges via activation patching.
Although effective, ACDC suffers from slow runtimes. More recent work has focused on improving the efficiency, for example, through linear approximating activation patching with attribution patching (Syed et al., ; Hanna et al., ), or training sparse autoencoders (SAEs) to learn attention head output patterns (O’Neill & Bui, ). While faster variants have been leveraged with some success, they still suffer from several drawbacks, such as approximation errors (Syed et al., ; Hanna et al., ), and the need of model training and carefully designed examples by human (O’Neill & Bui, ).

Our main contribution in this work is the introduction of contextual decomposition for transformers (CD-T), a mathematical or analytical method to efficiently compute the contribution of model features to other model features within a transformer.
This work builds on CD methods for CNNs and RNNs to examine local importance proposed in prior work  (Murdoch et al., ; Singh et al., ; Jumelet et al., ).
We significantly extend CD by defining novel mathematical decomposition principles which can be applied to transformers of all types (e.g., decoder-based, encoder-based), because these principles are indispensable for CD to scale to modern state-of-the-art (SOTA) deep learning models as they are mostly based on transformers.

We are also the first, to our knowledge, to show the utility of CD methods for mechanistic interpretability by proposing an automated circuit discovery algorithm utilizing CD-T.
That is, CD-T is able to discover circuits with finer granularity by further splitting into sequence positions while at the same time reduce runtime from hours to *seconds*, when it is compared with prior work (Syed et al., ; Conmy et al., ).
CD-T is automatic or does not require any training or manually-crafted examples by humans. CD-T is fundamentally compatible with all common transformer architectures: in particular, CD-T supports circuit discovery in both BERT-like (Devlin et al., ) and GPT-like (Brown et al., ) architectures.

Specifically, for evaluation, we use three standard circuit evaluation datasets: indirect object identification (IOI), greater-than comparisons (Greater-than), and docstring completion (Docstring). We compare recovered circuits by CD-T on these datasets with two SOTA baselines, ACDC (Conmy et al., ) and EAP (Syed et al., ). We demonstrate that CD-T outperforms the baselines by better identifying attention heads in the manual circuits with an average of 97% ROC AUC due to its outstanding ability in recovering negative and supporting heads efficiently. EAP obtains similar runtime as CD-T but performs slightly worse in ROC AUC for Greater-than and IOI. ACDC is the least efficient as it often takes hours to identify a circuit, yet with no better ROC AUC on all tasks compared to other methods.
In addition, we demonstrate faithfulness of CD-T circuits is not due to random chance by showing our circuits are 80% more faithful than random circuits of up to 60% of the original model size.
Finally, we show CD-T circuits are able to perfectly replicate original models’ behavior (faithfulness $=1$) using fewer nodes (^2^22In our experiments, ”nodes” refers to attention heads or attention heads’ output at a specific sequence. See the general definition in Section 3.) than the baselines for all tasks.

## 2 Related work

### 2.1 Causal Interpretation

Previous efforts to interpret neural networks have frequently drawn on ideas from causal inference (Pearl, ). Much of this work emphasizes counterfactual reasoning or enforcing causal constraints on computations to explain model outputs (Pearl, ; Feder et al., ; Geiger et al., ; Wu et al., ; Kaddour et al., ). A related line of research (Vig et al., ; Stolfo et al., ; Feng & Steinhardt, ) takes a different approach by treating internal components (or nodes, if the model is viewed as a computational graph) as mediators and conducting causal mediation analysis, also called *activation patching*. In these studies, the contribution of nodes is assessed by ablating subsets of them and measuring direct and indirect effects, with the direct effects serving as proxies for their overall contribution. In contrast to viewing nodes as mediators, Goldowsky-Dill et al. () and Wang et al. () propose treating input-to-output paths as more expressive mediators. They introduce the technique of *path patching* to explore how different edge subsets in a model’s computational graph affect its behavior.

### 2.2 Circuit discovery methods

Ablation-based methods are essential for identifying key components within models. Through activation patching, prior research on circuit discovery in language models has uncovered subgraphs of attention heads and MLPs that explain model behavior (Wang et al., ; Hanna et al., ; Heimersheim & Janiak, ). However, most of these methods require extensive manual inspection, limiting their application to smaller models (Lindner et al., ).
To address this, Conmy et al. () introduced Automated Circuit Discovery (ACDC), which scales to models of any size by automatically calculating edge importance between attention heads and MLPs based on model performance on a chosen metric via activation patching. While ACDC is effective, it suffers from slow runtimes and fails to recover certain attention head subsets within the manual circuits.
Syed et al. () introduced Edge Attribution Patching (EAP, or ACDC++ as later renamed), which improves ACDC’s efficiency by using a linear approximation of activation patching. However, this reliance on linear approximations can lead to overestimation of edge importance and a weaker correlation with true causal effects. EAP also struggles when the gradient of the performance metric is zero.
To resolve these issues, Hanna et al. () proposed Edge Attribution Patching with Integrated Gradients (EAP-IG) for more faithful circuit recovery, although it increases runtime by a constant factor.
Another approach trains sparse autoencoders (SAEs) on attention head outputs and uses the SAE-learned features to map attention head contributions to identified circuits. However, SAE-based methods either require carefully crafted human-generated examples (O’Neill & Bui, ) or require an impractical amount of compute to train an SAE (He et al., ; Marks et al., ).
Another common limitation of prior methods is the limited level of granularity of the circuits they discover, because most work either doesn’t support further attention head splitting at specific sequence positions (Syed et al., ; Hanna et al., ; O’Neill & Bui, ), or it is fundamentally feasible but requires a prohibitive amount of time to do so (Conmy et al., ), which drastically hinders more fine-grained mechanistic interpretability analysis.
Our method CD-T addresses the limitations of previous circuit discovery approaches by significantly reducing runtime, using mathematical decomposition equations with no approximations, allowing for arbitrary level of abstraction of circuits, and avoiding both model training and manually crafted examples (i.e. CD-T achieves all the above in an automated manner).

### 2.3 Contextual Decomposition in Neural Networks

Contextual Decomposition (CD), introduced by Murdoch et al. (), attributes local importance to input features in LSTMs by analytically decomposing the output without any changes to the underlying model.
For each word in a sentence, a forward pass computes activations for all cells and gates while partitioning each neuron’s activation into parts influenced by a selected token or phrase (denoted as the *relevant*) and those that are not (denoted as the *irrelevant*), using a factorization of the update equations for hidden states and cell states.
Jumelet et al. () propose Generalized Contextual Decomposition (GCD) for unidirectional LSTMs, applying it to phenomena like number agreement and pronoun resolution.
Singh et al. () extend CD from LSTMs to RNNs and CNNs and introduce hierarchical interpretation with feature clustering for improved input feature attributions.
A distantly related work, contextual explainability (Bertolini et al., ), focuses on feature attributions for each convolutional layer of CNNs by utilizing gradients as the feature importance measure.
We further extend CD to transformers (CD-T) and we argue that our development of CD-T is a significant advance with two novel contributions.
First, we carefully design mathematical decomposition principles tailored for transformers of all types (e.g., decoder-based, encoder-based), which makes it able to scale to larger models.
Second, given prior work on CD only involves input feature attribution, to our knowledge, we are the first to demonstrate the utility of CD for mechanistic interpretability by proposing an automated circuit discovery algorithm that leverages CD-T.

## 3 Definitions for circuit discovery algorithms

Before delving into our method, we first formally define a *circuit*. To maintain consistency in notations with prior work (Shi et al., ; Elhage et al., ), we view a model as a computational graph $M$, where nodes are activations of the model components in its forward pass (e.g. attention heads and MLPs) and edges are the interactions between those components, and a circuit $C$ is a subgraph of $M$ responsible for certain behavior of interest, such as solving for a specific task. Input embeddings and output logits are treated as input node and output node separately, and are both included in $M$ and $C$.

Naturally, the first step of circuit discovery is to define a *task* of interest to investigate. This requires specifying a dataset and a task-specific metric to measure the performance of $M$ and $C$. Here we define a task as a tuple $\tau=(D,s)$ of a dataset $D=\{(x_{i},y_{i})\}_{i=1}^{n}$ and a score $s\in\mathbb{R}$. The dataset $D$ contains $n$ pairs of inputs $x_{i}\in X$ and outputs $y_{i}\in Y$. The score is a scalar calculated against a task-specific metric, such as the logit difference between IO and S positions in the IOI task. Given an input $x_{i}$, similarly as to how the entire model defines a function $M(x_{i})$ from inputs to logits, we also associate each circuit $C$ with a function $C(x_{i})$, defined by ablating away the effect of all components in $M\backslash C$ (i.e. the components not included in $C$).

## 4 Our method: CD-T and mechanistic interpretability

In this section, we first recall the basic operations of transformers (Section 4.1). We then provide technical detail on contextual decomposition and its application to transformers (section 4.2). Finally, we conclude with a circuit discovery algorithm based on CD-T (section 4.3).

### 4.1 Transformers

The main technical element unifying transformer-based architectures is the attention mechanism. This mechanism allows contextually useful information to be transmitted from one position in the sequence to any other position. We start by describing the operation of a general attention mechanism before discussing the modifications needed to specialize it for generation, exemplified by models such as GPT (Brown et al., ), and for sequence-to-sequence transformation models such as BERT (Devlin et al., ). Formally, the attention mechanism computes a vector of contributions from a series of vectors $\{x_{i}\}_{i=1}^{l}$ to a target vector $x$ at position $p$ with $x_{i},x\in\mathbb{R}^{d}$. An attention layer typically consists of a small number of heads, with each head parameterized by three functions, the key function $f_{k}$, value function $f_{v}$, and query function $f_{q}$. The output of the attention head to $x$ is computed as follows where $d_{k}$ denotes the output dimension of $f_{k}$:

$$ $\displaystyle\forall i\in[l]:k_{i}=f_{k}(x_{i}),q_{x}=f_{q}(x),v_{i}=f_{v}(x_{ i})$ (1) $\displaystyle\forall i\in[l]:\alpha_{i}=\frac{\exp(q_{x}^{\top}k_{i}/\sqrt{d_{ k}})}{\sum_{j\in[l]}\exp(q_{x}^{\top}k_{j}/\sqrt{d_{k}})}$ (2) $\displaystyle y_{x}=\sum_{j=1}^{l}\alpha_{i}v_{i}.$ (3) $$

The functions $f_{k},f_{v},f_{q}$ are commonly simple learnable transformations such as linear transformations or one hidden-layer MLPs. An attention layer is composed of a series of attention heads applied to every position in the sequence such that the concatenation of their outputs equals the input dimension $d$. As the output of an attention layer is a series of vectors of the same length and dimensionality as its input, this allows for stacking of these layers to build increasingly complex representations of the input so far.

The two classes of transformer models we consider in this work, sequence-to-sequence models such as BERT and generative models such as GPT mainly differ in the set of positions that the attention mechanism operates on. Concretely, BERT maps a *fully specified* input sequence to an output sequence of the sample length while generative models produce increasingly larger sequences autoregressively with the $(t+1)^{th}$ token generated based on the previous $t$ tokens. Since *all* the input tokens are specified for BERT-based models, the computation of the attention vector at position $t$ utilizes representations at *every* position of the sequence. On the other hand, when generating the $(t+1)^{th}$, the positions at $t+1$ and beyond are not determined. As a consequence, only the representations for tokens before $t$ are used to compute the attention vectors.

### 4.2 Contextual decomposition for transformers (CD-T)

We will now describe the general method of contextual decomposition (CD), as well as our extension to the transformer architecture and applications to mechanistic interpretability.

CD (Murdoch et al., ) was first proposed to compute contributions of input tokens to the output of LSTMs as a local interpretation method.
CD divides each cell and hidden state into a sum of two parts: a $\beta$ part (said to be *relevant*), which contains the part of this particular state that stems from the input tokens of interest, and a $\gamma$ part (said to be *irrelevant*), which contains information coming from tokens outside of the list of interest. $\beta$ is often initialized with masked input embeddings ($1$’s specify where token positions of interest are, and 0 0’s the opposite), and $\gamma$ the complement of $\beta$.

Given a decomposition of a vector $x\in\mathbb{R}^{d}$ into *relevant* and *irrelevant* *constituents* $\beta+\gamma=x$ (here we use the word *constituents* instead of “components” to avoid name collision with the “components of a neural network”, and a module $f:\mathbb{R}^{d}\to\mathbb{R}^{k}$, CD consists of a set of mathematical equations to determine the decomposition of the output of a module $f$ when given $x$ as input: $f(x)\in\mathbb{R}^{k}=\beta_{o}+\gamma_{o}$. The general principle is to find a symbolic expression for $f(x)$ in terms of $\beta$ and $\gamma$, and group the terms in the expression according to whether they are solely a function of $\beta$ or not: the sum of terms solely relying on $\beta$ becomes $\beta_{o}$, and the sum of the remaining terms becomes $\gamma_{o}$.
We refer interested readers to Murdoch et al. () and Singh et al. () for further examples of decomposition formulas of various modules.
These decomposition rules can be composed through multiple modules (i.e, if $f(\beta_{1},\gamma_{1})=\beta_{2},\gamma_{2}$, and $g(\beta_{2},\gamma_{2})=\beta_{3},\gamma_{3}$, then $g(f(\beta_{1},\gamma_{1}))=\beta_{3},\gamma_{3}$) in order to define decomposition rules for larger computational blocks, including entire neural networks.

Other authors have heuristically adjusted the decomposition rules for a module to reflect specific mechanisms by which (the relevance term in an earlier layer $i-1$) can affect (the relevance term in a later layer $i$) that are unaccounted for by this rule.
For example, Singh et al. () adjust the decomposition rule for affine transformations $f(x)=Wx+b$, so that the bias term does not affect the relative magnitudes of $\beta$ and $\gamma$:

$$ $\displaystyle\beta_{i}$ $\displaystyle=W\beta_{i-1}+\frac{\left|W\beta_{i-1}\right|}{\left|W\beta_{i-1} \right|+\left|W\gamma_{i-1}\right|}\cdot b,$ (4) $\displaystyle\gamma_{i}$ $\displaystyle=W\gamma_{i-1}+\frac{\left|W\gamma_{i-1}\right|}{\left|W\beta_{i- 1}\right|+\left|W\gamma_{i-1}\right|}\cdot b.$ (5) $$

We succinctly represent the above equations as $\beta_{i},\gamma_{i}=LinearDecomp(\beta_{i-1},\gamma_{i-1})$, where $W$ and $b$ are understood to be tunable parameters of a neural network module.

Given the query, key, and value functions mentioned in Section 4.1 are linear transformations, the only module not accounted for in the context of transformers is the self-attention module. We handle the decomposition of the attention equations as follows:

$$ $\displaystyle\beta_{query},\gamma_{query}$ $\displaystyle=LinearDecomp(\beta_{in},\gamma_{in})$ (6) $\displaystyle\beta_{key},\gamma_{key}$ $\displaystyle=LinearDecomp(\beta_{in},\gamma_{in})$ (7) $\displaystyle\beta_{value},\gamma_{value}$ $\displaystyle=LinearDecomp(\beta_{in},\gamma_{in})$ (8) $\displaystyle\beta_{attention}$ $\displaystyle=\beta_{key}^{T}\beta_{query}/\sqrt{d_{k}}$ (9) $\displaystyle\gamma_{attention}$ $\displaystyle=((\beta_{key}+\gamma_{key})^{T}(\beta_{query}+\gamma_{query})/ \sqrt{d_{k}})-\beta_{attention}$ (10) $\displaystyle\beta_{probs}$ $\displaystyle=Softmax(Mask(\beta_{attention}))$ (11) $\displaystyle\gamma_{probs}$ $\displaystyle=Softmax(Mask(\beta_{attention}+\gamma_{attention}))-\beta_{probs}$ (12) $\displaystyle\beta_{z}$ $\displaystyle=\beta_{probs}*\beta_{value}$ (13) $\displaystyle\gamma_{z}$ $\displaystyle=(\beta_{probs}+\gamma_{probs})*(\beta_{value}+\gamma_{value})- \beta_{z}$ (14) $\displaystyle\beta_{output},\gamma_{output}$ $\displaystyle=LinearDecomp(\beta_{z},\gamma_{z})$ (15) $$

where above $Mask$ represents a causal mask in appropriate architectures and $Softmax$ is the softmax along the same dimension as in a standard implementation of self-attention.

Although in the original setting of CD, “input” and “output” refer to an input sequence $x$ and a model’s output logits, the method generalizes to arbitrary *source* and *target* activations in the model.
Furthermore, in the original context, the “relevant” portion of the decomposition was equal to 1 on a subsequence of the input and 0 elsewhere, but in a mechanistic interpretability context, the CD framework allows us to decompose activations into any choice of $\beta,\gamma$.
Thinking of the network again as a computational graph, this gives us a method of calculating the effect of an arbitrary constituent of the activations of an arbitrary node on the activations of another node.

#### 4.2.1 Choosing a decomposition of source nodes for mechanistic interpretability

In a manner analogous to the choice of ablation method elsewhere in mechanistic interpretability, the correct choice of starting decomposition can be subtle, and sometimes specific to the task. A general principle is that the relevant constituent of the activations at some node is the constituent that we hypothesize has a counterfactual effect on the task; often, this means the deviation from the mean activations over some distribution. (It follows that the irrelevant constituent of the activation at this node is the mean activation, which matches intuitions about mean-ablation.) We provide details on the evaluation tasks and name the distributions used for ablation in Appendix A.

### 4.3 Automating circuit discovery

Next, we present our algorithm for discovering circuits with CD-T. Although it is possible to perform CD at arbitrary granularity and on arbitrary components of a transformer (e.g., MLPs, query/key/value vectors), in this paper we focus solely on finding circuits consisting of attention heads in a transformer (in particular, we always leave the MLPs fully intact), and our unit of analysis is either *the output of an attention head* or *the output of an attention head at a specific sequence position*. For clarity, our method focuses on including or excluding specific nodes from a computational graph, rather than edges; CD-T at once implicitly models both direct and indirect effects of one node to another, and our algorithm does not make a distinction between the two.

In each iteration, we search over potential source nodes to find the ones with highest relevance to a set of target nodes (initialized to the task objective). Once we find the set of nodes which are most relevant to the task, we prune them by some simple heuristic (in our experiments, simply greedily removing the nodes with the least magnitude of impact up to a threshold was often sufficient, or sometimes pruning was not necessary), and designate these most important nodes the target nodes for the next iteration of the algorithm. We halt when the collection of nodes achieves adequate performance (e.g, comparable to the whole network), or has not improved from the previous iteration.

We need to specify how “highest relevance to a set of target nodes” is determined. As described above, we start with a decomposition $\beta_{s},\gamma_{s}$ of the output of a source attention head $s$, and we are interested in its relevance $R(s,T)$ to a set of target nodes $T$. Starting from $s$, we propagate the decomposition forward through the network using the CD-T equations for each module in the computational graph between $s$ and $T$; again, if there is some series of functions $f_{n},...f_{1}$, so that $T=f_{n}(f_{n-1}(...f_{1}(s)))$, and $f_{1}(\beta_{s}+\gamma_{s})=\beta_{1}+\gamma_{1},f_{2}(\beta_{1}+\gamma_{1})=
\beta_{2}+\gamma_{2},...,f_{n}(\beta_{n-1}+\gamma_{n-1})=\beta_{T}+\gamma_{T}$, we can straightforwardly apply the modules in order to obtain $\beta_{T},\gamma_{T}$. In the case that $T$ is the output of the network, $\beta_{T}$ will have matching dimensions, and it is usually appropriate to let $R(s,T)$ be the task-specific objective as evaluated on $\beta_{T}$. (One intuitive instance of this is the case where $T$ is a single “score” whose value we care about, so that $\beta_{T}$ is the contribution of $s$ to that score.) However, when $T$ is an arbitrary set of nodes in the network’s internals, we must define a proxy metric for the relevance of $s$ to $T$; in this case, we define the relevance of a source node to a set of target nodes to be the sum of the relevances to the target nodes $t\in T$, and in this paper for $R(s,t)$ we choose a quick-to-compute measure of the size of $\beta_{t}$:

$$ $R(s,T)\coloneqq\sum_{t\in T}R(s,t)=\sum_{t\in T}\frac{\|\beta_{t}\|_{l_{1}}}{ \|\gamma_{t}\|_{l_{1}}}.$ (16) $$

Careful readers may have noticed that this particular metric measures the magnitude of one node’s contribution to another, but not the “direction”, so that nodes which maximize this metric may actually contribute negatively to the task under investigation. We found in our experiments that this does not practically preclude us from finding functioning circuits anyway; it is relatively efficient to greedily prune away nodes by just measuring if doing so improves a circuit’s task performance. On the other hand, this allows us to straightforwardly find nodes which have significant negative contribution to a task metric, such as the “Negative Name Mover Heads” in the IOI task. In principle, it is also possible that different choices of target relevance metric can be developed to adjust the behavior of the circuit discovery algorithm for specific purposes; we do not explore this in the paper but identify it as a possible avenue for further investigation.

Empirically we find, for a fixed target node, the relevance scores of source nodes in different layers to this target node may have different expected magnitudes, due to the numerical effects of propagation through the network. To account for this, we simply normalize the relevance scores by dividing by the average magnitude across scores found in a given layer.
Our complete algorithm is described in Algorithm 1, presented is the specific case where we have chosen to decompose our source nodes $s$ so that $\beta_{s}$ is the activation’s deviation from the mean over some distribution.

Figure: Algorithm 1 Building a Circuit using CD-T

## 5 Evaluating circuit discovery algorithms

### 5.1 Experimental setup

We compare the performance of circuits recovered by CD-T with those obtained by two SOTA baselines: ACDC (Conmy et al., ) (^3^33We adopt the ACDC optimized for KL diveregence instead of task-specific metrics because it’s shown to perform better in Conmy et al. ().) and EAP (Syed et al., ) (^4^44We consider EAP but not EAP-IG because EAP-IG performs comparably to EAP on the tasks we evaluate (Hanna et al., ), but with a runtime increase by a constant factor in theory.). SAE-based methods (O’Neill & Bui, ; Marks et al., ; He et al., ) and classic neural network pruning methods are not considered here because they either require model training or focus more on compressing neural networks for faster inference to reduce storage requirements. Furthermore, pruning for interpretability is also shown to be outperformed by ACDC (Conmy et al., ).
We evaluate the circuit performance on three tasks: indirect object identification (IOI) (Wang et al., ), greater-than (Greater-than) (Hanna et al., ), and docstring completion (Docstring) (Heimersheim & Janiak, ) (see Appendix A for details).
They are three standard evaluation tasks to benchmark circuit discovery methods as prior work (Hanna et al., ; Wang et al., ; Heimersheim & Janiak, ) has *manually* found circuits in language models explaining for those tasks, which often serve as an imperfect reference standard for circuit discovery work (Conmy et al., ; Syed et al., ; O’Neill & Bui, ) due to the lack of ground-truth circuits.
For both baselines, we perform node-level patching to be comparable since CD-T is a node-level patching technique. All experiments are conducted on an NVIDIA A100 GPU.

### 5.2 CD-T excels in identifying the circuit responsible for the underlying algorithm efficiently

Figure: Figure 1: Left: Log algorithm runtime and ROC AUC comparison. Each dot represents an average measurement on one task with specific methods differentiated by colors.; Right: The relative faithfulness of CD-T circuits compared to random circuits from the reference distribution of varying sizes (x-axis). Dotted vertical lines indicate the actual size of the circuits. $C$ denotes a CD-T circuit. $C^{r}$ denotes a random circuit. $M$ denotes the full model.
Refer to caption: x1.png

#### 5.2.1 Runtime and the recovery of manual circuits

Leveraging the *manual* circuits as a reference standard, we measure how much overlap there is between the recovered circuits and the reference circuits using ROC AUC.
Specifically, we measure the ROC AUC by sweeping through a range of thresholds that determine cutoff scores to keep nodes in a circuit.
For CD-T, we test by varying the percentile of top nodes to extract in every iteration, in the range of $[90,99]$.
At the same time, we also compute the average algorithm runtime across runs when sweeping through the thresholds for each method.
Although CD-T supports arbitrary granular representations of the circuit, the runtime is measured by building a “coarse node-level” circuits by not splitting into sequence positions to be comparable with EAP and ACDC because EAP doesn’t support measuring node importance at sequence positions, and ACDC takes an impractical amount of time to achieve sequence-position splitting.
To demonstrate CD-T’s capability to identify circuits responsible for underlying algorithm for tasks with great efficiency, we plot the algorithm runtime against the ROC AUC and report the result in Figure 1-Left.
From Figure 1-Left we can see CD-T outperforms the baselines by achieving high ROC AUC and low runtime at the same time. EAP obtains similar runtime as CD-T but performs slightly worse in recovering *manual* circuits for Greater-than and IOI. Even under this coarse nodel-level abstraction, ACDC is the least efficient as it often takes *hours* to identify a circuit, yet with no better ROC AUC on all tasks compared to other methods.
To explain the better ROC AUC performance of CD-T, we find that CD-T is especially good at identifying negative heads and supporting heads, such as the negative name-mover heads and backup name-mover heads in IOI, which other algorithms struggle to do (more discussion in Appendix A).

#### 5.2.2 Probability of identified circuits to be more faithful than random circuits

Given a circuit and a task, one can evaluate how well the circuit performs on the task by measuring the task-specific metric, such as the logit difference on IOI. However, in mechanistic interpretability, instead of aiming to identify the best-performing circuits on the task-specific metric, an ideal circuit should be able to *faithfully replicate the full model behavior*. Following prior work (O’Neill & Bui, ; Marks et al., ), faithfulness is defined as the proportion of the full model’s performance that a circuit explains, relative to the baseline performance when no specific input information is provided. Given a task, faithfulness is computed as $\frac{m(C)-m(\emptyset)}{m(M)-m(\emptyset)}$, where $m(C)$, $m(\emptyset)$, and $m(M)$ are the average of the task-specific metric performance over the dataset for the circuit, the corrupted model with all heads ablated, and the full model, respectively.
To complement the imperfect nature of the *manual* circuits, leveraging faithfulness, here we provide another evidence to validate CD-T’s mechanism perservation of the original model.
We adopt a circuit hypothesis testing framework proposed in Shi et al. () to test whether a circuit preserves the original model’s performance by measuring the probability of a circuit $C$ to be more faithful to the original model $M$, compared to random circuits $C^{r}$ of the same size sampling from a reference distribution. This test verifies that the circuit is not a simple lucky draw from the distribution of random circuits, and ensures that it is better than at least a fraction $q^{*}$ of random circuits.
In our experiment, we sample random circuits from all possible nodes in the full model, and we repeat the process for 10 times.
Figure 1-Right shows the results of the best CD-T circuits (as measured by faithfulness and among different granularity) on the 3 tasks. Our best circuit for IOI is from attention heads at specific sequence positions, while for Greater-than and Docstring, they are from attention heads at a coarse level without the splitting. The results show CD-T circuits are 80%  ($q^{*}=0.8$) more faithful than random circuits of up to around 60% of the original model’s size for IOI and Greater-than, and even those of up to 80% of the original model’s size for Docstring, with just a small percentage of nodes in the CD-T circuits ($0.4\%$ of full model size for IOI, $5.5\%$ for Greater-than, and $31.3\%$ for Docstring). This finding suggests the faithfulness of CD-T circuits is not due to random chance, and that a more fine-grained abstraction of circuits sometimes is necessary to obtain better faithfulness with an extremely small and specific set of nodes, as in the IOI case.

### 5.3 Identified circuits match full model performance

#### 5.3.1 Faithfulness of circuits of varying sizes

Figure: Figure 2: Faithfulness of CD-T circuits, EAP circuits, and randomly selected circuits of equivalent size for IOI, Greater-than and Docstring tasks. CD-T circuits obtains the full model’s performance (faithfulness of 1) faster than EAP as attention heads are added in order of importance.
Refer to caption: x3.png

To understand how faithfulness changes as the size of the circuits grows, we compute node importance before pruning for each method (^5^55ACDC is omitted in this experiment because of the prohibitive amount of runtime in hours to obtain individual node importance.), and track faithfulness performance as we add circuit nodes in order of importance, while ablating all other nodes with corrupted activations (see Appendix A). We sample attention heads in the original model one by one in random order to construct the random baseline, and report the results in Figure 2.
Since the goal of mechanistic interpretability is to replicate the original model’s behavior, namely achieving faithfulness $=1$ (^6^66Faithfulness can go beyond 1 because there are both positively and negatively contributing heads in the model. The original model performance relies on a certain dynamic of interactions between them. When the dynamic is not perfectly replicated and there’s extra positively contributing effect, the circuit might give a higher prediction score than the original model would give, which results in a faithfulness $>1$.), with the smallest amount of nodes possible, this is naturally the first question we ask.
First, we find CD-T circuits are able to perfectly replicate model behavior (faithfulness $=1$) using fewer nodes than EAP for all the three tasks. Second, manual circuits outperform both CD-T and EAP of the same size, indicating room of improvement of advanced circuit discovery methods. Finally, we compare the effect of ablation methods on CD-T circuits’ performance by including CD-T circuits obtained from zero-ablation in the plot, and find that mean-ablation tend to yield higher faithfulness compared to zero-ablation on Greater-than and Docstring, except for IOI where they are quite comparable. Mean-ablation is often viewed as a preferred way of ablation because it preserves a general concept of the task, hence is less destructive than zero-ablation. Similar finding is also discussed in other circuit discovery work (Conmy et al., ; Wang et al., ).

## 6 Discussion and Limitations

Despite having shown both qualitative and quantitative evidence of the strengths of CD-T in automated circuit discovery, our results is limited in scale and to the datasets and prior interpretation methods evaluated. More work is needed to generalize these findings to a broader set of models, datasets, and interpretation methods.
Although our methods can be applied to construct circuits comprised of any of a neural network’s internal components, we only discussed and interpreted circuits built with purely attention heads. Automating the discovery of circuits with different/heterogeneous components, particularly the MLPs of a transformer, is a promising direction for further investigation.
Finally, our circuit discovery is itself limited in three major ways: first, as mentioned in 4.3, the metric which measures the relevance of a source node to a target node still lacks certain properties (e.g, its scale does not correspond to an interpretable quantity, and it does not measure the sign of its contribution) that would be broadly useful in an interpretability context.
Second, although the algorithm is capable of finding circuits with reasonable default settings, some amount of effort is still applied to tune the heuristic elements, such as the number of heads to select in each iteration, and the method of pruning away negative heads.
Third, and more fundamentally, although our method finds circuits, it offers relatively little in the way of distilling an interpretation of the circuits which is compatible with human understanding.

## 7 Conclusions

We have proposed contextual decomposition for transformers (CD-T), a set of mathematical decomposition principles to efficiently compute the contribution of model features to other model features within a transformer. CD-T can produce circuits of arbitrary granularity, and is the first able to produce circuits as fine-grained as attention heads at specific sequence positions while at the same time reduce runtime from hours to *seconds*, when it is compared with prior work (Syed et al., ; Conmy et al., ).
CD-T improves upon limitations of prior work, as it is automatic and does not require any training or manually-crafted examples by humans. CD-T is fundamentally compatible with all common transformer architectures.
On three standard circuit evaluation datasets (indirect object identification, greater-than comparisons, and docstring completion), we demonstrate that CD-T outperforms ACDC and EAP by better recovering the manual circuits with an average of 97% ROC AUC under low runtimes. In addition, we provide evidence that faithfulness of CD-T circuits is not due to random chance by showing our circuits are 80% more faithful than random circuits of up to 60% of the original model size. Finally, we show CD-T circuits are able to perfectly replicate original models’ behavior (faithfulness = 1) using fewer nodes than the baselines for all tasks. We hope that our future open-source implementation of CD-T benefits mechanistic interpretability research community by opening up a window from enabling more fine-grained analysis within a practical runtime.

## Acknowledgments

GZ would like to thank Shawn Hu for valuable discussions that contributed greatly to mathematical formalization, experiment design, and important heuristic components of our method. We gratefully acknowledge partial support from NSF grant DMS-2413265, NSF grant 2023505 on Collaborative Research: Foundations of Data Science Institute (FODSI), the NSF and the Simons Foundation for the Collaboration on the Theoretical Foundations of Deep Learning through awards DMS-2031883 and 814639, NSF grant MC2378 to the Institute for Artificial CyberThreat Intelligence and OperatioN (ACTION), and a Berkeley Deep Drive (BDD) grant from BAIR and a Dean’s fund from CoE, both at UC Berkeley.

## Appendix A Experiment details

### A.1 Task Description

Here we describe each task and how the experiments are performed in more detail; we also specify the exact model components which comprise the circuits we analyzed in the main paper. In this section, we use the convention of specifying attention heads in a model by the tuple (layer index, head index). For the IOI task, we add a third entry for output of the head at a specific sequence position, and follow their nomenclature for semantically labeling the sequence positions.

#### A.1.1 Indirect Object Identification (IOI) (Wang et al., 2023 )

The indirect object identification (IOI) task is to predict the indirect object in a sentence with two entities, such as identifying “Mary” in the sequence “When Mary and John went for a walk, John gave an apple to …”. The objective originally defined for this task, which we also use, is the difference between the predicted logit of the indirect object (IO) token and the subject (S) token. Wang et al. () carefully design their experiment and dataset code so that mean ablation occurs using the mean activations over the corrupted “ABC dataset”, which replaces the three name tokens in the task with random names. Likewise, when setting the decomposition of a node, we set the “relevant” component to the deviation from the mean activations on the ABC dataset.
This task was performed on GPT2-small. GPT-2 correctly performs this task about 99 percent of the time, so we did not take special measures to account for those samples where it doesn’t in our circuit analysis.
We identify circuits using 25 IOI samples drawn from mixed templates, and mean ablation is conducted using the corrupted ABC dataset. Another set of 100 IOI samples are used in evaluation.
Manual circuit of IOI compared against in the ROC AUC experiment is: [(2, 2), (4, 11), (0, 1), (3, 0), (0, 10), (5, 5), (6, 9), (5, 8), (5, 9), (7, 3), (7, 9), (8, 6), (8, 10), (10, 7), (11, 0), (9, 9), (9, 6), (10, 0), (9, 0), (9, 7), (10, 1), (10, 2), (10, 6), (10, 10), (11, 2), (11, 9)], which is from Figure 2 in Wang et al. ().

##### Details of Circuit Analysis

For this task, in order to provide some intuitions about what CD-T calculates, we provide a number of heatmaps of the relevance scores at specific positions during various iterations of the circuit analysis. In this section, we don’t follow our automated circuit discovery algorithm exactly, but instead partially follow Wang et al. ()’s analysis to decide what sequence positions to search over and visualize.

The first iteration of the algorithm finds the Name Mover Heads: (9, 9, end), (10, 0, end), and (9, 6, end); the Negative Name Mover Heads: (10, 7, end), (11, 10, end); and some Backup Name Mover Heads: (10, 2, end), (10, 6, end), (10, 10, end), described by Wang et al. ()

[Uncaptioned image]: x6.png

Following Wang et al. ()’s analysis further, and deviating from the normal course of our circuit-finding algorithm, we compute the relevance of nodes to the Name Mover Heads on just the end position, and find what they named the “S-Inhibition Heads”, at (8, 10, end), (7, 9, end), and (7, 3, end), though there are some other relevant-looking contenders:

[Uncaptioned image]: x7.png

Continuing to follow their analysis, Wang et al. ()’s analysis further, we compute the relevance of nodes to the S-Inhibition Heads at the S2 position, and find our first minor disagreement with their process: though we find two of what they named the Induction Heads at (5, 5, S2), (5, 8, S2), (5, 9, S2) we don’t find the one at (6, 9, S2) but instead find one at (5, 10, S2), and find a head their later analysis called a Duplicate Token Head, at (3, 0, S2).

[Uncaptioned image]: x8.png

Next we compute the relevance of nodes to the Induction Heads at the S2 position, and find that (3, 0, S2) mostly drowns out the signal of the other Duplicate Token Heads:

[Uncaptioned image]: x9.png

Finally, we compute the relevance of nodes to the Induction Heads at the S1+1 position, and find (4, 11, S1+1), with the other Previous Token Head they found at (2, 2, S1+1) a top contender, though there are other heads not accounted for in their analysis:

[Uncaptioned image]: x10.png

Overall, there is significant but not perfect agreement with the results and analysis of the IOI paper. We also attempted analysis by computing relevance to intermediate matrices (i.e., the key, query, value vectors) in the attention calculation, but found the plots to be qualitatively similar, possibly due to the fact that CD-T by design propagates relevances from these vectors to the attention head outputs as well.

Another fact of note is that the scales on these plots vary substantially. It would be desirable to find an interpretable normalization method to put these on the same scale with some intrinsic meaning, or otherwise explain the causes of this phenomenon.

#### A.1.2 Greater-Than (Hanna et al., 2023 )

The Greater than task is to predict the last two digits in an incomplete sentence following the template “The [noun] lasted from the year XXYY to the year XX”. And we expect the model to assign higher probability to years greater than YY.
The objective originally defined for this task, which we also use, is the sum of probabilities assigned to tokens corresponding to greater years, minus the sum of probabilities assigned to tokens corresponding to lesser years. (Some probability is assigned to tokens which don’t correspond to numbers at all.) For our “mean-ablation”, we simply take the mean over the activations over 100 negative datapoints (impossible completions, with the ending year preceding the starting century), and as above, when setting the decomposition at a source node, define the relevant component to be the deviation from the mean activation over this distribution.
This task was performed on GPT2-small. GPT-2 correctly performs this task about 99 percent of the time, so we did not take special measures to account for those samples where it doesn’t in our circuit analysis.
We identify circuits using a random sample of 100 datapoints provided by Hanna et al. (), and mean ablation is conducted using the negative impossible completion samples. Another set of 100 samples are used in evaluation.

It should be noted that our result and the results in Hanna et al. () are not directly comparable, since they also attempt to investigate the influence of MLPs and their resulting circuit removes most of the MLPs in GPT-2. For the comparisons found in the main paper, we have compared our result (with all MLPs) to the circuit which is obtained by taking all the attention heads named in Hanna et al. () (also with all MLPs), which is a circuit distinct from the one found in their work.

Manual circuit of Greater-than compared against in the ROC AUC experiment is: [(5, 1), (5, 5), (6, 1), (6, 9), (7, 10), (8, 8), (8, 11), (9, 1)].

Independently of the comparison, it remains true that keeping the relatively small proportion of attention heads in our circuit results in recovering almost all of GPT-2’s capability on this task; see below.

|  | Task-specific metric | Correct guess rate |
| --- | --- | --- |
| Full model | 0.817 | 0.992 |
| All attention heads ablated | -2.095 | 0 |
| Their circuit | 0.768 | 0.989 |
| Their circuit (most MLPs ablated) | 0.727 | N/A |
| Our circuit | 0.761 | 0.981 |

#### A.1.3 Docstring (Heimersheim & Janiak, 2023 )

The goal of the Docstring task is to predict the next variable name in a Python docstring. For example, given a function with variable names load, size, files, and last, the task is to predict the word after one :param. According to docstring conventions, this should be the variable name in the function definition which follows the most recent variable name to appear after :param.
The objective originally defined for this task is the logit assigned to the correct variable name, minus the max logit assigned to all other variable name tokens found in the function signature. For our “mean-ablation”, we use their random_random dataset which randomize both the variable names in the function definition and in the docstring of prompts, and correspondingly, when setting the decomposition at a source node, define the relevant component to be the deviation from the mean activation over this distribution. We identify circuits using a 100 datapoints sampling for the dataset provided by Heimersheim & Janiak (), and mean ablation is conducted using the corrupted random_random dataset. Another set of 100 samples are used in evaluation.
This task was performed on a 4-layer attention-only transformer trained on natural language and Python code (attn-only-4l) released with the TransformerLens library for the express purpose of facilitating mechanistic interpretability research.
Another complication is that the toy model only guesses the correct token between 60 and 65 percent of the time. To account for this, we perform our circuit analysis and evaluation on the subset of input examples for which the model performs the task correctly.

Our circuit consists of the nodes (3, 0), (3, 6), (1, 4), (0, 5), (0, 0), (1, 0), (1, 4), (0, 1), (2, 3), (1, 2). Heimersheim & Janiak () find the circuit (0, 2), (0, 4), (0, 5), (1, 2), (1, 4), (2, 0), (3, 0), (3, 6) with their initial analysis, and heuristically observe that three heads help to obtain the “augmented circuit” (0, 2), (0, 4), (0, 5), (1, 2), (1, 4), (2, 0), (3, 0), (3, 6), (1, 0), (0, 1), (2, 3).

Manual circuit of Docstring compared against in the ROC AUC experiment is: [(0, 2), (0, 4), (0, 5), (1, 2), (1, 4), (2, 0), (3, 0), (3, 6)].

|  | Task-specific metric | Correct guess rate |
| --- | --- | --- |
| Full model | 3.661 | 1.0 |
| All attention heads ablated | -2.095 | 0 |
| Their circuit | 2.884 | 0.54 |
| Augmented circuit | 3.524 | 0.66 |
| Our circuit | 3.235 | 0.57 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Line 20: "CD-T is the first ---?" there seems to be a missing word here
soundness: 3
strengths: 1. Motivation, relevant scientific terms and prior works are well-written, making the paper accessible to researchers who are non-experts in this area. 

2. The proposed method can be easily used for all transformer architectures (encoder-decoder and decoder-only, as per line 72). Further, the proposed method significantly reduces the runtime required for inference.
summary: This paper introduces a novel method CD-T that leverages contextual decomposition for mechanistic interpretability in transformers. CD-T's granularity is fine-grained up to attention heads (does not include MLP layers). They report the results of their method (classification metrics and faithfulness of the circuit as compared to randomly sampled circuits) for three tasks, namely indirect object identification, greater-than comparisons, and docstring completion.
weaknesses: 1. Manual circuits are not fully explained (definition, how they are computed, cost of computation, etc.) - manual circuits are used as reference during evaluation (line 405), and it is necessary to provide these details regarding them. 

2. It is unclear how CD-T works for inference - is there a distinct circuit discovered for each inference datapoint, or is the circuit found on a broader scale, so as to tradeoff size vs. performance for a larger test set?

3. The authors are encouraged to discuss the practical utility of CD-T. Once their method yields a circuit, could additional analyses, such as on attention heads, be conducted? For instance, are there heads that consistently produce positive or negative outputs across all data points, or do the importance of attention heads vary based on input data characteristics?

4. What transformer architecture is used for the experimental results?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. See W1; In addition, what’s the computational complexity of the algorithm in terms of the input parameter?
2. How does pruning affect the mechanistic interpretability of the model?
soundness: 3
strengths: 1.	The paper is relatively well presented. It provides a clear description of the method, experiment details, and contribution. 
2.	Using the decomposition approach for designing/scaling LLM models is relatively novel and effective. Results show promising improvements in computational efficiency while maintaining circuit quality, and the method seems to be agnostic to the specific types of transformer models.
summary: The paper introduces Contextual Decomposition for Transformers (CD-T) for building interpretable circuits in llms for better mechanistic interpretability in transformers efficiently. In contrast to other methods, CD-T employs a mathematical decomposition that isolates the contributions of specific features, allowing CD-T to discover circuits at various levels of granularity. Results show significant improvements in runtime and interpretability over existing methods such as ACDC and EAP on several tasks.
weaknesses: 1.	The description of the algorithm can be improved. In particular, the algorithm description is relatively informal and could benefit from more details and formalism. For “Prune nodes from S for which doing so increases the task metric”, how is increasing the task metric evaluated?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1) I would suggest clearly presenting model architectures under evaluation and assumptions being made on input data for the approach - the mechanistic relevance metrics are ultimately calculated from the outputs derived from input data.
2) How would this scale to large training datasets in LLMs? Is there a minimum dataset required? Can the authors comment on this?
3) What is the effect of autoregressive models and sequence to sequence decoding on this approach? Is there a difference? I don't see a clear analysis in Section 5, and while that is not necessary it would be nice to have the authors comment on this in Section 6.
Questions addressed in comments.
soundness: 3
strengths: 1) The paper's originality is in applying a novel formulation of an existing method - Context Decomposition - to transformer models, albeit to the attention heads of the transformer models. While novel, I am not in the space of circuit discovery in neural networks and unable to judge how novel this is. 
2) The presentation is clear and the limitations in data dependence and precomputation are made clear to the reader in algorithm pseudocode, which is also concise and clear. Experimentation is presented to clearly establish the work's superior performance over the baselines. Notation remains consistent through the paper to my eyes.
3) Significance: The work presents the an automatic circuit discovery approach using CD (extending it as CD-T) in transformer models, and clearly outperforms its chosen baselines. The approach is moreover mechanistic and iterative rather than probabilistic and therefore also comes with a guarantee of circuit discovery given certain assumptions on available data and time.
summary: The paper proposes an automatic, mechanistic interpretability approach for finding circuits in transformer models. Their approach propagates backwards from the model output, finding a circuit of nodes most relevant to the given output, and from there the nodes most relevant to those nodes, and so on. These circuits are task-specific subgraphs in the larger model compute graph that are most stimulated (and most relevant) to the input type or task under consideration. 

The authors consider circuit discovery for the attention heads of transformer models, choosing to leave the MLP layers in an attention block for later work. Their linear decomposition model is mechanistic in that it solely depends on the weights and biases of each layer of the network and the relevance metric calculated for the previous layer. It does appear to be data-dependent in that it still relies on a dataset for calculating these relevance metrics. It also relies on precomputed mean activations to do so.
weaknesses: 1) I am unsure of the significance of the work due to the iterative nature of the process and the need for data to adequately stimulate different 'tasks' (or behavior modes, perhaps?) and therefore discover circuits in the model. This is to my eyes not clearly explained. [Addressed]
2) Prior work is noted to have conducted circuit discovery on large language models, and I can't easily find the models used for circuit discovery here. That lack of clarity is concerning, especially when using the manually discovered circuits as a target and prior art as baselines. [Addressed]
3) I am also uncertain of scalability of this approach, considering that the network architectures under evaluation are not clear and that it has been restricted to attention heads rather than MLP layers. I am also uncertain of the effects of sequence length, autoregressive or non-autoregressive decoders. Section 5 does not make these things clear. [Addressed in part; prior art remains deficient as well here - future work?]

Meta-review: Thank you for your submission to ICLR. This paper presents CD-T (contextual decomposition for transformers), a method which aims to build interpretable circuits in large language models. It computes these circuits efficiently, with possible reductions in runtime from hours to seconds, in comparison with existing baselines. In addition, CD-T performs well empirically, compared to baselines, on multiple circuit evaluation datasets.

Reviewers appreciate the clarity of presentation, the method novelty, and the performance of the method. While there was some concern about certain relevant background descriptions and definitions, as well as the method’s scalability, the authors eventually addressed all of the reviewers’ concerns.

Decision: Accept (Poster)