Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control

## Abstract

Abstract Disentangling model activations into meaningful features is a central problem in
interpretability. However, the absence of ground-truth for these features in
realistic scenarios makes validating recent approaches, such as sparse
dictionary learning, elusive. To address this challenge, we propose a framework for
evaluating feature dictionaries in the context of specific tasks, by comparing
them against supervised feature dictionaries. First, we demonstrate that
supervised dictionaries achieve excellent approximation, control, and
interpretability of model computations on the task. Second, we use the
supervised dictionaries to develop and contextualize evaluations of unsupervised
dictionaries along the same three axes. We apply this framework to the indirect object identification (IOI) task using
GPT-2 Small, with sparse autoencoders (SAEs) trained on either the IOI or
OpenWebText datasets. We find that these SAEs capture interpretable features for
the IOI task, but they are less successful than supervised features in
controlling the model. Finally, we observe two qualitative phenomena in SAE
training: feature occlusion (where a causally relevant concept is robustly
overshadowed by even slightly higher-magnitude ones in the learned features),
and feature over-splitting (where binary features split into many smaller,
less interpretable features). We hope that our framework will provide a useful
step towards more objective and grounded evaluations of sparse dictionary
learning methods.

## 1 Introduction

While large language models (LLMs) have demonstrated impressive
(Vaswani et al., ; Devlin et al., ; Radford et al., ; Brown et al., ; OpenAI, )
results, the mechanisms behind their successes and failures
largely remain a mystery (Olah, ). A central problem in
this area is how to *disentangle* internal model representations into
meaningful concepts or *features*. If successful at scale, this research
could provide significant scientific and practical value, enabling enhanced
model robustness, controllability, interpretability, and debugging
(Gandelsman et al., ; Nanda et al., ; Marks et al., ).

A leading hypothesis for how LLMs represent and use features is the *linear
representation hypothesis*
(Mikolov et al., ; Grand et al., ; Li et al., ; Abdou et al., ; Nanda et al., ).
A strong version of this hypothesis posits that individual activations of a model can be
decomposed into sparse linear combinations of features from a large,
shared *feature dictionary*. Recently, a series of works has proposed
applying the (unsupervised) *sparse autoencoder* (SAE) framework to find
such dictionaries
(Olshausen & Field, ; Faruqui et al., ; Goh, ; Arora et al., ; Yun et al., ; Cunningham et al., ; Bricken et al., ).

Figure: Figure 1: Overview of our evaluation pipeline. We begin by selecting a specific model capability and then disentangling model activations into capability-relevant features using supervision. Then, we evaluate a given feature dictionary w.r.t. this capability, using the supervised features as a benchmark. We test the extent to which (1) the feature dictionary’s reconstructions of the activations are necessary and sufficient for the capability, (2) the features can be used to edit capability-relevant information in internal model representations (agnostic of feature interpretations), and (3) the features can be interpreted w.r.t. the capability in a manner consistent with their causal role.
Refer to caption: x1.png

While there are promising initial results (Bricken et al., ),
this research area faces a key obstacle: we cannot directly evaluate the
usefulness of features learned by an SAE, as we do not know the hypothetical
‘true’ features to begin with; indeed, finding them is the reason we use
SAEs in the first place. This has two consequences: (1) the training objective
of an SAE – balancing $\ell_{2}$ reconstruction with an $\ell_{1}$ penalty on
feature coefficients – is only a substitute for the true goal of finding meaningful features, and (2)
the metrics used to evaluate SAEs are indirect, relying on proxies for the
features, success in toy models, or non-trivial assumptions on SAE learning
(Elhage et al., ; Bricken et al., ; Sharkey et al., ). This hampers
progress in the field, as there is no simple, objective and direct way to
compare trained SAEs. In this paper, we take steps towards addressing this
challenge. Our contributions are outlined as follows:

- •
We propose a principled method for computing sparse feature dictionaries to
decompose language model activations on realistic tasks, using supervision from
attributes describing the prompts.
- •
We apply this method to the IOI task, demonstrating that these dictionaries exhibit
three desirable properties in the task’s context: (1) sufficiency and necessity of
activation reconstructions, (2) sparse controllability of model behavior via
feature editing, and (3) interpretability of the features consistent with their
causal role(^1^11 As a by-product, our supervised feature dictionaries also demonstrate that task activations can be usefully disentangled in a way that adheres to the linear representation hypothesis and exhibits superposition. To the best of our knowledge, this is the first time such a disentanglement has been achieved in a realistic LLM task.).
- •
We use these feature dictionaries to design and contextualize evaluations
of *unsupervised* feature dictionaries along the same three axes.
Importantly, we aim to develop evaluations that are agnostic to whether the
unsupervised dictionaries use the same concepts as the supervised
ones.
- •
We apply this methodology to feature dictionaries learned by SAEs trained on
either the IOI dataset (task-specific SAEs) or the LLM’s pre-training dataset
(full-distribution SAEs). We find that both types of SAEs find
interpretable features for the task, but task-specific SAEs allow us to edit
attributes by changing fewer features compared to full-distribution SAEs.
However, both kinds of SAEs fall short of supervised dictionaries.
- •
Finally, we dive deeper into some qualitative phenomena observed in
task-specific SAEs: feature occlusion (where SAEs have a tendency to learn only
the higher-magnitude of two task-relevant attributes, even when the attributes
have slightly different magnitudes), and feature over-splitting (where SAEs tend
to learn $\gg 2$ features without a clear interpretation for binary concepts). We
reproduce both phenomena in simple toy models, suggesting they may generalize
beyond our specific setting.

Our results underscore the need for more principled training and evaluation
methods in this active area and suggest that supervised feature dictionaries
can be a valuable tool for automating aspects of this process.

## 2 Preliminaries

The linear representation hypothesis and sparse autoencoders. A
central hypothesis in interpretability is the *linear representation
hypothesis*. A strong variant of this hypothesis posits that model activations can be decomposed into meaningful features
using a *sparse feature dictionary*: given a location in the model (e.g., an
attention head output), there exists a set of vectors
$\{\mathbf{u}_{i}\}_{i=1}^{m}$ such that each activation $\mathbf{a}$ at this location can
be approximated as a sparse linear combination of the $\mathbf{u}_{j}$ with
non-negative coefficients. In particular, recent work suggests that
$n$-dimensional activations $\mathbf{a}\in\mathbb{R}^{n}$ may be best described by $m\gg n$ such features in
*superposition* (Elhage et al., ; Gurnee et al., ).
Recently, SAEs have been proposed as a way to disentangle these features. Following the
setup of Bricken et al. () here and in the rest of this work, a sparse autoencoder (SAE) is an
unsupervised model which learns to reconstruct activations
$\mathbf{a}\in\mathbb{R}^{n}$ as a weighted sum of $m$ features
with non-negative weights. Specifically, the autoencoder computes a hidden
representation

$$ $\displaystyle\mathbf{f}=\operatorname{ReLU}\left(W_{enc}\left(\mathbf{a}- \mathbf{b}_{dec}\right)+\mathbf{b}_{enc}\right)$ $$

and a reconstruction

$$ $\widehat{\mathbf{a}}=W_{dec}\mathbf{f}+\mathbf{b}_{dec}=\sum_{j=1}^{m}\mathbf{ f}_{j}(W_{dec})_{:,j}+\mathbf{b}_{dec}$ (1) $$

where $W_{enc}\in\mathbb{R}^{m\times n}$, $W_{dec}\in\mathbb{R}^{n\times m},\mathbf{b}_{dec}\in\mathbb{R}^{n},\mathbf{b}_
{enc}\in\mathbb{R}^{m}$ are learned
parameters. The rows of $W_{enc}$ are the *encoder directions*, and the columns
of $W_{dec}$ are the *decoder directions*.
Similarly, $\mathbf{b}_{enc}$ is the encoder bias and $\mathbf{b}_{dec}$ is the decoder bias.
The decoder directions determine the features we decompose the activations into,
while the encoder directions compute the coefficients of these features for a
given activation. The decoder directions are constrained to have unit norm:
$\left\|(W_{dec})_{:,i}\right\|_{2}=1$. The training objective over examples
$\{\mathbf{a}^{(k)}\}_{k=1}^{N}$ is the sum of the MSE between the
activations $\mathbf{a}^{(k)}$ and their reconstructions
$\widehat{\mathbf{a}}^{(k)}$, and the $\ell_{1}$ regularization term
$\lambda\sum_{k=1}^{N}\left\|\mathbf{f}^{(k)}\right\|_{1}$, where $\lambda$ is
the $\ell_{1}$ regularization coefficient.

The IOI task.
In Wang et al. (), the authors analyze how the decoder-only
transformer language model GPT-2 Small (Radford et al., ) performs the
Indirect Object Identification (IOI) task.
In this task, the model is required to complete sentences of the form ‘When Mary
and John went to the store, John gave a book to’ (with the intended
completion in this case being ‘ Mary’). We refer to the repeated name (John) as
S (the subject) and the non-repeated name (Mary) as IO (the
indirect object). For each choice of the IO and S names, there
are two patterns the sentence can have: one where the IO name comes
first (we call these ‘ABB examples’), and one where it comes second (we call
these ‘BAB examples’). We refer to this binary attribute as the Pos
attribute (short for position). Additional details on the data distribution,
model and task performance are given in Appendix A.4.

Wang et al. () discover several classes of attention heads in GPT2-Small that
collectively form the *IOI circuit* solving the IOI task. Specifically,
Wang et al. () argue that the circuit implements the
algorithm:

- 1.
detect the (i) position in the sentence and (ii) identity of the repeated
name in the sentence (i.e., the S name). This information is computed
and moved by *duplicate token*/*induction* and *S-Inhibition*
heads;
- 2.
based on the two signals (i) and (ii), exclude this name from the
attention of the *name mover heads*, so that they copy the remaining name
(i.e., the IO name) to the output.

We refer the reader to Appendix A.1 and Appendix
Figure 7 for more details on the IOI circuit.

*The logit difference metric.*
To discover the circuit, Wang et al. () used the logit
difference: the difference in log-probabilities assigned by the model to the
IO and S names. This metric is more sensitive than accuracy,
which makes possible the detection of individual model components with a
consistent but non-pivotal role in the task. Accordingly, we also use the logit
difference throughout this work to evaluate the causal effect of fine-grained
model interventions.

## 3 Methods for Evaluating Feature Dictionaries

### 3.1 Overview and Motivation

In this section, we describe and motivate our methodology for evaluating sparse
feature dictionaries in the context of a specific task an LLM can perform.
Throughout, let $\mathcal{D}$ be a distribution over input prompts for the task.

Step 1: Parametrize inputs via (task-relevant) attributes. First, we
choose attributes $a_{i}:\operatorname{support}(\mathcal{D})\to S_{i}$ taking values in some finite sets
$S_{i}$. For example, in the IOI task we will focus on the attributes
IO, S, and Pos described in Section
2, with IO and S taking values in the set of
names in our dataset, and Pos taking values in the set $\{\text{ABB},\text{BAB}\}$.

Step 2: Compute high-quality supervised dictionaries using the attributes.
The second step is to learn and validate *supervised* dictionaries computed
using the attribute labels $a_{i}(p),p\sim\mathcal{D}$ (methodology described in
Section 4). In particular, not all attribute
configurations will result in dictionaries with good approximation, control and
interpretability for the task(^2^22 To get high-quality supervised dictionaries, we should choose attributes that completely capture the task-relevant information in the prompt, and are moreover compatible with the intermediate states of the model’s computation on the task. Validating that our attributes satisfy this is a non-trivial but key prerequisite for our evaluation to be meaningful. We do this for the IOI task in Section 4, where we show the supervised dictionaries score highly on all tests described in the current section.).

Step 3: Evaluate (unsupervised) dictionaries using the supervised
dictionaries as a benchmark. Finally, we evaluate the usefulness of a given
feature dictionary for the chosen task, using the supervised features as a
reference point in various ways (described in the next subsections). The
supervised dictionaries are a necessary part of this evaluation because they
allow us to *contextualize* performance differences between feature
dictionaries for the chosen task.

Why are supervised dictionaries a meaningful benchmark? On the one
hand, supervised dictionaries’ features are hard-coded to
correspond to the ‘human-salient’ attributes $a_{i}$. So, our evaluations risk
overlooking useful feature dictionaries that do not align with this human-chosen
ontology. On the other hand, the saliency of the attributes is precisely what
makes them desirable targets for controlling and interpreting the model; for
example, in the IOI task it is natural to want to change e.g. the representation
of the IO name in the model’s computation.
Thus, our evaluations aim to balance two opposing goals: establish properties
useful from a human standpoint, while still being agnostic to the exact concepts
represented in the unsupervised dictionary.

### 3.2 Test 1: Sufficiency and Necessity of Dictionary Reconstructions for the Task

Figure: Figure 2: Sufficiency (left) and necessity (right) evaluations of reconstructions of cross-sections of the IOI circuit computed using supervised feature dictionaries, task- and full-distribution SAEs. Left: average logit difference when replacing activations in cross-sections of the IOI circuit with their reconstructions, normalized by the average logit difference over the data distribution in the absence of intervention (a $y$-axis value of $1$ is best). Right: drop in logit difference when deleting reconstructions, normalized by the respective drop when performing mean-ablation, and linearly rescaled so that values close to 1 are best. See Appendix A.3 for details.
Refer to caption: x2.png

Our first test checks if activation reconstructions $\widehat{\mathbf{a}}$ *as
a whole* are sufficient and necessary for the model to perform the
task(^3^33This test is analogous to the faithfulness/completeness test from Wang et al. (); the sufficiency test is also widely used in the literature to evaluate feature dictionaries.).
To evaluate sufficiency, we intervene by replacing
internal activations $\mathbf{a}$ with their reconstructions
$\widehat{\mathbf{a}}$, and measure the drop in performance on the task. To evaluate
necessity, we intervene by replacing activations
$\mathbf{a}$ with $\mathbb{E}_{\mathcal{D}}\left[\mathbf{a}\right]+\left(\mathbf{a}-\widehat{
\mathbf{a}}\right)$, and compare the resulting drop in performance to the
effect of replacing activations with $\mathbb{E}_{\mathcal{D}}\left[\mathbf{a}\right]$
alone (also known as *mean ablation*; see Appendix
A.2 for motivation)
(^4^44 This test has very few assumptions: it is independent of the attributes we have chosen to describe inputs with, and of any interpretation assigned to the features. However, it only measures the quality of the dictionary as a whole, and not how effectively and sparsely it disentangles any hypothetical concepts in the activations. We include this test because poor performance indicates a fundamental failure of the feature dictionary relative to the task.).

### 3.3 Test 2: Sparse Controllability of Attributes

With this test, we want to measure the degree to which the feature dictionary
can be used to control the model’s behavior on the task by editing intermediate
representations of attribute values.
To use a feature dictionary for editing, we write an activation as a linear combination of dictionary vectors, and try to remove/add dictionary elements to the combination in order to achieve the desired change.
To evaluate such an intervention, we consider both (1) the number of features in the dictionary that need to be changed
in order to achieve a given change in outputs (editing is trivial if e.g. activations are dense sums of random features and we change many of them), and
(2) the geometric magnitude of the edit (editing is trivial if you are willing
to throw away the entire activation vector and replace it with the target). Jump ahead to Figure 4 to see what trade-offs for these parameters look like in our experiments.

Being agnostic to feature interpretations.
Importantly, we require this test to be agnostic to any human interpretation of
the feature dictionary. This is valuable for several reasons: (1) even if our
attributes are compatible with model computations, the feature dictionary may
not be directly interpretable in terms of these
attributes (see Appendix A.10 for hypothetical
examples in the IOI task); (2) human-generated feature interpretations, even
with meticulous methodology, may be subjective; (3) sparse
control may be possible to a useful degree even if the features are inherently
non-interpretable from a human perspective.
Thus, this test should evaluate the degree to which the feature dictionary
sparsely disentangles the attributes, but is independent of whether the features
are interpretable in terms of the attributes (or at all)
(^5^55Note that, to meaningfully conclude disentanglement of attributes from this test, the task must be one where at least some of the activations represent multiple attributes simultaneously. This is true in the IOI task, where many circuit locations represent the S and Pos information simultaneously. We also find that one particular location, the queries of the L10H0 name mover head, represent all three attributes IO, S and Pos.).

Implementation design space.
To achieve independence of interpretations, we frame the problem as a
combinatorial optimization problem over the feature dictionary: given an
activation $\mathbf{a}_{s}$ for prompt $p_{s}$ and a counterfactual activation
$\mathbf{a}_{t}$ for a prompt $p_{t}$ which differs from $p_{s}$ only in the values of
the attributes we wish to edit, we can optimize over subsets of features active
in $\widehat{\mathbf{a}_{s}}$ to subtract from $\mathbf{a}_{s}$, and subsets of
features active in $\widehat{\mathbf{a}_{t}}$ to add to $\mathbf{a}_{s}$. There are
multiple optimization objectives possible, such as making the edited activation
similar to $\mathbf{a}_{t}$ geometrically, or making model behavior on the edited
activation similar to model behavior on $\mathbf{a}_{t}$.
The optimization may be constrained by the number of features changed and/or the
magnitude of edits
(^6^66 There are many possible ways to instantiate such an optimization; in Section 5 we present one such way based on a greedy algorithm minimizing $\ell_{2}$ distance in activation space. Further methodological notes, such as how we implement edits in multiple model components at once, are given in Appendix A.2.).
Importantly, this optimization proceeds on a
per-prompt basis, which means that it can select different features to
edit depending on the prompt being edited. This means it will not disadvantage
dictionaries that do not dedicate a uniform w.r.t. all prompts set of features
to each attribute (which would be a major assumption to impose).

Evaluation. To measure how well an edit changed model behavior, we
compare against the ‘ground truth’ change in behavior that would be achieved by
intervening on the model to replace $\mathbf{a}_{s}$ with $\mathbf{a}_{t}$ directly.
To contextualize the magnitude of the edit, we can compare the contribution of
the changed features to the reconstruction against the analogous quantity for
our supervised feature dictionary (see Appendix A.2 for
details), as we do later in Section 4.

### 3.4 Test 3: Interpretability

With our final test, we want to assess the degree
to which the feature dictionary can be interpreted in terms of the task-relevant
attributes and the features are causally relevant for the model’s
behavior in a way consistent with their interpretations. While failing this test
does not necessarily mean that the feature dictionary is not useful (e.g., it
could still be useful for control, or interpretable with respect to different
attributes), passing it would increase our confidence in the feature
dictionary’s utility and our understanding of the model’s computation. More
broadly, passing this test on a wide range of important tasks would represent a
qualitative improvement over control alone in applications such as auditing,
debugging and verification.

Assigning interpretability scores. Given a feature $\mathbf{u}_{j}$
from the dictionary, its *active set* $F\subset\operatorname{supp}\left(\mathcal{D}\right)$ is the subset of the support of
$\mathcal{D}$ where $\mathbf{f}_{j}>0$. Given
a binary property of inputs $P:\mathcal{D}\to\{0,1\}$, following
Bricken et al. (), we say that $P$ is a good interpretation of
the feature if $F$ has high precision and recall relative to $P$
(^7^77Clearly, the best interpretation of $F$ according to these metrics alone is the indicator $\mathbf{1}_{F}$ itself; for $P$ to be a useful interpretation from a human perspective, it must in addition be ‘simple’ enough, a property harder to formalize.).
We combine the precision and recall metrics into a single number using the $F_{1}$
score.

For example, each attribute defines $\left|S_{i}\right|$ binary properties
$\mathbf{1}_{a_{i}(p)=v}$ for $v\in S_{i}$ that we can use to try to interpret the
feature dictionary. When considering a set of binary properties $\{P_{j}\}_{j\in J}$ as possible interpretations for a feature, we pick the one with the highest
$F_{1}$ score and assign this as the interpretation of the feature. Other
interpretability measures are possible; for a discussion of the limitations of
the $F_{1}$ score, see Appendix A.2.

Which interpretations to consider? As with controllability, we should
not assume that the features correspond 1-to-1 with the attributes we have
chosen. Thus, evaluating the $F_{1}$ scores only for the binary properties
$\mathbf{1}_{a_{i}(p)=v}$ corresponding to a single value for a single attribute
may be subjective. On the other hand, when the chosen attributes result in
a good supervised dictionary (required by Step 2 of our method), the attribute
values will be sufficient to deduce the causally important information in model
activations. This suggests that causally-relevant features in an arbitrary
feature dictionary will correspond to particular subsets of the cartesian
product $\prod_{i}S_{i}$ of the values the attributes can take.

This motivates us to look for interpretations that are expressible as
intersections and unions of the indicator sets $\mathbf{1}_{a_{i}(p)=v}$ of the
chosen attributes. We use heuristics to navigate this search space. We present
one possible implementation of this in Section 5 for
the IOI task. As we will see, this choice, while possibly arbitrary/subjective,
is to some extent validated empirically: we find that many features can be
interpreted in this way for a significantly high $F_{1}$ score threshold. We further discover some
interpretable structure in the ways attribute values group together in unions.

Causal evaluation of interpretations. Finally, while our previous
interpretability methods look for correlations with properties of the input
prompts, we also want to know if the interpretations we assign to features in
the dictionary are consistent with their causal role in the model’s computation.
There are two increasingly demanding ways to check this that mirror our first
and second tests (Subsections 3.2 and 3.3):

- •
interpretation-aware sufficiency/necessity of reconstructions:
similar to our first test (Subsection
3.2), we can (1) subtract
non-interpretable features from activations and see if the model is still able
to perform the task; (2) subtract highly-interpretable features and see if
the model’s performance degrades to the same extent as with mean ablation. This
evaluates whether our interpretability method as a whole flags the important
features for the task;
- •
interpretation-aware sparse controllability: like our second test
(Subsection 3.3), but explicitly
using highly-interpretable features (with respect to a given attribute) to
remove/add in order to edit an activation. This evaluates whether
interpretations that specifically relate features to attributes find the
attribute-relevant features.

## 4 Computing and Validating Supervised Feature Dictionaries

We first present our methods and results for computing *supervised* feature
dictionaries, in which features correspond 1-to-1 with the possible values of
attributes we have chosen. This is a key prerequisite for our evaluation of SAEs
later on, as it (1) verifies that the attributes we have chosen are compatible
with the model’s internal states; (2) demonstrates the existence of high-quality
sparse feature dictionaries for the task; and (3) provides an ‘ideal’
reference for SAE evaluation.
Specifically, we will establish that feature dictionaries for all locations in
the IOI circuit exist with the following properties:

- •
they approximate activations well using as few as 3 active features per
activation;
- •
any of the IO, S, and Pos attributes can be
edited precisely by replacing only 1 active feature with a different one;
- •
the features in the dictionary are by construction interpretable w.r.t. the attributes, and furthermore, interactions in the IOI circuit exhibit
sparsity when decomposed into feature-to-feature interactions.

### 4.1 Computing Supervised Feature Dictionaries

Algorithms. Motivated by the linear representation hypothesis, we
conjecture that given activations $\mathbf{a}\left(p\right)\in\mathbb{R}^{d}$ of a given
model component (e.g., outputs of an attention head) for prompts $p\sim\mathcal{D}$, there exists a choice of attributes $\{a_{i}:\mathcal{D}\to S_{i}\}_{i\in I}$ such that

$$ $\mathbf{a}(p)\approx\mathbb{E}_{p\sim\mathcal{D}}\left[\mathbf{a}(p)\right]+ \sum_{i\in I}\mathbf{u}_{a_{i}(\cdot)=v}:=\widehat{\mathbf{a}}$ (2) $$

where $\widehat{\mathbf{a}}$ is the *reconstruction* of $\mathbf{a}$, and
$\mathbf{u}_{a_{i}(\cdot)=v}\in\mathbb{R}^{d}$ is a feature corresponding to the
$i$-th attribute having value $v\in S_{i}$
(^8^88This formulation is less expressive than SAE reconstructions, as it effectively requires a given feature to always appear with the same coefficient in reconstructions; we discuss this choice in Appendix A.3.).

Given a dataset of prompts $\{p_{k}\}_{k=1}^{N}$ with associated activations
$\{\mathbf{a}\left(p_{k}\right)\}_{k=1}^{N}$, how should we compute ‘good’ values for the
vectors $\mathbf{u}_{a_{i}(\cdot)=v}$? While we considered several ways to do so,
our best method for the IOI task is simple: we average all activations for
prompts for which $a_{i}(p)=v$. Formally,

$$ $\displaystyle\mathbf{u}_{a_{i}(\cdot)=v}:=\frac{1}{\left|\{k:a_{i}(p_{k})=v\} \right|}\sum_{k:a_{i}(p_{k})=v}\mathbf{a}\left(p_{k}\right)-\overline{\mathbf{ a}}$ $$

where $\overline{\mathbf{a}}=\frac{1}{N}\sum_{k=1}^{N}\mathbf{a}\left(p_{k}\right)$ is
the empirical mean activation. We refer to these dictionaries as mean
feature dictionaries. One can prove that, in the limit of infinite data, the
mean features for an attribute not linearly detectable in the activations will
converge to zero (Appendix A.5).

The main alternative method we considered was MSE feature dictionaries,
which use a least-squares linear regression to predict the activations from the
attribute values. We note that mean feature dictionaries work well in our
setting because the attributes we choose in the IOI task are
probabilistically independent in the IOI distribution; we recommend using MSE
dictionaries in general (and see Appendix A.6 for more details on
MSE dictionaries and comparisons to mean dictionaries).

Choosing attributes for the IOI task. Not every set of attributes
will result in a good approximation of the model’s internal activations; in
fact, we find that the choice of attributes is a crucial modeling decision.
Recall that, according to Wang et al. (), each IOI prompt $p$
is described by three properties influencing how $p$ is processed in the IOI
circuit: the subject (S) and indirect object (IO) names, and
their relative position (Pos).
Motivated by this,
we chose the attributes S, IO, and Pos to describe
each prompt. We experimented with other choices of attributes, but did not find
them to be more successful in our tests (see Appendix
A.8 for details(^9^99 In particular, we found that another set of attributes, though more expressive in principle, learns to approximate the features we get using the S, IO and Pos attributes through a change-of-variables-like transformation.)). We
emphasize that there are many other imaginable choices of attributes; see
Appendix A.10 for further discussion.

Figure: Figure 3: Accuracy when editing IO, S and Pos for circuit cross-sections using our supervised feature dictionaries and task-specific SAEs; the outcome in the absence of intervention is shown in blue for reference. When using task-specific SAEs, we edit either 2, 4 or 6 features (which means we in total add and/or remove up to that many features from activations). For comparison, supervised edits always involve removing 1 feature and adding 1 feature. Accuracy is measured as the proportion of examples for which the model’s prediction agrees with the ground-truth prediction for the edit; see Section 5.1 and Appendix A.3 for details.
Refer to caption: x4.png

### 4.2 Evaluation Results

Our evaluations are carried out at the main cross-sections of the IOI circuit.
These cross-sections, as well as the information processing they perform and the
expected effect of editing attributes in them, are described in detail in
Appendix A.3. These considerations guide our choice
of the attributes to edit in each given cross-section.

Sufficiency/necessity plots using the mean feature dictionaries are shown in
Figure 2 (orange bars); we find that
our supervised dictionaries are quite successful. For controllability, note
that attribute editing can naturally be defined in closed form, because we have
a 1-to-1 correspondence between attribute values and features by construction.
We find that simple feature arithmetic works quite well. Formally, given prompt
$p$ with $a_{i}(p)=v$, we can edit the attribute $a_{i}$ to have value $v^{\prime}$ via
$\mathbf{a}_{edited}(p):=\mathbf{a}(p)-\mathbf{u}_{a_{i}(\cdot)=v}+\mathbf{u}_{
a_{i}(\cdot)=v^{\prime}}$. Results for editing in cross-sections of the IOI
circuit are shown in Figure 3 (orange bars), where we
report the fraction of the time our intervention predicts the same token as the
ground-truth intervention that substitutes counterfactual activations in the
corresponding cross-section (a value of 1 is best). We observe that, when
editing the S attribute, not performing any intervention often already
mostly agrees with the ground-truth edit, effectively reducing the resolution of
our evaluation results for this attribute. Otherwise, we find our supervised
dictionaries to perform well, always achieving $>80\%$ agreement with the
ground-truth intervention.

Finally, the supervised feature dictionaries tautologically pass the
interpretability test, as they were defined to have a single feature activating
for each possible attribute value, achieving perfect $F_{1}$ scores. Accordingly, we
performed a more demanding test of interpretability: decomposing internal model
computations in terms of interactions between individual features. We find that
pre-softmax attention scores and composition between heads can be decomposed in
terms of feature-level interactions, such that many interactions are close to
zero, and the few non-zeros correspond to those expected based on the high-level
IOI circuit description from Wang et al. (); see Appendix
A.7 for details.

## 5 Evaluating Task-Specific and Full-Distribution Sparse Autoencoders

### 5.1 Methodology

SAE training on the task-specific and full pretraining distributions.
We trained SAEs on all IOI circuit locations, using activations from either the
IOI dataset (‘task SAEs’) or OpenWebText distribution
(Gokaslan & Cohen, ) (‘full-distribution SAEs’). While performance varied
strongly across circuit locations, most full-distribution SAEs had an
$\ell_{0}$-loss between 2 and 12 and a recovered loss fraction (against a mean ablation baseline)
between $0.4$ and $0.9$ (both measured on OpenWebText). Similarly, most
task-SAEs had an $\ell_{0}$-loss below $25$ and a recovered logit difference
fraction against mean ablation $>0.8$ (both measured on the IOI dataset)
(^10^1010Importantly, we did not perform exhaustive hyperparameter tuning to train these SAEs, as our main goal was to evaluate the methodology and how it can distinguish between different classes of feature dictionaries, rather than to achieve state-of-the-art performance. Thus it is possible that significantly better performance could be achieved with more tuning. Indeed, it is our hope that the methods we present here will be useful for tuning SAEs in the future.).
Further details are given in Appendix A.11.

Sparse controllability implementation in the IOI task.
We instantiate the sparse controllability test from Subsection
3.3 as follows. Suppose our SAE has a dictionary of decoder
vectors $\{\mathbf{u}_{j}\}_{j=1}^{m}$, and the original and counterfactual
activations $\mathbf{a}_{s},\mathbf{a}_{t}$ have reconstructions respectively

$$ $\displaystyle\widehat{\mathbf{a}}_{s}=\sum_{i\in S}\alpha_{i}\mathbf{u}_{i}+ \mathbf{b}_{dec},\quad\widehat{\mathbf{a}}_{t}=\sum_{i\in T}\beta_{i}\mathbf{u }_{i}+\mathbf{b}_{dec}$ $$

for $S,T\subset\{1,\ldots,m\}$ and $\alpha_{i},\beta_{i}>0$. Consider the
optimization problem

$$ $\displaystyle\min_{R\subset S,A\subset T,\left|R\cup A\right|\leq k}\left\| \mathbf{a}_{s}-\sum_{i\in R}\alpha_{i}\mathbf{u}_{i}+\sum_{i\in A}\beta_{i} \mathbf{u}_{i}-\mathbf{a}_{t}\right\|_{2}$ $$

In words, this problem asks for at most $k$ features to remove ($R$) from and/or
add ($A$) to the original activation(^11^1111We also experimented with using the reconstructions instead of the actual activations in this algorithm, but results were worse.) to bring it as close as possible to the
counterfactual activation, where the features to add are taken directly from the
counterfactual one. In general, this problem has no polynomial-time (in
$k,\left|S\right|,\left|T\right|$) solution, as the NP-hard problem
SubsetSum reduces to it; instead, we use a greedy algorithm to find a
solution.

*Measuring the magnitude of edits.*
To measure the magnitude of the edit, we compare the contribution of the changed
features to the reconstruction against the analogous quantity for our ‘ideal’
supervised feature dictionary. Namely, for each summand in the reconstruction
we assign a measure of its contribution $\operatorname{weight}(i)=(f_{i}\mathbf{u}_{i})^{\top}\left(\widehat{\mathbf{a}
}-\mathbf{b}_{dec}\right)/\left\|\widehat{\mathbf{a}}-\mathbf{b}_{dec}\right\|
_{2}^{2}$ so that
$\sum_{i=1}^{k}\operatorname{weight}(i)=1$(^12^1212 While weights can in general take any real value, we find that in practice they are almost always approximately in $[0,1]$; see Appendix A.16 for empirical details.). Note that weights are
additive in the features, so that the sum of weights of some subset of features
is the weight for these features’ total contribution to the reconstruction. We
then measure the magnitude of an edit by the total weight of the features
removed during the edit.

Interpretability implementation for the IOI task. We instantiate the
interpretability test from Section 3 as follows. Starting
with our primary attributes IO, S, and Pos, we
consider intersections of each of IO and S with the
Pos attribute. Then, for each resulting attribute that represents
variation over the IO or S name, we also consider unions of up
to 10 attribute values (30 for OpenWebText). Finally, we add some other
attributes of interest, such as the gender commonly associated with names in our
dataset. Details on all the possible interpretations we consdered are given in
Appendix
A.12.
Additionally, we evaluate the causal role of highly interpretable features as
described in Subsection 3.4:

- •
sufficiency/necessity of interpretable features: for
sufficiency, we intervene by subtracting from
activations only the features for which *no* interpretation has an $F_{1}$
score above some threshold. To test necessity, we instead subtract the features
for which their chosen interpretation has an $F_{1}$ score above a threshold.
This
experiment is directly comparable with the sufficiency/necessity of
reconstructions experiments, and uses the same baselines.
- •
interpretability-aware sparse control: we attempt to edit a given
attribute by removing/adding only $\leq k$ features with highest $F_{1}$ score for
the value of this attribute in the original/counterfactual prompt.
This experiment is directly comparable to the (interpretation-agnostic) sparse
control experiment and uses the same baseline.

Additional details are given in Appendix A.12.

Figure: Figure 4: Trade-offs between edit magnitude and edit success for attribute editing using task-specific SAEs for select IOI circuit cross-sections. The x-axis measures the weight (see Subsection 5.1) of the features removed by the edit (features added are not reported in this plot), averaged over the attention heads in the cross-section. This metric is affine-transformed so that a value of 0 indicates the weight removed by the corresponding supervised edit, and a value of 1 indicates that the edit removed all features in the reconstruction. The y-axis is an affine transform of the fraction of examples for which the edit results in the same next-token prediction as the ground-truth edit, with a value of 0 0 corresponding to no intervention, and a value of $1$ corresponding to the supervised edits. Results for our interpretation-agnostic/interpretation-aware editing methods are shown as thick/dashed lines respectively. For both methods, we edit 2, 4 or 6 features (a higher magnitude score indicates editing more features).
Refer to caption: x5.png

### 5.2 Results

Test 1: Sufficiency/necessity of reconstructions. The
sufficiency/necessity plots for the task-specific and full-distribution SAEs are
shown in Figure 2 (green and red).
We find that the task-specific SAEs offer a significantly worse, but not
catastrophically bad, approximation compared to the supervised feature
dictionaries. Meanwhile, the full-distribution SAEs fare notably worse than the
task SAEs.

Test 2: (Interpretation-agnostic) sparse controllability.

*Task-specific SAEs.* Results for sparse controllability using our
task-specific SAEs are shown in Figure 3 as well as Figure 4 for $k=2,4,6$. We
find that our SAEs can edit the Pos attribute and, to a small extent,
the IO attribute, even though this requires changing more features
compared to the supervised dictionaries. For the S attribute, the
results are less clear, because the range of performance between ‘no
intervention’ and the supervised edit is often within the margin of error; the
exception is the queries of the name mover heads, where results indicate failure
of the SAE features. Regarding the magnitude of edits, the most successful edits
introduce higher-magnitude changes as measured by the weight (recall Subsection
5.1) than the corresponding supervised
edits (results in Figure 4 as well as Appendix Figure 22). On the positive
side, the edits don’t overwrite the SAE features entirely.

*Full-distribution SAEs.* Full-distribution SAEs require a significantly
larger number of features to achieve a statistically significant level of
control compared to task-specific ones (often 32 or more); results are shown in
Appendix Figure 25. We also found that
the magnitude of these edits surpsasses that of supervised edits significantly,
often up to the point of throwing away total weight approaching $1$.

*A baseline: task-specific SAEs with decoder directions frozen at
initialization.* Do these results demonstrate *any* non-trivial
controllability afforded by the SAE features? To check, we run the same
controllability pipeline on task-specific SAEs which were trained with frozen at
initialization decoder directions; thus, these SAEs are forced to approximate
activations using sparse sums of random features. Results in Appendix Figure
24 show that our evaluation
distinguishes between these two types of SAEs: the frozen-decoder task SAEs perform
much worse than the ordinary task SAEs. In fact, we find that the
frozen-decoder task SAEs perform on par with the full-distribution SAEs.

Test 3: Interpretability.

Figure: Figure 5: Interpreting the IOI features learned by SAEs trained on OpenWebText. For each node in the IOI circuit, we show the distribution of interpretations for the features which have any interpretation with $F_{1}$ score above a threshold. The numbers in the right column indicate the number of features with an assigned interpretation by our method, and the color bars show the overall distribution of the SAE features (conditioned on the feature not being dead on the SAE training distribution). See Section 5 for methodology; details on the interpretations considered are given in Appendix A.12.
Refer to caption: x6.png

*Correlational evaluation.* Full-distribution SAEs must capture variation
in activations across a large set of text, of which IOI-like prompts are only a
small subset. Consistent with this, we found that only a subset of
full-distribution SAE features activates on IOI prompts, with the number of
features that fire on IOI prompts varying strongly between components. We scored
the features that do fire on IOI prompts and found a significant amount of
feature descriptions with high $F_{1}$-score. We summarize the number of
high-$F_{1}$-score features per type in Figure
5. Remarkably, we find that the
interpretable features in the full-distribution SAEs and the task-specific SAEs
are qualitatively similar; corresponding task-specific results are given in
Appendix Figures 26 (showing the most
interpretable SAEs at each node of the IOI circuit) and
27 (showing the SAEs chosen to optimize the
tradeoff between the $\ell_{0}$ loss and the logit difference reconstruction, which we use throughout the main body of the paper).

In practice, we want to use feature explanations to get insight into the more
general computation of a component. Thus, we investigated whether the features
found are consistent with the previously established function of the heads from
Wang et al. (). We found that this was true for all heads and
that simply looking at the number of features with a given interpretation draws
a clear picture; examples are provided in Appendix
A.13.

Our results also suggests several details about the IOI circuit that weren’t
reported previously, which we summarize in Appendix
A.13. We were also curious about how the detected
features behave on arbitrary text of the model’s training distribution. As
creating a rigorous test for this is difficult, we report some anecdotal
evidence in support of feature generalization in Appendix
A.13.

*Causal evaluation: sufficiency/necessity of interpretable features.*
Results here are encouraging: keeping/removing the features with $F_{1}$ score
$\geq 0.6$ often goes a long way towards preserving/degrading the model’s
performance on the task. Appendix Figures 30 and
31 show the results of these experiments for task SAEs.

*Causal evaluation: sparse control via interpretable features.* Here,
results are also moderately encouraging. We find that, for the task-specific
SAEs, editing using the high-$F_{1}$-score features w.r.t. a given attribute as a
guide performs not much worse than the interpretation-agnostic editing method. Results
are provided in Figure 4 in the form of a comparison
with interpretation-agnostic editing, as well as Appendix Figure
28. However, for full-distribution SAEs, we
again need to edit a high number of features to achieve a noticeable effect
(results in Appendix Figure 29).

## 6 Qualitative Phenomena in SAE Learning

Figure: Figure 6: Left: Fraction of IO (left subplot) and S (right subplot) names in our dataset for which a feature with $F_{1}$ score $\geq 0.5$ is found, as a function of dictionary size (x-axis) and effective $\ell_{1}$ regularization coefficient (y-axis), over a wide hyperparameter sweep for the queries of L10H0. Right: fraction of IO and S names in our dataset for which a feature with $F_{1}$ score $\geq 0.5$ is found, as a function of $\alpha$ ($x$-axis), the fraction of supervised IO features we subtract from the activations.
Refer to caption: x7.png

### 6.1 Feature Occlusion

Our experiments suggest that when two (causally relevant)
attributes are represented in the same activation, but one attribute’s (supervised) features have overall
higher magnitude, SAEs have a tendency to robustly learn more interpretable
features for the attribute with higher magnitude. We observe this in the queries
of the L10H0 name mover head at the END token in the IOI circuit, where the
IO and S attributes are both represented and causally
relevant (ablating either leads to a change of $\approx 0.5$ in logit difference). We find that our task SAEs consistently find features with high $F_{1}$ score
for individual IO names, but fail to find a significant number of
features for individual S names. In Figure 6
(left), we show interpretability results from training SAEs over a wide grid of
hyperparameters that confirm this observation; more details in Appendix
A.14.

*Hypothesis: feature magnitude is a driver of occlusion.* We noticed that
the supervised features for IO and S names in the L10H0
queries have a small but significant difference in norm (see Appendix Figure
13 (left)). We then hypothesized that feature magnitude
is a factor in this phenomenon. To check this, we surgically reduce the
magnitude of IO features in the activations using our supervised
feature dictionaries, and observe that the number of S features
discovered in these modified activations monotonically increases as we remove
larger fractions of the IO features (Figure 6
(right); see Appendix A.14 for methodology). We furthermore
constructed a simple toy model based on i.i.d. isotropic random features that
mimic the norms of supervised features in the L10H0 queries, and find that a
similar phenomenon occurs for the distribution of features with high $F_{1}$ score
(e.g. higher than $0.9$; see Appendix A.14).

### 6.2 Feature Over-splitting

We also found in our experiments that task SAEs have a tendency to split a single binary
attribute into multiple features, even when the number of features available
could in principle be spent on other attributes. Note that, while this behavior
may be counter-intuitive from a human standpoint, it does not necessarily mean
that the SAE failed; it may be that the binary attribute is not part of an
optimal sparse description of the model’s internal states. We observe this
phenomenon with the Pos attribute in the IOI task (again in the queries
of the L10H0 name mover), which is robustly split into many (e.g. $\geq 10$)
features by our SAEs that activate for small, mostly non-overlapping subsets of
examples sharing the same Pos value that have no clear semantic
interpretation.

*Is over-splitting a form of over-fitting?* To investigate whether this is due
to overfitting, we compared Pos features between (1) different random
seeds for the same training dataset and (2) different training datasets. In both
cases, we found that the Pos features discovered are similar above
chance levels, suggesting that the over-splitting is not due solely to
overfitting to randomness in the training algorithm and/or dataset.

*Reproducing the over-splitting phenomenon in a simple toy model.* On the
other hand, we show empirical evidence that in a toy model where activations are
a uniform mixture of two isotropic Gaussian random variables, an appropriately
*randomly initialized* SAE with enough hidden features will achieve lower
total loss than an ideal SAE with just two features corresponding to the two
components of the mixture. Such randomized constructions exist for *any*
$\ell_{1}$ regularization coefficient, even in the limit of infinite training
data. Details are given in Appendix A.15.

## 7 Related Work

Learning and evaluating SAE features. The SAE paradigm predates the recent surge of interest in LLMs. Early work
in ML focused on the analysis of word embeddings (Mikolov et al., ),
with works such as
Faruqui et al. (); Subramanian et al. (); Arora et al. () finding
sparse linear structure. Elhage et al. () proposed the use
of sparse autoencoders to disentangle features in LLMs.
Sharkey et al. () used SAEs to learn an over-complete dictionary in a
toy model and in a one-layer transformer, and follow-up work by
Cunningham et al. () applied this technique to residual stream
activations(^13^1313We adopt the terminology of (Elhage et al., ) when discussing internal activations of transformer-based language models.) of a
6-layer transformer from the Pythia family (Biderman et al., ).
Bricken et al. () trained SAEs on the hidden MLP activations of
a 1-layer language model, and performed several thorough evaluations of the
resulting features. Similarly to us, Gould et al. () trained SAEs
on a *narrow* data distribution instead of internet-scale data. Tamkin et al. () incorporated sparse feature
dictionaries (without per-example weights for the features) into the model
architecture itself, and fine-tuned the model on its pre-training distribution
to learn the dictionaries. More recently, Kissane et al. () used SAEs
on attention layer outputs of GPT-2 Small and found learned features that are
consistent with the IOI circuit from Wang et al. ().

Sparse autoencoders seek to (approximately) decompose
activations into meaningful features, and are thus a stronger form of
interpretability than linear probing for individual concepts
Alain & Bengio (), or finding individual subspaces with causal
effect (Geiger et al., ).

Throughout this line of work, the evaluation of learned SAE features has been a
major challenge. The metrics used so far can be broadly categorized as follows:

- •
indirect geometric measures: Sharkey et al. () proposed
use of the mean maximum cosine similarity (MMCS) between two different SAEs’
learned features to evaluate their quality. However, this metric relies on the
assumption that convergence to the same set of features is equivalent to
interpretability and having found the ‘true’ features.
- •
auto-interpretability:
Bricken et al. (); Bills et al. (); Cunningham et al. () used a
frontier LLM to obtain natural-language descriptions of SAE features based on
highly activating examples, and use the LLM to predict feature activations on
unseen text; the prediction quality is then used as a measure of
interpretability. However, the use of maximum (or even stratified by activation
value) activating examples has been criticized as potentially giving an illusory
and subjective sense of interpretability (Bolukbasi et al., ).
- •
manually crafted proxies for ground truth:
Bricken et al. () manually formed hypotheses about a handful
of SAE features and defined computational proxies for the ground truth features
based on these hypotheses. This method may be less prone to blind spots than
auto-interpretability, but still relies on the correctness of the computational
proxy.
- •
toy models: Sharkey et al. () used a toy model where
ground-truth features are explicitly defined; however, it is unclear whether toy
models miss crucial aspects of real LLMs. Similar objections apply to manually
injecting ground-truth features into a real model.
- •
direct logit attribution: Bricken et al. ()
additionally considered the direct effect of a feature on the next-token
distribution of the model; this method is valuable because it tells us about the
causal role of a feature, but it cannot detect its indirect effects via other
features.

Beyond the evaluation challenges, there is debate about whether SAEs find
computationally non-trivial, compositional features, or merely clusters of
similar examples in the data Olah et al. ().

Mechanistic interpretability and circuit analysis. Mechanistic interpretability (MI) aims to reverse-engineer the internal workings
of neural networks (Olah et al., ; Elhage et al., ). In particular,
MI frames model computations as a collection of *circuits*: narrow,
task-specific algorithms (Olah et al., ). So far, circuit analyses of
LLMs have focused on the component level, mapping circuits to collections of
components such as attention heads and MLP layers
(Wang et al., ; ).

However, the linear representation hypothesis suggests that component
activations can be broken down further into (sparse) linear combinations of
meaningful feature vectors; thus, the eventual goal of MI is to give a precise,
*subspace-level* understanding of the model’s circuits. Initial steps in
this direction have been taken using methods distinct from SAEs.
Geiger et al. () propose finding meaningful subspaces using an
optimization-based method; Nanda et al. () discover linear subspaces
in emergent world-models on a toy task; Tigges et al. () discover linear
subspaces corresponding to sentiment in a LLM. However, while these works focus
on finding individual subspaces representing specific concepts, the SAE paradigm
is more ambitious, as it aims to fully decompose activations as a sum over
meaningful features. This is a stronger property than identifying individual
meaningful subspaces, and would accordingly provide a more exhaustive form of
interpretability.

More broadly, MI has found applications in several downstream tasks: removing
toxic behaviors from a model (Li et al., ), changing factual knowledge
encoded by models (Meng et al., ), improving the truthfulness of LLMs
at inference time (Li et al., ), studying the mechanics of gender bias
in language models (Vig et al., ), and reducing spurious correlations by
intervening on model internals (Gandelsman et al., ).

## 8 Limitations and Conclusion

We have taken steps towards more principled and objective evaluations of the
usefulness of sparse feature dictionaries for disentangling LLM activations. In
particular, we have demonstrated that:

- •
simple supervised methods can be used as a principled way to compute
high-quality feature dictionaries in a task-specific context;
- •
these dictionaries can be used as ‘skylines’ to evaluate and contextualize
the performance of unsupervised methods, such as SAEs.

Limitations.
The central conceptual limitation of our work is that our method relies on
supervision in the form of a potentially subjective choice of variables used to
parametrize task-relevant information in model inputs. We mitigate this to some
extent by requiring this parametrization to be *consistent* with the
internal computations of the model, as quantified by our tests for
approximation, control and interpretability of model computations on the task.
However, in principle there could be many parametrizations that are just as
consistent, but fundamentally different (recall the discussion in Appendix
A.8 and
A.10). Thus, we risk making the proverbial ‘judging
a fish by its ability to climb a tree’ mistake. We have mitigated this problem
further by devising evaluations that are, when possible, agnostic to the precise
features in a dictionary, as long as they allow us to disentangle and control
our chosen variables in a sparse manner.

Finally, we find it likely that features used by LLMs in tasks of practical
interest will be quite complex from a human standpoint, and we believe it is
useful to be able to assess the degree to which these features can be harnessed
towards controlling model behavior along more top-down, concise and
human-understandable concepts. Our evaluations provide such an assessment.

Our work is also limited in that we only consider a single task, and a single
language model. We hope that in future work we will reduce these limitations.

Conclusion. Sparse dictionary learning, and SAEs in particular,
represent an interesting and promising avenue for disentangling internal
representations of LLMs. However, in order to measure progress in this area, it
is imperative to have nuanced conceptual understanding of the goals and
downstream applications of such disentangling, and benchmarks faithful to this
understanding. We hope that our work will inspire further research in this
direction, and that our methods will be useful for practitioners in the field.

## Acknowledgements

We thank Sonia Joseph and Can Rager for feedback on earlier versions of this
manuscript. We thank the SERI MATS program for making this collaboration possible. We
also thank Atticus Geiger for valuable discussions and supervision during some
early versions of this project. AM was supported by grants from AI Safety
Support and Effective Ventures. GL was supported by grants from AI Safety
Support.

## Author Contributions

AM designed and ran the algorithms for computing supervised feature dictionaries
and observed their theoretical properties; trained task-specific SAEs; designed
the tests for sufficiency/necessity of reconstructions, sparse controllability
and causal faithfulness of feature descriptions; ran these tests on full-distribution and task-specific SAEs; observed the occlusion/oversplitting phenomena in task-specific SAEs and developed the toy models for them; wrote all sections of the paper unless otherwise specified.

GL trained full-distribution SAEs; designed and ran the automatic feature scoring algorithms for them, and ran most of the interpretability evaluations for full-distribution SAEs; contributed to the sections on interpretability of full-distribution SAE features, created Figures 1 and 5.

NN provided guidance on research goals and prioritization, as well as writing, throughout the entire project; proposed the initial idea of finding supervised feature dictionaries for the IOI task, and using them to evaluate (unsupervised) feature dictionaries; refined the search for algorithms to compute the supervised dictionaries.

## Appendix A Appendix

### A.1 Additional details on the IOI circuit

Circuit structure.
To refer to individual token positions within the sentence, we use the notation
of Wang et al. (): IO denotes the position of the IO
name, S1 and S2 denote respectively the positions of the first and second
occurrences of the S name (with S1+1 being the token position after
S1), and END denotes the last token in the sentence (at the word ‘to’).

Wang et al. () suggest the
model uses the algorithm ‘Find the two names in the sentence, detect the
repeated name, and predict the non-repeated name’ to do this task. Specifically,
they discover several classes of heads in the model, each of which performs a
specific subtask of this overall algorithm. A simplified version of the circuit
involves the following three classes of heads and proceeds as follows:

- •
Duplicate token heads: these heads detect the repeated name in
the sentence (the S name) and output information about both its
position and identity to the residual stream(^14^1414We follow the conventions of Elhage et al. () when describing internals of transformer models. The residual stream at layer $k$ is the sum of the output of all layers up to $k-1$, and is the input into layer $k$.)
- •
S-Inhibition heads: these heads read the identity and position of
the S name from the residual stream, and output a signal to the effect
of ‘do not attend to this position / this token identity’ to the residual stream
- •
Name Mover heads: these are heads that attend to names in the
sentence. Because the signal from the S-Inhibition heads effectively removes the
S name from the attention of these heads, they read the identity of the
IO name from the input prompt, and copy it to the last token position
in the residual stream.

In reality, the circuit is more nuanced, with several other classes of heads
participating: previous token heads, induction heads (Olsson et al., ),
backup name mover heads, and negative name mover heads.
In particular, the
circuit exhibits *backup behavior* (McGrath et al., ) which poses
challenges for interpretability methods that intervene only on single model
components at a time. We refer the reader to Figure 7 for a
schematic of the full circuit, and to Wang et al. () for a
more complete discussion.

Figure: Figure 7: A reproduction of Figure 2 from Wang et al. (), showing the internal structure of the IOI circuit. Original caption: *The input tokens on the left are passed into the residual stream. Attention heads move information between residual streams: the query and output arrows show which residual streams they write to, and the key/value arrows show which residual streams they read from.*
Refer to caption: x9.png

### A.2 Additional details for Section 3

Why do we use mean ablation as a baseline for necessity of
reconstructions? Using the mean instead of simply a zero value is intended to
not take the model away from the task distribution. The information that is
constant across the task distribution – such as grammar and syntax – will
remain in the mean ablation, while task-specific information will be averaged
out. If the residual $\mathbf{a}-\widehat{\mathbf{a}}$ is not task-relevant,
we expect that the model will perform the task as well as with the mean
ablation; conversely, if the reconstructions $\widehat{\mathbf{a}}$ leave out
task-relevant information, we expect that the model will perform better with our
intervention than with the mean ablation.

Editing in multiple model locations at once. We do this by
pre-computing edited activations for each location, then patching all of them
into the model’s forward pass at once. This means that we are only editing
computational cross-sections of the model; however, applying the edits
sequentially as the model is run is prone to taking activations
off-distribution, since an edit will propagate to downstream activations, where
making the same edit again may be the wrong intervention; we have observed this
in practice, in the interaction of the name mover heads in layers 9 and 10 in
the IOI circuit.

Evaluating edits. Our edits are intended to only change the values of
specific attributes, and leave all other information in the activations
unchanged. A ground-truth baseline for the effect of an edit would be to take
the model’s activation on a *counterfactual* prompt: one which differs from
the original prompt only in the value of the attribute being edited, in the
precise way that the edit changes it. The existence of counterfactual prompts in
the support of $\mathcal{D}$ is not guaranteed in general, but it holds in our
IOI distribution.

Precision, recall, and the $F_{1}$-score.
Given a set of examples $S$ used for evaluation, a learned feature $f$ active on
a subset $F\subset S$ of examples, and a binary attribute of a prompt which is
true for the subset $A\subset S$, we define
$\operatorname{recall}(F,A)=\left|A\cap F\right|/\left|A\right|$ and
$\operatorname{precision}(F,A)=\left|A\cap F\right|/\left|F\right|$. Following
Bricken et al. (), we consider a feature meaningful for a given
property if it has both high recall and high precision for that property, and we
combine them into a single number using the F-score:

$$ $\displaystyle F_{1}(F,A)=\frac{2\operatorname{precision}(F,A)\operatorname{ recall}(F,A)}{\operatorname{precision}(F,A)+\operatorname{recall}(F,A)}.$ $$

An $F_{1}$-score of $\alpha$ guarantees that both precision and recall are at least
$\frac{\alpha}{2-\alpha}$. For example, when $\alpha=0.8$ (the value we use in
most evaluations), both precision and recall are at least $0.8/1.2\approx 0.67$.
Requiring a sufficiently high $F_{1}$ value is important in order to avoid
labeling a trivial feature as meaningful for attributes where $\left|A\right|$
is large, because then a feature active for all examples can have a high
$F_{1}$-score.

The $F_{1}$ score has some limitations in the context of our work:

- •
it does not take into account the magnitude of the feature activations;
for instance, a feature that is active for all examples in $S$ but only has high
activation values on the examples in $A$ may have a low $F_{1}$ score, even though
it is in some sense highly informative for the attribute $A$.
- •
it is a very conservative metric, in that it requires both high precision
and high recall to be high. For example, a feature with precision $0.5$ but
recall $0.02$ will have an $F_{1}$ score of $\sim 0.04$, heavily skewed towards the
lower of the two metrics, even though it is in some sense informative for the
attribute $A$.

We hope to address these limitations in future work.

### A.3 Additional details for Section 4

Why do we use fixed coefficients for our supervised activation
reconstructions?
Note that the formulation from Equation 2 is
*less expressive* than the reconstruction provided by an SAE (Equation
1), as it requires the coefficients of decoder vectors
to be fixed (similar to Tamkin et al. (), and unlike in the SAE, where
they are computed from the input via a linear function and a ReLU). This is a
reasonable assumption in settings where we expect the relevant features to
behave as binary, on/off switches (as opposed to having continuous degrees of
activation). The IOI task is an example of such a setting, as we expect that
there is no ‘degree’ to which a given name in the sentence is present or not, or
to which a given name is repeated or not. See also the discussion of ‘features
as directions’ vs ‘features as points’ in Tamkin et al. ().

Computing and evaluating supervised feature dictionaries. For each
parametrization and each method to compute feature dictionaries, we use 20,000 prompts sampled
from our IOI distribution (see Appendix A.4) to
compute feature dictionaries for the query, key, value, and attention output (i.e.,
attention-weighted values) of the relevant token positions of all 26 heads
identified in Wang et al. () (recall Figure
7). We use another sample of 5,000 prompts to validate the
quality of the feature dictionaries.

Cross-sections of the circuit. Based on the understanding of the IOI
circuit from Wang et al. (), we identify several
cross-sections of the computational graph of the IOI circuit where feature
editing is expected to have effects meaningful for the task:

- •
*outputs of (backup) name mover heads at END ((B)NM out)*: these activations
encode the IO name and write it to the END token of the residual
stream. We expect that editing the IO name in these activations will
directly affect the model’s prediction, while editing other attributes will not
have a significant effect.
- •
*queries+keys of (backup) name movers at END ((B)NM qk)*: the queries represent the
S name and Pos information, but they are mainly used as
*inhibitory* signals for the model, decreasing the attention to the
S token(^15^1515In addition, we later find that the queries of the L10H0 name mover head also represent the IO attribute, and serve an *inhibitory* role for it as well, decreasing the attention to the IO token.). The
keys represent information about the IO and S names: in
particular, the S information combines with the query to inhibit
attention to the S token.
We expect that editing the S and Pos attributes in *both*
the keys and queries will not significantly hurt model performance, because as a
result attention to the S token will again be inhibited. By contrast,
it is unclear what editing the IO name is expected to do, since its
role in the attention computation is not fully described in
Wang et al. ().
- •
*outputs of S-Inhibition heads at END (S-I out), values of
S-Inhibition heads at S2 (S-I v), and outputs of duplicate token and
induction heads at S2 (Ind+DT out)*: these activations transmit the
inhibitory signal to the name mover heads through the residual stream. We expect
that editing S and Pos in these activations will lower the
model’s logit difference by disrupting the inhibitory signal, while editing
IO will have no effect.

Evaluating necessity of feature reconstructions: When we intervene on the model by removing
reconstructions from activations in cross-sections of the circuit, model
performance on the IOI task (as measured by the logit difference) goes down from the clean value $\operatorname{logitdiff}_{\mathbf{clean}}$ to a lower value $\operatorname{logitdiff}_{\mathbf{intervention}}$. As
we describe in the main text, the ground-truth intervention for removing the
features from the activations is mean ablation of the corresponding
cross-section, which also results in a lower value of the logit difference, $\operatorname{logitdiff}_{\mathbf{mean\ ablation}}$. We want to measure the degree to which $\operatorname{logitdiff}_{\mathbf{intervention}}$ approximates $\operatorname{logitdiff}_{\mathbf{mean\ ablation}}$, in a way that normalizes for different values of $\operatorname{logitdiff}_{\mathbf{mean\ ablation}}$ across cross-sections of the circuit. We use the following metric to do this:

$$ $\displaystyle\operatorname{necessity\ score}=1-\frac{\left|\operatorname{ logitdiff}_{\mathbf{mean\ ablation}}-\operatorname{logitdiff}_{\mathbf{ intervention}}\right|}{\left|\operatorname{logitdiff}_{\mathbf{mean\ ablation} }-\operatorname{logitdiff}_{\mathbf{clean}}\right|}.$ $$

Evaluating accuracy of attribute edits. In our figures on attribute
editing (e.g., Figure 3), we report the proportion of
examples (in a test set not used to compute feature dictionaries) for which the
model’s prediction when intervening via a given edit equals the model prediction
when we intervene by the ground-truth editing intervention described in
3.3. This metric’s ideal value is
$1$, and its worst value is zero. In many cases, simply not intervening on the
model already achieves a nontrivial (and sometimes very high) value of this
score; this is why we also report the value in the absence of intervention.

### A.4 Dataset, Model and Evaluation Details for the IOI Task

We use GPT2-Small for the IOI task, with a dataset that spans 216 single-token names, 144 single-token objects and 75 single-token places, which are split $1:1$ across a training and test set.
Every example in the data distribution includes (i) an initial clause introducing the indirect object (IO, here ‘Mary’) and the subject (S, here ‘John’),
and (ii) a main clause that refers to the subject a second time.
Beyond that, the dataset varies in the two names, the initial clause content, and the main clause content.
Specifically, use three templates as shown below:

Then, [ ] and [ ] had a long and really crazy argument. Afterwards, [ ] said to

Then, [ ] and [ ] had lots of fun at the [place]. Afterwards, [ ] gave a [object] to

Then, [ ] and [ ] were working at the [place]. [ ] decided to give a [object] to

and we use the first two in training and the last in the test set. Thus, the test set relies on unseen templates, names, objects and places. We used fewer templates than the IOI paper Wang et al. () in order to simplify tokenization (so that the token positions of our names always align), but our results also hold with shifted templates like in the IOI paper.

On the test partition of this dataset, GPT2-Small achieves an accuracy of
$\approx 91\%$. The average difference of logits between the correct and
incorrect name is $\approx 3.3$, and the logit of the correct name is greater
than that of the incorrect name in $\approx 99\%$ of examples. Note that, while
the logit difference is closely related to the model’s correctness, it being
$>0$ does not imply that the model makes the correct prediction, because there
could be a third token with a greater logit than both names.

### A.5 Properties of mean feature dictionaries

Mean feature dictionaries enjoy several convenient properties:

- •
The vectors $\mathbf{u}_{iv}$ for an attribute $a_{i}$ do not depend on which
other attributes $a_{l}\neq a_{i}$ we have chosen to describe the prompt $p$ with.
- •
If an attribute $i$ is not linearly represented in the activations, the
mean code features $\mathbf{v}_{iv}\to 0$ in the limit of infinite data (see
below). In particular, this also holds if the attribute is not represented
*at all* in the activations.

This suggests that mean feature dictionaries are robust to the inclusion of irrelevant or
non(-linearly)-represented attributes, which is a desirable property in real
settings where we may not know the exact attributes present in each activation.
However, mean feature dictionaries are *not* robust to the inclusion of redundant
attributes, as the lack of interaction between the attributes means that
redundant attributes cannot ‘coordinate’ to reduce the reconstruction error
$\left\|\mathbf{a}-\widehat{\mathbf{a}}\right\|_{2}^{2}$.

#### A.5.1 Mean features are zero for non-linearly-represented attributes.

Suppose we have a random vector $\mathbf{x}$ for a $k$-way classification task
with one-hot labels $\mathbf{z}\in\mathcal{Z}=\{\mathbf{z}\in\{0,1\}^{k}\text{ s.t. }\left\|\mathbf
{z}\right\|_{1}=1\}$. In Section 3 of Belrose et al. (), it is
shown that the following are equivalent:

- •
the expected cross-entropy loss of a linear predictor
$\widehat{\mathbf{z}}=\mathbf{w}^{\top}\mathbf{x}+\mathbf{b}$ for $\mathbf{z}$
is minimized at a *constant* linear predictor. In other words, the optimal
logistic regression classifier (in the limit of infinite data) is no better than
the optimal constant predictor (which, at best, always predicts the majority
class).
- •
the class-conditional mean vectors
$\mathbb{E}\left[\mathbf{x}|\mathbf{z}=e_{i}\right]$ are all equal to the overall mean
$\mathbb{E}\left[\mathbf{x}\right]$ of the data.

If we translate this to the context of mean feature dictionaries from Subsection
4, we have that logistic regression for the value
of an attribute $a_{i}$ will degenerate to the majority class predictor if and
only if the mean feature dictionaries for all values of this attribute are zero. In the finite
data regime, this gives us some theoretical grounds to expect that the mean
feature dictionaries will be significantly away from zero if and only if the
attribute’s values can be non-trivially recovered by a (logistic) linear probe.
As a special case, if an attribute is not represented in the data at all, we
expect the mean feature dictionaries for this attribute to be zero.

### A.6 Definition and Properties of MSE Feature Dictionaries

MSE feature dictionaries compute $\mathbf{u}_{a_{i}(\cdot)=v}$ by directly minimizing
the $\ell_{2}$ reconstruction error over the centered activations:

$$ $\{\mathbf{u}_{a_{i}(\cdot)=v}\}_{i\in I,v\in S_{i}}=\operatorname*{arg\,min}_{ \mathbf{u}_{a_{i}(\cdot)=v}}\frac{1}{N}\sum_{k=1}^{N}\left\|\left(\mathbf{a}(p _{k})-\overline{\mathbf{a}}\right)-\sum_{i\in I}\mathbf{u}_{a_{i}(\cdot)=a_{i} (p_{k})}\right\|_{2}^{2}$ (3) $$

This objective is convex, and is equivalent to a least-squares regression
problem; in fact, the optimal solutions take a form very similar to the mean
feature dictionaries (see below). Furthermore, this objective closely mimics the
SAE objective: here, the sparsity is hard-coded, leaving only the $\ell_{2}$
objective.

We next discuss some properties of the MSE feature dictionaries. For brevity, in
the remainder of this section we write $\mathbf{u}_{iv}$ instead of
$\mathbf{u}_{a_{i}(\cdot)=v}$.

#### A.6.1 MSE feature dictionaries as a multivariate least-squares regression problem.

Let $S=\sum_{i=1}^{N_{A}}\left|S_{i}\right|$ be the total number of possible values for
all attributes. For each attribute $i$, consider the characteristic matrix
$C_{i}\in\mathbb{R}^{N\times S_{i}}$ of the dataset for this attribute, where

$$ $\displaystyle C_{kj}=\begin{cases}1&\text{if }a_{i}(p^{(k)})=v_{j}\\ 0&\text{otherwise}\end{cases}$ $$

for some ordering $(v_{1},\ldots,v_{\left|S_{i}\right|})$ of the values in $S_{i}$,
and let $C=\begin{bmatrix}C_{1}&C_{2}&\cdots&C_{N_{A}}\end{bmatrix}\in\mathbb{R}^{N
\times S}$ be the concatenation of all characteristic matrices.
Also, let $A\in\mathbb{R}^{N\times d}$ be the matrix of activations with rows
$\mathbf{a}^{(k)}$. Then the objective function for the MSE feature dictionaries can be written
as the multivariate least-squares regression problem

$$ $\displaystyle\min_{U\in\mathbb{R}^{S\times d}}\frac{1}{N}\left\|A-CU\right\|_{ F}^{2}$ $$

where the rows of $U$ are the vectors $\mathbf{u}_{iv}$ across all $i$ and
$v\in S_{i}$, with the optimal solution given by

$$ $U^{*}=\left(C^{\top}C\right)^{+}C^{\top}A$ (4) $$

#### A.6.2 MSE feature dictionaries as averaging over examples.

Using the special structure of the
objective, we can also derive some information about the optimal solutions
$\mathbf{u}_{iv}^{*}$. Namely, at optimality we should not be able to decrease
the value of the objective by changing a given $\mathbf{u}_{iv}^{*}$ away from
its optimal value. The terms containing $\mathbf{u}_{iv}^{*}$ in the objective
are

$$ $\displaystyle\frac{1}{N}\sum_{k\in P_{iv}}\left\|\mathbf{a}^{(k)}-\sum_{l\neq i }\mathbf{u}_{lv_{l}^{(k)}}^{*}-\mathbf{u}_{iv}^{*}\right\|_{2}^{2}$ $\displaystyle=\frac{1}{N}\sum_{k\in P_{iv}}\left\|\left(\mathbf{a}^{(k)}-\sum_ {l\neq i}\mathbf{u}_{lv_{l}^{(k)}}^{*}\right)-\mathbf{u}_{iv}^{*}\right\|_{2}^ {2}$ $\displaystyle=\frac{1}{N}\sum_{k\in P_{iv}}\left\|\overline{\mathbf{a}}^{(k)}- \mathbf{u}_{iv}^{*}\right\|_{2}^{2}$ $$

where recall that $P_{iv}=\{k\,|\,a_{i}(p^{(k)})=v\}$, and
$\overline{\mathbf{a}}^{(k)}$ is the residual of $\mathbf{a}^{(k)}$ after
subtracting the reconstruction using all other attributes $l\neq i$. Since this
value cannot be decreased by changing $\mathbf{u}_{iv}^{*}$, we have that it
equals the minimizer of this term (holding $\overline{a}^{(k)}$ fixed). In other
words, if we define

$$ $\displaystyle f\left(\mathbf{u}\right)=\frac{1}{N}\sum_{k\in P_{iv}}\left\| \overline{\mathbf{a}}^{(k)}-\mathbf{u}\right\|_{2}^{2}$ $$

we have that $\mathbf{u}_{iv}^{*}=\operatorname*{arg\,min}_{\mathbf{u}}f\left(\mathbf{u}\right)$. Since
$f$ is a sum of convex functions, it is itself convex, and so the first-order
optimality condition is also sufficient for optimality. We have

$$ $\displaystyle\nabla f\left(\mathbf{u}\right)\propto\sum_{k\in P_{iv}}\left( \overline{\mathbf{a}}^{(k)}-\mathbf{u}\right)\propto\frac{1}{\left|P_{iv} \right|}\sum_{k\in P_{iv}}\overline{\mathbf{a}}^{(k)}-\mathbf{u}$ $$

and so

$$ $\mathbf{u}_{iv}^{*}=\frac{1}{\left|P_{iv}\right|}\sum_{k\in P_{iv}}\overline{ \mathbf{a}}^{(k)}$ (5) $$

Note that this is very similar to the definition of mean feature dictionaries,
but also importantly different, because the optimal $\mathbf{u}_{iv}^{*}$ depends
on the optimal values of the feature dictionaries for the other attributes.

#### A.6.3 MSE feature dictionaries with independent attributes.

Finally, we can prove that,
under certain conditions, attributes for which $\mathbb{E}\left[\mathbf{a}|a_{i}(p)=v_{i}\right]=\mathbb{E}\left[\mathbf{a}\right]$, i.e. the conditional
mean of activations over values of the attribute is the same as the overall mean
(assuming both means exist), will have (approximately) constant MSE feature dictionaries
$\mathbf{u}_{iv}=\mathbf{u}_{i}\forall v$. This is a counterpart to the result
from Appendix A.5 for MSE feature dictionaries:

###### Lemma A.1 .

Suppose that all conditional means $\mathbb{E}_{p\sim\mathcal{D}}\left[\mathbf{a}|a_{i}(p)=v\right]$ exist for all $i,v\in S_{i}$. Let $a_{i}$ be an attribute such its values
appear independently from the values of all other attributes, i.e.

$$ $\displaystyle\mathbb{P}_{p\sim\mathcal{D}}\left[a_{i}(p)=v_{i},a_{l}(p)=v_{l} \right]=\mathbb{P}_{p\sim\mathcal{D}}\left[a_{i}(p)=v_{i}\right]\mathbb{P}_{p \sim\mathcal{D}}\left[a_{l}(p)=v_{l}\right]\quad\forall v_{i}\in S_{i},v_{l} \in S_{l},l\neq i$ $$

Then, in the limit of infinite training data, the conditional means
$\mathbb{E}\left[\mathbf{a}|a_{i}(p)=v\right]$ are all equal to the overall
mean $\mathbb{E}\left[\mathbf{a}\right]$ if and only if the optimal MSE feature dictionaries
$\mathbf{u}_{iv}^{*}$ for this attribute are constant with respect to the value
$v$ of the attribute, i.e. $\mathbf{u}_{iv}^{*}=\mathbf{u}_{i}$ for all $v\in S_{i}$.

###### Proof.

From Equation
5, we have

$$ $\displaystyle\mathbf{u}_{iv}^{*}$ $\displaystyle=\frac{1}{\left|P_{iv}\right|}\sum_{k\in P_{iv}}\overline{\mathbf {a}}^{(k)}=\frac{1}{\left|P_{iv}\right|}\sum_{k\in P_{iv}}\left(\mathbf{a}^{(k )}-\sum_{l\neq i}\mathbf{u}_{lv_{l}^{(k)}}^{*}\right)$ $\displaystyle=\frac{1}{\left|P_{iv}\right|}\sum_{k\in P_{iv}}\mathbf{a}^{(k)}- \frac{1}{\left|P_{iv}\right|}\sum_{k\in P_{iv}}\sum_{l\neq i}\mathbf{u}_{lv_{l }^{(k)}}^{*}$ $$

The first term converges to $\mathbb{E}\left[\mathbf{a}|a_{i}(p)=v\right]$.
The second term is a sum of terms of the form

$$ $\frac{1}{\left|P_{iv}\right|}\sum_{k\in P_{iv}}\mathbf{u}_{lv_{l}^{(k)}}^{*}= \frac{1}{\left|P_{iv}\right|}\sum_{v_{l}\in S_{l}}\mathbf{u}_{lv_{l}}^{*}\left |\{k\text{ s.t. }a_{i}(p_{k})=v,a_{l}(p_{k})=v_{l}\}\right|$ (6) $$

for $l\neq i$. Since we are assuming $a_{i}$ is uncorrelated with $a_{l}$, in the
limit of the size $N$ of the dataset $\mathbf{a}^{(1)},\mathbf{a}^{(2)},\ldots,\mathbf{a}^{(N)}$ going to infinity, $\left|\{k\text{ s.t. }a_{i}(p_{k})=v,a_{l}(p_{k})=v_{l}\}\right|$ will approach $\left|P_{iv}\right|\mathbb{E}_{p\sim\mathcal{D}}\left[\mathbf{1}_{a_{l}(p)=v_{
l}}\right]$. Moreover, note that in the
closed-form solution $U^{*}=\left(C^{\top}C\right)^{+}C^{\top}A=\left(\frac{C^{\top}C}{N}\right)^{+}
\frac{C^{\top}}{N}A$ from Equation 4, the
matrix $\frac{1}{N}C^{T}C$ converges to some limit $\Sigma\in\mathbb{R}^{S\times S}$ as $N\to\infty$, and the matrix $\frac{1}{N}C^{\top}A$ similarly converges to
some limit $M\in\mathbb{R}^{S\times d}$ by the assumption that all conditional
means for all attributes exist. Thus, the optimal feature dictionaries $\mathbf{u}_{iv}^{*}$ will
also converge as $N\to\infty$. So we see that the sum in Equation
6 will converge to a value that is independent of the
value $v$ for the attribute $a_{i}$.

Thus, if the conditional means $\mathbb{E}\left[\mathbf{a}|a_{i}(p)=v\right]$
are all equal to the overall mean $\mathbb{E}\left[\mathbf{a}\right]$, we get
that $\mathbf{u}_{iv}^{*}$ is independent of $v$; conversely, if
$\mathbf{u}_{iv}^{*}$ is independent of $v$, we get that the conditional means are
all equal to the overall mean. This completes the proof.

∎

###### Lemma A.1 .

###### Proof.

### A.7 Feature-level mechanistic analyses for Section 4

Since each activation is approximated as the sum of several vectors from a
finite set, it becomes possible to decompose the model’s internal operations in
terms of elementary interactions between the learned vectors themselves. In the
current paper, we are particularly interested in attention heads, as they are
the building blocks of the IOI circuit. We consider the following subspace-level
analyses:

- •
Attention scores: The attention mechanism is considered to be a crucial
reason for the success of LLMs (Vaswani et al., ), but a
subspace-level understanding of it is mostly lacking (but see
Lieberum et al. ()). How do the features in the keys and queries of
attention heads combine to produce the attention scores? Which feature pairs are
most important for the head’s behavior?
- •
Head composition: If we are to understand a circuit on the
subspace level, we need to develop a subspace-level account of how the outputs
of one attention head compose with the queries, keys and values of a downstream
head in the circuit. Each head adds its output to the residual stream, and
downstream heads’ query/key/value matrices read from the residual stream. We can
thus examine the contribution, or *direct effect*, of a head’s output to
another head’s queries/keys/values. We can decompose this direct effect in terms
of the features of the source head to calculate contributions of each feature to
the direct effect.

Implementation details for these analyses follow.

Attention scores. Given feature dictionary reconstructions for the keys
and queries of an attention head at certain positions

$$ $\displaystyle\mathbf{k}\approx\sum_{i\in I}\mathbf{u}_{a_{i}(\cdot)=a_{i}(p)}, \quad\mathbf{q}\approx\sum_{i\in I}\mathbf{v}_{a_{i}(\cdot)=a_{i}(p)}$ $$

we can decompose the attention scores as a sum of pairwise dot products between
the dictionary features

$$ $\displaystyle\mathbf{q}^{T}\mathbf{k}/\sqrt{d_{head}}\approx\sum_{i,j\in I} \mathbf{v}_{a_{i}(\cdot)=a_{i}(p)}^{T}\mathbf{u}_{a_{j}(\cdot)=a_{j}(p)}/\sqrt {d_{head}}$ $$

where $d_{head}$ is the dimension of the attention head. This allows us to
examine which feature combinations are most important for the head’s attention
according to the learned dictionaries. Variants of this decomposition can also
be applied to e.g. the difference in attention scores at two different token
positions.

Head composition. Following the terminology and results from Elhage et al. (), the
residual stream $\mathbf{r}_{l,t}$ of a transformer at a given layer $l$ and
token position $t$ is the sum of the input embedding and the outputs of all
earlier MLP and attention layers at this position. The residual stream is in
turn the input to the next attention layer; so, for example, we can write the
query vector for the $h$-th head at layer $l$ and token $t$ as

$$ $\displaystyle\mathbf{q}_{l,t,h}=W_{l,h}^{Q}\operatorname{LayerNorm}\left( \mathbf{r}_{l,t}\right)=W_{l,h}^{Q}\operatorname{LayerNorm}\left(\overline{ \mathbf{r}}_{l,t}+W^{O}_{l^{\prime},h^{\prime}}\mathbf{z}_{l^{\prime},t,h^{ \prime}}\right)$ $$

where $\mathbf{z}_{l^{\prime},t,h^{\prime}}$ is the attention-weighted sum of values of the $h^{\prime}$-th
head at layer $l^{\prime}<l$ and token $t$, $\overline{\mathbf{r}}_{l,t}$ is the remainder
of the residual stream after removing the contribution of this head, and LayerNorm is
the model’s layer normalization operation (Ba et al., ) before the
attention block in layer $l$. By treating the layer normalization as an approximately linear
operation (taking
the scale from an average over the dataset(^16^1616This is justified by the empirical observation that the layer normalization scales across the dataset are well concentrated around their mean.)), we can derive an approximation of
the *(counterfactual) direct effect* of the output of the $h^{\prime}$-th head at
layer $l^{\prime}$ and token $t$ on the query vector of the $h$-th head at layer $l$ and
token $t$:

$$ $\displaystyle\mathbf{q}_{l,t,h}\approx W_{l,h}^{Q}\left(\gamma_{l}\odot\frac{ \mathbf{r}_{l,t}-\mu_{l,t}}{\sqrt{\widehat{\sigma}_{l}^{2}+{\epsilon}}}+\beta_ {l}\right)$ $$

where $\gamma_{l},\beta_{l}$ are the learned scale and shift parameters of the LN
operation, $\mu_{l,t}$ is the average of the vector $\mathbf{r}_{l,t}$ over its
coordinates, and $\widehat{\sigma}_{l}$ is an average over the dataset of the
standard deviation of the residual stream at this position. Alternatively, we
can use the exact layernorm scale from the forward pass over a large sample to
compute the statistics of the exact direct effect over observed data.

With either approach, we obtain a decomposition

$$ $\displaystyle\mathbf{q}_{l,t,h}\approx\sum_{l^{\prime}<l,h^{\prime}}\mathbf{u} _{t,(l^{\prime},h^{\prime})\to(l,h)}+\overline{r}_{l,t}$ $$

of direct contributions from the outputs of earlier heads at this position, plus
some residual terms $\overline{r}_{l,t}$ (which are the contributions of all
previous MLP layers and the input embedding to the query vector). We can then
further decompose $\mathbf{u}_{t,(l^{\prime},h^{\prime})\to(l,h)}$ by replacing it with its
reconstruction from our feature dictionary.

For either way to treat the layer normalization, we can use the learned feature
dictionaries for the outputs, keys, queries and values of attention heads in a
number of ways to decompose the direct effect further. Here, we consider head-and-feature attribution: fixing the head $(l,h)$, we can vary the head $(l^{\prime},h^{\prime})$ and break down the direct effects (projected on the query vector) by
feature.

Results. An interesting location to examine is the attention of
the name mover heads from END to the IO and S1 positions, where (according to
the analysis in Wang et al. ()) the signal from the
S-Inhibition heads effectively removes the S name from the attention of
these heads.

We show the results for the head L10H0 in Figure
8. Crucially, we observe that most interactions
are tightly clustered around zero, which suggests that these feature
dictionaries provide a sparse and interpretable account of the attention
mechanism. The only significantly nonzero interactions are (1) between the S
features in the query and the key at the S1 position; (2) between the Pos
features in the query and the key at both positions; and (3) between the IO
features in the query and the key at the IO position. The first two interactions
are expected given the findings of Wang et al. (). More
interesting is the third interaction, which is negative, suggesting that the
L10H0 head inhibits both the S and IO name tokens, and
effectively relies only on the Pos attribute to distinguish between the
two names. Notably, this is in contrast with the other two name movers L9H6 and
L9H9 (for which analogous plots are shown in Figure
16 and
17), where the inhibition of the IO
attribute is absent. We found that using other methods to compute feature
dictionaries result in less sparse and interpretable patterns.

Figure: Figure 8: Decomposing the attention scores of the name mover head L10H0 from END to the S1 (left) and IO (right) positions. The y-axis ranges over the combinations of features from the query (first element) and the key (second element). The boxplots show the distribution of dot products between the corresponding feature vectors. The interaction between the bias terms (i.e., the means of the respective queries/keys) provides a sense of the scale of the effects.
Refer to caption: x10.png

We further investigated the queries of the L10H0 head, by looking at which
features from upstream head outputs at the END token have a large direct effect
on these queries. Following the methodology detailed in Appendix
A.7, we plot the direct effect from the
outputs of the S-Inhibition heads, as well as the two name mover heads L9H6 and
L9H9 in layer 9 in Figure 9. We find that the
S-Inhibition heads’ IO features have no significant contribution to the
queries, but the IO features from the two name mover heads in layer 9
have a significant direct effect (aligned with the overall centered query
vector). This suggests that, having already computed a representation of the
IO attribute, these heads transmit it to the next layer, where it gets
picked up by the L10H0 head’s query. Conversely, the S-Inhibition heads
contribute significantly with their Pos and S features,
whereas the name mover heads in layer 9 do not.

Figure: Figure 9: Direct effect of supervised features in the output of S-Inhibition heads, and Name Mover heads in layer 9, on the queries of the L10H0 name mover head at the END token.
Refer to caption: x12.png

### A.8 Alternative parametrizations for the IOI task

We mostly experimented with two possible parametrizations of prompts via
attributes:

- •
independent parametrization: we use the three independently
varying attributes – S, IO and Pos – to describe
each prompt. This is the parametrization used in the main text.
- •
coupled parametrization: we couple position with name, and use
the two attributes (S, Pos) and (IO, Pos) to
describe each prompt. This parametrization is more expressive than the
independent one, as it allows for different features for the same name depending
on whether it comes first or second in the sentence. At the same time, the
coupled parametrization can express the independent one as a special case
(Appendix A.9).

We find that these parametrizations arrive at highly similar activation
reconstructions $\widehat{\mathbf{a}}$. In fact, we find an even stronger
property: the coupled parametrization essentially simulates the features in the
independent one; details are given in Appendix
A.9.1.

Finally, we note that the fact that we find parametrizations that result in good
approximation is not trivial. Not every ‘natural-seeming’ parametrization will
lead to a good approximation of model behavior; we show an example of this with
a ‘names’ parametrization in Appendix Figure
15, where we instead use an
attribute for the 1st, 2nd and 3rd name in the sentence.

##### What about other parametrizations?

Note that there are combinatorially many possible ways to pick attributes to
disentangle the activations into, and a priori any specific choice is arbitrary.
We justify our choice of parametrizations in several ways: (1) they pass our
tests for model approximation, control and interpretability given later in this
section; (2) they correspond to the internal states of the IOI circuit
identified in Wang et al. (); (3) we experimented and/or
considered other parametrizations, but found they either perform the same or
worse on our tests. In Appendix A.10, we provide a
more detailed discussion of different possible parametrizations in the IOI task
and their relative strengths and weaknesses.

### A.9 Comparing the coupled and independent parametrizations

#### A.9.1 The coupled parametrization captures the independent one

Idealized model. We first
note that the coupled parametrization can express all reconstructions
expressible by the independent parametrization. Suppose we have an IOI
distribution using a set of available names $S_{names}$, and let
$\mathbf{pos}_{ABB},\mathbf{pos}_{BAB},\{\mathbf{io}_{a}\}_{a\in S_{names}},\{
\mathbf{s}_{a}\}_{a\in S_{names}}$ be feature dictionaries for the independent parametrization
at some model activation. Then, we can define the following feature dictionaries for the
coupled parametrization:

$$ $\displaystyle\mathbf{io}_{a,ABB}$ $\displaystyle=\mathbf{io}_{a}+\frac{1}{2}\mathbf{pos}_{ABB},\quad\mathbf{io}_{ a,BAB}=\mathbf{io}_{a}+\frac{1}{2}\mathbf{pos}_{BAB},\quad$ $\displaystyle\mathbf{s}_{a,ABB}$ $\displaystyle=\mathbf{s}_{a}+\frac{1}{2}\mathbf{pos}_{ABB},\quad\mathbf{s}_{a, BAB}=\mathbf{s}_{a}+\frac{1}{2}\mathbf{pos}_{BAB}$ $$

Then for a prompt $p$ of the form ABB (the BAB case is analogous), with the
IO name being $a$ and the S name being $b$, we have that the
reconstruction of an activation $\mathbf{a}$ using the independent
parametrization is

$$ $\displaystyle\widehat{\mathbf{a}}_{independent}=\mathbf{io}_{a}+\mathbf{s}_{b} +\mathbf{pos}_{ABB}$ $$

and the reconstruction using our coupled parametrization is

$$ $\displaystyle\widehat{\mathbf{a}}_{coupled}$ $\displaystyle=\mathbf{io}_{a,ABB}+\mathbf{s}_{b,ABB}=\mathbf{io}_{a}+\mathbf{s }_{b}+\frac{1}{2}\mathbf{pos}_{ABB}+\frac{1}{2}\mathbf{pos}_{ABB}$ $\displaystyle=\mathbf{io}_{a}+\mathbf{s}_{b}+\mathbf{pos}_{ABB}=\widehat{ \mathbf{a}}_{independent}$ $$

Empirical evaluation.
We evaluated whether this occurs empirically with the MSE features for the two
parametrizations. First, we plot the fraction of variance explained by the
reconstructions using the independent parametrization in the reconstructions
using the coupled parametrization. We find very high agreement (Figure
10); results in the other direction
are almost identical and are not shown here for brevity. Next, we check if the
coupled parametrization essentially simulates the independent one as described
analytically above. We do this by measuring the cosine similarity between the
vector $\mathbf{pos}_{ABB}-\mathbf{pos}_{BAB}$ from the independent
parametrization and vectors $\mathbf{io}_{a,ABB}-\mathbf{io}_{a,BAB}$ and
$\mathbf{s}_{a,ABB}-\mathbf{s}_{a,BAB}$ from the coupled parametrization. In
our idealized simulation of the independent parametrization using the coupled
one, these values would be exactly $1$ for all names. We find that in all
circuit locations that represent both IO and Pos, the
similarities w.r.t the IO differences are significant; similarly, in all
circuit locations that represent both S and Pos, the
similarities w.r.t the S differences are significant (Figure
11). For reference, in a space
of this dimensionality (64), the expected magnitude of the cosine similarity
between two random vectors is $1/8$.

Figure: Figure 10: Variance explained by the reconstructions using the independent parametrization, with respect to the reconstructions using the coupled parametrization, averaged over combinations of class of heads in the circuit and activation types.
Refer to caption: x13.png

Figure: Figure 11: Cosine similarity between the vector $\mathbf{pos}_{ABB}-\mathbf{pos}_{BAB}$ from the independent parametrization and vectors $\mathbf{io}_{a,ABB}-\mathbf{io}_{a,BAB}$ and $\mathbf{s}_{a,ABB}-\mathbf{s}_{a,BAB}$ from the coupled parametrization, averaged over several classes of circuit locations. When evaluating similarity for the $\mathbf{io}_{a,ABB}-\mathbf{io}_{BAB}$ vectors, we only include locations where both the IO and Pos attributes are represented; and similarly for the $\mathbf{s}$-vectors.
Refer to caption: x14.png

#### A.9.2 Other parametrizations expressible by the coupled parametrization.

Consider the parametrization with attributes (S, Pos) and
IO. Again, let $\{\textbf{io}_{a}\}_{a\in S_{names}},\{\textbf{s}_{a,ABB}\}_{a\in S_{names}},
\{\textbf{s}_{a,BAB}\}_{a\in S_{names}}$ be feature dictionaries for this parametrization at some model activation.
Then, we can define the following feature dictionaries for the coupled parametrization:

$$ $\displaystyle\mathbf{io}^{\prime}_{a,ABB}=\mathbf{io}^{\prime}_{a,BAB}=\mathbf {io}_{a},\quad\mathbf{s}^{\prime}_{a,ABB}=\mathbf{s}_{a,ABB},\quad\mathbf{s}^{ \prime}_{a,BAB}=\mathbf{s}_{a,BAB}$ $$

Then for a prompt $p$ of the form ABB (the BAB case is analogous), with the
IO name being $a$ and the S name being $b$, we have that the
reconstruction of an activation $\mathbf{a}$ using the (S,
Pos) + IO parametrization is

$$ $\displaystyle\widehat{\mathbf{a}}_{(\textbf{S},\textbf{Pos})+\textbf{IO}}= \mathbf{io}_{a}+\mathbf{s}_{b,ABB}=\mathbf{io}^{\prime}_{a,ABB}+\mathbf{s}^{ \prime}_{b,ABB}=\widehat{\mathbf{a}}_{coupled}$ $$

and so the coupled parametrization can express all reconstructions expressible
by the (S, Pos) + IO parametrization. Analogously, it
can express all reconstructions expressible by the (IO, Pos) +
S parametrization.

#### A.9.3 Editing methodology with the coupled parametrization.

For each activation, we may choose to edit one or several of the IO,
S and Pos properties of the prompt. With the independent
parametrization, this is straightforward, since the attributes match these
properties. With the coupled parametrization, suppose we are given a prompt of
the form ABB (the BAB case is analosous) with the IO name being $a$ and
the S name being $b$. Given an activation $\mathbf{a}$ with
corresponding reconstruction under the coupled parametrization

$$ $\displaystyle\widehat{\mathbf{a}}=\mathbf{io}_{a,ABB}+\mathbf{s}_{b,BAB}$ $$

we can perform edits as follows:

- •
to change the IO name from $a$ to $a^{\prime}$: $\mathbf{a}_{edit}=\mathbf{a}-\mathbf{io}_{a,ABB}+\mathbf{io}_{a^{\prime},ABB}$
- •
to change the S name from $b$ to $b^{\prime}$: $\mathbf{a}_{edit}=\mathbf{a}-\mathbf{s}_{b,ABB}+\mathbf{s}_{b^{\prime},ABB}$
- •
to change the Pos property from ABB to BAB: $\mathbf{a}_{edit}=\mathbf{a}-\mathbf{io}_{a,ABB}-\mathbf{s}_{b,ABB}+\mathbf{io
}_{a,BAB}+\mathbf{s}_{b,BAB}$

### A.10 On Possible Feature Dictionaries for the IOI Task

In this section, we compare the properties of several *a priori* possible
ways in which the activations of the IOI circuit could be disentangled via
sparse feature dictionaries. The main goal is to illustrate that different
feature dictionaries can have similar usefulness in terms of model control and
interpretability, even if they fail natural tests that directly look for similar
features in the two dictionaries. This motivates evaluations that are agnostic
to the specific features in a dictionary, as long as the features can
parsimoniously disentangle the model’s internal computations.

The independent parametrization from the main text. It
is worth explicitly describing the properties of the supervised feature
decomposition we constructed in Section
LABEL:subsection:manual-ioi-implementation, which uses the IO,
S and Pos attributes to describe the prompts; it serves as an
idealized example against which to compare other possible decompositions. In
this decomposition, we can approximate an activation $\mathbf{a}$ for a prompt
$p$ where the IO name is $a$ and the S name is $b$, with the
IO name appearing first, as follows:

$$ $\displaystyle\mathbf{a}\approx\mathbf{io}_{a}+\mathbf{s}_{b}+\mathbf{pos}_{ABB}$ $$

where the vector $\mathbf{io}_{a}$ is the feature for the IO name $a$,
the vector $\mathbf{s}_{b}$ is the feature for the S name $b$, and the
vector $\mathbf{pos}_{ABB}$ is the feature for the Pos attribute when
the IO name appears first in the sentence (and analogously for
$\mathbf{pos}_{BAB}$).

Imagine now that we are given an ‘unlabeled’ feature dictionary that corresponds
to this decomposition (i.e., we don’t know which attribute each feature
corresponds to). We want to evaluate the usefulness of this decomposition for
model control and interpretability relative to the ‘human-legible’ attributes
IO, S and Pos. This will be tautologically successful:

- •
the reconstructions are a faithful and complete representation of the
model’s internal computations;
- •
the dictionary can (by definition) express edits to the IO,
S and Pos attributes very efficiently, as we only need to
change a single feature vector to change the corresponding attribute’s value.
- •
the features are fairly interpretable: we can understand the meaning of
each feature in terms of the attribute it represents.
- •
moreover, using metrics such as recall and precision (following the
evaluation methodology of Bricken et al. ()) will readily
surface the features that are most important for each attribute.

Using per-gender vectors to describe names. Another possible
decomposition is

$$ $\displaystyle\mathbf{a}\approx\mathbf{io}_{a}+\mathbf{io_gender}_{g(a)}+ \mathbf{s}_{b}+\mathbf{s_gender}_{g(b)}+\mathbf{pos}_{ABB}$ $$

where $g:\mathbf{Names}\to\{M,F\}$ is some labeling function that roughly
classifies names according to how they are typically gendered(^17^1717We experimented with this decomposition in our supervised framework, but did not find it to confer additional benefits for the purposes of our tests.).
Here, we hypothesize that the model may use features of high norm that sort
names into genders (which may be useful to the model for various reasons), and
then add a small-norm per-name correction to obtain a name-specific representation.
In particular we imagine that $\mathbf{io}_{a}+\mathbf{io_gender}_{g(a)}$ in
this representation would correspond to $\mathbf{io}_{a}$ in the supervised
decomposition, and similarly for the S name.

- •
This decomposition is also fairly sparse and interpretable, and it can
express edits almost as parsimoniously as the supervised decomposition (we only
need to change *two* feature vectors to edit a name, and one to edit
Pos).
- •
Moreover, metrics such as recall and precision will pick up on the
per-name features;
- •
If we compare the prompts for which a feature activates for this
dictionary and our ‘independent parametrization’ feature dictionary (discussed
above), we will easily discover the per-name features that correspond to the
IO and S names.
- •
However, if we instead directly use cosine similarity to the
supervised features $\mathbf{io}_{a},\mathbf{s}_{b}$ as a metric, we may be misled,
because if the per-gender features have sufficiently higher norm than the
per-name corrections, the cosine similarity may be low.

Features for small, overlapping subsets of names. Going further, we can
imagine a decomposition where we have features that correspond to pairs of
names, such that each name is in exactly two pairs (this can be achieved by
partitioning all names into pairs along a cycle). We can express a name as a sum
of the features for the two pairs it is in, with some superposition. Note that
more sophisticated constructions with more features per name / more names per
feature are possible by e.g. picking subsets at random or using expander graphs
(Hoory et al., ), as they will ‘spread out’ the superposition more
evenly.

- •
This decomposition is *somewhat* sparse and interpretable, and can
likely be used for feature editing in a reasonable way, as long as the sets of
features associated with each name are not too large. Even though we would need
to change several feature vectors to edit a name, there should also be a fair
amount of disentanglement so that we don’t also need to throw away all the
features active in an example to change a single attribute.
- •
However, comparing our supervised decomposition against this one using
geometric metrics such as cosine similarity may be misleading, because while a
sum of a few feature vectors associated with the same name may point in the same
direction as our supervised feature, any individual feature may not.
- •
Furthermore, it also has significantly reduced precision for the
features, because each of the few features associated with a name will also
activate for several other names. This can make directly looking for features
whose activation patterns resemble the ones in our supervised decomposition
misleading.

Our experiments suggest that both task-specific and full-distribution SAEs
trained on IOI circuit activations learn a decomposition resembling this
abstract construction more than any other decomposition discussed here.

Overfitting dictionaries. Finally, a worst-case decomposition would be
to have a single feature for each possible set of values of the S,
IO and Pos attributes.

- •
This decomposition is not interpretable, and it is not editable in any
non-trivial way: to change a single attribute, the entire decomposition must be
replaced;
- •
Features of this form will have maximum precision, but very low
recall for the attributes.

### A.11 Details for training Sparse Autoencoders

Task SAEs.
We followed the methodology of Bricken et al. (), with the
exception that our neuron re-initialization method is not as sophisticated as
theirs: we simply re-initialize the encoder bias, encoder weights and decoder
weights for the dead neurons every 500 training epochs.

Training SAEs on the IOI distribution alone allows us to do a more extensive
hyperparameter search. Importantly, we normalized SAE inputs across attention
heads so that activations have an $\ell_{2}$ norm of $1$ on average in order to
make it easier for the same set of hyperparameters to work well across different
heads. In our main experiments, we use SAEs that were trained with a $16\times$
hidden expansion factor, (effective) $\ell_{1}$ regularization coefficient in
$(0.01,0.05,0.1,0.3)$, batch size of $1024$, and learning rate of $0.001$.

We evaluated two key test-set metrics across training epochs: the average number
of active features per example (i.e. the average $\ell_{0}$ norm of activations),
and the fraction of the logit difference recovered when using the SAE
reconstructions at the given model location instead of the original activations,
scaled against a mean-ablation baseline (which is more stringent than the
zero-ablation baseline employed in most other work). We chose the regularization
coefficient and training checkpoint for each node that provided a good trade-off
between these two metrics; in particular, for almost all nodes, we recover logit
difference to within $20\%$ with respect to the mean-ablation baseline, and
there are $<25$ active features per example. We provide the results of this
sweep in Figure 18. While most SAEs seemed adequate, some
still have poor approximation as measured by the reconstructed logit difference.

We use a training set of 20,000 examples and an evaluation set of 8,000 examples
(for the purposes of automatic interpretability scoring, we need a large enough
evaluation sample so that each property in the distribution appears a
significant number of times). Since our training regime is significantly
distinct from that of Bricken et al. () (we use a much smaller
dataset), we first experimented extensively with different hyperparameters,
focusing on training SAEs on the queries of the name mover heads. We observed
that the most important hyperparameters are the dictionary size and the
effective $\ell_{1}$ regularization coefficient. We found that the batch size did
not influence the eventual quality of the learned features, only the speed of
convergence, and that a learning rate of $10^{-3}$ (as in
Bricken et al. ()) was a good choice throughout. The runs
reported here used a dictionary size of $1024$ (a $16\times$ increase over the
dimensionality of attention head activations in GPT-2 Small), an effective
$\ell_{1}$ regularization between $0.05$ and $0.3$, and a batch size of $1024$.

We normalized activations across the circuit to make it
easier for the same range of hyperparameters to give good results, and ran a
sweep over $\ell_{1}$ regularization coefficients in $(0.01,0.05,0.1,0.3)$.

Full-distribution SAEs.
We trained full-distribution SAEs on every IOI component using
OpenWebText as training data. We mostly followed the method outlined in
Bricken et al. (). We added a
standardization procedure to be able to train SAEs on components with different
activation scales using the same l1-coefficient. Before training, we calculated
the mean and the mean l2-norm over 10 million activations. These values were
then frozen and used to standardize all activations as a preprocessing step and
to rescale the SAE reconstructions to match the original scale as a
post-processing step. We generated the training dataset by extracting a buffer
of 10 million activation vectors from the shuffled OpenWebText dataset
at a time with a maximal context window of $512$ tokens. We then trained the SAEs
for 250 million activation vectors and resampled dead neurons after 50000 steps
(around 100 million activations) as outlined in
Bricken et al. (). We used a batch size of 2048 and 8192
features per SAE. Post-training, we excluded dead and ultra-low frequency
neurons that we define as neurons who activate less than once per million
activations. The amount of dead neurons varies across SAEs between 20 and 90%.
We plot the fraction of dead neurons versus $\ell_{0}$ loss in Figure
20, and the loss recovered versus $\ell_{0}$ loss in
Figure 21.

We used an $\ell_{1}$ coefficient of $0.006$ initially for all SAEs, and retrained SAEs
with a different $\ell_{1}$ coefficient for crosssections whose SAE metrics were
undesired (e.g. very low $\ell_{0}$ / bad reconstruction or very high $\ell_{0}$).
For the name mover outputs, we used $0.025$, and for S-Inhibition keys we used
$0.005$. The test $\ell_{0}$ and loss-recovered metrics were calculated on $81920$
unseen activation vectors.

We trained fewer SAEs on the full pre-training distribution compared to the IOI
distribution, as the computational cost is higher. We observe that SAEs with a
lower number of active features per example generally perform better for
IOI-related tests, even if their other metrics (such as loss recovered on
OpenWebText) are worse.

SAE training loss metrics. The most important loss metrics to track
during SAE training are the $\ell_{0}$ loss (measuring the average number of
active features per activation) and the language model loss recovered when using
the learned features to reconstruct the model’s logits
(Bricken et al., ). To turn the loss recovered into a meaningful
quantity, it is rescaled against a baseline; both zero ablation and mean
ablation have been used for this purpose in the literature
(Bricken et al., ; Kissane et al., ). In this work, we used
mean ablation, as it is a more strict test of the quality of approximation.

### A.12 Additional notes on methodology for SAE interpretability

Interpretations considered. Let Names be
the set of names in our IOI dataset. We consider the following binary
predicates over prompts as possible interpretations for SAE features in the
activations at the S2 and END tokens:

- •
IO is 2nd name: the Pos attribute having
value corresponding to BAB-type prompts;
- •
IO is 1st name: the Pos attribute having value
corresponding to ABB-type prompts;
- •
S is $<$name$>$: the S attribute has a certain value in Names;
- •
S is $<$name$>$ and at 1st position: same as above, but also the
S name is at 1st position in the prompt (i.e., this is a BAB-type
prompt);
- •
S is $<$name$>$ and at 2nd position: same as above, but for
ABB-type prompts;
- •
IO is $<$name$>$: the IO attribute has a certain value in Names;
- •
IO is $<$name$>$ and at 1st position: same as above, but also the
IO name is at 1st position in the prompt (i.e., this is a ABB-type
prompt);
- •
IO is $<$name$>$ and at 2nd position: same as above, but for
BAB-type prompts;
- •
S is male: the S name is labeled as a male name under
our labeling of Names provided by GPT-4;
- •
S is female: same as above for female names;
- •
$<$name$>$ is in sentence: a certain name in Names;
- •
$<$name$>$ is at 1st position: same as above, but the name is the
first name in the sentence;
- •
$<$name$>$ is at 2nd position: same as above, but the name is the
second name in the sentence;

The next several interpretations are only defined for the keys and values of the
name mover heads. We collect together activations for the keys according to
their role in the IOI circuit as opposed to absolute position: we group together
all activations at the IO token position (these are the IO
keys/values), even though they come from different absolute positions across IOI
prompts, because in ABB prompts the IO name comes first, while in BAB
prompts it comes second. The same applies for gathering the S
keys/values.

The key/value activations described above have not yet ‘seen’ the repeated name
in the sentence, so there is no meaningful concept of IO and S
for them. Instead, the only potentially task-relevant information contained in
them is about the name(s) seen so far in the sentence, and the position (1st
name or 2nd name) where the activation is taken from. Accordingly, the
applicable interpretations for features contained in these activations are
different:

- •
current token is $<$name$>$: the token from which the activations
are taken holds a certain value in Names.
- •
token is $<$name$>$ and at 1st position: the activation was taken
from a token with a certain value in Names, and in addition it comes
from the first name in the sentence;
- •
token is $<$name$>$ and at 2nd position: same as above, but
activation is from the second name in the sentence;
- •
current token is at 1st position: the activation is from the first name in the sentence;
- •
current token is at 2nd position: same as above, but from second
name;
- •
current token is female: the token the activation is from is
female under our labeling of Names.

*Unions of interpretations*. In addition, for each type of predicate that
has a free parameter in Names, we considered unions of up to 10 such
predicates (recall that we have a total of 216 names in our dataset). We ordered
the individual predicates according to their $F_{1}$ score, and chose the union of
the first $k\leq 10$ predicates with the highest $F_{1}$ score as a possible
interpretation. Note that the $F_{1}$ score is not in general a monotone function of $k$ in this setup; indeed, we find that for many features the highest-$F_{1}$-score explanation uses $k<10$ features.

Sufficiency/necessity of interpretable features. We take the
interpretations of the features described above and their respective $F_{1}$
scores, and for each threshold $t\in[0,1]$ over $F_{1}$ scores consider two
interventions:

- •
sufficiency: we subtract from the respective activation all
active features with $F_{1}$ score $<t$;
- •
necessity: we subtract from the respective activation all
active features with $F_{1}$ score $\geq t$;

Interpretation-aware sparse control. The goal of this experiment is to
evaluate the usefulness of our feature interpretations for editing the
attributes IO, S and Pos we have chosen to describe
prompts with. In particular, this is a different goal from our exploratory
interpretability experiment, where we were concerned with assigning
interpretations to each feature in a way agnostic to whether the features
correspond 1-to-1 with the attributes.

Correspondingly, for this experiment we use the $F_{1}$ score with respect to each
attribute as a guide for the potential relevance of a feature to this attribute.
Specifically, recall from Subsection 3.4
that we can assign an $F_{1}$ score to each combination of a feature and a
possible value of one of the attributes IO, S and
Pos.

To perform editing, let our SAE have a dictionary of decoder vectors
$\{\mathbf{u}_{j}\}_{j=1}^{m}$, and the original and counterfactual prompts
$p_{s},p_{t}$ have respective activations $\mathbf{a}_{s},\mathbf{a}_{t}$ with
reconstructions

$$ $\displaystyle\widehat{\mathbf{a}}_{s}=\sum_{i\in S}\alpha_{i}\mathbf{u}_{i}+ \mathbf{b}_{dec},\quad\widehat{\mathbf{a}}_{t}=\sum_{i\in T}\beta_{i}\mathbf{u }_{i}+\mathbf{b}_{dec}$ $$

for $S,T\subset\{1,\ldots,m\}$ and $\alpha_{i},\beta_{i}>0$. Suppose the
original and counterfactual prompts differ only in the value of the attribute
$a$ we wish to edit, with $a(p_{s})=v_{s},a(p_{t})=v_{t}$.
Using a test set, we estimate the $F_{1}$ score of each possible value $v$ of the
attribute $a$, and for each $v$ we order the SAE features in decreasing order of
the $F_{1}$ score, obtaining a list $\operatorname{top-features}\left(v\right)=\left[j_{1},j_{2},\ldots\right]$ with $1\leq j_{k}\leq m$.

Then, fixing some value of $k$, we compute the edited activation as

$$ $\displaystyle\mathbf{a}_{edited}=\mathbf{a}_{s}-\sum_{j\in(S\cap\operatorname{ top-features}(v_{s}))\left[:k\right]}\alpha_{j}\mathbf{u}_{j}+\sum_{j\in(T\cap \operatorname{top-features}(v_{t}))[:k]}\beta_{j}\mathbf{u}_{j}.$ $$

Note that this is a somewhat *less expressive* intervention than our
per-prompt agnostic feature editing, because here we require that there is some
fixed ordering from which we choose features, instead of being free to pick
features for each prompt independently.

### A.13 Additional observations on feature interpretations

Interpretable features agree with head roles identified in the IOI
circuit by Wang et al. ().
For example, duplicate token heads attend to a previous occurrence of the
previous token and write information about this to the residual stream.
Consistent with this, we found that SAEs trained on them contain features that
indicate the duplicated name, the subject in case of IOI. This information is
then used by the induction heads that determine whether the subject is the first
or second name. The subject position is subsequently copied to the END position
by the S-Inhibition heads, where it will query the name movers to *not*
attend to the subject, and to copy and predict the IO name. Indeed, we find
features of the outputs of induction heads, in the outputs and values of the S-I
heads, and in the queries of the name movers that inform about the position of
the indirect object. Lastly, the name movers attend to the IO position and copy
the name to predict it. As anticipated, the name mover values and outputs
contain features that specify the concrete IO name. In summary, the type of
features detected informs well about the function of the head on a certain task.

New insights from the feature dictionaries.

- •
The first layer DT-heads almost exclusively contain S
features but the third layer duplicate token head also has positional
features, suggesting more sophisticated text processing happening there.
- •
Induction head encompass different positional features. L6H9 features
inform about what name is at the second position, while L5H5 and L5H8
activate when the IO is at the first position. L5H9 is comprised of
different features including positional features that don’t inform about the
role (IO vs S) of the name.
- •
L7H3 is the only head that contains a significant amount of gender
features.
- •
S-Inhibition heads include primarily features that are a combination
of names and S, while the name mover queries seem to only contain
Pos features.
- •
The keys and values of name movers both inform about the token at the
current position, but there is an important difference in the type of
features: while the values primarily contain features that contain the name
(that later gets moved to the END position), the keys consist of positional
features and combinations of position and name. This hints at an important
mechanism where the name mover query contains positional information about
the S name that gets matched with the corresponding key,
effectively shifting the attention towards the IO token such that the value
with the IO gets copied to the END position.

Editing interpretable features

While the feature descriptions generated through our automatic scoring predict well when a feature is active, it is still unclear whether they also have an interpretable causal role, i.e. whether activating or deactivating a certain feature leads to a change in output logits that would be expected from the feature’s description. To test this, we propose two experiments to judge the faithfulness and completeness of our interpreted features that involve patching activations from a counterfactual prompt and calculating the effect on the model’s output:

- •
Estimating Faithfulness:
To estimate faithfulness, we construct SAE activations where we fix features with a test F1-score smaller than a threshold and patch activations of features with a high F1-score from the counterfactual prompt. We then calculate reconstructions of this new SAE activation vector, patch cross-sections, and record whether the model successfully predicts the correct counterfactual name.
- •
Estimating Completeness:
To estimate completeness, we propose a similar experiment where we fix features with a high test F1-score and patch all remaining features. This intervention should not change the output logits if our features are complete.

We run this experiment on cross-sections of name mover outputs and repeat this experiment for different thresholds. We observe that for a threshold F1-score of 0.6, the SAE features are both faithful and complete to a high degree. We observe that the faithfulness metric significantly decreases for higher F1-scores of 0.7 and 0.8 but remarkably, we also observe that only fixing features with a very high F1-score of $>0.8$ while patching all other features from the counterfactual prompt is sufficient to keep the model predicting the base prompt’s output.

Generalization of features to the full distribution.
We sample prompts from OpenWebText and visualize prompts that highly activate a
feature. We do this for the name mover queries and outputs. For the outputs, we
calculate the direct-feature-attribution (DFA) metric
first proposed by Krzyzanowski et al. ()
that calculates per position how large the contribution of its
values to activating the feature is. Thus, it informs what position led to the
feature being activated.

We find that features mostly generalize. For example, we investigated a feature
that activated if the IO name starts with the letter ”E” and we found that on
full distribution, this feature fires at tokens preceeding words starting with
”E”. DFA suggests that previous tokens starting with ”E” activate this feature,
and calculating the unembed $W_{dec}[j]W_{O}W_{U}$, denoted with
”Positive Logits” and ”Negative Logits” in Figure
14 shows that activating the feature increases
logits for words starting with ”E”. Together, this hints at a general name
moving mechanisms to predict words that previously occurred in the context that
drives in-context learning where the head’s QK-circuit drives attention to the
position of the word to predict, and the OV circuit copies the token from the
previous to the current position to predict it.

Figure: Figure 12: Two representative features discovered in the output name mover SAE L9H9 to illustrate the features behavior or webtext. Both features are IO name features, the upper one containing 23 names, the lower one only a single name. Left: Feature activation per position; Middle: Direct Feature Attribution (DFA) that tracks the position whose values contribute most to activating the given feature; Right: The output tokens whose logits get increased (positive) or decreased (negative) when the feature is active, calculated by $\textbf{v}\textbf{W}_{O}\textbf{W}_{U}$ with v being the decoder weight vector of the feature of interest, $\textbf{W}_{O}$ the output weight of the head and $\textbf{W}_{U}$ the unembed
Refer to caption: x15.png

### A.14 Feature occlusion details

Training an exhaustive set of SAEs. Focusing on the L10H0 queries, we
found that our SAEs consistently find a single feature for almost all
IO names, but fail to find a significant number of features for
individual S names.

To investigate further whether this is a result of poor hyperparameter
choices, we trained SAEs on the queries of L10H0 over a wide grid of
hyperparameters, so that performance deteriorates/plateaus at the edges of the
grid. We pushed the dictionary size, training dataset size, $\ell_{1}$
regularization coefficient and number of training epochs significantly beyond
the values used in our other experiments. Specifically, we used a training set
of 100,000 examples (this is more datapoints than all $\sim 93,000$ possible
combinations of S, IO and Pos in our data); we
trained dictionaries of up to $128*64=8192$ features (our supervised feature
dictionaries contain $\sim 500$ features); we varied the effective $\ell_{1}$
regularization coefficient across two orders of magnitude; and we trained for up
to $6,000$ epochs.

Results on the number of IO/S features with $F_{1}$ score $>0.5$
are reported in Figure 6 (left). We observe that
across hyperparameters, we often find as many IO features as there are
names in our dataset; however, the number of S features is consistently
low, never exceeding $22$.

Magnitudes of the IO and S features. As a first step, we investigate
the distribution of the norms of the features for IO and S
across the names in the IOI dataset in the L10H0 queries in our supervised
feature dictionaries from Section 4; results are shown in
Figure 13 (left). We observe that the norms of the
IO features are significantly (but not by much) higher than those of
the S features.

Figure: Figure 13: Left: distributions of the $\ell_{2}$ norms of the feature vectors for the IO and S attributes from our supervised feature dictionary for the queries of the L10H0 name mover. Right: the results of the toy model experiment, where we investigate whether a disparity in feature magnitudes alone can lead to the occlusion phenomenon. The $x$-axis shows the fraction of high-magnitude ground-truth features for which we find an SAE feature with $F_{1}$ score $>0.9$; the $y$-axis shows the same for the low-magnitude ground-truth features.
Refer to caption: x16.png

Surgically reducing the magnitude of IO features. To examine
the role of feature magnitude in the occlusion phenomenon, we continuously
reduce the magnitude of IO features. Namely, given our supervised
feature dictionary with features $\{\mathbf{io}_{a}\}_{a\in\text{Names}},\{\mathbf{s}_{b}\}_{b\in\text{Names}},
\{\mathbf{pos}_{v}\}_{v\in\{\text{ABB},\text{BAB}\}}$ and an activation $\mathbf{a}(p)$ for a prompt $p$ where the
IO name is $a$, we construct a new activation

$$ $\displaystyle\mathbf{a}_{\alpha}\left(p\right)=\mathbf{a}\left(p\right)-\alpha \mathbf{io}_{a}$ $$

for $\alpha\in[0,1]$. We find that, applying this intervention without any
hyperparameter tuning (with a modest dictionary size of $1024$, and $\ell_{1}$
regularization coefficient of $0.2$), increasing $\alpha$ from 0 0 to $1$
gradually makes the number of IO features with $F_{1}$ score $>0.5$
to decrease, while the number of S features with $F_{1}$ score $>0.5$
increases; results are shown in Figure 6 (right).

Reproducing the occlusion phenomenon in a toy model. Finally, we wanted
to know if a disparity in feature magnitudes alone could lead to the occlusion
phenomenon. We constructed a simple toy model closely based on the empirical
setup in the queries of the L10H0 head to test this hypothesis.

We form synthetic activations $\mathbf{a}=\mathbf{u}_{i}+\mathbf{v}_{j}$ for
random pairs $i,j\in\{1,\ldots,\left|\text{Names}\right|\}$, where
$\mathbf{u}_{i},\mathbf{v}_{j}\in\mathbb{R}^{d_{head}}$ are sampled independently
from a standard normal distribution centered at zero, and then $\mathbf{u}_{i}$
are rescaled so that their mean norm matches the mean norm of IO
features in the L10H0 queries, and similarly for $\mathbf{v}_{j}$ and S
features. We train SAEs on these activations over a wide grid of hyperparmeters:
dictionary sizes in $(512,1024,2048)$, $\ell_{1}$ regularization in $(0.0125,0.025,0.4,0.5,1.0,2.0)$, batch size in $(256,1024)$ and learning rate in
$(0.001,0.003,0.0003)$. We trained for $1000$ epochs, saving checkpoints in a
geometric progression of epochs. Results for the number of high- and
low-magnitude features with $F_{1}$ score $\geq 0.9$ discovered are shown in
Figure 13 (right); we observe that we easily find one SAE
feature per each high-magnitude ground-truth feature, but it is more difficult
to find an SAE feature for each low-magnitude feature.

However, we note that with lower $F_{1}$ thresholds, this effect is less
pronounced and eventually disappears.

### A.15 Feature over-splitting in a mixture of gaussians toy model

Figure: Figure 14: Main experiment for our toy model of feature oversplitting. The data distribution is a uniform mixture of two standard multivariate Gaussian random variables in $100$ dimensions. Blue: the (approximate) best possible total loss (in the infinite data limit) achieved by a class of ‘ideal’ SAEs that use two features pointing towards the means of the two components of the mixture. Orange: an approximate upper bound on the best possible total loss achieved by an SAE with $m=1,000$ hidden features (in the infinite data limit). The $x$-axis is the $\ell_{1}$ regularization coefficient $\lambda$. The cutoff on the $x$-axis is chosen so that the idealized solution activates only for a vanishing fraction ($<2\%$) of the examples in the mixture.
Refer to caption: x18.png

Our goal in this section is to demonstrate that there *exist* setups where
an SAE with a large number of hidden features $m\gg 2$, when trained on a uniform
mixture of two isotropic gaussian variables, will prefer a solution with $\gg 2$
features (as opposed to the ‘ideal’ solution with only two features, one per
component of the mixture), for *any* value of the $\ell_{1}$ regularization
coefficient $\lambda$ and *any* amount of training data.

Setup.
We consider a simple toy model where activations are
distributed according to a uniform mixture $\mathcal{D}_{toy}$ of two isotropic
gaussians $\pm\bm{\mu}+\mathcal{N}\left(0,\mathbf{I}_{d}\right)$ in $\mathbb{R}^{d}$
(i.e., we first flip a fair coin to determine the sign of $\bm{\mu}$, and then
sample an extra additive term from $\mathcal{N}(0,\mathbf{I}_{d})$). We sample an
i.i.d. dataset from this mixture. We used $d=100,\left\|\bm{\mu}\right\|_{2}=2$ in the
experiments below; this choice guarantees that the two components of the mixture
are separable with high ($>95\%$) probability.

The ‘ideal’ solution with 2 features.
We might hope that, with the right $\ell_{1}$ regularization, an SAE trained on
such a distribution will discover only two interpretable features, one for each
component in the mixture, with the encoder/decoder vectors aligned with
$\bm{\mu}$ and $-\bm{\mu}$ respectively, and each feature activating for
(approximately) the examples in its corresponding component.

We find this is the case empirically when we train an SAE with only two hidden
features on this toy distribution. Specifically, there is a range of $\lambda$
values $1\leq\lambda\leq 3$ where the SAE reliably approximately recovers this
ideal solution, with the encoder bias controlling the trade-off between the two
loss terms: when $\lambda$ increases, the encoder bias changes so that fewer
examples in each component of the mixture are activated (and the active examples
have a lower $\ell_{1}$ loss). Beyond this range, the SAE often fails to activate
any feature on any examples ($\lambda>3+\varepsilon$), or activates both
features on almost all examples ($\lambda<1-\varepsilon$).

Analyzing the ideal solution across $\ell_{1}$ coefficients.
To study the properties of the ‘ideal’ solution analytically, we make the
following assumptions (borne out empirically with 2-feature SAEs) using only
symmetries of the data distribution:

- •
the decoder vectors are $\pm\bm{\mu}$, normalized to have
unit $\ell_{2}$ norm (by symmetry of each component around its mean);
- •
the respective encoder vectors are $\pm k\bm{\mu}$ for some $k>0$ (again
by symmetry of each component around its mean);
- •
the decoder bias is zero (by symmetry of the mixture around zero);
- •
both encoder biases are set to $-\gamma$ for some $\gamma>0$ (again by
symmetry of the mixture around zero).

This leaves only two parameters to tune: the encoder bias $\gamma$ and the
encoder scale $k$. We can thus use the following strategy to analytically
approximate the best loss of this class of solutions for a given $\lambda$:

- •
approximate the expected $\ell_{1}$ and $\ell_{2}$ losses over a fine grid of
values for $\gamma$ and $k$, for a large dataset of samples from the mixture;
- •
given a $\lambda$ value, find the point in the grid that minimizes the
total loss $\ell_{2}+\lambda\ell_{1}$.

We implemented this using $10^{5}$ samples, with a grid of 100 values for $\gamma$
in $[0,5]$ and 20 values for $k$ in $[0,2]$, over a grid of 100 values of
$\lambda$ in $[0,20]$. We verify that the best values chosen for each
$\lambda$ are not on the edges of the grid; the resulting curve of best total
loss values versus $\lambda$ is shown in Figure 14 (blue), cut off at
$\lambda\approx 7$, beyond which the selected SAE activates for $<2\%$ of
examples in the components of the mixture.

SAEs with $m\gg 2$ features prefer other solutions even with infinite data.
Next, we want to show that with enough features, SAEs will prefer solutions
different from the class of 2-feature solutions described above.
How can we give an *empirical* argument that applies to *any* amount
of training data and any $\lambda$? After all, a *trained* SAE is a
function of the data it is trained on, so no experiment on datasets of bounded
size can establish properties of SAEs trained on arbitrarily large datasets.

We get around this by defining a class of SAEs that is competitive with the
class of ideal 2-feature SAEs *upfront*, independent of the training
sample, by using a randomized construction that works w.h.p., and then
estimating the expected loss of this SAE in the infinite data limit empirically.
To give evidence of our result for arbitrary $\lambda$, we consider a fine
enough grid of $\lambda$ values, and for each $\lambda$ we construct an SAE that
is competitive with the best ideal 2-feature SAE.

Our randomized SAE construction proceeds as follows:

- •
Sample $m$ encoder vectors $W_{enc}\in\mathbb{R}^{m\times d}$ from
$\beta*\mathcal{D}_{toy}$ (i.e. a version of $\mathcal{D}_{toy}$ scaled by
$\beta$) where $\beta>0$ is a hyperparameter that we will tune;
- •
Set the decoder vectors $W_{dec}\in\mathbb{R}^{d\times m}$ to be the same
as the encoder vectors $W_{enc}^{\top}$, but normalized so that each column of
$W_{dec}$ has unit $\ell_{2}$ norm;
- •
Set all encoder biases to $-\gamma$ (where $\gamma>0$ is a hyperparameter
that we will tune), and the decoder bias to $\mathbf{0}\in\mathbb{R}^{d}$.

We used $m=1,000$, a grid of 100 values for $\beta$ in $[0,0.1]$ to search for
the best $\beta$ for each $\lambda$, and fixed $\gamma\approx 2.35$.

Empirical confirmation. Finally, we actually trained SAEs with many
hidden features on $\mathcal{D}_{toy}$, and observed that these SAEs reliably
learned solutions with many active features across $\lambda$ values.

### A.16 Additional details for Section 5

Feature weights are mostly in the interval $[0,1]$.
Recall that given a reconstruction $\widehat{\mathbf{a}}=\sum_{i}\mathbf{u}_{i}$, we defined the feature weight for the $i$-th feature as

$$ $\displaystyle\operatorname{weight}\left(i\right)=\mathbf{u}_{i}^{\top}\widehat {\mathbf{a}}/\left\|\widehat{\mathbf{a}}\right\|_{2}^{2}.$ $$

For our supervised feature dictionaries, we find that $10\%$ of all weights are
negative, and that the average value of all negative weights across all nodes in
the IOI circuit and all three attributes is $-0.037$. Similarly, for our
task-specific SAE feature dictionaries, even though $31\%$ of all weights are
negative, the average value of all negative weights is $-0.002$. The number and
magnitude of weights higher than $1$ are even smaller.

Causal evaluation using interpretability.
While the feature descriptions generated through our automatic scoring predict
well when a feature is active, it is still unclear whether they also have an
interpretable causal role, i.e. whether activating or deactivating a certain
feature leads to a change in output logits that would be expected from the
feature’s description. To test this, we propose two experiments to judge the
sufficiency and necessity of our interpreted features that involve patching
activations from a counterfactual prompt and calculating the effect on the
model’s output:

- •
Estimating sufficiency:
To estimate sufficiency, we construct SAE activations where we fix features with a test F1-score smaller than a threshold and patch activations of features with a high F1-score from the counterfactual prompt. We then calculate reconstructions of this new SAE activation vector, patch cross-sections, and record whether the model successfully predicts the correct counterfactual name.
- •
Estimating necessity:
To estimate necessity, we propose a similar experiment where we fix features with a high test F1-score and patch all remaining features. This intervention should not change the output logits if our features are complete.

We run this experiment on cross-sections of name mover outputs and repeat this
experiment for different thresholds. We observe that for a threshold F1-score of
0.6, the SAE features are both faithful and complete to a high degree. We
observe that the faithfulness metric significantly decreases for higher
F1-scores of 0.7 and 0.8 but remarkably, we also observe that only fixing
features with a very high F1-score of ¿0.8 while patching all other features
from the counterfactual prompt is sufficient to keep the model predicting the
base prompt’s output.

### A.17 Additional figures

Figure: Figure 15: Fraction of recovered logit difference for several different methods to compute feature dictionaries, across cross-sections of the circuit. For a definition of the ‘names’ parametrization, see Appendix A.10.
Refer to caption: x19.png

Figure: Figure 16: Attention score decomposition for the L9H6 name mover (see Figure 8 for explanation). Notice that, in contrast with L10H0 attention socres, there is no significant (inhibitory) interaction between the IO features in the query and key.
Refer to caption: x20.png

Figure: Figure 17: Attention score decomposition for the L9H9 name mover (see Figure 8 for explanation).
Refer to caption: x22.png

Figure: Figure 18: Metrics for our chosen task-specific SAEs for each relevant node in the IOI circuit. The x-axis shows the absolute value of the difference in logit differences between a clean run of the model, and a run where the activations at the given node are replaced by the SAE’s reconstructions, normalized by the difference between the clean logit difference and the logit difference when the node is mean-ablated instead. The y-axis shows the average number of features active per prompt.
Refer to caption: x24.png

Figure: Figure 19: Counterpart to Figure 18 where the decoder vectors are frozen during SAE training.
Refer to caption: x25.png

Figure: Figure 20: $\ell_{0}$ loss versus fraction of dead neurons for our full-distribution SAEs.
Refer to caption: x26.png

Figure: Figure 21: $\ell_{0}$ loss versus loss recovered (against a mean ablation) for our full-distribution SAEs.
Refer to caption: x27.png

Figure: Figure 22: Distribution of the average feature weight removed by our interpretation-agnostic edits using SAE feature dictionaries, over all locations in a given cross-section, normalized by the corresponding weight for the supervised feature dictionaries. Weight removed values are transformed linearly so that a value of 0 0 indicates that the weight removed by the edit equals the weight removed by the corresponding ‘ground truth’ supervised edit; and a value of $1$ indicates that the edit removed a total weight of $1$, meaning that the edit essentially overwrites all SAE features present in the activation. Negative values are clipped at $-5$ to preserve readability.
Refer to caption: x28.png

Figure: Figure 23: Counterpart of Figure 22 for full-distribution SAEs, when editing using features with high F1 score for the attribute
Refer to caption: x29.png

Figure: Figure 24: Counterpart to Figure 3, where the decoder vectors are frozen during SAE training.
Refer to caption: x30.png

Figure: Figure 25: Counterpart to Figure 3, where we use full-distribution SAEs instead. Here, we need to change a much higher number of features in order to have a noticeable effect (and sometimes editing even 32 features fails)
Refer to caption: x31.png

Figure: Figure 26: Interpreting the features learned by the task SAEs. For each node in the main IOI circuit (without backup/negative name movers), we show the distribution of the features which have an explanation with $F_{1}$ score above a threshold. The SAE chosen at each node is the one with the most interpretable features out of all SAEs trained on this node during our hyperparameter sweep.
Refer to caption: x32.png

Figure: Figure 27: Interpreting the features learned by the task SAEs. This is a counterpart to Figure 26 for the SAEs chosen based on the $\ell_{0}$ and logit difference recovered metrics.
Refer to caption: x33.png

Figure: Figure 28: Interpretation-aware sparse control, using task SAE features with the highest $F_{1}$ score with respect to the given attribute.
Refer to caption: x34.png

Figure: Figure 29: Counterpart of Figure 28 with full-distribution SAEs.
Refer to caption: x35.png

Figure: Figure 30: Measuring the sufficiency of interpretable features for task SAEs: effect of subtracting features with the lowest $F_{1}$ score from activations on logit difference. A value of 1 is best.
Refer to caption: x36.png

Figure: Figure 31: Measuring the necessity of interpretable features for task SAEs: effect of removing features with the highest $F_{1}$ score from activations on the logit difference. Values are rescaled linearly so that a value of 1 corresponds to perfect recovery of the logit difference achieved by mean ablation (i.e., ideal intervention removing all features). A value of 1 is best.
Refer to caption: x37.png

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: In the title of Figure 1, could you make a clear connection of the text with precise parts of visuals, to facilitate the understanding?

Taking into account the size of the paragraph on related work, it should be possible to describe the related work without much terminology and thus shifting it closer to the beginning of the manuscript. This would allow the reader to better position the framework with respect to the state of the art.
soundness: 3
strengths: In general the manuscript is well written and "strong".

The question which is really to ask is how much this finding is relevant for the literature. Here I should say my expertise is perhaps too limited to provide a proper judgment.
summary: The authors propose a (principled) method allowing to create supervised dictionaries for space features, which allow for evaluating the degree of disentanglement of SAEs. The developed method is then applied to SAEs and LLMs, witnessing not only interpretable later variables, but also providing possibility of editing attributes. Metrics of sufficiency, necessity and control are used for this.
weaknesses: Though this has most probably costed a lot of work, the empirical validation of the proposed methodology is rather scarce. Whether the constructed dictionaries would also function for other tasks/semantics is not clear.

The mathematics are well explained, in clear and simple way.

Some (not that many) parts of the manuscript I had to read several times, e.g., the title under Figure 1 or the paragraph on interpretability at the end of Section 3.2. But in general formulas aid much understanding.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weakness section.
soundness: 3
strengths: This paper is well-written and easy to follow.
summary: This paper studies the Sparse autoencoders to capture interpretable features for the IOItask and the expeirment results show that the proposed approach achieves the best performance.
weaknesses: There are several weaknesses:

1. The motivation of this section should be enhanced.

2. The English language should be improved.

3. The main idea seems not very novel. This paper should provide a strong motivation.

4. The experiment can be further improved by providing more results and analysis.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 4
questions: - In section 4.3, if I understand correctly, the SAE latents are found by simply optimizing/searching for the features that perform the task (move one latent to the specified counterfactual latent). This seems a bit roundabout to me - wouldn’t this propose that you need to know the features you are looking for in order to label SAE features? How would one do this searching or interpretation without access to the counterfactual activations? Wouldn’t it be more realistic to interpret or label each SAE feature and then use the features that are labelled to be relevant to the task at hand?
- Please see the above weaknesses!
soundness: 3
strengths: - The writing and paper structure are very clear and easy-to-follow.
- The focus of this paper is highly relevant and interesting. The problem of finding a ground truth with which to evaluate interpretability and explainability methods has remained an issue for decades, and this work works towards solving this problem by exploring using human-generated groundtruths that have been backed up by prior work.
- The experiments are well-defined, inutitive, and easy to understand.
- I believe the results are interesting and useful - they reveal that task-specific SAEs are more useful in practice than full-distribution SAEs, hinting that data quality is of utmost importance when training SAEs. Further, this suggests that human priors may be useful when developing interpretability methods/SAEs.
summary: This paper focuses on evaluating sparse autoencoders (SAEs) for their ability to recover known ground-truth features learned by a model. To do so, the authors first train a supervised dictionary on the indirect object identification (IOI) task, for which model computation is already relatively known due to prior interpretability and circuit discovery work. Both IOI task-specific SAEs and full-distribution SAEs are trained and evaluated with respect to the supervised dictionaries to understand if SAEs allow for the same level of approximation, control, and interpretability as the supervised dictionaries. Results reveal that more recent SAE architectures improve these capabilities and task-specific SAEs are much more effective than full-distribution SAEs.
weaknesses: - I find the “Why use these attributes?” paragraph in Section 3.1 to be confusing. If prior work had not proposed the IO, S, and Pos attributes, how would one go about defining the supervised dictionary? If the evaluation pipeline described in this paper were to be used for a different task, I’m not sure whether this section would be generalizable. In particular, when there are many choices of attributes, what is the manner of choosing between them without using another interpretability method, which would then prevent the need of using an SAE in the first place?
- It would have been significantly more convincing to me if the authors had considered more than one task in their evaluation. At the moment, it’s unclear to me how the proposed methodology and results from this work could be applied to future works that want to evaluate trained SAEs.
- The section on interpretability (section 4.4) is also a bit confusing to me - I would find it very helpful if the authors provided interpretations of the SAE latents, and a visualization of how these features could then be used to explain the LLM’s computation on a single example. Some examples of /possible/ interpretations are provided in Appendix 7.13-7.14, but if I understand correctly these are not the actual labels of the SAE features.
- It is my understanding that the authors wish to propose the use od supervised dictionaries an evaluatory baselines for SAEs. However, in practice, this paper reads more as an exploration of whether SAEs can recover the IOI task. While the authors discuss the limitations of hardcoding the attributes to compare SAEs against and only considering a single task and model, I believe these drawbacks fundamentally limit the work in its general utility.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: - Equation 2. Reason for the "bias" term is $E[a(p)]$? Does this mean $E[u_{IO}] + E[u_S] + E[u_{POS}] \approx 0$ if we take the expected value on $a(p)$ and $\hat{a}(p)$? If it is by design, can we make a comment on this design? What is a brief explanation behind this design?
- What is F and A in F1 Score? In the text, it seems F=the set of examples activating a specific SAE latent "f". A=binary attribute of a prompt. It seems that the F1 Score is applied **on each SAE latent**, as described in Section 4. How do we get "A" in this case? How do we know which "binary" attribute of a prompt that the latent f corresponds to? Can we give a more detailed explanation in the text? It would be nice to include some examples of F and A (specifically A)?
soundness: 3
strengths: - Thorough study in the case of IOI. The use of IOI in the study mainly because we know the "feature" that more or less directly affect the outputs. This minimizes the possibilities of the cases that "SAE features" may not coincide with the features that human understands. This was also implied in Section 4.2.
- Extensive study on the paper that includes a lot of results in the appendix.
- Some proposed methods such as Necessity and sufficiency, to the control method proposed can be applied to a more general case, as described in Section 4. 
- This paper also addresses a lot of the details on small nuances in evaluation of SAEs. This includes the discussion in Session 4.2, and the paragraph on "Preventing..." in Section 4.3.
summary: This paper introduces a way to compute sparse feature dictionaries that disentangle model computations similar to an SAE fashion using a supervised method; as well as introduces several ways to evaluate these feature dictionaries from different perspective such as necessity, sufficiency and controls of the feature dictionaries towards a specific task. The author applies the work to the indirect object identification (IOI) using GPT-2 Small, and compare the feature dictionaries obtained by their method, vs some other recent SAE variants such as Gated SAE adn Top-K SAE.
weaknesses: **Overall**

- As this paper is a case study to IOI, it is somewhat restrictive. I don't think it is necessarily a weakness of the paper though as I don't think anything can be done to address this "weakness". It could be viewed as a stepping stone for future work as well.
- Presentation and Readability are the main weakness of this work. For example, experiment results in Section 3 includes a lot in section 4 such as description of FullDistribution SAE, Task SAE, TopK, Gated SAE etc. There are crucial formulae and methods that should be in the main text, but instead they are in the Appendix. For instance, the necessity formula in Section 7.4 ~line 878 in the appendix.
- The section on "Interpretability" such as in Section 3.2, Figure 5 and Section 4.4 are out of place. The author even mentioned that "we stress that this experiment is of a rather exploratory and qualitative nature". Putting these sections in the appendix would be much more coherent.
- Section 6 on discussions and limitations is too short. There should be more discussions on the experiment results. Possibilities of applying similar techniques to a general settings (as IOI provides the "known features"). Including a discussion on future work may be useful.

**Detailed issues**

- On Sufficiency in "Figure 3. Left" - as the experiment is to test whether the reconstructions are sufficient, we would hope to compare the logit difference of the original and the reconstruction. The method in Figure 3 shows the ratio of the *average* logit difference with and without the intervention. This may not be the best because the averaging may hide the changes in logit difference with the intervention for each example. A simpler method like the average changes (absolutized) of the logit difference _may_ work. The authors can also opt to include a brief discussion on this so that it does not seem like they were hiding something. For example, showing the distribution of  absolute changes of the logit difference in a histogram, or some statistics on it.
- Necessity. The experiment is to test whether the reconstructions are necessary. This means that we want to show that without the reconstructions, the model cannot do so well, resulting a drop in model performance - i.e. logit difference. Thus there are three quantities
1) The reconstruction $\hat{a}(p)$
2) The proposed quantity; average plus SAE error term
3) The average.

Showing necessity should be showing the difference between 1) and 2) are large. However, in the main text of the paper, it opts to show the difference between 2) and 3) only. A crucial formula and description of the necessity score directly addressing this problem is in the appendix (Section 7.4, around line 878), which in my opinion, should be in the main text.
- "Probing accuracy" in "Control": Seems out of place? Is it referenced somewhere else in the paper? Also no results were shown? In the absence of results, this section on probing accuracy does not seem to achieve the goal of the section: "measures the degree to which the supervised feature dictionaries disentangle the different attributes". This is because the probe is linear, which itself can "disentagle" the attributes. I think it would be better to either remove this section (put this in the appendix), or show some experiment results with discussions on the way to disentangle the attributes.

**Minor**
- Section 4.3 Expressing... line ~361. edit 3 --> (3) or Equation 3.
- Results (line ~424). objective 4 --> (4) or Equation 4.
- missing parenthesis at line 434 (resp. $a(p_t)$ **)** by their...
- broken references in appendix (line 883, 935, 1849)

Meta-review: This paper explores sparse autoencoders on language tasks that have a known ground truth to evaluate whether SAEs can provide similar interpretability and control as supervised feature dictionaries.

Since most language tasks do not have a single ground truth, the authors originally used only one highly specialized experimental setting. Reviewers found this to be lacking, and also gave advice on how to improve the presentation, and better communicate the limitations of the work. After discussion, the authors heavily revised their presentation, included better discussion of limitations, and added additional experimental settings. All these changes were well-received by the reviewers, and the paper now stands as a clear and interesting contribution to studies on interpretability techniques for language models. This is a valuable research direction, and one in desperate need of more investigation by the community.

As such, I am recommending acceptance as a Spotlight.

Decision: Accept (Poster)