Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control
Authors: Aleksandar Makelov
Georg Lange
Neel Nanda
Venue: ICLR 2025 Poster (2025)
Keywords: mechanistic interpretability
sparse autoencoders
evaluations
Primary area: interpretability and explainable AI
TL;DR: We compute and validate *supervised* sparse feature dictionaries on the IOI task, and then compare SAEs against them

Abstract: Disentangling model activations into human-interpretable features is a central
problem in interpretability. Sparse autoencoders (SAEs) have recently attracted
much attention as a scalable unsupervised approach to this problem. However, our
imprecise understanding of ground-truth features in realistic scenarios makes it
difficult to measure the success of SAEs. To address this challenge, we propose
to evaluate SAEs on specific tasks by comparing them to supervised
feature dictionaries computed with knowledge of the concepts relevant to the
task. 

Specifically, we suggest that it is possible to (1) compute supervised sparse
feature dictionaries that disentangle model computations for a specific task;
(2) use them to evaluate and contextualize the degree of disentanglement and
control offered by SAE latents on this task. Importantly, we can do this in a
way that is agnostic to whether the SAEs have learned the exact ground-truth
features or a different but similarly useful representation.

As a case study, we apply this framework to the indirect object identification
(IOI) task using GPT-2 Small, with SAEs trained on either the IOI or OpenWebText
datasets. We find that SAEs capture interpretable features for the IOI task, and
that more recent SAE variants such as Gated SAEs and Top-K SAEs are competitive
with supervised features in terms of disentanglement and control over the model.
We also exhibit, through this setup and toy models, some qualitative phenomena
in SAE training illustrating feature splitting and the role of feature
magnitudes in solutions preferred by SAEs.

Published as a conference paper at ICLR 2025
TOWARDS PRINCIPLED EVALUATIONS OF SPARSE AU- TOENCODERS FOR INTERPRETABILITY AND CONTROL
Aleksandar Makelov
Georg Lange' ma @georg ange.com
Neel Nanda
a aksandar.maaee ovv8maa coom
neel nanda2 78 78mm 11
ABSTRACT
Disentangling model activations into buman-interpretable features is a central problem in interpretability. Sparse autoencoders (SAEs) have recently attracted much attention as scalable unsupervised approach to this problem. However, our limited understanding of ground-truth features in realistic scenarios makes it difficult to measure the success of SAFs. To address this challenge, we propose to evaluate SAEs on specific tasks by comparing them to supervised feature dictio- naries computed with knowledge of the concepts relevant to the task.
Specifically,, we suggest that is possible (I) compute supervised sparse feature dictionaries that disentangle model computations for specific task: (2) use them to evaluate and contextualize the degree of disentanglement and control offered by SAE latents on this task. Importantly, we can do this in a way that is agnostic to whether the SAEs have learned the exact ground-truth features or a different but similarly useful representation.
As case study, we apply this framework to the indirect object identification (IOI) task using GPT-2 Small, with SAES trained on either the IOI or OpenWebText datasets. We find that SAES capture interpretable features for the IOI task, and that more recent SAE variants such as Gated SAEs and Top-K SAEs are competitive with supervised features terms of disentanglement and control over the model. We also exhibit, through this setup and toy models, some qualitative phenomena in SAE training illustrating feature splitting and the role of feature magnitudes solutions preferred by SAES.
INTRODUCTION
While large language models (LLMs) have demonstrated impressive (Vaswani et al. 2017 Devlin al. 2019 Radford et al.) 2019 Brown et al. 2020 OpenAl, 2023) results, the mechanisms be- hind their successes and failures largely remain a mystery (Olah (023). A prominent bottom-up approach this problem taken by mechanistic interpretability (MI, which aims to disentangle model activations into units with faithful, human-interyn intmannitterpretable roles in an LLM'' computation (Olah) 2022]. Recently, the MI community has focused on sparse autoencoders (SAEs) as promising approach to this problem (Olshausen & Field 1997 Faruqui et al. 2015 Goh 2016 Arora et al. 2018 Yun
Yun et al. [202T] Cunningham et al. | [002]] Bricken et al. [002)). successful scale, this re-
search could provide significant scientific and practical value, enabling enhanced model robustness,
controllability. interpretability, and debugging (Gandelsman et al. 2023 Nanda et al. 2023 Marks
al. 2024
Commy & Nanda 2024).
However, realistic soound-truth evaluations of the SAE paradigm have been lacking, which hinders progress in this area. While recent works have proposed various metrics to evaluate SAE quality (detailed in Section[]]. they all rely on indirect metrics only conjecturally correlated with recovering the 'true' features used by LLMs.
To address this challenge, we propose the first principled method for evaluating SAES on realistic linguistic tasks. Our method directly compares SAES supervised feature dictionaries computed using knowledge of task-relevant concepts. By comparing SAES to objects of the same 'type signa- ture", we provide fair comparison where the supervised dictionaries serve as a canonical skyline
"Joint contribution. Correspondence al lessandar www.nndrr..aaeeeememm c com

---

Published as a conference paper at ICLR 2025
Select Prompts with Known Attributes
Test 1: Approximation
Test 2: Sparse Controllability
Nocerally Suttwienyy
201 Yar wet the Hery COE
Purren
Aecandizto varn uoo Auo diicc
New Pai-
Ieatunss
Compute Supervised Feature Dictionaries
Test 3: Faithfulneee of Feature Interpretations
Mackon 10
Soo destirrdeeciiiiiire roing inowe sttritunss
Figure 1: Overview of our evaluation pipeline. We begin by selecting a specific model capability (IOI task, Section 3 and then disentangling model activations into capabilityrrelevant features using supervision (Section 4.2 Then, we evaluate given feature dictionary w.r.t this capability, using the supervised features as a benchmark (Section 4.3) We test the extent to which (1) the feature dictionary's s reconstructions of the activations are necessary and sufficient for the capability (Section 5.3), (2) the features can be used to edit sspability-relevnn information in internal model represen- tations agnostic of feature interpretations (Section 5.4), and (3) the features can be interpreted w.r.t. the capability in manner consistent with their causal role (Section 5.5)
for SAE performance. The IOI task (Wang et al., 2022) a non-trivial natural language task that has been extensively studied in the mechanistic interpretability literature serves as a natural case study on which we base this paper. However, we stress that our framework is applicable to other tasks, as we illustrate in Appendix 7.1 Our contributions are outlined as follows:
We propose a principled method to compute supervised sparse feature dictionaries w.r.t. a specific task an L.LM can do, using supervision via task-relevant prompt attributes.
We apply this method to the IOI task, demonstrating that these dictionaries exhibit three desirable properties: (1) dictionaries' activation reconstructions are both sufficient and nec- essary for the task; (2) attributes relevant to the task are disentangled in way that allows precise control over model behavior; and (3) the features are interpretable in a way consis- tent with their causal role in the model's computation.
We use these feature dictionaries to design and contextualize evaluations of any feature dictionaries along the same three axes, in a way agnostic to whether they use the same latent directions.
We apply this methodology to SAEs trained on either the IOI dataset speciffpecific SAEs) the LLM's pre-training dataset (full- distribbtiinio SAEs), finding that both types contain interpretable latents for the task, and that some task-specific SAE variants allow us to edit attributes about as effectively as supervised dictionaries
We also briefly remark on two qualitative phenomena we observed in SAE training and reproduced in toy models: tendency for SAEs to learn higher-magnitude features, and a tendency to 'over-split" an intuitively single concept into many features. Due to space constraints, we defer full discussion Appendix 77.2]
Our results suggest that more detailed and controlled SAE evaluations are possible and informative, and that SAES may hold promise for disentangling model computations in realistic scenarios. A limitation of our approach is that it requires potentially substantial per-task effort to independently identify the relevant attributes, which is proportional to the complexity of the task and the number of attributes we want to consider. However, as we show in Appendix 7.. our framework allows for the targeted evaluation of a few attributes few locations of the model, which can substantially reduce this effort.
2 RELATED WORK
Most related to this paper. Several recent works propose controlled SAE evaluations related to approach. Karvonen et al. used board game model testbed, comparing SAE latents to
2

---

Published as a conference paper at ICLR 2025
Legend
When
10
Mary
Key Valuue
and
Dohri
Query
Class of Heads syeer Head
Outout
Previous Token Heads 22
S1+1 went
store
Negative Name Mover Heads 107 1110
Duplicate Token Heads on 1000)
Induction Heads 69
S2
Dohn
gave
drink
9. 9.0 100
S-inhibition Heads 79 66 8.1
END
Harkup Name Mover Heads
5. 97 101 102 106 10:00 112 119
Figure 2: A reproduction of Figure 2 from Wang et al. (2022), showing the internal structure of the IOI circuit. Original caption: The input tokens on the left are passed into the residual stream. Attention heads move information between residual streams: the query and output arrows show which residual streams they write to, and the key/value arrows show which residual streams they read from.
supervised features based on atomic board state components and simple strategic configurations. Huang et al. (2024); Chaudhary & Geiger (2024) evaluated SAEs for disentanglement of factual attributes in templated dataset, while Gao al. (2024) proposed evaluating SAE latents as binary classifiers over human-defined tasks (e.g., sentiment classification). A concern with the choice of "ground-truth" concepts these works that, while meaningful from a human perspective, they may not align with internal model computations. We also differ from these works in that we focus on linguistic tasks in pre- mre-traneed LLMs.
Other SAE evaluations. Prior works have also proposed various indirect metrics proxies for assessing recovery of "ground-tutth' features. These include automated interpretability methods based highly activating examples (Bills et al. 2023 Choi al. 2024 Juang et al., 2022)), though using maximum activating examples has been criticized as potentially misleading (Bolukbasi et al.
2021
Other approaches the trade-off between reconstruction error and sparsity (Bricken al.
2023
Cunningham et al.) [0023] Raaaamanhharan et al., 2024), toy models (Elhage et al.) 2022b),
computational proxies (Bricken al.] [2023] Templeton et al. 2024), or geometric measures like
mean maximum cosine similarity between different SAEs' features (Sharkey et al. 2023).
3 PRELIMINARIES
The linear representation hypothesis and sparse autoencoders. A central hypothesis in inter- pretability is the linear representation hypothesis. A strong variant of this hypothesis posits that model activations can be approximately decomposed into meaningful features using a sparse fea- ture dictionary: given location in the model (c.g., an attention head output), there exists set of vectors {u; J such that each activation a at this location can be approximated as sparse linear combination of the u, with non non-negative coefficients. In particular, recent work suggests that n- dimensional activations a E R" may be best described by m >>>>>> such features in superposition (Elhage al. 2002aa] Gurnee al., 2023). Recently, SAES have been proposed as way disen- tangle these features. Following the setup of Bricken et al. (203) here and in the rest of this work, sparse autoencoder (SAE) is an unsupervised model which learns to reconstruct activations e Th weighted sum of m features with non-negative weights. Specifically, the autoencoder computes a hidden representation f = ReLU (Wene (a bile)) + bene) and a reconstruction
â = Wdeof f + bdee = Σ (W + batec
(1)
where Wene R Wdes Raxm bdec E RR,beene Rd are learned parameters. The rows of Wene are the encoder directions, and the columns of Wa are the decoder directions. Similarly, bene is the encoder bias and Dadc is the decoder bias. The decoder directions determine
3

---

Published as a conference paper at ICLR 2025
the features decompose the activations into, while the encoder directions compute the coefficients of these features for a given activation. The decoder directions are constrained to have unit norm: ((Weee)..lll 1. In the simplest setup (Bricken et al. 2023). the training objective over examples (a(*)) 1==1 is the sum of the MSE between the activations (k) and their reconstructions (()), and the l regularization term À Ex=1
where À is the L regularization coefficient.
The IOI task. In Wang et al. 2022). the authors analyze how the decoder-only transformer language model GPT-2 Small (Radford et al. 2019) performs the Indirect Object Identification (IOI) task. In this task, the model is given sentences of the form 'When Mary and John went the store, John gave a book to" (with the intended completion in this case being Mary"). ) We refer to the repeated name (John) as (the subject) and the non-repeated name (Mary) as I0 (the indirect object). For each choice of the 10 and S names, there are two patterns the sentence can have: one where the IO name comes first (we call these 'ABB examples"), ) and one where it comes second (we call these 'BAB examples"). We refer to this binary attribute as the Pos attribute (short for position). Additional details on the data distribution, model and task performance are given in Appendix 7.6
Wang et al. (2022) discover several classes of attention heads in GPT2-Small that collectively form the 101 circuit solving the IOI task (Figure 2 see Appendix 7.3 for more details on the circuit structure). Specifically, Wang et al. (2022) provide multiple lines of evidence that the circuit ap- proximately implements the algorithm:
1. detect the (i) position in the sentence and (ii) identity of the repeated name in the sentence (i.e., the name). This information is computed and moved by duplicate token/induction and S-Inhibition heads;
2. based on the two signals (i) and (ii), exclude this name from the attention of the name mover heads, so that they copy the remaining name (i.e., the 10 name) to the output.
The logit difference metric. To discover the IOI circuit, Wang et al. 2022) used the logit difference: the difference in og-probabillites assigned by the model to the 1O and S names. This metric more sensitive than accuracy, which makes possible the detection of individual model components (or interventions thereof) with a consistent but non-pivotal role in the task. Accordingly, we also use the logit difference throughout this work evaluate the causal effect of fine-grained model interventions.
Measuring interpretability with the F1 score. Following prior work (Bricken et al. 2023)
assign interpretability scores for SAE latents using precision and recall, which we combine in the SAE latent active
Fi score. Specifically, given set of examples S used for evaluation, subset F C S of examples, and a potential interpretation in the form of a subset A C S (e.g.. the 10 name being 'Mary"), we can evaluate the interpretation's precision and recall with respect f: recall(F,, A) = |A n F1/ / |A] and precision( F, A) = |A n F1 [| We combine them into a single number using the F1 score (see Appendix 7.4 for some discussion of this metric):
procision (F, A) recall(F, A) F1F A) precision( F, A) + recall F,AA)
4 COMPUTING AND EVALUATING SUPERVISED FEATURE DICTIONARIES
This section is a self-contained presentation of our supervised feature dictionary methods and results. In the next section we will extend these methods to SAES.
4.1 PARAMETRIZING THE IOI TASK VIA ATTRIBUTES
Given a distribution over IOI prompts ~ D, we can form the sets Names Namess of names that appear as, respectively, the IO and S name in the prompts from supp(D). In a slight abuse of notation, we define functions
10: p Namesto, S p Namess, Pos : {ABB, BAB}
that assign the values of 10, S and Pos each prompt ('ABB' means the 10 name appears first, 'AAB' means appears second). Together, these three attributes define a parametrization IOI prompts.
4

---

Published as a conference paper at ICLR 2025
Why use these attributes? Not every set of attributes will result in good approximation of the model's 's internal activations; in fact, we find that the choice of attributes is a crucial modeling de- cision. We chose the I0, S and Pos attributes because they are the ones found to be key to the IOI circuit in Wang et al. (2022). We experimented with other choices of attributes, but did not find them to be more successful in our tests (see Appendix 77.0 for ) We emphasize that there are many other imaginable choices of attributes; see Appendix 7.12 for further discussion.
4.2 COMPUTING SUPERVISED FEATURE DICTIONARIES
Motivated by the linear representation hypothesis, we conjecture that model activations can be lin- early decomposed as sums of features corresponding to the attributes. Specifically, given activations (p) == ae (p) E IRd of given model component C. (e.g., outputs of an attention head) for IOI prompts p ~ D, there exist vectors (specific to the component C, which we omit from the notation for brevity) Uro : Ug my Upas=z E Rd for each possible value x, u = of the respective attributes, such that
(p) E [a()]] 110=10(p) + Us=S(p) + PPso=PPs(pp) := a(p)
where ä(p) is the reconstruction of a(p). Note that Equation 2 is quite similar, but less expressive, than SAE reconstructions (Equation D. it effectively requires given feature always appear with the same coefficient in reconstructions. This is reasonable assumption in settings like IOI, where we expect the relevant features behave as binary, on/off switches, as opposed to having continuous degrees of activation The expectation of a(p) is analogous to the decoder bias in an SAF.
To compute optimal values for the attribute vectors U10=z.-.. we considered several approache + Our most effective method for the IOI task is straightforward: to compute e.g. u1o==. we take the conditional expectation over the event 10 (p) = x and subtract the overall expectation. Formally,
uIo= Ep-D |a (p) Ip(p) 2] - Ep-D (a (p)].
We refer to this method as mean feature dictionaries. One can prove that, in the limit of infinite data, the mean features for an attribute not linearly detectable the activations will converge to zero (Appendix 7.7
4.3 EVALUATING SUPERVISED FEATURE DICTIONARIES
Intervention notation and locations. As above, denote by a(p) ag (p) the activation of a single model component C (c.g., output of a single attention head) that we omit for brevity. We denote intervention that runs the model a(p) f v and continues the forward pass (often called activation patching (Vig et al. 2020)). Note that such interventions can be applied simultaneously to multiple model components, but upstream interventions may be 'overwritten' by downstream ones. In what follows, given an intervention intv, denote the logit difference of the model after the intervention logitdiff (intv).
prompt p but replaces the activation a (p) with v as
In all evaluations, we focus on cross-sections of the IOI circuit as identified by Wang et al. (002)) These cross- secsionn are groups of attention heads that serve shared computational role, such as detecting name repetitions or moving information between positions. They represent key informa- tion bottlenecks in the model's computation where variables are passed between different parts of the circuit (described in more detail in Appendix 7.3 also see Figure 2) We intervene on entire cross-sections instead of just single heads in order to observe stronger causal effect.
Sufficiency and necessity of reconstructions. The first set of evaluations measures the faithfulness of activation reconstructions ā(p). It is closely related to prior work (e.g.. the 'cross-entropy loss recovered' metric used in many SAE works (Bricken et al. 2023; Raaaaaanhharan et al.) (024))- Here, we consider natural IOI-specific versions of this metric, and complement it with a measure of how much information is lost when reconstructions are removed.
In particular, we found that another set of attributes, though more expressive in principle, Icarss approx- imate the features get using the S. 10 and Pos attributes through -ariaaee--aaiaabll--iiie transformation.
"See also the discussion of 'features as directions' vs 'features as points' in Tamkin et al. (023).
See Appendix 7.8 for more details other approaches we considered.
5

---

Published as a conference paper at ICLR 2025
1.0
1.0
0.8
0.6
Intervention Supervised
0.2
0.2
Task SAE distribution
0.0
0.0
In DT out
S-1v
S-1
[UINM
(UINM ok.
18INM
Ind-DD
150NM
(50NM
111NM
Figure 3: Sufficiency (left) and necessity (right) evaluations of reconstructions of cross-sections of the IOI circuit computed using supervised feature dictionaries, task- and full-distribution SAEs. Left: average logit difference when replacing activations cross-sections of the IOI circuit with their reconstructions, normalized by the average logit difference over the data distribution in the absence of intervention (a y-axis value of is best). Right: drop in logit difference when deleting reconstructions, normalized by the respective drop when performing mean-ablation, and linearly rescaled so that values close to are best. See Appendix 7.5 for details.
Sufficiency. We check if reconstructions are sufficient to do the task using the intervention a(p) <- (p). i.e. replacing activations a(p) with their reconstructions a(p) at circuit cross-sections. To normalize values, we report the metric
sufficinncyscore:: = E [logitdiff (a(p) + a(p)))
EpwD [logiddiff (id)]
where logitdiff (id) is the logit difference of the model in the absence of intervention, with expec- tation 2 3.3 over our dataset. Results are shown Figure (left, orange bars
Necessity. We check if the reconstructions are necessary for the model to do the task. To do this, we intervene by replacing activations ap)) with their average plus the SAE error term:
necessity-ine -nttvvtyyyntvvvie a(p) Ep-D [a(p)] + (a(p) a(p))
We measure the drop in logit difference that results from this intervention compared to interven- tion, and compare to the corresponding drop when using the mean-ablation intervention to normal- ize
E [logitdiff (necessity-intyy - logitdiff (a + Ep-D [a(p)))
necessity score
E [logitdifT (id) - logitdilf (a <- Ep [a())))]
Results are shown in Figure absence of intervention.
3 (right, orange bars), rescaled linearly versus the logit difference in the
Control. This evaluation measures the degree to which the supervised feature dictionaries disentan- gle the different attributes | Specifically, supervised dictionaries suggest a straightforward way to edit the model's internal representation of the I0, S and Pos attributes via feature arithmetic. For
*We note that report fractions of average logit differences; this ignores the distribution of logit dif- ferences across prompts, and raises the possibility that few examples with very high logit differences may dominate the final score. However, in practice we observe that logit differences are concentrated close to their means.
3 'The intuition for this is mean-ablation over the distribution keeps shared information grammar, syntax and non- 1O1-specific semantics intact, which keeps the model somewhat on nn-distribution, but erases I01- specific data. If the error term has useful information for the task, we expect the logit difference to drop less when intervene with the error term than when we intervene with the mean-ablation.
"Note that, to meaningfully conclude disentanglement of attributes, they must be 'entangled" in the first place. This is true in the 10I task, where many circuit locations represent the S and Pos information simultane- ously. We also find that particular location, the queries of the L.0HH name mover head, represent all three attributes I0, and Pos.

---

Published as a conference paper at ICLR 2025
example, to change the value of the 10 attribute in some activation (p), can define the edited activation
a10~ (p) == ap) + (μio=z 110=100p))
We want to check the extent to which this edit changes the 10 attribute and only the 10 attribute. We measure this in two ways:
probing accuracy (correlational effect): we train linear probes to predict the I0, S and Pos attributes from the activations a(p) of a single circuit location, and then we check whether their predictions are consistent with the edits. We find these probes have high accuracy overall w.r.t. both the attribute being edited-in, as well as the attributes that should be fixed (see Appendix Figure Tog the results).
edit accuracy (causal effect): we measure the fraction of edits where an intervention such as ap) <- a10 (p) results in the same next-token prediction as the 'ground truth' intervention app) f a(Pio-z). Here Pio-= is the prompt p where the IO attribute is set to z, but all other data is kept the same. This is 'ground truth' because it directly patches the 'correct' activation our edit attempts to approximate. Results are shown in Figure 4 (red shapes, x-axis); some additional results are given in Appendix Figure 28 (orange bars).
Sparsity of supervised feature interactions. Supervised feature dictionaries are in way emph- tautologically interpretable: they were defined to have a single feature activating for each possible value of each of the 3 attributes. In particular, the Fi score of a supervised feature with respect to its ground-truth interpretation (e.g., Io='John') will be 1 by definition. Accordingly, we performed a more demanding test of interpretability: decomposing internal model computations in terms of interactions between individual fcatures. We find that pre-softmax attention scores and composition between heads can be decomposed in terms of feature-level interactions, such that many interactions are close to zero, and the few non-zeros correspond to those expected based on the high-level IOI circuit description from Wang et al. (2022); see Appendix 7.9 for details, and especially Figures and 9
Having established our supervised feature dictionary framework and demonstrated effectiveness on the IOI task, we turn to the challenge of evaluating SAEs using this methodology.
5 EVALUATING SPARSE AUTOENCODERS
5.1 SAE TRAINING METHODOLOGY
We trained SAES using range of architectures and datasets. A central distinction in our experiments is between task SAES trained on IOI dataset activations, and full-distribution SAEs trained on activations from the LLM's pre-training dataset OpenWebTes. (Gokaslan & Cohen 2019).
For most experiments reported in this paper, we used the 'vanilla' SAE architecture following Bricken et al. (2023) with task SAEs. On the whole, we found vanilla task SAES good at suffi- ciency/necessity, but struggling to control the model effectively.
This prompted us to experiment with more recent SAE architectures such as gated (Rajamanoharan al. (224)) and topK (Gao et al., 2024) SAES, as well as better optimize how we choose SAES for control. The specific details of these architectures are not relevant for this paper, and we refer the reader to the original papers for more details. For our purposes, SAE architectures used come with notion of reconstruction
bias term plus a linear combination of a subset of SAE latents. For the training methodology of these newer variants, we adopted standard practices from recent works (Bricken et al.) 2023 Rajamanoharan al. 2024 Gao al. 2024). Exhaustive details on SAE training algorithms and hyperparameters are given in Appendix 7.13 We found that these new architectures significantly increased performance in the sparse control evaluation.
5.2 CHALLENGES IN SAE EVALUATION
Our goal now is to extend the methodologies used to evaluate supervised feature dictionaries from Subsection 4.3 to deal with SAE latents. This involves two main challenges:
7

---

Published as a conference paper at ICLR 2025
interpretation-agee edits
Interpretation-warre edits
1.0
Edit Magnitude (Welght Removed)
Vanilla (inteppretationawaard
Gated (Inteppretation aw ware)
Topk (Interpretation aware)
36
0.6
Supervised
and Location
(B)NM out Pos in (B)NM ak
0.4
Pos in (BJNM
Pos out
0.2
Pos
Pos in Ind--D out
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Edit Accuracy
Edit Accuracy
Figure 4: Activation editing results, showing the trade-offs between edit accuracy and edit mag- nitude (better is down and to the right). Left: results in the interpretation-agnosiie editing regime, where we edit using SAE latents best each individual edit. Right: results the interpretation- aware editing regime, where edit using SAE latents with high FI score for the attribute values being edited. Each color represents an SAE variant, and each shape cross-section of the IOI circuit. See subsection 5.4 for more details on the editing regimes.
No built-in interpretations: unlike our supervised features, which by definition correspond val- ues of the IOI attributes, SAE latents do not come with built-in interpretation. This means that we must perform some kind of labeling/search over SAE latents when interested e.g. using SAE latents to control the model;
Multiple good solutions may exist: even if the SAE latents not correspond a 1-to-1 manner to the supervised features, they may still be useful for approximating and controlling the IOI circuit w.r.t. the attributes I0, and Pos we have chosen. A qualitative discussion of some concrete ways in which this might occur given in Appendix 7712] This means that we should not compare SAE latents to supervised features in a head-to-head manner, but rather allow for some flexibility in how SAE latents can be leveraged to achieve the outcomes want.
With these challenges in mind, we now generalize the evaluations from Subsection in the next sections.
5.3
SUFFCIENCY/NECESSIT
We note that the sufficiency/necessityy evaluation applies as-is to SAE latents, as only depends on the unambiguous notion of reconstruction. Results are shown in Figure (green and red bars) for vanilla SAEs. We find that vanilla task SAES are good at sufficiency/necessitty but full-distribution SAEs are not. We hypothesize this because the task SAEs' training dataset (i.e., the IOI dataset) is a much better fit for the IOI task than the full pre- pre-training dataset; this is consistent with our other observations that full-distribution SAEs do worse on the control evaluations.
5.4 SPARSE CONTROL VIA SAE LATENTS
Expressing activation editing as an optimization problem over SAE latents. To generalize the supervised edit from Equation B to the SAE setting, we need a way to pick SAE latents to re- move/add to the clean activation a ap)). In the supervised setting, always have a single feature for e.g. 1Jh'John' which we always add to the activation to change the 10 attribute.
However, in the SAE setting, it may be too demanding expect single feature with this property to exist. For example, multiple SAE latents might be relevant to representing IO='John', but each might activate only in specific contexts.
Thus, to remain agnostic to the exact decomposition the SAE latents, we propose to allow more flexibility to the SAE latents in the control evaluation. Suppose have prompts ps; Pt where
8

---

Published as a conference paper at ICLR 2025
P. is the 'source' prompt that want to intervene on, and pt is the 'target' prompt which expresses the edit we want to make; for example, we could have p: = (ps (Ps)10== if we want to change the I0 attribute to Let the SAE have a dictionary of decoder vectors {u, J- and the original and counterfactual activations a(ps), a(p:) have reconstructions respectively a(p.) Eies a + baec; ā(pt) Eier 3,u, + bate for S,T C [l.....m and a; > 0. We frame activation editing as the optimization problem
min RC R.CSAACT,IRAASS
a(ps) Σa,u, + Σ B.u. - a(p:) EEA
(4)
iER
In words, this problem asks for at most & features to remove (R) from and/or add (AA) to the original activation to bring it as close as possible to the counterfactual activation, where the features to add taken directly from the counterfactual one (we also experimented with using the reconstructions instead of the actual activations in this algorithm, but results were worse). In general, this problem has no polynomial-time it; instead, we use a greedy algorithm to find a solution.
(in k. [S] (T') solution, as the NP-hard problem SUBSETSUM reduces to
We stress that this formulation is heuristic, but can be readily replaced with a more principled one; we chose it because it is intuitive and cheap to optimize. Another natural choice would be to optimize the logit difference downstream of the intervention to match that of patching the true counterfactual activation directly.
Preventing complete activation overwrites by measuring edit magnitude. When editing acti- vations, we must ensure our edits are minimal and targeted. A concerning failure mode would be to completely overwrite the original activation's 's features with those from the target, which would demonstrate no true disentanglement of attributes. Furthermore, depending on the SAE's structure, this situation can arise even if we remove latent and add latent back
To measure the magnitude of the edit, we compare the contribution of the changed features to the reconstruction against the analogous quantity for our 'ideal' supervised feature dictionary. Namely, for each summand f;u in reconstruction á, we assign a measure of its contribution
(5) Note that weights are additive in the features, so that the sum of weights of some subset of features is the weight for these features' total contribution to the recon- struction. We then measure the magnitude of an edit by the total weight of the features removed during the edit.
(1i == (fru.) (a bdec) / | baeell
so that Ex= w1i =
Results. We instantiate the optimization objective from Equation in two main ways:
interpretation-agnonne here, optimize the objective over all SAE latents active in the source and target activations. This is the most flexible regime.
Interpretation-awware here, pick SAE latents in the order of F1 score w.r.t. the relevant attribute value. For instance, if we want to change activation a (p&) with I0='John' to activation a (pr) with MOr'M,ry', we first order the SAE latents active in a(p,) (resp. a (p:)) by their Fl score for 10='John' (resp. 10='Mary"), ) and then remove the top k/2 features from a(p.) and add the top k/2 features from a(p.).
Results on the trade-off between edit accuracy and edit magnitude for l: = 2 are shown in Figure 4 for task SAE variants. Note that A = 2 is the smallest meaningful value we can use, and it has The same expressive power as our supervised edits, leading to more fair comparison. Furthermore, topk SAES allow us to directly set the sparsity level of the latents, and we chose 3 active latents per activation for this evaluation. This further matches the supervised feature dictionaries, which also have 3 'active" features per activation. Results for probing accuracy with topK SAES (k 3) are shown in Appendix Figure 16
7 As simple but extreme example, consider an SAE which has a dedicated latent for each possible combi- nation of values of the 10. S and Pos attributes (this can be achieved with ~ 10 features in setup, far more than any SAF we train). Such SAF can represent every IOI example perfectly (modulo random variation beyond the attributes) with just active feature.
'While weights can in general take any real value, we find that in practice they are almost always approxi- mately in (0, 1}; see Appendix (.nenndix7.18faa for empirical details.
9

---

Published as a conference paper at ICLR 2025
We find that vanilla SAEs are markedly worse at this evaluation than supervised dictionaries, but gated and especially topK SAEs are competitive with supervised dictionaries, especially in the iecepretation-agnostit
regime. Further results for vanilla task SAEs with higher k are shown in Appendix Figure 28 (for task SAES). We find that higher k values lead to only marginally better performance. Full-distrbuttin vanilla SAEs do even worse on this task (Appendix Figure (27)- As a control condition, we also consider running the same experiment with an SAE where the decoder vectors were frozen initialization; this leads significantly worse performance (Appendix Figure indicating that the SAE latents are indeed learning useful information for the task.
26
5.5
INTERPRETABILITY
To evaluate SAE latent interpretability, we frame an 'interpretation' as a subset A of the IOI dis- tribution's ' support, assigning interpretability scores via the Fi score between a latent's active set F and A. To avoid trivial interpretations where each latent simply matches its own active set, we restrict A to meaningful human-defined subsets, including our primary attributes (IO, S, Pos), their intersections, and unions of attribute values. Given a latent, we pick the interpretation among these with the highest F1 score. Details on the interpretation methodology are given in Appendix 7.14
Our analysis reveals that many SAE latents have clear interpretations (F1 > 0.8) consistent with their circuit locations, as shown in Appendix Figure 15 for full-distribution SAEs and 29 (among others) for task SAEs. We note that these experiments are of a rather exploratory and qualitative nature; the main conclusion we draw is that there wide variety of interpretations for the SAE latents.
6 DISCUSSION, LIMITATIONS AND CONCLUSION
Discussion. We have taken steps towards more principled and objective evaluations of the usefulness of sparse feature dictionaries for disentangling LLM activations. In particular, we have demonstrated that simple supervised methods can be used as a principled way to compute high-quality feature dictionaries in a task-specific context, and that these supervised dictionaries can be used as 'skylines' to evaluate and contextualize the performance of unsupervised methods, such as SAEs. Moreover, while we have focused on the IOI task because of how well-studied is, we have shown proof of concept that our methods can be generalized straightforwardly to new tasks and models, as well as new distributions (Appendix 7.1.
Limitations and Future Work. A central limitation of our approach is that we impose the set of attributes we want to evaluate SAEs against. This influences many of our evaluations, in particular the 'sparse control evaluation' from Section [5.4] which relies on these attributes to compute the counterfactual activations used to evaluate edit accuracy This is somewhat mitigated by the fact that we use attributes shown to be consistent with the computation of the IOI circuit, and that we aim to be agnostic to the precise SAE representation of the task. We believe this assumption (conditioned on the supervised dictionaries successfully passing tests) is reasonable middle ground between assessing the usefulness of SAES for model behaviors humans understand and wish to control, and evaluating SAEs fairly.
Another major limitation of our framework is that it deals with quite templatic tasks. We hope that future work can bridge this gap. For example, we find the approach of Gao al. (2024) for evalu- ating SAE latents as probes for binary classification tasks to be a promising direction for building more fine-grained supervised feature dictionaries.
Finally, a third limitation is that it takes work to identify in way independent of SAEs what good attributes for given task are. We show through two case studies in Appendix 77. that, by targeting only key attributes in a single activation location, we can reduce this effort substantially. This comes the cost of analyzing more limited, but still potentially interesting slice of the task. However, we admit that these tasks are still somewhat contrived/templaicc For truly open- tasks, we believe that methods to generate 'concept vectors' at scale, such as the ones given in Luo et al. (2024); Zou et al. (2023), may be promising direction for obtaining starting point for "ground-truthh feature dictionaries.
However, note that the sufficiency and necessity evaluations are independent of the choice of attributes, as they evaluate SAE reconstructions as whole.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank Sonia Joseph and Can Rager for feedback on earlier versions of this manuseript. We thank SERI MATS program for making this research possible. A.M. and G.L. were supported by grants from AI Safety Support and Effective Ventures.
AUTHOR CONTRIBUTIONS
AM designed and ran the algorithms for computing supervised feature dictionaries and observed their theoretical properties; trained task-specific SAEs; designed the tests for suffcienccy/eccessity of reconstructions, sparse controllability and causal faithfulness of feature descriptions; ran these tests full-distribution and task-specific SAEs; observed the acclusion/overrplittngg phenomena in task-specific SAEs and developed the toy models for them; wrote all sections of the paper unless otherwise specified.
GL trained full hull-distribution SAEs; designed and ran the automatic feature scoring algorithms for them, and ran most of the interpretability evaluations for full-distribution SAEs; contributed the sections on interpretabilityy of full-distribution SAE features, created Figures [nd and 15
NN provided guidance on research goals and prioritization, well writing, throughout the entire project; proposed the initial idea of finding supervised feature dictionaries for the I0I task, and using them to evaluate (unsupervised) feature dictionaries; refined the search for algorithms to compute the supervised dictionaries.
REFERENCES
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic struc- ture of word senses, with applications to polysemy. Transactions of the Association for Compu- tational Linguistics, 6:483-495, 2018.
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/ 1677.66450 2016.
http://wwa...ccoooooooooooooooaa ott.accc.....a.o.ooooor.aa.aa11. 82336311
Nora Belrose, David Schneider-Josephh, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure closed form. ArXiv, 0ts/2306.038199 2023.
URL https::/appl www.::eeeaanncchheeeeeeeeee
25908885449
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. pttep:://openapplie feeuroa-expllainer/pppprrr///
blob. core. windows.
html 2023.
Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wattenberg. An interpretability illusion for bert. arXiv preprint 1IXi::2104.07143, 2021.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Tumner Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, 7ac Hatfield-Dodss Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https:/ttrassformer sicuuis.pub22223mmoooeemma htwp://22/mnosssamiaiaaaaaauaida
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Maheep Chaudhary and Atticus Geiger. Evaluating open-source sparse autoencoders on disentan- gling factual knowledge in gpt-2 small. arXiv preprint ArXiv:2409.04477 2024.

---

Published as a conference paper at ICLR 2025
Dami Choi, Vincent Huang, Kevin Meng. Daniel D Johnson, Jacob Steinhardt, and Sarah Schwettmann. Scaling automatic neuron description. neezonndescriptioon October 2024.
Tom Conerly, Adly Templeton, Trenton Bricken, Jonathan Marcus, and Tom Henighan.
Update on how we train saes. Transformer Circuits Thread, 2024. URL https:
//Ezansformerrcirooiii htt.://wwwaaaaaaaooooooo//////aa training-saes
html#
Arthur Conmy and Neel Nanda. Activation steering with SAEs. Alignment Forum,
2024. URL https: //t::://wwwww/ http::/wwwwalilnnnnm.m alignmentforum org. /posts/ cXxAzoib3zzzpeyrg progress-update-1 Progress Update #1 from the GDM Mech Interp Team.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600) 2023.
Jacob Devlin, Ming-Wer Weig Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 4171-4186 Minneapolis, Min- nesota, 2019. Association for Computational Linguistics. doi: 10. 10.18663/11/111-42222 URL https: http://acaantthhho woa.thooogy..o.. /oggNN9-1/900
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Gan- guli, Zac Hatfield-oodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. /ttauu//222/// htwe///frmwwor....o wnnex.ndiiee html
URL
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac HatfieldDDodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of super- position. Transformer Circuits Thread, html
/
circuits.
pub/2022/ ttt_ttoooomoooooeeeeeeeoeeete
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodd Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposi- tion. arXiv preprint arXiv:2209. 10652, 2022b.
Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A. Smith. Sparse over- complete word vector representations. In Annual Meeting of the Association for Computational Linguistics, 2015. URL https://api. semanticscoolar: wwwpaarro//ooooooooppp 9397697
Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's image representation via text-based decomposition. arXiv preprint 1xXiv:23100059// 2023.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406. .,0934 2024.
Gabriel Goh. Decoding the representation of code in the brain: an fmri study of code review and expertise. 2016. URL http://///aaaggggggg.......o//aa
Aaron Gokaslan and Vanya Cohen. Openwehtext corpus. http::///k/11oo0000.ggtto.. @penWeebTextCrrpus 2019.
io/
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bert- simas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint 1oXiv:23050016/0 2023.
12

---

Published as a conference paper at ICLR 2025
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Inter- preting mathematical abilities in a pre- te-itraind language model. Advances in Neural Information Processing Systems, 36, 2024.
Shlomo Hoory, Nathan Linial, and Avi Wigderson. Expander graphs and their applications. Bulletin of the American Mathematical Sociery, 43(4):439-565 5611 2006.
Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. Ravel: Evaluat-
ing interpretability methods on disentangling language model representations. arXiv preprint 27X1::2402.17700, 2024.
Caden Juang, Gonçalo Paulo, Jacob Drori, and Nora Belrose. Open source automated interpretability for sparse autoencoder features. EleutherAl Blog, July 30 2024.
Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Riggs Smith, Claudio Mayrink Verdun, David Bau, and Samuel Marks. Measuring progress in dic- tionary learning for language model interpretability with board game models. In ICML 2024 Workshop on Mechanistic Interpretability.
Connor Kissane, Robert Krzyzanowski, Arthur Conmy, and Neel Nanda. Attention saes scale to gpt- small. Alignment Forum, 2024. URL http:p///www...aiaoomenttttttro.
F ESTRedt 4Gfabb
Robert Krzyzanowski, Connor Kissane, Arthur Conmy, and Neel Nanda. We inspected every head in gpt-2 small using saes so you don't have to. Alignment Forum, httpp::/ww.....gneenteoo..
wttp//psste/eeeeeeeotosste 2024. URL https: wwwp. al www.aanennnnnnnaaaeee
we eee--nppeceeed-eery--yeeee head-
sma l-us www.uuiinnnsssssseeeee so-
Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Miku- lik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.09456 2023.
Jinqi Luo, Tianjiao Ding, Kwan Ho Ryan Chan, Darshan Thaker, Aditya Chattopadhyay, Chris Callison-Burch and René Vidal. Pace: Parsimonious concept engineering for large language models. arXiv preprint arXiv:2406.04331, 2024.
Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint SttT:22003.1944777 2024.
Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. arXiv preprint -xXx::2307.15771, 2023.
Neel Nanda, Andrew Lee, and Martin Wattenberg, Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941, 2023.
Chris Olah. Mechanistic interpretabilityy variables, and the importance of inter-
pretable bases. https//www.rranssoorrrrrrruttuts 2022. URL https: www.
aaaaaeforoo-crrruis...p..u 2022 http:///2222mmchhiiieeee http:///....../////mmeeeeeeeeeae
Christopher Olah. Interpretability dreams. Transformer Circuits Thread, 2023. https:/transformer- hrtp://shhhh222//mmmeeeeeiiieeee
Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A
strategy employed by v1? Vision Research, 37:3311-3325, 1997. URL https://api.
semant inmmantcscholioo ar. org/ Co: 14086692
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Haffield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Ka- plan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Trans- former Circuits Thread, 2022. URL httpsss//traassooneeeee wwweerrsrrrrsss... in-context -emexxlllearae learning and incl-iiducciin htw.:/aaanddddddnnnioooeeee////e pou.r//mmmoorrrrrurruouuo///22 pub/2022/
html
13

---

Published as a conference paper at ICLR 2025
OpenAl. Gpt-4 technical report, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoen- coders. arXiv preprint arXiv:2404. 16014, 2024.
Lee Sharkey, Dan Braun, and Beren Millidge. of superposition with sparse autoencoders.
Taking
features
out
2023.
URL https:
www. www.:allggiitt /tw..alignneet..o.o...mmmoo htt.s//www....ooooptt////ttttaaa interime rww.arch-eeaeeeeeeeee report---epttttttaaaaaaae -ng featureseeeeeeese out-of----oooo www.:ofsssprppsttiiiooo ott.ruurrrrottto----peppppppaaaa
Alex Tamkin, Mohammad Taufeeque, and Noah D Goodman. Codebook features: Sparse and
discrete interpretability for neural networks. arXiv preprint arXiv:2310.172.30 2023.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Trans- reuutst...... pttpttppppbb222400
former Circuits Thread, 2024. URL phttps://trassffiiet
scaling www.mmoonoooiiieeeteeeee hte///inxxxxx/////// html
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL /tttpps neuri 3f5ee24354 7e Fbd053c cetpo0053c1444488530000 c4a845aa Abst htm
Jesse Vig. Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias. arXiv preprint 2XXv:2004412263: 2020.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In
The Eleventh International Conference on Learning Representations, 2022. URL https:
//openreviee ew. net./ watt/forrm?/// wtt.:e/oooommdddaa///eaaaaeee N-NSVSSN6NNN
Zeyu Yun, Yubei Chen, Bruno A. Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: contextualized embedding as linear superposition of transformer factors. In Workshop Knowledge Extraction and Integration for Deep Learning Architectures; Deep Learning Inside Out, 2021. URL lhttps://app sewanticscheieeee wwwphhhooo....rraoroooo Oowg/ooorroIii...oooooo
232417301
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazcika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405,2000 2023.
2023.
7 APPENDIX
7.1 GENERALIZING OUR METHODS TO OTHER MODELS, DISTRIBUTIONS, AND TASKS
In this section, we detail two new tasks and datasets we used to showcase the generalization of our methods.
14

---

Published as a conference paper at ICLR 2025
7.1.1 THE GREATER-THAN TASK
First, we ran -rconeet- for the "greater-than" task (Hanna al., 2024 to edit the attribute denoted "YY" in the outputs of 7 attn heads in the greater-than circuit.
Task, dataset, and model. We the greater-than task as-is, with the template
'The SNOUN lasted from the year 17$YY to the year 17
where YY is two-digit number that concatenates with 17 to form year, and restrict years to be between 1701 and 1720. We intervene on the outputs of the attention heads L5H1, LSH5, L6H1, L6H9, L7H10, L8H8, and L8H1 I in order change the attribute YY. The LLM used is the same as in the main paper, the GPT-2 Small model. We sample several thousand prompts for training and testing.
SAE training. We train vanilla task SAEs with an & regularization coefficient of 5.0; we follow the methodology Bricken et (2023) Conerly al. (2024) for training the SAEs.
Sparse control results. Exchanging as few as 2 SAE latents matched next-token predictions 90% of the time when patching attention heads with counterfactual activations (compared 38% for no intervention; this is binary classification task). Results are in Figure 3
Edit accuracy
0.6
0.4
0.2
4
8
16
Number of features exchanged
Figure 5: Results for sparse control on the greater-than task. We show the accuracy of next-token predictions (against counterfactual labels) after patching attention heads with SAE latents, compared to patching with ground-truth counterfactual activations.
7.1.2 THE 'BOTH' TASK
To showcase the flexibility of our methods, we have evaluated them in setup that departs in several major ways from the main results reported in the paper:
We evaluate on a new task in-context learning task we created for this paper. The task is based setting up prompt where the model can apply induction (i.e., predict that the next token is the same as some earlier token in the context) in one of two ways; our experiments are centered on disentangling the two different ways the model can apply induction.
We evaluate a new model and pre-training dataset, the 4-layer 33M-parameter model trained on the Tiny Stories dataset (Eldan Li 2023).
Whereas for most of the paper we have focused task-specific SAEs, here we evaluate topK SAES trained on the same pre-training dataset as the model (Tiny Stories).
This section is proof of concept that methods be straightforwardly generalized to models and datasets.
Task. The task uses template of the form
Once upon a time, there live a kid named SNAME. SNAME really loved SANI- MALSS. SNAME also really loved SSPORT. So $NAME really loved both
15

---

Published as a conference paper at ICLR 2025
where correct next-token prediction is either 'SANIMALS' or 'SSPORT'; we refer to these the two attributes for the task.
In our experiments, we sample set of 1000 prompts where we randomly and independently sample from a set of 25 names, 6 animal species, and 9 sports. We observe the following key properties of the task:
While at temperature 0 the model will always prefer to start its completion with one of the two attributes (e.g.. generate "cats and football"), we can still measure that at the "both" token the model places higher probability on the correct attribute values compared to in- correct ones.
Specifically, we define the restricted logit accuracy with respect to an attribute, which is the fraction of examples where the correct attribute value has the highest logit among all values for this attribute. For example, for the 'animal' attribute, an example where SANIMALS='cats' will be considered accurate if the logit for "cats" is higher than the logit for each of the other 5 animal species in our dataset.
We find that in our dataset, the restricted logit accuracy for sport 85.5%, and for animal is 75.9%. Moreover, we find that the residual stream representation of the 'both' token at the last (4th) layer of the model is causal w.r.t. this metric. Specifically, we can activation- patch this location from prompt where we keep one attribute fixed but change the other. We observe that the accuracy w.r.t. the attribute value being patched-in is 77.1% for animal and 85.6% for sport, while the accuracy w.r.t. the attribute that is fixed remains mostly unchanged (at most 1 2% difference from clean).
Supervised feature dictionaries. We create supervised features for the two attributes like in the main paper by using conditional expectation over examples in the dataset having a given value of the attribute.
We observe that the supervised features are able to recover the causal effect of the attribute value being patched-in on the restricted logit accuracy, suggesting that we have found meaningful features for the task. Specifically, when we edit the animal attribute, the restricted logit accuracy w.r.t. the animal being edited-in is 73%, while the accuracy w.r.t. the (fixed) sport attribute remains high at 82.8%. Similar results are found when editing the sport attribute.
Sparse control with TopK SAEs. We train topK SAFs with & E (32, 64) and number of latents in (8192, 16384). We find that these SAEs achieve a good value of variance explained in the activa- tions.
We perform a variant of the sparse control evaluation from Section 5.4 where we edit each attribute in isolation, using counterfactual prompts where only this attribute has changed. This is analogous to the intervention we use to evaluate our supervised feature dictionaries.
We explore changing number of SAE latents in (2,4, 2, sufficient to recover the full causal effect of our supervised feature dictionaries. Results are shown in Figure 6
6, 8}. and observe that this intervention is not
7.2
QUALITATIVE PHENOMENA IN SAE LEARNING
7.2.1 FEATURE OCCLUSION
Our experiments suggest that when two (causally relevant) attributes are represented in the same activation, but one attribute's (supervised) features have overall higher magnitude, SAEs have tendency to robustly learn more interpretable features for the attribute with higher magnitude. We observe this in the queries of the I.10HO name mover head at the END token in the IOT circuit, where the 10 and S attributes are both represented and causally relevant (ablating either leads to change of x5 0.5 in logit difference). We find that our task SAEs consistently find features with high Fi score for individual I0 names, but fail find significant number of features for individual S names. In Figure (left), we show interpretability results from training SAES over a wide grid of hyperparameters that confirm this observation; more details in Appendix 7.16
Hypothesis: feature magnitude is a driver of occlusion. We noticed that the supervised features for 10 and S names in the L10HO queries have small but significant difference in norm (see Appendix
16

---

Published as a conference paper at ICLR 2025
Accuracy Changes When Editing Animal
Accuracy wrt. original labels (K-32, D-8192)
Accuracy counterfactual labels D-8192)
Accuracy wxt. original labels (K-32. D-16384)
Accuracy counterfactual
D-16384)
Accuracy w.r.. original labels (K=64, D=8192)
0.0
Accuracy
Accuracy w.r.t original labels (Km64, D=16384)
Accuracy counterfactual
D=16384)
0.5
0.2
0.1
Number of Exchanges
Accuracy Changes When Editing Sport
0.8
0.6
Accuracy wrt. original labels (K-32, D-8192)
Accuracy counterfactual
Accuracy w.t.. original labels (K-32, D-16384)
Accuracy
D=16384)
.... Accuracy wrt. original labels (K=64, D=8192)
Accuracy
Accuracy wrt. original labels (K-64, D-16394)
Accuracy wxt.. counterfactual (K=64, D-16384)
Number of Exchanges
Figure 6: Sparse control results for the new task. We plot how the restricted logit accuracy w.r.t. the original (blue) and counterfactual (orange) attribute values changes as edit more SAE latents. The accuracy w.r.t. the other attribute is not shown as it remains high and nearly unchanged.
Figure 13 (left)). We then hypothesized that feature magnitude is a factor in this phenomenon. To check this, we surgically reduce the magnitude of IO features in the activations using our supervised feature dictionaries, and observe that the number of S features discovered in these modified activa- tions monotonically increases as we remove larger fractions the 10 features (Figure 7 (right); see Appendix 7.16 for methodology). We furthermore constructed simple toy model based on i.i.d. isotropic random features that mimic the norms supervised features in the L10HO queries, and find that similar phenomenon occurs for the distribution of features with high F; score (e.g. higher than 0.9; see Appendix 7.16).
7.2.2 FEATURE -VER-SPLITTING
We also found in our experiments that task SAEs have tendency to split single binary attribute into multiple features, even when the number of features available could in principle be spent on other attributes. Note that. while this behavior may be counter-intuitiv from human standpoint, it does not necessarily mean that the SAE failed; may that the binary attribute not part of an optimal sparse description of the model's internal states. We observe this phenomenon with the Pos
17

---

Published as a conference paper at ICLR 2025
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.4
1.0
0.81
0.82
0.84
0.85
0.05
0.05
0.09
0.10
0.2
0.38
0.57
0.63
0.63
0.01
0.01
0.06
0.06
0.2
0.04
0.04
0.13
0.44
0.54
0.00
0.00
0.00
0.00
0.1
10
0.0
0.2
Figure 7: Left: Fraction of 10 (left subplot) and S (right subplot) names in our dataset for which feature with score 0.5 found, function of dictionary size (x-axis) and effective ε regularization coefficient (y-axis), over a wide hyperparameter sweep for the queries of LIOHO. Right: fraction of 10 and names our dataset for which feature with F score 2 0.5 is found, as function of a (z-axis), the fraction of supervised IO features we subtract from the activations.
attribute in the IOI task (again in the queries of the L10HO name mover), which is robustly split into many (e.g. 2 10) features by our SAEs that activate for small, mostly non-overlapping subsets of examples sharing the same Pos value that have no clear semantic interpretation.
Is over-splitting a form of over-fitting? To investigate whether this is due overfitting, com- pared Pos features between (1) different random seeds for the same training dataset and (2) different training datasets. In both cases, found that the Pos features discovered similar above chance levels, suggesting that the over-splitting is not due solely to overfitting to randomness in the training algorithm and/or dataset.
Reproducing the over-splitting phenomenon in simple toy model. On the other hand, we show empirical evidence that in toy model where activations are uniform mixture of isotropic Gaussian random variables, an appropriately randomly initialized SAE with enough hidden features will achieve lower total loss than ideal SAE with just two features corresponding the two com- ponents of the mixture. Such randomized constructions exist for any & regularization coefficient, even in the limit of infinite training data. Details are given in Appendix 7.17 7.17
7.3 ADDITIONAL DETAILS ON THE IOI CIRCUIT
Circuit structure. To refer to individual token positions within the sentence, we use the notation of Wang al. (2022) IO denotes the position of the IO name, S1 and S2 denote respectively the positions of the first and second occurrences of the name (with S1+1 being the token position after S1), and END denotes the last token in the sentence (at the word 'to').
Wang et al. 2022) suggest the model uses the algorithm 'Find the two names in the sentence, detect the repeated name, and predict the non-repeated name' do this task. Specifically, they discover several classes of heads in the model, each of which performs a specific subtask of this overall algorithm. A simplified version of the circuit involves the following three classes of heads and proceeds as follows:
Duplicate token heads: these heads detect the repeated name in the sentence (the S name) and output information about both its position and identity to the residual stream 1o
S-Inhibition heads:
these heads read the identity and position of the name from the residual stream, and output signal to the effect of 'do not attend to this position / this token identity" to the residual stream
Name Mover heads: these are heads that attend to names in the sentence. Because the signal from the S-Inhibition heads effectively removes the S name from the attention of these heads, they read the identity the TO name from the input prompt, and copy the last token position in the residual stream.
1e follow the conventions of Elhage et al. 2021 when describing internals of transformer models. The residual stream layer & the sum of the output of all layers up 1, and the input into layer k.
18

---

Published as a conference paper at ICLR 2025
In reality. the circuit is more nuanced, with several other classes of heads participating: previous token heads, induction heads (Olsson et al. 2022), backup name mover heads, and negative name mover heads. In particular, the circuit exhibits backup behavior (McGrath al. 2023) which poses
challenges for interpretability methods that intervene only on single model components at a time.
We refer the reader to Figure for schematic of the full circuit, and to Wang al. (2022) for more complete discussion.
7.4 Fi SCORE NOTES
An F1 Fi-score of a guarantees that both precision and recall are least 2-a' For example, when a = 0.8 (the value we use in most evaluations), both precision and recall are at least 1.8/1.22 = 0.67. Requiring sufficiently high F1 value is important in order to avoid labeling a trivial feature as meaningful for attributes where |A| is large, because then a feature active for all examples can have high -core..
The Fr score has some limitations in the context of our work:
it does not take into account the magnitude of the feature activations; for instance, a feature that is active for all examples in S but only has high activation values the examples in may have a low Fl score, even though it is in some sense highly informative for the attribute A.
it is very conservative metric, in that it requires both high precision and high recall to be high. For example, a feature with precision 0.5 but recall 0.02 will have an F1 score of 0.04, heavily skewed towards the lower of the two metrics, even though it is in some sense informative for the attribute A.
We hope to address these limitations future work.
7.5 ADDITIONAL DETAILS FOR SECTION
Computing and evaluating supervised feature dictionaries. For each parametrization and each method to compute feature dictionaries, we use 20,000 prompts sampled from our 101 distribution (see Appendix 7.6) to compute feature dictionaries for the query, key, value, and attention output (i.e., attention-weighe weighted values) of the relevant token positions of all 26 heads identified in Wang et al. (2022) (recall Figure 2). We another sample of 5,000 prompts to validate the quality of the feature dictionaries.
Cross-sections of the circuit. Based on the understanding of the IOI cireuit from Wang al. (2022) we identify several cross-sections of the computational graph of the IOI circuit where feature editing expected to have effects meaningful for the task:
outputs of (backup) name mover heads END ((B)MM out): these activations encode the 10 name and write it to the END token of the residual stream. We expect that editing the 10 name in these activations will directly affect the model's ''s prediction, while editing other attributes will not have a significant effect.
queries+ keys of (backup) name movers END (BBNMM qk): the queries represent the name and Pos information, but they are mainly used inhibitory signals for the model, decreasing the attention to the S token II The keys represent information about the I0 and S names: in partieular, the S information combines with the query to inhibit attention the S token.
We expect that editing the S and Pos attributes in both the keys and queries will not signif- icantly hurt model performance, because as result attention to the token will again be inhibited. By contrast, it is unclear what editing the 10 name is expected to do, since its role in the attention computation is not fully described in Wang et al. (0022).
outputs of S-Inhibition heads END (S-I out), values of S-Inibbition heads at S2 (S-I v), and outputs of duplicate token and induction heads at S2 (Ind+DT out): these activations
In addition, we later find that the queries of the LIHHH name mover head also represent the 10 attribute, and serve inhibitory role for it well, decreasing the attention the 1O token.
19

---

Published as a conference paper at ICLR 2025
transmit the inhibitory signal the name mover heads through the residual stream. We expect that editing S and Pos in these activations will lower the model's 's logit difference by disrupting the inhibitory signal, while editing 10 will have no effect.
Evaluating necessity of feature reconstructions: When we intervene the model by removing reconstructions from activations in cross-scctions of the circuit, model performance on the IOI task (as measured by the logit difference) goes down from the clean value Dogitdfffceeau a lower value As we describe in the main text, the ground-truth intervention for removing the features from the activations mean ablation of the corresponding cross-section, which also re- sults in a lower value of the logit difference, logitdiff mean ablation We want to measure the degree to which logildiff intervention approximates logildilf mean ablation in a way that normalizes for different values of logitdill mean ablation across cross-sections of the circuit. We use the following metric to do this:
necessity score = 1 |logitdiff mean ablation logitdiff intervention!
|logitdiff mean ablation llgiiddiff clean
7.6 DATASET, MODEL AND EVALUATION DETAILS FOR THE IOI TASK
We use GPT2-Small for the IOI task, with dataset that spans 216 single-token names, 144 single- token objects and 75 single-token places, which are split 1 across training and test set. Every example in the data distribution includes (i) an initial clause introducing the indirect object (IO, here 'Mary") ) and the subject (S. here 'John') ), and (ii) a main clause that refers to the subject second time. Beyond that, the dataset varies in the two names, the initial clause content, and the main clause content. Specifically, three templates as shown below:
Then, [] and [] had long and really crazy argument. Afterwards, [] said to Then, [] and [] had lots of fun at the [place]. Afterwards, [] gave a [object] to Then, [] and [] were working the [place]. [] decided give [object] to
and we use the first two in training and the last in the test set. Thus, the test relies on unseen templates, names, objects and places. We used fewer templates than the IOI paper [Wang et al. (2022) in order to simplify tokenization (so that the token positions of our names always align), but our results also hold with shifted templates like in the IOI paper.
On the test partition of this dataset, GPT2-Small achieves an accuracy of 91%. The average difference of logits between the correct and incorrect name is as 3.3, and the logit of the correct name greater than that of the incorrect name in & 99% of examples. Note that, while the logit difference is closely related to the model's correctness, being 0 does not imply that the model makes the correct prediction, because there could be a third token with a greater logit than both names.
7.7 PROPERTIES OF MEAN FEATURE DICTIONARIES
Mean feature dictionaries enjoy several convenient properties:
The vectors Uie for an attribute a; do not depend on which other attributes # a, we have chosen to describe the prompt p with.
If an attribute is not linearly represented in the activations, the mean code features Vi. + 0 in the limit of infinite data (see below). In particular, this also holds if the attribute not represented at all in the activations.
This suggests that mean feature dictionaries are robust to the inclusion of irrelevant non(-linearly)- represented attributes, which a desirable property in real settings where we may not know the exact attributes present in each activation. However, mean feature dictionaries the inclusion of redundant attributes, as the lack of interaction between the attributes means that redundant attributes cannot 'coordinate' to reduce the reconstruction emror ||a -
robust to
20

---

Published as a conference paper at ICLR 2025
7.7.1 MEAN FEATURES ARE ZERO FOR NON-LINEARLY REEEEEEEEEERREEEE ATTRIBUTES.
Suppose we have random vector for k-way classification task with one-hot labels Z {z E {0, 13 s.t. ||z|| = 1}. In Section of Belrose et al. (2023), it is shown that the following are equivalent:
the expected cross-entropy loss of a linear predictor T + b for is minimized at constant linear predictor. In other words, the optimal logistic regression classifier (in the limit of infinite data) is no better than the optimal constant predictor (which, at best, always predicts the majority class).
the class-conditional mean vectors E [x the data.
ex] are all equal to the overall mean E [x] of
If we translate this to the context of mean feature dictionaries, we have that logistic regression for the value of attribute a, will degenerate to the majority class predictor if and only if the mean feature dictionaries for all values of this attribute are zero. In the finite data regime, this gives us some theoretical grounds expect that the mean feature dictionaries will be significantly away from zero if and only if the attribute's 's values can be non-trivially recovered by a (logistic) linear probe. As special case, if attribute is not represented in the data at all, we expect the mean feature dictionaries for this attribute to be zero.
7.8 DEFINITION AND PROPERTIES OF MSE FEATURE DICTIONARIES
The main alternative method considered to compute supervised feature dictionaries from at- tributes was MSE feature dictionaries, which use a least-squares linear regression to predict the activations from the attribute values. We note that mean feature dictionaries work well in our set- ting because the attributes we choose in the IOI task are probabilistically independent w.r.t. the IOI distribution we work with. We recommend using MSE dictionaries when attributes are not proba- bilistically independent.
MSE feature dictionaries compute Ua)()= by directly minimizing the Eg reconstruction error over the centered activations:
N
()=,)))))e arg min (nz()=m
Σ A=1
(a(mk) a) Σ Ua.(()=a(()) iEl
N
This objective convex, and is equivalent a least-squaree regression problem: fact, the optimal solutions take a form very similar to the mean feature dictionaries (see below). Furthermore, this objective closely mimics the SAE objective: here, the sparsity is hard-coded, leaving only the & objective.
We next discuss some properties of the MSE feature dictionaries. For brevity, in the remainder of this section we write Uio instead of Ua (+)=
7.8.1
MSE FEATURE DICTIONARIES AS A MULTIVARIATE LEAST-SQUARES REGRESSION PROBLEM.
Let S MM |S be the total number of possible values for all attributes. For each attribute consider the characteristic matrix C₁ E R. Sx of the dataset for this attribute, where
{
if (x()())) = 0j
Cky {
otherwise
for ordering (U.,..., 1 of the values in S,, and let C |C1 C2
CNal RNS
be the concatenation of all characteristic matrices. Also, let A E Nxd be the matrix of activations with rows (*), Then the objective function for the MSE feature dictionaries can be written as the multivariate least-squares regression problem
min UERaxA xd
|A
CUI?
N
where the rows of U are the vectors U.y across all i and E S., with the optimal solution given by
U
(C= CTA
21

---

Published as a conference paper at ICLR 2025
7.8.2 MSE FEATURE DICTIONARIES AVERAGING OVER EXAMPLES.
Using the special structure of the objective, we can also derive some information about the optimal solutions ". Namely, at optimality we should not be able to decrease the value of the objective by changing given u away from its optimal value. The terms containing u in the objective are
1 N
Σ &EP
(k)
(8)
N
a
l
RE
PEP
a (b)
i
REP
where recall that P.. {Ir (p)) = v), and (*) is the residual of a after subtracting the reconstruction using all other attributes & i. Since this value cannot be decreased by changing u we have that it equals the minimizer of this term (holding ((k) fixed). In other words, if we define
f (u)
1 N
Σ kEP..
we have that u = arg min f (u). Since f a sum of convex functions, is itself convex, and so the first-order optimality condition is also sufficient for optimality. We have
Σ kEP.
a (k) --
Σ kEP..
a -u
P
and
Σ
a kk
(8)
P
kEP.
Note that this is very similar to the definition of mean feature dictionaries, but also importantly different, because the optimal u depends on the optimal values of the feature dictionaries for the other attributes.
7.8.3 MSE FEATURE DICTIONARIES WITH INDEPENDENT ATTRIBUTES.
Finally, we can prove that, under certain conditions, attributes for which E (ala:()) = v/l = E [a], ie.. the conditional mean of activations over values of the attribute is the same the overall mean (assuming both means exist), will have (approximately) constant MSE feature dictionaries u, V.Vr. This a counterpart to the result from Appendix 7.7 for MSE feature dictionaries:
Lemma 7.1. Suppose that all conditional means Ep-D Jala, e] exist for all i, St. Let as be an allribute such its values appear independently from the values of all other altributes, i.e.
PPPDD (a=(p) us ap(p) P (a:(p) PAWD (ar(p) mi]
Voe E St, uh E St, 1
Then, in the limit of infinite training data, the conditional means E la a; (p) = v] are all equal to the overall mean E [a] if and only if the optimal MSE feature dictionaries ut,, for this attribute are constant with respect to the value u of the attribute, i.e. u for all E S..
Proof. From Equation g we have
a (k)
a(k)
Σ 1z1
P
P
kEP
P
a
P
u
(4)
kEP
kEP biii
The first term converges to E (ala(()) = v). The second term is sum of terms of the form
Σ
uia H| s.t. a(pp)) v, vp)( =v =..
Piol
(1()))
Piol
kEA.
22

---

Published as a conference paper at ICLR 2025
for 1 # i. Since are assuming a, is uncorrelated with at in the limit of the size N of the dataset a al2) ......(M) going to infinity, l[ s.t. (s(ps) = v, (pp)) = mi)l will approach |Pul E EpWD [1a((n)=m]] Moreover, note that in the closed-form solution U (CTC)* CTA=
C C from Equation 7 the matrix N CT C converges some limit Σ RSxS as N 0C, and the matrix 1/ C A similarly converges to some limit M E 5x by the assump- tion that all conditional means for all attributes exist. Thus, the optimal feature dictionaries uf,, will also converge N + oc. So we see that the sum in Equation 9 will converge to a value that is independent of the value e' for the attribute O,,.
Thus, if the conditional means [p)o;()) u] are all equal to the overall mean E [a]. we get that u is independent of v; conversely, if ui, is independent of v, we get that the conditional means are all equal the overall mean. This completes the proof.
7.9 FRATURE-LEVEL MRCHANISTIC ANALYSES FOR SECTION 4.3
Since each activation is approximated as the sum of several vectors from a finite set. it becomes possible to decompose the model's internal operations in terms of elementary interactions between the learned vectors themselves. In the current paper, we are particularly interested in attention heads, as they are the building blocks of the IOI circuit. We consider the following subspace-level analyses:
Attention scores: The attention mechanism is considered to be a crucial reason for the success of LLMs (Vaswani et al. 2017), but subspace-leve understanding of it is mostly lacking (but see Lieberum et al. (2023)). How do the features in the keys and queries of attention heads combine to produce the attention scores? Which feature pairs are most important for the head's behavior?
Head composition: If we are to understand circuit on the subspace level, we need to develop subspace-level account of how the outputs of one attention head compose with the queries, keys and values of a downstream head in the circuit. Each head adds its out- put to the residual stream, and downstream heads' query/key/value matrices read from the residual stream. We can thus examine the contribution, or direct effect, of head's output another head's queries/keys/values We can decompose this direct effect in terms of the features of the source head to calculate contributions of each feature to the direct effect.
Implementation details for these analyses follow.
Attention scores. Given feature dictionary reconstructions for the keys and queries of an attention head at certain positions
k ~
Un,
Va(()=a((r)
KET
SET
we can decompose the attention scores as a sum of pairwise dot products between the dictionary features
q k/ dhead se Σ V Me (2,())a; (p) dhead
i.jE!
where dhead is the dimension of the attention head. This allows us to examine which feature com- binations are most important for the head's attention according to the learned dictionaries. Variants of this decomposition can also be applied to e.g. the difference in attention scores at two different token positions.
Head composition. Following the terminology and results from Ellhae et al. 2201]) the residual stream rı., of a transformer given layer 7 and token position is the sum of the input embedding and the outputs of all earlier MLP and attention layers at this position. The residual stream is in turn the input to the next attention layer; so, for example, we can write the query vector for the h-th head at layer and token i as
A... = W LayerNorm (ri,e) = w& LayerNorm (Fi + W W.NZZ (2
23

---

Published as a conference paper at ICLR 2025
where z ,e,'''' is the attention- attention-weighted weighted sum of values of the h'-th head layer I' < l and token t, Fl, the remainder of the residual stream after removing the contribution of this head, and LayerNorm is the model's layer normalization operation (Ba et al. 2016) before the attention block in layer l. By treating the layer normalization as an approximately linear operation (taking the scale from an average over the datase we can derive an approximation of the (counterfactual) direct effect of the output of the /'-th head at layer l' and token on the query vector of the h-th head at layer and token
Pr & WO 1.6
2 O
Fi,t M1 of +
+ B1
where "ts 3t are the learned scale and shift parameters of the LN operation, f4, is the average of the vector ri, its coordinates, and ōn an average over the dataset of the standard deviation of the residual stream at this position. Alternatively, we can use the exact layernorm scale from the forward pass over large sample to compute the statistics of the exact direct effect over observed data.
With either approach, we obtain a decomposition
q1athh
(L(()))0---))) + Fi.e
PELK
of direct contributions from the outputs of earlier heads at this position, plus some residual terms Fi. (which the contributions of all previous MLP layers and the input embedding to the query vector). We can then further decompose (M.(),')) by replacing it with its reconstruction from our feature dictionary.
For either way to treat the layer normalization, we can use the learned feature dictionaries for the outputs, keys, queries and values of attention heads in number of ways decompose the direct effect further. Here, we consider head-and-feature attribution: fixing the head (1, h). we can vary the head (I", h() and break down the direct effects (projected on the query vector) by feature.
Results. An interesting location to examine is the attention of the name mover heads from END to the 10 and SI positions, where (according to the analysis in Waan et al. (2022)) the signal from the S-Inhibition heads effectively removes the S name from the attention of these heads.
We show the results for the head LI0HO in Figure X Crucially, we observe that most interactions are tightly clustered around zero, which suggests that these feature dictionaries provide sparse and interpretable account of the attention mechanism. The only significantly nonzero interactions are (1) between the S features in the query and the key at the $1 position; (2) between the Pos features in the query and the key both positions; and (3) between the IO features in the query and the key the IO position. The first two interactions interesting is the third interaction, which is negative, suggesting that the LIOHO head inhibits both the S and 10 name tokens, and effectively relies only on the Pos attribute to distinguish between the two names. Notably, this is in contrast with the other two name movers L9H6 and L9H9 (for which analogous plots shown in Figure [8 and (99). where the inhibition of the 10 attribute is absent. We found that using other methods to compute feature dictionaries results in less sparse and interpretable patterns.
expected given the findings of [ang et al. (2022). More
We further investigated the queries of the L10HO head, by looking at which features from upstream head outputs at the END token have large direct effect on these queries. Following the methodol- ogy detailed in Appendix 7.9. we plot the direct effect from the outputs of the S-Inhibition heads, as well as the two name mover heads L9H6 and L9H9 in layer 9 in Figure 9 We find that the S S-Ihibiitoon heads' 10 features have no significant contribution to the queries, but the IO features from the two name mover heads layer 9 have a significant direct effect (aligned with the overall centered query vector). This suggests that, having already computed a representation of the 10 at- tribute, these heads transmit it to the next layer, where gets picked up by the L.1HH head's query. Conversely, the S-Inhibition heads contribute significantly with their Pos and S features, whereas the name mover heads in layer do not.
12ih is justified by the empirical observation that the layer normalization scales across the dataset are well concentrated around their mean.
24

---

Published as a conference paper at ICLR 2025
Attention to S1
Attention to IO
Bias X Bias
Bias x 10
Bias X Pos
Bias S Bias
10
Pos
10 S
Pos Bias
Pos x I0
Pos x Pos
Pos X S
S X Bias
10
$ X Pos
$ S
-2
0
2
-2
2
Figure 8: Decomposing the attention scores of the name mover head L10HO from END to the SI (left) and 1O (right) positions. The y-axis ranges over the combinations of features from the query (first element) and the key (second element). The boxplots show the distribution of dot products between the corresponding feature vectors. The interaction between the bias terms (i.e., the means of the respective queries/keys) provides sense of the scale of the effects.
Direct effect on query of L10HO at END
Source Feature
L7H3 (S-I)
10
S
Pos
L7H9 (S-1)
L8H10 (S-1)
L8H6 (S-1)
L9H6 (NM)
L9H9 (NM)
0.01
0.00
0.01
0.02
0.03
0.04
0.05
Figure 9: Direct effect of supervised features in the output of S-Inhibition heads, and Name Mover heads in layer 9, on the queries of the L.1HH name mover head at the END token.
7.10 ALTERNATIVE PARAMETRIZATIONS THE IOI TASK
We mostly experimented with two possible parametrizations of prompts via attributes:
independent parametrization: we use the three independently varying attributes S, 10 and Pos to describe each prompt. This the parametrization used in the main text.
coupled parametrization: we couple position with name, and use the two attributes (S, Pos) and (IO, Pos) to describe each prompt. This parametrization is more expressive than
25

---

Published as a conference paper at ICLR 2025
the independent one, as it allows for different features for the same name depending whether it comes first or second in the sentence. At the same time, the coupled parametriza- tion can express the independent one special case (Appendix 7.11.
We find that these parametrizations arrive at highly similar activation reconstructions a. In fact, we find even stronger property: the coupled parametrization essentially simulates the features in the independent one; details are given in Appendix 7.11.1
Finally, we that the fact that we find parametrizations that result in good approximation is not trivial. Not every natural-seeminng parametrization will lead to a good approximation of model behavior; we show an example of this with 'names' parametrization in Appendix Figure where we instead use an attribute for the 1st, 2nd and 3rd name in the sentence.
What about other parametrizations? Note that there are combinatorially many possible ways to pick attributes to disentangle the activations into, and a priori any specific choice is arbitrary. We justify our choice of parametrizations in several ways: (I) they pass our tests for model approxi- mation, control and interpretabilityy given later in this section; (2) they correspond to the internal states of the IOI circuit identified in Wang et al.] (2022); (3) we experimented and/or considered other parametrizations, but found they either perform the same or worse our tests. In Appendix we provide a more detailed discussion of different possible parametrizations in the IOI task and their relative strengths and weaknesses.
7.12
7.11
COMPARING THE COUPLED AND INDEPENDENT PARAMETRIZATIONS
7.11.1 THE COUPLED PARAMETRIZATION CAPTURES THE INDEPENDENT ONE
Idealized model. We first note that the coupled parametrization can express all reconstructions expressible by the independent parametrization. Suppose we have an IOI distribution using a set of available names Snames, and let 1B BAS {io, (oD))eeSamee {Sa Saame be feature dictionaries for the independent parametrization at some model activation. Then, we can define the following feature dictionaries for the coupled parametrization:
1
AB.BABB io,
2 DOSABB
ABB
BAB io, POSBAB BAB
ABB Sa 2 ABB
BA = S. + BAB
Then for a prompt p of the form ABB (the BAB case is analogous), with the 10 name being and the S name being b, we have that the reconstruction of activation using the independent parametrization
Bindependent io. sb + AOS
and the reconstruction using our coupled parametrization is
acoupled = 10a, ABB + Sb, ABB = io, + Sb + POS ARB + PPP ARB
io, + Sb + AOS aandependent
Empirical evaluation. We evaluated whether this occurs empirically with the MSE features for the two parametrizations. First, we plot the fraction of variance explained by the reconstructions using the independent parametrization in the reconstructions using the coupled parametrization. We find very high agreement (Figure 10); results in the other direction are almost identical and are not shown here for brevity. Next, we check if the coupled parametrization essentially simulates the independent one as described analytically above. We do this by measuring the cosine similarity between the vec- tor pos ABB posp, B AB from the independent parametrization and vectors iO,,ABB - io,, and Sa.ABB - So, BBB from the coupled parametrization. In our idealized simulation of the independent parametrization using the coupled one, these values would be exactly 1 for all names. We find that in all circuit locations that represent both 10 and Pos, the similarities w.r.t the 10 differences are significant; similarly. in all circuit locations that represent both S and Pos, the similarities w.r.t the S differences are significant (Figure [T]. For reference, in a space of this dimensionality (64), the expected magnitude of the cosine similarity between random vectors is 1/8.
26

---

Published as a conference paper at ICLR 2025
Backup Mover
1.00
0.95
1.00
Duplicate
1.00
1.00
Induction
1.00
1.00
1.00
Name Mover
0.98
1.00
0.96
1.00
Negative Name Mover
0.98
1.00
0.98
1.00
Previous Taken
1.00
1.00
1.00
1.00
S-Inhibition
0.97
0.97
0.99
Output
Key
Query
Figure 10: Variance explained by the reconstructions using the independent parametrization, with respect to the reconstructions using the coupled parametrization, averaged over combinations of class of heads in the circuit and activation types.
Duplicate Taken Attn Output (5)
Induction Output (5)
(S)
I-hiiiitionn (S)
Name Mover Query (5)
Name Mover Query (IO)
Backup Name Mover Query (ID)
Backup Query
Cosine Similarity
Figure 11: Cosine similarity between the vector poS POS from the independent parametrization and vectors іо,,ABB iO,,BAB and SaABB BaBBAB from the coupled parametrization, averaged over several classes of circuit locations. When evaluating similarity for the ABB io AB vectors, we only include locations where both the 10 and Pos attributes are represented; and similarly for the s-vectors.
7.11.2 OTHER PARAMETRIZATIONS EXPRESSIBLE BY THE COUPLED PARAMETRIZATION.
Consider the parametrization with attributes (S, Pos) and 10. Again, let (io.)aeS..mess [Sc,ABB] JaE.naerr (S,,ABB JoESmames be feature dictionaries for this parametrization at some model activation. Then, we can define the following feature dictionaries for the coupled parametriza- tion:
i0' ABB
-io i io.. 8 SBBBB = St,ABB
S BAB Sa,BAB
27

---

Imagine now that are given an 'unlabeled' feature dictionary that corresponds this decom- Published as a conference paper at ICLR 2025
Then for prompt p of the form ABB (the BAB case is analogous), with the 10 name being and the S name being b, we have that the reconstruction an activation a using the (S, Pos) + I0 parametrization is
PSs 10 io, + ABB io A,BB S,ABB acoupted
and the coupled parametrization express all reconstructions expressible by the (S, Pos) I0 parametrization. Analogously, it can express all reconstructions expressible by the (IO, Pos) + S parametrization.
7.11.3 EDITING METHODOLOGG WITH THE COUPLED PARAMETRIZATION
For each activation, we may choose to edit one or several of the IO, S and Pos properties of the prompt. With the independent parametrization, this is straightforward, since the attributes match these properties. With the coupled parametrization, suppose we are given a prompt of the form ABB (the BAB case is analogous) with the IO name being and the S name being b. Given activation a with corresponding reconstruction under the coupled parametrization
a ABB + Sb,BAB
we can perform edits as follows:
to change the I0 name from a to d': aedit = a ii.,ABBB + iio,'ABB
to change the S name from b to bb: aedit a Sh,ABB + SKAB
to change the Pos property from ABB to BAB: aedit a AB..ABB -B St.BB +io,
Sb, BAB
7.12 ON POSSIBLE FEATURE DICTIONARIES FOR THE IOI TASK
In this section, we compare the properties of several a priori possible ways in which the activations of the IOI circuit could be disentangled via sparse feature dictionaries. The main goal is to illustrate that different feature dictionaries can have similar usefulness in terms of model control and inter- pretability, even if they fail natural tests that directly look for similar features in the two dictionaries. This motivates evaluations that are agnostic to the specific features a dictionary, as long as the features parsimoniously disentangle the model's internal computations.
The independent parametrization from the main text. is worth explicitly describing the prop- erties of the supervised feature decomposition constructed in Section 4.2 which uses the 10, S and Pos attributes to descrihe the prompts; it serves as an idealized example against which to com- pare other possible decompositions. In this decomposition, prompt where the 10 name is and the S name is 6, with the IO name appearing first, as follows:
approximate activation for
a ~ io Sb + pos
where the vector io, is the feature for the 10 name the vector S, is the feature for the name and the vector pos aRB is the feature for the Pos attribute when the 10 name appears first in the sentence (and analogously for BAS)AB).
position (i.e., we don't know which attribute each feature corresponds to). We want to evaluate the usefulness of this decomposition for model control and interpretability relative to the 'human- legible' attributes 10, S and Pos. This will be tautologically successful:
the reconstructions are faithful and complete representation of the model's internal com- putations;
the dictionary can (by definition) express edits to the 10, S and Pos attributes very ficiently, as we only need to change single feature vector to change the corresponding attribute's 's value.
the features are fairly interpretable: we can understand the meaning of each feature in terms of the attribute it represents.
28

---

Published as a conference paper at ICLR 2025
moreover, using metrics such recall and precision (following the evaluation methodology of Bricken et al. (2023)) will readily surface the features that are most important for each attribute.
Using per-gender vectors to describe names, Another possible decomposition
a R io, io geeggeeddgr.... g(a) + + s.gender s(b) + ABB
g())
where g: : Names + [M, F) is some labeling function that roughly classifies names according to how they are typically gendered [i Here, we hypothesize that the model may use features of high norm that sort names into genders (which may useful to the model for various reasons), and then add a small-norm per-name correction to obtain name-specific representation. In particular we imagine that io, + io gender g(a) in this representation would correspond to io, in the supervised decomposition, and similarly for the S name.
This decomposition is also fairly sparse and interpretable, and it can express edits almost parsimoniously as the supervised decomposition (we only need to change two feature vectors to edit a name, and one to edit Pos).
Moreover, metries such recall and precision will pick up the per-name features;
If we compare the prompts for which a feature activates for this dictionary and our 'inde- pendent parametrization' feature dictionary (discussed above), we will easily discover the per-name features that correspond to the 10 and S names.
However, if we instead directly use cosine similarity to the supervised features io,, Sp as a metric, may be misled, because if the per per-gender features have sufficiently higher norm than the per-name corrections, the cosine similarity may be low.
Features for small, overlapping subsets of names. Going further, we can imagine a decomposition where we have features that correspond to pairs of names, such that each name in exactly two pairs (this can be achieved by partitioning all names into pairs along a cycle). We can express name as a sum of the features for the two pairs it is in, with some superposition. Note that more sophisticated constructions with more features per name more names per feature are possible by e.g. picking subsets at random or using expander graphs (Hoory et al., 2006), as they will 'spread out' the superposition more evenly.
This decomposition is somewhat sparse and interpretable, and can likely be used for feature editing in a reasonable way, as long as the sets of features associated with each name are not too large. Even though we would need to change several feature vectors to edit a name, there should also be fair amount of disentanglement so that we don't also need to throw away all the features active in an example to change a single attribute.
However, comparing our supervised decomposition against this one using geometric met- rics such as cosine similarity may be misleading, because while a sum of a few feature vectors associated with the same name may point in the same direction as our supervised feature, any individual feature may not.
Furthermore, it also has significantly reduced precision for the features, because each of the few features associated with a name will also activate for several other names. This can make directly looking for features whose activation patterns resemble the ones in our supervised decomposition misleading.
Our experiments suggest that both task-specific and full-distribution SAEs trained on IOI circuit activations learn a decomposition resembling this abstract construction more than any other decom- position discussed here.
Overfitting dictionaries. Finally, a worst-case decomposition would be to have a single feature for each possible set of values of the S, 10 and Pos attributes.
This decomposition is not interpretable, and it is not editable in any non-trivial way: to change a single attribute, the entire decomposition must be replaced;
Features of this form will have maximum precision, but very low recall for the attributes.
We experimented with this decomposition in our supervised framework, but did not find it to confer addi- tional henefits for the purposes of our tests.
29

---

Published as a conference paper at ICLR 2025
7.13 DETAILS FOR TRAINING SPARSE AUTOENCODERS
7.13.1 VANILLA TASK SAES
We followed the methodology of Bricken et al. (2023), with the exception that our neuron re- initialization method is not as sophisticated as theirs: we simply re-initialize the encoder bias, en- coder weights and decoder weights for the dead neurons every 500 training epochs.
Training SAEs on the IOI distribution alone allows us to do a more extensive hyperparameter search. Importantly, we normalized SAE inputs across attention heads so that activations have an E norm of 1 on average in order to make it easier for the same set of hyperparameters to work well across different heads. In our main experiments, we use SAEs that were trained with a 16x hidden expan- sion factor, (effective) E regularization coefficient in (0.01, 0.05, 0.1, 0.3), batch size of 1024, and learning rate of 0.001.
We evaluated two key test-set metrics across training epochs: the average number of active fea- tures per example (i.e. the average &o norm of activations), and the fraction of the logit difference recovered when using the SAE reconstructions at the given model location instead of the original activations, scaled against a mean-ablation baseline (which is more stringent than the zero-ablation baseline employed in most other work). We chose the regularization coefficient and training check- point for each node that provided a good trade-off between these two metrics; in particular, for almost all nodes, we recover logit difference to within 20% with respect the mean-ablation base- line, and there are < 25 active features per example. We provide the results of this sweep in Figure 20] While most SAEs seemed adequate, some still have poor approximation as measured by the reconstructed logit difference.
We use training set of 20,000 examples and an evaluation set of 8,000 examples (for the purposes of automatic interpretability scoring, we need a large enough evaluation sample so that each property in the distribution appears a significant number of times). Since our training regime significantly distinct from that of Bricken et al. (2023) (we use a much smaller dataset), we first experimented extensively with different hyperparameters. focusing on training SAEs on the queries of the name mover heads. We observed that the most important hyperparameters are the dictionary size and the effective & regularization coefficient. We found that the batch size did not influence the eventual quality of the learned features, only the speed of convergence, and that learning rate of 10-3 (as in Bricken et al. (2023) was a good choice throughout. The runs reported here used a dictionary size of 1024 (a 16x increase over the dimensionality of attention head activations in GPT-2 Small), an effective & regularization between 0.05 and 0.3, and a batch size of 1024.
We normalized activations across the circuit to make it easier for the same range of hyperparameters to give good results, and ran sweep over & regularization coefficients in (0.01, 0.05, 0.1, 0.3).
Results. While performance varied strongly across circuit locations. most full-distribution SAEs had an fn-loss between 2 and 12 and a recovered loss fraction (against mean ablation baseline) between 0.4 and 0.9 (both measured on OEET). Similarly, most task-SABs had an l-- loss below 25 and a recovered logit difference fraction against mean ablation > 0.8 (both measured on the IOI dataset)
7.13.2 ADDITIONAL TASK SAE VARIANTS: GATED AND TOPK
Training schedule. We incorporate several training tactics from recent literature. For all SAE variants considered, we used the same (small) learning rate of 3 x 10 trained for 2000 epochs in total, and applied resampling followed by learning rate warmup over 100 epochs (roughly following Rajamanohaaan et al. (2024)) at epochs 501 and 1001. Our resampling methodology closely follows that of Bricken et al. (2023). In addition, we decay the learning rate linearly to zero over the last 25% of training (following Conerly et al. (2024)) For topK SAFs, we initialize the encoder to the transpose of the decoder, as suggested by Gao al. (2024); however, we do not
"Importantly, we did not perform exhaustive hyperparameter tuning to train these SAES, as our main goal was to evaluate the methodology and how it can distinguish between different classes of feature dictionaries, rather than to achieve state-of the-art performance. Thus it is possible that significantly better performance could be achieved with more tuning. Indeed, it is our hope that the methods we present here will be useful for tuning SAFs in the future.
30

---

Published as a conference paper at ICLR 2025
the auxiliary loss term suggested in section 2.4 ('Preventing dead latents') from that work (which is the only difference of this paper from the implementation of Gao et al. (2024)).
We checkpoint all models at 14 epochs: (1, 2, 4, 8, 16, 32, 64, 128, 500, 750, 1000, 1250, 1500, 2000). This checkpoint schedule is chosen to ensure that we have a dense enough sampling of the early stages of training, while also capturing the state of the model right before resampling, and after the learning rate warmup that done post-resampling is sufficiently in the past.
Preprocessing. We normalize all IOI circuit activations prior passing them through our SAES, following the scaling methodology in Conerly et al. (2024), so that they on average have & norm of Vihaad- This helps share hyperparameters across sites of the circuit, and reduces the range of hyperparameters to search over.
Hyperparamettrs. We sweep over values À E (0.5, 1.0, 5.0, for the l regularization penalty for vanilla and gated SAEs, and over values k E (3, 6, 12, 24) for topK SAEs. This is a reasonable range: we found that the highest value of X leads to about 3-4 active features per example on average; the supervised dictionaries have 3. Conversely, the lowest value of À lead to very good Ez loss and recover close to 100% of the logit difference, but have too many (about 20-30) active features per example on average. The range for k is chosen to include values equal and close to our expectation for the 'true' number of necessary features (3), while still allowing significantly more features to be active.
7.13.3 FULL-DISTRIBUTION SAES
We trained full-distribution SAES on every IOI component using OPENWERTEET as training data. We mostly followed the method outlined in [Bicken et al. (2023). We added a standardization procedure to be able to train SAES on components with different activation scales using the same ll-coefficient. Before training, we calculated the mean and the mean 12-norm over 10 million ac- tivations. These values were then frozen and used to standardize all activations as a preprocessing step and to rescale the SAE reconstructions to match the original scale as a post-processing step. We generated the training dataset by extracting buffer of 10 million activation vectors from the shuffled OPENWEBTEEX dataset at time with a maximal context window of 512 tokens. We then trained the SAEs for 250 million activation vectors and resampled dead neurons after 50000 steps (around 100 million activations) as outlined in Brickkn et al. (2023). We used a batch size of 2048 and 8192 features per SAE. Post-training, we excluded dead and ultra-low frequency neurons that we define as neurons who activate less than once per million activations. The amount of dead neurons varies across SAEs between 20 and 90%. We plot the fraction of dead neurons versus Eo loss in Figure 22 and the loss recovered versus to loss in Figure 23
We used an & coefficient of 0.006 initially for all SAES, and retrained SAEs with different & co- efficient for cross-sections whose SAE metrics were undesired (e.g. very low &o bad reconstruction or very high fo). For the name mover outputs, we used 0.025, and for S S-Inhibition keys we used 0.005. The test Eo and loss-recovered metrics were calculated on 81920 unseen activation vectors.
We trained fewer SAEs on the full pre-training distribution compared the IOI distribution, as the computational cost is higher. We observe that SAEs with lower number of active features per example generally perform better for 10I-related tests, even if their other metries (such as loss recovered on OPEN WEBTEERRE TEXTT) are worse
SAE training loss metrics. The most important loss metrics to track during SAE training are the to loss (measuring the average number of active features per activation) and the language model loss recovered when using the learned features to reconstruct the model's logits (Bricken et al.) 2023). To turn the loss recovered into a meaningful quantity, it is rescaled against baseline; both zero ablation and mean ablation have been used for this purpose the literature (Bricken et al. 2023 Kissane et al. 2024). In this work, used mean ablation, as it is more strict test of the quality of approximation.
7.14 ADDITIONAL NOTES ON METHODOLOGY FOR SAE INTERPRETABILITY
Interprefations considered. Let Names be the set of names in our IOI dataset. We consider the following binary predicates over prompts as possible interpretations for SAE features in the activations at the $2 and END tokens:
31

---

Published as a conference paper at ICLR 2025
10 is 2nd name: the Pos attribute having value corresponding BAB-type prompts;
10 is 1st name: the Pos attribute having value corresponding to ABB-type prompts;
S <name>: the attribute has certain value Names;
S is <name>> and at 1st position: same above, but also the S name is at 1st position in the prompt (i.e., this is BAB-type prompt);
S is <name>> and 2nd position:
same as above, but for ABB-type prompts;
10 is <name>: the 10 attribute has certain value in Names;
10 is <name>> and at 1st position: same as above, but also the 10 name at 1st position in the prompt (i.e., this is ABB-type prompt);
10 is <name>> and at 2nd position: same as above, but for BAB-type prompts;
S is male: the S name is labeled as male name under our labeling of Names provided by GPT-4;
S is female: same as above for female names;
<ame>>> in sentence: a certain name in Names;
<name>> is at 1st position:
above, but the name is the first name in the sentence;
<name>> is at 2nd position: same above, but the name is the second name in the
sentence;
The next several interpretations are only defined for the keys and values of the name mover heads. We collect together activations for the keys according to their role in the IOI circuit as opposed to absolute position: we group together all activations at the IO token position (these are the IO keys/values), even though they come from different absolute positions across IOI prompts, because in ABB prompts the 10 name comes first, while in BAB prompts it comes second. The same applies for gathering the keys/values.
The key/value activations described above have not yet 'seen' the repeated name in the sentence, so there is no meaningful concept of IO and S for them. Instead, the only potentially task-relevant information contained in them is about the name(s) seen so far in the sentence, and the position (Ist name or 2nd name) where the activation is taken from. Accordingly, the applicable interpretations for features contained in these activations different:
current token is name>> the token from which the activations are taken holds a certain value in Names.
token is <amee> and at 1st position: the activation was taken from token with certain value in Names, and addition it comes from the first name the sentence;
token is <nmee>> and at 2nd position: same above, but activation is from the second
name in the sentence;
current token at 1st position: the activation is from the first name in the sentence;
current token is at 2nd position: same above, but from second name;
current token is female: the token the activation is from is female under our labeling of Names.
Unions of interpretations. In addition, for each type of predicate that has free parameter in Names, considered unions of up to 10 such predicates (recall that we have total of 216 names in our dataset). We ordered the individual predicates according to their Fi score, and chose the union of the first i S predicates with the highest F; score as possible interpretation. Note that the Fj score is not in general a monotone function of k in this setup; indeed, we find that for many features the highest- -core explanation uses k < 10 features.
Sufficiency/necessitty of interpretable features. We take the interpretations of the features de- scribed above and their respective Fi scores, and for each threshold E [0, 1] over Fi scores con- sider two interventions:
sufficiency: we subtract from the respective activation active features with Fi score < f; necessity: we subtract from the respective activation all active features with F, score I:
32

---

Published as a conference paper at ICLR 2025
7.15 ADDITIONAL OBSERVATIONS ON SAE LATENT INTERPRETATIONS
Correlational evaluation. Full-distribution SAES must capture variation in activations across a large set of text, of which IOI-like prompts are only small subset. Consistent with this, we found that only a subset of full-distribution SAE latents activates on IOI prompts, with the number of latents that fire on IOI prompts varying strongly between components. We scored the latents that do fire on IOI prompts and found a significant amount of latent descriptions with high Fi-score. We summarize the number of high-Fi-score latents per type in Figure [5] Remarkably, we find that the interpretable latents in the full-distribution SAEs and the task-specific SAEs are qualitatively similar; corresponding task-specific results are given in Appendix Figures 29] (showing the most interpretable SAEs at each node of the IOI circuit) and 30 (showing the SAEs chosen to optimize the tradeoff between the Ey loss and the logit difference reconstruction, which we use throughout the main body of the paper).
In practice, we want to use latent explanations to get insight into the more general computation of component. Thus, we investigated whether the latents found are consistent with the previously established function of the heads from Wang et al. (2022). We found that this was true for all heads and that simply looking at the number of latents with given interpretation draws clear picture; examples are provided in Appendix 7715
Our results also suggest several details about the IOI circuit that weren't reported previously, which we summarize in Appendix [7.15] We were also curious about how the detected latents behave on arbitrary text of the model's s training distribution. As creating rigorous test for this is difficult, we report some anecdotal evidence in support of latent generalization in Appendix 77.15
Causal evaluation: sufficiencyhecessityy of interpretable latents. Results here are encourag- ing: keeping/removing the latents with F1 score 2 0.6 often goes long way towards preserv- ing/degrading the model's performance on the task. Appendix Figures 33 and 33 show the results of these experiments for task SAES.
Causal evaluation: sparse control via interpretable latents. Here, results are also moderately en- couraging. We find that, for the task-specifee SAEs, editing using the high-Fi-score latents w.r.t. given attribute as a guide performs not much worse than the inerpretationnagnossi editing method. Results are provided in Appendix Figure 31 However, for full-distribution SAEs, again need to edit a high number of latents to achieve a noticeable effect (results in Appendix Figure (2..
Interpretable latents agree with head roles identified in the IOI circuit by Wang et al. 20022) For example, duplicate token heads attend to a previous occurrence of the previous token and write information about this to the residual stream. Consistent with this, we found that SAEs trained on them contain latents that indicate the duplicated name, the subject in case of IOL. This information is then used by the induction heads that determine whether the subject is the first or second name. The subject position is subsequently copied to the END position by the S-Inhibition heads, where it will query the name movers to not attend to the subject, and to copy and predict the 10 name. Indeed, we find latents of the outputs of induction heads, in the outputs and values of the S-I heads, and in the queries of the name movers that inform about the position of the indirect object. Lastly, the name movers attend to the I0 position and copy the name to predict it. As anticipated, the name mover values and outputs contain latents that specify the concrete 10 name. In summary, the type of latents detected informs well about the function of the head on a certain task.
New insights from the latent dictionaries.
The first layer DTThheds almost exclusively contain S latents but the third layer duplicate token head also has positional latents, suggesting more sophisticated text processing hap- pening there.
Induction heads encompass different positional latents. L6H9 latents inform about what name is at the second position, while L5H5 and L.5H8 activate when the 10 is at the first position. I.5H9 comprised of different latents including positional latents that don't. 't inform about the role (10 vs S) of the name.
L7H3 is the only head that contains a significant amount of gender latents.
S- S-Inhibition heads include primarily latents that are a combination of names and S, while the name mover queries seem to only contain Pos latents.
33

---

Published as a conference paper at ICLR 2025
The keys and values of name movers both inform about the token at the current position, but there is an important difference in the type of latents: while the values primarily contain latents that contain the name (that later gets moved to the END position), the keys consist of positional latents and combinations of position and name. This hints at an important mechanism where the name mover query contains positional information about the S name that gets matched with the corresponding key, effectively shifting the attention towards the 10 token such that the value with the IO gets copied the END position.
Editing interpretable latents
While the latent descriptions generated through our automatic scoring predict well when latent is active, it is still unclear whether they also have an interpretable causal role, i.e. whether activating or deactivating certain latent leads to a change in output logits that would expected from the latent's description. To test this, propose two experiments to judge the faithfulness and completeness of our interpreted latents that involve patching activations from a counterfactual prompt and calculating the effect on the model's output:
Estimating Faithfulness: To estimate faithfulness, we construct SAE activations where we fix latents with test Fl-score smaller than a threshold and patch activations of latents with high Fi-score from the counterfactual prompt. We then calculate reconstructions of this new SAE activation vector, patch cross-sections, and record whether the model successfully predicts the correct counterfactual name.
Estimating Completeness: To estimate completeness, we propose a similar experiment where fix latents with high test Fi-score and patch all remaining latents. This inter- vention should not change the output logits our latents are complete.
We run this experiment on cross-sections of name mover outputs and repeat this experiment for different thresholds. We observe that for a threshold Fi-score of 0.6, the SAE latents are both faithful and complete to a high degree. We observe that the faithfulness metric significantly decreases for higher Fl-scores of 0.7 and 0.8 but remarkably, we also observe that only fixing latents with a very high F1-score of > 0.8 while patching all other latents from the counterfactual prompt sufficient to keep the model predicting the base prompt's 's output.
Generalization of latents the full distribution. We sample prompts from Deb'eeeTexxt and vi- sualize prompts that highly activate latent. We do this for the name mover queries and outputs. For the outputs, we calculate the finettllatenttattributionn (DFA) metric first proposed by Krzyzanowski et al. (2024) that calculates per position how large the contribution of its values activating the latent is. Thus, it informs what position led to the latent being activated.
We find that latents mostly generalize. For example, we investigated a latent that activated if the 10 name starts with the letter "E" and we found that on full distribution, this latent fires at tokens preceding words starting with "E". DFA suggests that previous tokens starting with "E" activate this latent, and calculating the unembed W W We Wrr denoted with "Positive Logits" and "Negative Logits" in Figure 14 shows that activating the latent increases logits for words starting with "E". Together, this hints at general name moving mechanisms to predict words that previously occurred in the context that drives in-context learning where the head's QK-circuit drives attention to the position of the word to predict, and the OV circuit copies the token from the previous to the current position to predict it.
7.16 FEATURE OCCLUSION DETAILS
Training an exhaustive set of SAFs. Focusing on the L10H0 queries, we found that our SAEs consistently find a single feature for almost all 10 names, but fail find significant number of features for individual S names.
To investigate further whether this is a result of poor hyperparameter choices, we trained SAEs on the queries of LIOHO over wide grid of hyperparameters, so that performance deteriorates/plateaus at the edges of the grid. We pushed the dictionary size, training dataset size, & regularization coef- ficient and number of training epochs significantly beyond the values used in other experiments.
Specifically, we used a training set of 100,000 examples (this is more datapoints than all ~ 93, 000 possible combinations of S, 10 and Pos in our data); trained dictionaries of up 12 -192
34

---

Published as a conference paper at ICLR 2025
Fl-score:
Names: Fijja,, Jeremiah, Andrew, Eric, Fihan, Evai, Emilyy Eit, Eizabeth, Eleanor, Paige, Andres, Keith, Alice,
Paddie logits
Regurtive Tegits
Aaron,
0.23
Start
-220
TOP ACTIVATIONS
Top
sre position
5.87
Miloomi
oway spaeedd interartt Ii seic djestiiteet tieo, (assin wi [ill rad 1ing
s'pawesf otterares sadd
(Jntettart in Ineed Sepel
5.85
Lindey
tix) talking souc sale sails ruche will Elarher
(nndeftant)) wnsteringg "Ceiitin hamraee prevecdda..
5:43
Spancelte
-312
5:41
our onlive trevern %4 tas leakted
connact rebera parttament support Esactrante tarn Austratic 141,, iedes
F1-score: 0.95
Powiive lngit
Merattiv lngit
Names Adhley
TOP ACTIVATIONS
Top position
PM
enyy ingaectt Altevoed- Angol (lentrfeett
Angel long (lancofbauej is Then, (AA358) and Trane rad tong
km
1.27
13.0
craty aremeet Pttenert asstes
4.5
4.21
Noc
-327
(estoriags ro-criiiin
Sartoned bry taproe rulue
Seartaae Bol botteting crtntral caeas againce conpastee stalating San
620
Two representative features discovered in the output mover SAE L9H9 to illustrate the features behavior or webtext. Both features are IO name features, the upper one containing 23 names, the lower only single name. Left: Feature activation per position; Middle: Direct Feature Attribution (DFA) that tracks the position whose values contribute most to activating the given feature; Right: The output tokens whose logits get increased (positive) or decreased (negative) when the feature is active, calculated by oWo with being the decoder weight vector of the feature of interest, Wo the output weight of the head and We the unembed
Figure 12:
features (our supervised feature dictionaries contain 500 features); we varied the effective & reg- ularization coefficient across two orders of magnitude; and we trained for up to 6, 000 epochs.
Results on the number of IO/S features with Fi score 0.5 are reported in Figure 7 (left). We observe that across hyperparameters, we often find as many 10 features as there are names in our dataset; however, the number of $ features is consistently low, never exceeding 22.
Magnitudes of the 10 and S features. As first step, we investigate the distribution of the norms of the features for IO and across the names in the IOI dataset the L10HO queries our supervised feature dictionaries from Section 6 results are shown in Figure 13 (left). We observe that the norms of the features are significantly (but not by much) higher than those of the features.
Norms of and IO feature vectors in L10H0 queries
1.0
% low-norm features found
coefficient
Attribute
10
0.8
0,013
0.025
0,400
0.6
0,500
1.000
0.4
2.000
0.2
0.0
0.2
0.4
0.6 0.8
1.5
2.0
2.5
3.0
3.5
% high-norm features found
Figure 13: Left: distributions the &2 norms of the feature vectors the IO and attributes from our supervised feature dictionary for the queries of the L10HH name mover. Right: the results of the toy model experiment, where we investigate whether a disparity in feature magnitudes alone can lead to the occlusion phenomenon. The z-axis shows the fraction of high- maggittg ground-truth features for which we find an SAE feature with F score 0.9; the y-axis shows the same for the low-magnitude ground-truth features.
Surgically reducing the magnitude of I0 features. To examine the role of feature magnitude in the occlusion phenomenon, we continuously reduce the magnitude of 10 features. Namely, given supervised feature dictionary with features {io. Names: (St) be {pos,) (ABB,BAB and
35

---

Published as a conference paper at ICLR 2025
an activation a(p) for prompt p where the 10 name is a, construct activation
au (p) = a (p) aio,
for a [0, 1]. We find that, applying this intervention without any hyperparameter tuning (with modest dictionary size of 1024, and i regularization coefficient of 0.2), increasing from to gradually makes the number of IO features with F1 score 0.5 to decrease, while the number of features with F1 score > 0.5 increases; results are shown in Figure (
Reproducing the occlusion phenomenon in a toy model. Finally, we wanted know if disparity in feature magnitudes alone could lead to the occlusion phenomenon. We constructed simple toy model closely based on the empirical setup in the queries of the L.10HO head to test this hypothesis.
u; + Vj for random pairs i, E , Names }, where V; E Rdiced are sampled independently from standard normal distribution centered at zero, and then u, are rescaled so that their mean norm matches tures in the L10HO queries, and similarly for vj and S features. We train SAEs on these activations over wide grid of hyperparmeters: dictionary sizes in (512, 1024, 2048). & reg- ularization in (0.0125, 0.025, 0.4,0.4.... 0.5, 1.0, 2.0), batch size in (256, 1024) and learning rate in (0.001, 0.003, 0.0003). We trained for 1000 epochs, saving checkpoints sion of epochs. Results for the number of high- and low-mannitude features with F1 score 2 0.9 discovered are shown in Figure 13 (right); we observe that we easily find one SAE feature per each high-magnitude ground-truth feature, but it is more difficult to find an SAE feature for each low-magnitude feature.
We form synthetic activations
mean norm of I0 fea-
a geometric progres-
However, note that with lower F1 thresholds, this effect is less pronounced and eventually dis- appears.
7.17
FEATURE OVER-SPLITTIN IN A MIXTURE OF GAUSSIANS TOY MODEL.
Best total loss lambda*LD
102.5
100.0
97.5
95.0
92.5
90.0
SAE class
Ideal 2-feature
87.5
Randomized many-feature
0
1
5 L1 regularization (lambda)
2 3 4
6
7
Main experiment for our toy model of feature oversplitting. The data distribution is uniform mixture of two standard multivariate Gaussian random variables in 100 dimensions. Blue: the (approximate) best possible total loss (in the infinite data limit) achieved by a class of 'ideal' SAES that use two features pointing towards the means of the two components of the mixture. Orange: an approximate upper bound on the best possible total loss achieved by an SAE with 1, 000 hidden features (in the infinite data limit). The z-axis is the é regularization coefficient X. The cutoff on the r-axis is chosen so that the idealized solution activates only for a vanishing fraction (< 2%) of the examples in the mixture.
Figure 14:
Our goal in this section is to demonstrate that there exist setups where SAE with a large number of hidden features m >>>>> 2, when trained on a uniform mixture of two isotropic gaussian variables, will prefer solution with > features (as opposed to the 'ideal' solution with only two features, one per component of the mixture), for any value of the f, regularization coefficient À and any amount of training data.
36

---

Published as a conference paper at ICLR 2025
Setup. We consider a simple toy model where activations are distributed according to uniform mixture Dioy of two isotropic gaussians = + W (0, I2) in R" (i.c., we first flip fair coin to determine the sign of μ,, and then sample extra additive term from Ia).. We sample an i..dd dataset from this mixture. We used = 100, ||/=||2 = 2 in the experiments below; this choice guarantees that the two components of the mixture separable with high (> 95%) probability.
The 'ideal' solution with 2 features. We might hope that, with the right & regularization, an SAE trained on such distribution will discover only two interpretable features, one for each component in the mixture, with the encoder/decoder vectors aligned with le and 1e respectively, and each feature activating for (approximately) the examples its corresponding component.
We find this is the case empirically when we train an SAE with only two hidden features on this toy distribution. Specifically, there range of À values 1 S S 3 where the SAE reliably approximately recovers this ideal solution, with the encoder bias controlling the trade-off between the two loss terms: when increases, the encoder bias changes so that fewer examples in each component of the mixture are activated (and the active examples have a lower & loss). Beyond this range, the SAE often fails to activate any feature on any examples (A > + E), or activates both features on almost all examples (A < 1 e).
Analyzing the ideal solution across l coefficients. To study the properties of the 'ideal' solution analytically, we make the following assumptions (borne out empirically with 2-feature SAEs) using only symmetries of the data distribution:
the decoder vectors are =u, normalized to have unit lg norm (by symmetry of each com- ponent around its mean);
the respective encoder vectors are =k for some k > 0 (again by symmetry of each com-
ponent around its mean);
the decoder bias is zero (by symmetry of the mixture around zero);
both encoder biases are set to -y for some 7 > 0 (again by symmetry of the mixture around zero).
This leaves only two parameters to tune: the encoder bias y and the encoder scale k. We can thus use the following strategy to analytically approximate the best loss of this class of solutions for a given A:
approximate the expected & and & losses over a fine grid of values for y and k. for large dataset of samples from the mixture;
given Ả value, find the point in the grid that minimizes the total loss E2 + Xe1.
We implemented this using 10s samples, with grid of 100 values for in [0, 5] and 20 values for in 2), over a grid of 100 values of À in [0, 20). We verify that the best values chosen for each A are not on the edges of the grid; the resulting curve of best total loss values versus À is shown in Figure 14 (blue), cut off at X ss 7, beyond which the selected SAE activates for < 2% of examples in the components of the mixture.
SAES with m >> 2 features prefer other solutions even with infinite data. Next, we want to show that with enough features, SAEs will prefer solutions different from the class of 2- 2-feature solutions described above. How can we give an empirical argument that applies to any amount of training experiment
data and any X? After all, trained SAE is a function of the data it trained on, on datasets of bounded size can establish properties of SAEs trained on arbitrarily large datasets.
We get around this by defining a class of SAES that competitive with the class of ideal f-aeuuere SAES upfront, independent of the training sample. by using a randomized construction that works w.h.p., and then estimating the expected loss of this SAE the infinite data limit empirically. To give evidence of our result for arbitrary A, we consider fine enough grid of À values, and for each we construct an SAE that is competitive with the best ideal 2-feature SAE.
Our randomized SAE construction proceeds as follows:
Sample m encoder vectors W Wene E R from 3 * Droy (i.c. a version of Droy scaled by 3) where B > 0 is a hyperparameter that we will tune;
37

---

Published as a conference paper at ICLR 2025
Set the decoder vectors Wdec E Rd to be the same the encoder vectors W ence but normalized so that each column of Wdec has unit le norm;
Set encoder biases to -y (where y > is a hyperparameter that we will tune), and decoder bias to 0 E R
We used m = 1, 000, a grid of 100 values for 3 in [0, 0.1) to search for the best 3 for each A, and fixed 7 2.35.
Empirical confirmation. Finally, we actually trained SAEs with many hidden features on Droy and observed that these SAEs reliably learned solutions with many active features across values.
7.18 ADDITIONAL DETAILS FOR SECTION
Feature weights are mostly in the interval [0, 1]. Recall that given reconstruction ä Σ; U. we defined the feature weight for the i-th feature as
weight (2) = u, a/ ||
For our supervised feature dictionaries, we find that 10% of all weights are negative, and that the average value of all negative weights across all nodes the IOI circuit and all three attributes is -0.037. Similarly, for our task-specific SAE feature dictionaries, even though 31% of all weights are negative, the average value of all negative weights is 0.002.. The number and magnitude of weights higher than are even smaller.
Causal evaluation using interpretability. While the feature descriptions generated through our automatic scoring predict well when feature is active, it is still unclear whether they also have an interpretable causal role, i.e. whether activating or deactivating certain feature leads change in output logits that would be expected from the feature's 's description. To test this, we propose two experiments to judge the sufficiency and necessity of our interpreted features that involve patching activations from a counterfactual prompt and calculating the effect on the model's output:
Estimating sufficiency: To estimate sufficiency, construct SAE activations where fix features with a test F1-score smaller than a threshold and patch activations of features with high F1-score from the counterfactual prompt. We then calculate reconstructions this new SAE activation vector, patch cross-sections, and record whether the model successfully predicts the correct counterfactual name.
Estimating necessity: To estimate necessity, we propose similar experiment where fix features with a high test Fi-score and patch all remaining features. This intervention should not change the output logits if our features complete.
We this experiment cross-sections of name ferent thresholds. We observe that for a threshold Fi-score of 0.6, the SAE features are both faithful and complete to a high degree. We observe that the faithfulness metric significantly decreases for higher F1-scores of 0.7 and 0.8 but remarkably, we also observe that only fixing features with very high F1-score of (0.8 while patching all other features from the counterfactual prompt sufficient to keep the model predicting the base prompt's 's output.
outputs and repeat this experiment for dif-
7.19 ADDITIONAL FIGURES
38

---

Published as a conference paper at ICLR 2025
celiaate Tahan Ouguus
LOH1
639
output
322
L3HO output
1SH
146
Induction
Ootpuns
output
10 is 2nd name
10
LSHB output
5 <names
LSH output
25
S <name> and at 1st position
output
173
position
Simolaiion Ougut
10 is <name>
L7HB output
10 is <names and 1st position
LBH6 output
434
<hhmee>
position
$ male
autput
239
female
L7H3 value
<name> sentence
Values
L7H99 valuue
1176
<name> 1st position
360
current token <name>
L8H10 value
position
Neme Mooeer
19919 query
53
current token 1st pesition
position
Quanties
query
ourrent 2nd position
LI0HO query
Nemo Moverr
L9H9
111
Kaps
key
L10HO key
Interpreted
Nemo Moverr
value
540
No good description found IDI
Values
L10HO value
710
Nemo Mover
L9H9
805
Ouguus
output
L10HO output
0.0
0.2
0.4
0.6
0.8
1.0
Figure 15: Interpreting the 10I features learned by SAEs trained on OPEN WEBTEXT. For each node in the IOI circuit, we show the distribution of interpretations for the features which have any interpretation with F1 score above threshold. The numbers in the right column indicate the num- ber of features with an assigned interpretation by our method, and the color bars show the overall distribution of the SAE features (conditioned on the feature not being dead on the SAE training distribution).
39

---

Published as a conference paper at ICLR 2025
Probe accuracy cross-section when probing for 10
0.8
0.7
0.3
0.2
supervised
0.1
beseline test
topk, edit
10 nm -onmi
10 in nm bomgokk
1D nos ba
ni -bonggg
mm brm@gg
Pos in sigz
sk@v
Pos in sagg-
Edit attribute and Cross-section
Probe accuracy by sectioo when probing
1.0
0.6
supervised edit
0.2
haseline teist accuracy
topk_edit
10 bnnge
nms bum@ek
10 in nn -nnngg
nm Bing
Poy in nm bringhy
Pos in sigz
Pos skaw
Pas in ind ange
Edit attribute and Cross-section
Probe accuracy by cross-section when probing Pos
0.8
0.4
supervised edit
Desellie test accurecy tapk, edit
0.0
in mm-tbrm@z
10 in mmn brm@ggk
in nms bnmngg
Pos mms kremiiik
Par in rm barnggg
Pos in skaz
Pos skdy
Pus in fnd 2
Edit attribute Cress-section
Figure 16: Accuracy for probes for the 10, S and Pos attributes in nodes of the 1OI circuit, as we apply feature editing methods via either supervised (blue) task-specific topK SAEs (green). For each cross-section, we train a probe on each location this cross-section, and report average accuracies. The "baseline test set accuracy' is the accuracy of each probe on test of activations w.r.t. the one used to train the probe. We always evaluate probe accuracy w.r.t the attribute value we expect to observe in an edit. This means that, for example, we are editing the IO attribute, we evaluate the 10 probes against the name we are editing to, but the S and Pos probes against the original values of these attributes.
40

---

Published as a conference paper at ICLR 2025
independent mean
coupled mean
names mean
(B)NM out
(BJNM qk
(BJNM
S-1
S-1 out
Dn out
independent MSE
coupled MSE
names MSE
(B)NM out
(B)NM qk
(B)NM
S-I
S-l
Dn out
0
Figure 17: Fraction of recovered logit difference for several different methods to compute feature dictionaries, across cross-sections of the circuit. For definition of the 'names" parametrization, Appendix 7.12
Attention to S1
Attention to IO
Bias Bias
Bias IO
Bias Pos
Bias 5
Bias
10
Pos
IO S
Pos Bias
Pos X I0
Pos X Pos
Pos X S
S X Bias
S x I0
5 X Pos
5 X
2
4
2
4
Figure 18: Attention score decomposition for the L9H6 name mover (see Figure for explanation). Notice that, in contrast with L10HO attention socres, there is no significant (inhibitory) interaction between the 10 features in the query and key.
41

---

Published as a conference paper at ICLR 2025
Attention to S1
Attention
Bias X Bias
Bias x IO
Bias X Pos
Bias X S 10 Blas
10 x I0
S
Bias
Pos X IO
Pos Pos
Pos S
$ Bias
$ X I0
5 X Pos
$ X S
2
2
Figure 19: Attention score decomposition for the L9H9 name mover (see Figure for explanation).
Head Class
Name Mover
Backup Name Mover
128
S-Inhibition
Induction
Duplicate Token
Features Per Example
Component
Output
32
Query
Value
Key
8
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
Logit Difference Recovered
Figure 20: Metrics for chosen task-specific SAEs for each relevant node in the IOI circuit. The x-axis shows the absolute value of the difference in logit differences between a clean run of the model. and where the activations at the given node replaced by the SAE's reconstructions, normalized by the difference between the clean logit difference and the logit difference when the node is mean-ablated instead. The y-axis shows the average number of features active per prompt.
42

---

Published as a conference paper at ICLR 2025
Head Class
Name Mover
Backup Name Mover
S-Inhibition
Induction
Features Per Example
Duplicate Token
32
Component
Output
Query
Value
Key
2
0.0
0.2
0.4 Logit Difference Recovered
0.6
0.8
1.0
Figure Counterpart to Figure 20 where the decoder vectors are frozen during SAE training.
0.9
v-110h0
4-19h9
q-110h10
z-18h6
0.8
V-I9h9
v-18h10
k-110h0
k-18h
2-15h9
q-110h2
k-18h6
v-18h6
2-17hv9
neurons
k-17h9
2-15h8
k--9h6
k-77h
k--11ho
2-17h3
0.6
z-110h10
v-17ha
q-110h1
2-1Gh9
0.5
2-110h2
k-111h2
z-10h10
V-17ha
k-110h10
k-110h1
e-110h0
0.3
2-41 Dh1
q-19h7
Loss Recovered
q-111n9
0.30
2-19hB
q-18h6
0.45
k-19h7
0.0
2.5
5.0
10.0
12.5
15.0
17.5
20.0
-h00h0
q-111h2
0.75
Test LO loss
q-19h6
k-19h0
0.90
Figure 22: to loss versus fraction of dead neurons for our full-distribution SAEs.
43

---

Published as a conference paper at ICLR 2025
v-110h0
q-19h6
k-J9h77
v-19h6
2-19h6
q-111h2
17.5
V-19ha
q-19h9
k-19ho
k-18h10
19h9
z-48h1a
K-18h6
v-J8h1D
q-110h10
k-17h9
2-15h9
z 2-18h6
12.5
v-58h6
k-110ho
Z-11 0h10
2-45h8
q-110h2
10.0
z- 1111h
1h-11h9
2 2-17h9
2-11 Dh2
V-17he
k--1966
7.5
1-11h
z-16h9
z-17h3
k-110h10
k-411h2
q-110h1
2-15h5
V-17h3
k-19h9
5.0
2-110h1
k-110h1
z-Iahia
Z-1hh7
k-110h2
2-43hD
z z-9hh
q-19h7
q-110ho
4-19h0
q-111h9
0.0
2h0
q-18h6
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Figure 23: to loss versus loss recovered (against a mean ablation) for our full-distribution SAEs.
IO in (B)NM out
Pos in (B)NM
Pos in (B)NM qk
Pos in Dn out
Pos in S-I out
Intervention
Edit
Pos in S-I
Edit 4
in (B)NM q
Edit
S in (B)NM qk
S Ind+DT out
S in S-I out
S in v
-2
-1
Distribution of the average feature weight removed by our interpretatioo-aanosiiic edits using SAE feature dictionaries, over all locations in a given cross-section, normalized by the cor- responding weight for the supervised feature dictionaries. Weight removed values are transformed linearly so that a value of ( indicates that the weight removed by the edit equals the weight removed by the corresponding 'ground truth' supervised edit; and value of indicates that the edit removed total weight of 1, meaning that the edit essentially overwrites all SAE features present in the acti- vation. Negative values are clipped -5 to preserve readability.
Figure 24:
44

---

Published as a conference paper at ICLR 2025
IO in (B)NM out
Pos in (B)NM q
Pos in (B)NM qk
Pos in DT out
Intervention
Pos in S-I out
Edit 8
Pos in S-I
Edit 16
Edit 32
S in (B)NM
Edit 64
in (B)NM qk
S in DT out
S in S-I out
S in v-1
-5
4
-2
0
1
Figure 25: Counterpart of Figure 24 for full-distribution SAEs, when editing using features with
high F1 score for the attribute
1.0
0.8
Intervention
0.6
No intervention
Edit
0.4
0.2
Edit
Supervised edit
0.0
10 in (BJNM out
Pos in (BINM
Pos in (BINM qk
Pos in Ind+DT out
Pos out
Pos in S-I
in (BINM a
5 (B)NM qk
i Ind+DT out
in out
in S4
Figure 26: Counterpart to Figure 28
where the decoder vectors frozen during SAE training.
45

---

Published as a conference paper at ICLR 2025
1.0
0.8
Intervention
intervention Edit
0.6
Edit
0.4
Edit 16
0.2
Edit 32
Supervised edit
0.0
10 in (B)NM out
Pos in (BJMM
Pos in (BINM gk
Pos Ind+DT out
Pos in 5-1 out
Pos in 5-1
5 (BJNM q
5 (BINM gk
i Ind+DT out
5 out out
5 i-I v
Figure 27: Counterpart to Figure 28 where we use full-distribution SAEs instead. Here, we need to
change a much higher number of features in order to have noticeable effect (and sometimes editing even 32 features fails)
1.0
0.8
Intervention
0.6
No intervention
Edit 2
0.4
Edit
0.2
Edit
Supervised edit
0.0
10 in (BJNM out
Pos in (B)NM
Pos in (BINM ak
Pas in Ind+DDT out
Pos in 5-1 out
Pos in In S-1v
S (BNNM q
in (B)NM ak
in Ind+OT out
5 out
S4 v
Figure 28: Accuracy when editing 10. S and Pos for circuit cross-sections using our supervised feature dictionaries and task-specific SAEs; the outcome in the absence of intervention is shown in blue for reference. When using task-specific SAEs, we edit either 2, 4 6 features (which means we in total add and/or remove up to that many features from activations). For comparison, supervised edits always involve removing feature and adding feature. Accuracy measured as the proportion of examples for which the model's prediction agrees with the ground-truth prediction for the edit; see Section and Appendix 7.5 for details.
46

---

Published as a conference paper at ICLR 2025
Duplicate Token
Lottl output
Outpung
LOH10 output
output
Intuction
Outpuds
16H0 output
LSH0 output
10 2nd name
I <namex
tobilmon
L7H3 output
is <name> and at 2nd position
position
Oudppty
17H9 output
AHH ourput
10 <nemee and at 1st position
output
position
L7HB value
is male
iimblaation
female
Valuns
L7H9
I0 is male
female
LBH10 value
<name>> sentence
Nome Nover
L9H9
taken <names and 1st positiin
Quants
query
taken
position
L10H0 query
current token is male
Mame Mover
L9HB keys
template
Neys
keys
Name Never
L9H9 output
Output
L9H6
Figure 29: Interpreting the features learned by the task SABs. For each node in the main IOI circuit (without backup/negative name movers), we show the distribution of the features which have an explanation with Fi score above threshold. The SAE chosen each node is the one with the most interpretable features out of all SAEs trained on this node during our hyperparameter sweep.
Dupicaate Taken Outpids
output
L3HD output
LSH5
Irsuudoon
Outpott
output
output
10 1st name
L5H9 output
L7H3
i <names and at 1st position
Simbiibtion
Outpuos
output
10 <name>
LSHG output
L8H10 output
<name> and st 2nd position
is female
Siobsotton
Values
L711 value
L8HG value
famale
L8H10
Hame Mover
query
taken
current token is <names
Quernes
19H6 query
taken <nnamee and at 2nd position
L10H1
Namo Mover
keys
current token is famale
Keys
L9H1 keys
L10HD keys
Nome Mover.
19H0 output
Outpars
output
10HD output
Figure 30: Interpreting the features learned by the task SAEs. This is a counterpart to Figure 29 for the SAEs chosen based the Eg and logit difference recovered metrics.
47

---

Published as a conference paper at ICLR 2025
1.0
0.8
Intervention
0.6
No intervention
Edit
0.4
Edit 4
0.2
Edit 6
Supervised edit
0.0
1O (BJNM out
Pos in (B)NM q
Pos in (BINM ak
Pos Ind+DT out
Pos in out
Pos in 5-1
in (BNNM a
in (B)NM gk
in Ind+DT out
5 in out
in S- 5
Figure 31: Interpretation-awee aware sparse control, using task SAE features with the highest F1 score
with respect to the given attribute.
1.0
0.8
Intervention
No intervention
0.6
Edit
Edit 16
0.4
Edit 32
0.2
Edit
Supervised edit
0.0
10 in (B)NM out
Pos in (BINM ak
Pas in Ind+DT out
Pos in 5-1 out
Pos in 5-1v
in (B)NM q
5 (B)NM ak
in Ind+DT out
in out
Pos in NB)
5
Figure 32: Counterpart of Figure 31 with full-distribution SAES.
48

---

Published as a conference paper at ICLR 2025
1.0
0.8
0.6
Intervention
0.4
Keep only features with F1 2 0.0
Keep only features with F = 0.2
0.2
Keep only features with F1 = 0.4
Keep only features with F1 z 0.6
Keep only features with F 0.8
0.0
Ind+DT out
S-I V
S-I out
(B)NM q (B)NM qk (B)NM out
Figure 33:
Measuring the sufficiency of interpretable features for task SAEs: effect of subtracting features with the lowest score from activations on logit difference. A value of I best.
1.0
0.8
0.6
0.4
Intervention
Remove features with F 2 0.0
Remove features with F 2 0.2
0.2
Remove features with F1 2 0.4
Remove features with F1 2 0.6
Remove features with F1 2 0.8
0.0
Ind+DT out
S-1 v
S-I out (B)NM q
(B)NM qk (B)NM out
Figure 34: Measuring the necessity of interpretable features for task SAEs: effect of removing features with the highest F, score from activations on the logit difference. Values are rescaled linearly that value of corresponds perfect recovery of the logit difference achieved by mean ablation (i.e., ideal intervention removing all features). A value of is best.
49


Reviews:
[{"review_id": "3Fh1ANhB3y", "reviewer": "Reviewer_jgpT", "summary": "The authors propose a (principled) method allowing to create supervised dictionaries for space features, which allow for evaluating the degree of disentanglement of SAEs. The developed method is then applied to SAEs and LLMs, witnessing not only interpretable later variables, but also providing possibility of editing attributes. Metrics of sufficiency, necessity and control are used for this.", "questions": "In the title of Figure 1, could you make a clear connection of the text with precise parts of visuals, to facilitate the understanding?\n\nTaking into account the size of the paragraph on related work, it should be possible to describe the related work without much terminology and thus shifting it closer to the beginning of the manuscript. This would allow the reader to better position the framework with respect to the state of the art.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In general the manuscript is well written and \"strong\".\n\nThe question which is really to ask is how much this finding is relevant for the literature. Here I should say my expertise is perhaps too limited to provide a proper judgment.", "weaknesses": "Though this has most probably costed a lot of work, the empirical validation of the proposed methodology is rather scarce. Whether the constructed dictionaries would also function for other tasks/semantics is not clear.\n\nThe mathematics are well explained, in clear and simple way.\n\nSome (not that many) parts of the manuscript I had to read several times, e.g., the title under Figure 1 or the paragraph on interpretability at the end of Section 3.2. But in general formulas aid much understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ttFGrS6I5I", "reviewer": "Reviewer_S53N", "summary": "This paper studies the Sparse autoencoders to capture interpretable features for the IOItask and the expeirment results show that the proposed approach achieves the best performance.", "questions": "Please see the weakness section.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and easy to follow.", "weaknesses": "There are several weaknesses:\n\n1. The motivation of this section should be enhanced.\n\n2. The English language should be improved.\n\n3. The main idea seems not very novel. This paper should provide a strong motivation.\n\n4. The experiment can be further improved by providing more results and analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cCZ8f2yWqG", "reviewer": "Reviewer_w15N", "summary": "This paper focuses on evaluating sparse autoencoders (SAEs) for their ability to recover known ground-truth features learned by a model. To do so, the authors first train a supervised dictionary on the indirect object identification (IOI) task, for which model computation is already relatively known due to prior interpretability and circuit discovery work. Both IOI task-specific SAEs and full-distribution SAEs are trained and evaluated with respect to the supervised dictionaries to understand if SAEs allow for the same level of approximation, control, and interpretability as the supervised dictionaries. Results reveal that more recent SAE architectures improve these capabilities and task-specific SAEs are much more effective than full-distribution SAEs.", "questions": "- In section 4.3, if I understand correctly, the SAE latents are found by simply optimizing/searching for the features that perform the task (move one latent to the specified counterfactual latent). This seems a bit roundabout to me - wouldn’t this propose that you need to know the features you are looking for in order to label SAE features? How would one do this searching or interpretation without access to the counterfactual activations? Wouldn’t it be more realistic to interpret or label each SAE feature and then use the features that are labelled to be relevant to the task at hand?\n- Please see the above weaknesses!", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The writing and paper structure are very clear and easy-to-follow.\n- The focus of this paper is highly relevant and interesting. The problem of finding a ground truth with which to evaluate interpretability and explainability methods has remained an issue for decades, and this work works towards solving this problem by exploring using human-generated groundtruths that have been backed up by prior work.\n- The experiments are well-defined, inutitive, and easy to understand.\n- I believe the results are interesting and useful - they reveal that task-specific SAEs are more useful in practice than full-distribution SAEs, hinting that data quality is of utmost importance when training SAEs. Further, this suggests that human priors may be useful when developing interpretability methods/SAEs.", "weaknesses": "- I find the “Why use these attributes?” paragraph in Section 3.1 to be confusing. If prior work had not proposed the IO, S, and Pos attributes, how would one go about defining the supervised dictionary? If the evaluation pipeline described in this paper were to be used for a different task, I’m not sure whether this section would be generalizable. In particular, when there are many choices of attributes, what is the manner of choosing between them without using another interpretability method, which would then prevent the need of using an SAE in the first place?\n- It would have been significantly more convincing to me if the authors had considered more than one task in their evaluation. At the moment, it’s unclear to me how the proposed methodology and results from this work could be applied to future works that want to evaluate trained SAEs.\n- The section on interpretability (section 4.4) is also a bit confusing to me - I would find it very helpful if the authors provided interpretations of the SAE latents, and a visualization of how these features could then be used to explain the LLM’s computation on a single example. Some examples of /possible/ interpretations are provided in Appendix 7.13-7.14, but if I understand correctly these are not the actual labels of the SAE features.\n- It is my understanding that the authors wish to propose the use od supervised dictionaries an evaluatory baselines for SAEs. However, in practice, this paper reads more as an exploration of whether SAEs can recover the IOI task. While the authors discuss the limitations of hardcoding the attributes to compare SAEs against and only considering a single task and model, I believe these drawbacks fundamentally limit the work in its general utility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0BDW41XtiK", "reviewer": "Reviewer_TRbA", "summary": "This paper introduces a way to compute sparse feature dictionaries that disentangle model computations similar to an SAE fashion using a supervised method; as well as introduces several ways to evaluate these feature dictionaries from different perspective such as necessity, sufficiency and controls of the feature dictionaries towards a specific task. The author applies the work to the indirect object identification (IOI) using GPT-2 Small, and compare the feature dictionaries obtained by their method, vs some other recent SAE variants such as Gated SAE adn Top-K SAE.", "questions": "- Equation 2. Reason for the \"bias\" term is $E[a(p)]$? Does this mean $E[u_{IO}] + E[u_S] + E[u_{POS}] \\approx 0$ if we take the expected value on $a(p)$ and $\\hat{a}(p)$? If it is by design, can we make a comment on this design? What is a brief explanation behind this design?\n- What is F and A in F1 Score? In the text, it seems F=the set of examples activating a specific SAE latent \"f\". A=binary attribute of a prompt. It seems that the F1 Score is applied **on each SAE latent**, as described in Section 4. How do we get \"A\" in this case? How do we know which \"binary\" attribute of a prompt that the latent f corresponds to? Can we give a more detailed explanation in the text? It would be nice to include some examples of F and A (specifically A)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Thorough study in the case of IOI. The use of IOI in the study mainly because we know the \"feature\" that more or less directly affect the outputs. This minimizes the possibilities of the cases that \"SAE features\" may not coincide with the features that human understands. This was also implied in Section 4.2.\n- Extensive study on the paper that includes a lot of results in the appendix.\n- Some proposed methods such as Necessity and sufficiency, to the control method proposed can be applied to a more general case, as described in Section 4. \n- This paper also addresses a lot of the details on small nuances in evaluation of SAEs. This includes the discussion in Session 4.2, and the paragraph on \"Preventing...\" in Section 4.3.", "weaknesses": "**Overall**\n\n- As this paper is a case study to IOI, it is somewhat restrictive. I don't think it is necessarily a weakness of the paper though as I don't think anything can be done to address this \"weakness\". It could be viewed as a stepping stone for future work as well.\n- Presentation and Readability are the main weakness of this work. For example, experiment results in Section 3 includes a lot in section 4 such as description of FullDistribution SAE, Task SAE, TopK, Gated SAE etc. There are crucial formulae and methods that should be in the main text, but instead they are in the Appendix. For instance, the necessity formula in Section 7.4 ~line 878 in the appendix.\n- The section on \"Interpretability\" such as in Section 3.2, Figure 5 and Section 4.4 are out of place. The author even mentioned that \"we stress that this experiment is of a rather exploratory and qualitative nature\". Putting these sections in the appendix would be much more coherent.\n- Section 6 on discussions and limitations is too short. There should be more discussions on the experiment results. Possibilities of applying similar techniques to a general settings (as IOI provides the \"known features\"). Including a discussion on future work may be useful.\n\n**Detailed issues**\n\n- On Sufficiency in \"Figure 3. Left\" - as the experiment is to test whether the reconstructions are sufficient, we would hope to compare the logit difference of the original and the reconstruction. The method in Figure 3 shows the ratio of the *average* logit difference with and without the intervention. This may not be the best because the averaging may hide the changes in logit difference with the intervention for each example. A simpler method like the average changes (absolutized) of the logit difference _may_ work. The authors can also opt to include a brief discussion on this so that it does not seem like they were hiding something. For example, showing the distribution of  absolute changes of the logit difference in a histogram, or some statistics on it.\n- Necessity. The experiment is to test whether the reconstructions are necessary. This means that we want to show that without the reconstructions, the model cannot do so well, resulting a drop in model performance - i.e. logit difference. Thus there are three quantities\n1) The reconstruction $\\hat{a}(p)$\n2) The proposed quantity; average plus SAE error term\n3) The average.\n\nShowing necessity should be showing the difference between 1) and 2) are large. However, in the main text of the paper, it opts to show the difference between 2) and 3) only. A crucial formula and description of the necessity score directly addressing this problem is in the appendix (Section 7.4, around line 878), which in my opinion, should be in the main text.\n- \"Probing accuracy\" in \"Control\": Seems out of place? Is it referenced somewhere else in the paper? Also no results were shown? In the absence of results, this section on probing accuracy does not seem to achieve the goal of the section: \"measures the degree to which the supervised feature dictionaries disentangle the different attributes\". This is because the probe is linear, which itself can \"disentagle\" the attributes. I think it would be better to either remove this section (put this in the appendix), or show some experiment results with discussions on the way to disentangle the attributes.\n\n**Minor**\n- Section 4.3 Expressing... line ~361. edit 3 --> (3) or Equation 3.\n- Results (line ~424). objective 4 --> (4) or Equation 4.\n- missing parenthesis at line 434 (resp. $a(p_t)$ **)** by their...\n- broken references in appendix (line 883, 935, 1849)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)