Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control

Authors: Aleksandar Makelov, Georg Lange, Neel Nanda
Primary area: interpretability and explainable AI
Keywords: mechanistic interpretability, sparse autoencoders, evaluations

Abstract:
Disentangling model activations into human-interpretable features is a central
problem in interpretability. Sparse autoencoders (SAEs) have recently attracted
much attention as a scalable unsupervised approach to this problem. However, our
imprecise understanding of ground-truth features in realistic scenarios makes it
difficult to measure the success of SAEs. To address this challenge, we propose
to evaluate SAEs on specific tasks by comparing them to supervised
feature dictionaries computed with knowledge of the concepts relevant to the
task. 

Specifically, we suggest that it is possible to (1) compute supervised sparse
feature dictionaries that disentangle model computations for a specific task;
(2) use them to evaluate and contextualize the degree of disentanglement and
control offered by SAE latents on this task. Importantly, we can do this in a
way that is agnostic to whether the SAEs have learned the exact ground-truth
features or a different but similarly useful representation.

As a case study, we apply this framework to the indirect object identification
(IOI) task using GPT-2 Small, with SAEs trained on either the IOI or OpenWebText
datasets. We find that SAEs capture interpretable features for the IOI task, and
that more recent SAE variants such as Gated SAEs and Top-K SAEs are competitive
with supervised features in terms of disentanglement and control over the model.
We also exhibit, through this setup and toy models, some qualitative phenomena
in SAE training illustrating feature splitting and the role of feature
magnitudes in solutions preferred by SAEs.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730315827176

thread_type:
official_review

<thread_object>
creator: reviewer_trba

<content>
summary:
This paper introduces a way to compute sparse feature dictionaries that disentangle model computations similar to an SAE fashion using a supervised method; as well as introduces several ways to evaluate these feature dictionaries from different perspective such as necessity, sufficiency and controls of the feature dictionaries towards a specific task. The author applies the work to the indirect object identification (IOI) using GPT-2 Small, and compare the feature dictionaries obtained by their method, vs some other recent SAE variants such as Gated SAE adn Top-K SAE.

strengths:
- Thorough study in the case of IOI. The use of IOI in the study mainly because we know the "feature" that more or less directly affect the outputs. This minimizes the possibilities of the cases that "SAE features" may not coincide with the features that human understands. This was also implied in Section 4.2.
- Extensive study on the paper that includes a lot of results in the appendix.
- Some proposed methods such as Necessity and sufficiency, to the control method proposed can be applied to a more general case, as described in Section 4. 
- This paper also addresses a lot of the details on small nuances in evaluation of SAEs. This includes the discussion in Session 4.2, and the paragraph on "Preventing..." in Section 4.3.

weaknesses:
**Overall**

- As this paper is a case study to IOI, it is somewhat restrictive. I don't think it is necessarily a weakness of the paper though as I don't think anything can be done to address this "weakness". It could be viewed as a stepping stone for future work as well.
- Presentation and Readability are the main weakness of this work. For example, experiment results in Section 3 includes a lot in section 4 such as description of FullDistribution SAE, Task SAE, TopK, Gated SAE etc. There are crucial formulae and methods that should be in the main text, but instead they are in the Appendix. For instance, the necessity formula in Section 7.4 ~line 878 in the appendix.
- The section on "Interpretability" such as in Section 3.2, Figure 5 and Section 4.4 are out of place. The author even mentioned that "we stress that this experiment is of a rather exploratory and qualitative nature". Putting these sections in the appendix would be much more coherent.
- Section 6 on discussions and limitations is too short. There should be more discussions on the experiment results. Possibilities of applying similar techniques to a general settings (as IOI provides the "known features"). Including a discussion on future work may be useful.

**Detailed issues**

- On Sufficiency in "Figure 3. Left" - as the experiment is to test whether the reconstructions are sufficient, we would hope to compare the logit difference of the original and the reconstruction. The method in Figure 3 shows the ratio of the *average* logit difference with and without the intervention. This may not be the best because the averaging may hide the changes in logit difference with the intervention for each example. A simpler method like the average changes (absolutized) of the logit difference _may_ work. The authors can also opt to include a brief discussion on this so that it does not seem like they were hiding something. For example, showing the distribution of  absolute changes of the logit difference in a histogram, or some statistics on it.
- Necessity. The experiment is to test whether the reconstructions are necessary. This means that we want to show that without the reconstructions, the model cannot do so well, resulting a drop in model performance - i.e. logit difference. Thus there are three quantities
1) The reconstruction $\hat{a}(p)$
2) The proposed quantity; average plus SAE error term
3) The average.

Showing necessity should be showing the difference between 1) and 2) are large. However, in the main text of the paper, it opts to show the difference between 2) and 3) only. A crucial formula and description of the necessity score directly addressing this problem is in the appendix (Section 7.4, around line 878), which in my opinion, should be in the main text.
- "Probing accuracy" in "Control": Seems out of place? Is it referenced somewhere else in the paper? Also no results were shown? In the absence of results, this section on probing accuracy does not seem to achieve the goal of the section: "measures the degree to which the supervised feature dictionaries disentangle the different attributes". This is because the probe is linear, which itself can "disentagle" the attributes. I think it would be better to either remove this section (put this in the appendix), or show some experiment results with discussions on the way to disentangle the attributes.

**Minor**
- Section 4.3 Expressing... line ~361. edit 3 --> (3) or Equation 3.
- Results (line ~424). objective 4 --> (4) or Equation 4.
- missing parenthesis at line 434 (resp. $a(p_t)$ **)** by their...
- broken references in appendix (line 883, 935, 1849)

questions:
- Equation 2. Reason for the "bias" term is $E[a(p)]$? Does this mean $E[u_{IO}] + E[u_S] + E[u_{POS}] \approx 0$ if we take the expected value on $a(p)$ and $\hat{a}(p)$? If it is by design, can we make a comment on this design? What is a brief explanation behind this design?
- What is F and A in F1 Score? In the text, it seems F=the set of examples activating a specific SAE latent "f". A=binary attribute of a prompt. It seems that the F1 Score is applied **on each SAE latent**, as described in Section 4. How do we get "A" in this case? How do we know which "binary" attribute of a prompt that the latent f corresponds to? Can we give a more detailed explanation in the text? It would be nice to include some examples of F and A (specifically A)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730568897845

thread_type:
official_review

<thread_object>
creator: reviewer_w15n

<content>
summary:
This paper focuses on evaluating sparse autoencoders (SAEs) for their ability to recover known ground-truth features learned by a model. To do so, the authors first train a supervised dictionary on the indirect object identification (IOI) task, for which model computation is already relatively known due to prior interpretability and circuit discovery work. Both IOI task-specific SAEs and full-distribution SAEs are trained and evaluated with respect to the supervised dictionaries to understand if SAEs allow for the same level of approximation, control, and interpretability as the supervised dictionaries. Results reveal that more recent SAE architectures improve these capabilities and task-specific SAEs are much more effective than full-distribution SAEs.

strengths:
- The writing and paper structure are very clear and easy-to-follow.
- The focus of this paper is highly relevant and interesting. The problem of finding a ground truth with which to evaluate interpretability and explainability methods has remained an issue for decades, and this work works towards solving this problem by exploring using human-generated groundtruths that have been backed up by prior work.
- The experiments are well-defined, inutitive, and easy to understand.
- I believe the results are interesting and useful - they reveal that task-specific SAEs are more useful in practice than full-distribution SAEs, hinting that data quality is of utmost importance when training SAEs. Further, this suggests that human priors may be useful when developing interpretability methods/SAEs.

weaknesses:
- I find the “Why use these attributes?” paragraph in Section 3.1 to be confusing. If prior work had not proposed the IO, S, and Pos attributes, how would one go about defining the supervised dictionary? If the evaluation pipeline described in this paper were to be used for a different task, I’m not sure whether this section would be generalizable. In particular, when there are many choices of attributes, what is the manner of choosing between them without using another interpretability method, which would then prevent the need of using an SAE in the first place?
- It would have been significantly more convincing to me if the authors had considered more than one task in their evaluation. At the moment, it’s unclear to me how the proposed methodology and results from this work could be applied to future works that want to evaluate trained SAEs.
- The section on interpretability (section 4.4) is also a bit confusing to me - I would find it very helpful if the authors provided interpretations of the SAE latents, and a visualization of how these features could then be used to explain the LLM’s computation on a single example. Some examples of /possible/ interpretations are provided in Appendix 7.13-7.14, but if I understand correctly these are not the actual labels of the SAE features.
- It is my understanding that the authors wish to propose the use od supervised dictionaries an evaluatory baselines for SAEs. However, in practice, this paper reads more as an exploration of whether SAEs can recover the IOI task. While the authors discuss the limitations of hardcoding the attributes to compare SAEs against and only considering a single task and model, I believe these drawbacks fundamentally limit the work in its general utility.

questions:
- In section 4.3, if I understand correctly, the SAE latents are found by simply optimizing/searching for the features that perform the task (move one latent to the specified counterfactual latent). This seems a bit roundabout to me - wouldn’t this propose that you need to know the features you are looking for in order to label SAE features? How would one do this searching or interpretation without access to the counterfactual activations? Wouldn’t it be more realistic to interpret or label each SAE feature and then use the features that are labelled to be relevant to the task at hand?
- Please see the above weaknesses!
</content>
</thread_object>

</thread>
<thread>
creation date:
1730599806002

thread_type:
official_review

<thread_object>
creator: reviewer_s53n

<content>
summary:
This paper studies the Sparse autoencoders to capture interpretable features for the IOItask and the expeirment results show that the proposed approach achieves the best performance.

strengths:
This paper is well-written and easy to follow.

weaknesses:
There are several weaknesses:

1. The motivation of this section should be enhanced.

2. The English language should be improved.

3. The main idea seems not very novel. This paper should provide a strong motivation.

4. The experiment can be further improved by providing more results and analysis.

questions:
Please see the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730850184949

thread_type:
official_review

<thread_object>
creator: reviewer_jgpt

<content>
summary:
The authors propose a (principled) method allowing to create supervised dictionaries for space features, which allow for evaluating the degree of disentanglement of SAEs. The developed method is then applied to SAEs and LLMs, witnessing not only interpretable later variables, but also providing possibility of editing attributes. Metrics of sufficiency, necessity and control are used for this.

strengths:
In general the manuscript is well written and "strong".

The question which is really to ask is how much this finding is relevant for the literature. Here I should say my expertise is perhaps too limited to provide a proper judgment.

weaknesses:
Though this has most probably costed a lot of work, the empirical validation of the proposed methodology is rather scarce. Whether the constructed dictionaries would also function for other tasks/semantics is not clear.

The mathematics are well explained, in clear and simple way.

Some (not that many) parts of the manuscript I had to read several times, e.g., the title under Figure 1 or the paragraph on interpretability at the end of Section 3.2. But in general formulas aid much understanding.

questions:
In the title of Figure 1, could you make a clear connection of the text with precise parts of visuals, to facilitate the understanding?

Taking into account the size of the paragraph on related work, it should be possible to describe the related work without much terminology and thus shifting it closer to the beginning of the manuscript. This would allow the reader to better position the framework with respect to the state of the art.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732412899539

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Top-level comment on motivation/novelty

comment:
We thank all reviewers for their feedback. Since the unclear presentation of the paper's motivation and novelty were shared concerns for multiple reviewers, we would like to respond with a top-level comment addressing this concern.

We have (in the attached revision) **reworked the introduction and related work sections of the paper to emphasize the motivation of our work and what sets it apart from other works in this area** (changes marked in red). In brief:
- the research community has invested a lot of effort in the SAE paradigm; the main goal of this paradigm is to recover features the model uses to perform computations on various tasks.
- However, prior evaluations have focused on *proxy metrics* that we only hope correlate with recovery of these features
- The novelty of our work is, by contrast, in being the first to compare SAEs to carefully validated approximations of ground-truth features in a realistic linguistic task (as well as two other tasks we added during the discussion period to showcase the flexibility of our framework; see the top-level comment on generalizability of results). Moreover, these ground-truth features provide a skyline of a supervised dictionary trained to predict the labels of features discovered by prior work, which allows a fair comparison grounded in the limits of the representational power of SAEs, and allows us to contextualize the outputs of our metrics. 

While we have added two other tasks to demonstrate broader applicability (see other top comment), we believe our framework provides significant value even with a limited task selection. Practitioners often need to evaluate SAEs across hyperparameter sweeps [1] or compare new architectures [2,3,4], and the IOI task better represents realistic downstream applications than standard proxy metrics (which we discuss in the related work of the paper). Even though IOI is a special case where prior work identified relevant features, evaluating full-distribution SAEs on this task provides evidence for their effectiveness on tasks where ground truth is unknown. The same applies to evaluating SAE training methods via task-specific SAEs.

We thank the reviewers for pointing out the poor readability of the paper's motivation, and hope that our revisions and comments will mitigate this issue.

[1] Lieberum, Tom, et al. "Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2." *arXiv preprint arXiv:2408.05147* (2024).

[2] Rajamanoharan, Senthooran, et al. "Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders." *arXiv preprint arXiv:2407.14435* (2024).

[3] Gao, Leo, et al. "Scaling and evaluating sparse autoencoders." *arXiv preprint arXiv:2406.04093* (2024).

[4] Rajamanoharan, Senthooran, et al. "Improving dictionary learning with gated sparse autoencoders." *arXiv preprint arXiv:2404.16014* (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732412994941

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Top-level comment on generalizability of results

comment:
We thank all reviewers for their feedback. Since the seemingly restricted nature of the paper's evaluation (which is based only on the IOI task) was a shared concern of most reviewers, we would like to respond with a top-level comment addressing this.

We have (in the attached revision) **added a new appendix section (as well as relevant pointers from the main text) on how our methods can be straightforwardly extended to a new natural language task we introduce on a different model and distribution, to test an SAEs ability to independently represent and disentangle different concepts**. We additionally apply our methods to another task from the mechanistic interpretability literature that has been studied in a way similar to IOI: the greater-than task [1]

In brief:
- We generalize the setup of the paper in multiple ways:
  - **dataset and model**: we use the Tiny Stories dataset and the 4-layer 33M parameter model from the Tiny Stories paper (https://arxiv.org/abs/2305.07759)
  - **full-distribution SAEs**: we use TopK SAEs trained on the full set of Tiny Stories activations over the 4 residual streams of the model (the dataset has on the order of 100s of millions of tokens)
- We consider sentences of the form "NAME really loves ANIMALS. NAME also really loves SPORT. So NAME really loves both" 
  - For example, "Lily really loves cats. Lily also really loves basketball. So Lily really loves both" should be followed by "cats and basketball" or "basketball and cats", so the model should be able to put high probability on both cats and basketball coming next. A well functioning SAE should be able to disentangle these, ie we should be able to damage the model's ability to say cats but not basketball, and vice versa
  - We consider several values for each of NAME, ANIMALS and SPORT to sample a dataset of 1k prompts of this form
- Intuitively, at the "both" token, the model "prepares" to output either the ANIMALS or SPORT. We verify experimentally that the next-token probability distribution after "both" prefers the correct SPORT compared to other sports, as well as the correct ANIMALS compared to other animals.
- We turn this into a task to which we apply (key parts of) our framework. Namely:
  - We compute supervised features corresponding to each given SPORT and ANIMALS value by taking conditional expectations (like in the paper)
  - We check that the supervised features work to edit the logit distribution in the expected way
  - We compare this to SAEs via a sparse control test to change the value of one variable at a time via an SAE latent exchange. 
  - This setup also allows us to measure the "disentanglement" between the two attributes (sport and animals) afforded by the SAE latents.

Overall, we see that our framework applies with minimal modification to this different setting, and yields interesting results about SAEs. 

We thank the reviewers for pointing out this weakness in the presentation of our paper, and hope that this new addition will serve as a convincing example of the generality of our evaluation framework.


[1] Hanna, Michael, Ollie Liu, and Alexandre Variengien. "How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model." Advances in Neural Information Processing Systems 36 (2024).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper explores sparse autoencoders on language tasks that have a known ground truth to evaluate whether SAEs can provide similar interpretability and control as supervised feature dictionaries.

Since most language tasks do not have a single ground truth, the authors originally used only one highly specialized experimental setting. Reviewers found this to be lacking, and also gave advice on how to improve the presentation, and better communicate the limitations of the work. After discussion, the authors heavily revised their presentation, included better discussion of limitations, and added additional experimental settings. All these changes were well-received by the reviewers, and the paper now stands as a clear and interesting contribution to studies on interpretability techniques for language models. This is a valuable research direction, and one in desperate need of more investigation by the community.

As such, I am recommending acceptance as a Spotlight.