Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models

Authors: Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, Aaron Mueller
Primary area: interpretability and explainable AI
Keywords: interpretability, mechanistic interpretability, circuits, spurious correlations, generalization, dictionary learning

Abstract:
We introduce methods for discovering and applying **sparse feature circuits**. These are causally implicated subnetworks of human-interpretable features for explaining language model behaviors. Circuits identified in prior work consist of polysemantic and difficult-to-interpret units like attention heads or neurons, rendering them unsuitable for many downstream applications. In contrast, sparse feature circuits enable detailed understanding of unanticipated mechanisms in neural networks. Because they are based on fine-grained units, sparse feature circuits are useful for downstream tasks: We introduce SHIFT, where we improve the generalization of a classifier by ablating features that a human judges to be task-irrelevant. Finally, we demonstrate an entirely unsupervised and scalable interpretability pipeline by discovering thousands of sparse feature circuits for automatically discovered model behaviors.

Decision: accept (oral)
Reviewer scores: 8, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730508803911

thread_type:
official_review

<thread_object>
creator: reviewer_teh3

<content>
summary:
This paper introduces a method that automatically discovers "sparse feature circuit", defined as subgraphs of a large language model that satisfy two desiderata: (1) responsible for implementing a model behavior, and (2) human-interpretable. As a by-product, the authors propose SHIFT, a method that removes non-causal features (e.g. gender bias) from a classifier. Finally, the authors scale their method to thousands of automatically discovered behaviors from SAEs, hinting at the possibility of automated interpretability.

strengths:
- This paper is very well written, and provides enough background on SAEs to motivate their approach. Each component of the method is also presented very clearly.
- This contribution is very timely. Most prior works on mechanistic interpretability suffers from two pitfalls: (1) Inability to understand / interpret dense MLP layers [1], and (2) what's next after interpretability. This work provides satisfactory answers to both, by (1) using more recent approaches of SAEs to disentangle polysemanticity and (2) using interpretable features to debias a classifier / remove non-causal relationships from it.
- Owning to the scalability of SAEs, this method can be scaled to automatically discover thousands of behaviors to billion scale parameters. The authors also supplement this ability with an interactive website, which offers many interesting insights and improves accessibility of this research.

[1] How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, https://arxiv.org/abs/2305.00586

weaknesses:
- While this approach is scalable with respect to model size and a broad range of behaviors, using this approach would still require contrastive sample pairs that is hard to get and / or define for tasks such as long-form generation and algorithmic reasoning. Nevertheless, I believe this work represents a step in the right direction, but it'd be great for the authors to discuss this type of dataset limitations and future directions to automatically construct contrastive pairs for more challenging tasks.
- As the authors have mentioned, most of these results are qualitative, and it would be great to establish a more quantitative benchmark to supplement the other interesting behaviors that the authors were able to mine from their unsupervised approach.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730561021514

thread_type:
official_review

<thread_object>
creator: reviewer_xplm

<content>
summary:
This paper introduces a method for discovering interpretable sparse feature circuits in LLMs using sparse autoencoders (SAEs). While previous mechanistic interpretability work focused on polysemantic neurons, MLP attention heads, this paper shows how to decompose model behaviors into monosemantic, human-interpretable features. The authors validate their approach through three key contributions: (1) a circuit discovery method combining SAEs with linear approximations of indirect effects, (2) a debiasing application called SHIFT that ablates task-irrelevant features, and (3) demonstration of scalable, automated circuit discovery for large sets of model behaviors.

strengths:
Strengths:

Successfully addresses a core challenge in MI: finding and scaling interpretable units of analysis. 


- Empirical Results:

Comprehensive subject-verb agreement circuits reveal clear compositional structure (e.g., distinct pathways for handling relative clauses vs. prepositional phrases)
Human evaluation shows significantly better feature interpretability (81.5 for BiB circuit features vs 36.0 for neurons)
SHIFT achieves 93.1% profession accuracy while reducing gender bias to near chance (52.0%)

- Validation & Reproducibility:

Thorough ablation studies on all key parameters (SAE width, layer selection, manipulation coefficients). Clear comparison against neuron-based approaches showing consistent advantages.

weaknesses:
Weaknesses:

- No code and data released yet. 

- Theoretical Analysis:

In terms of theoretical weaknesses, some formal characterization of when SAE features capture meaningful concepts vs memorized patterns would strength the paper. 


Most of these are noted in the paper but it's worth mentioning it here, SAE training requires significant compute (2 billion tokens) with potential instability (dead features). The current implementation limited to decoder-only models (Pythia-70M and Gemma-2-2B). 

- Evaluation Scope:

Subject-verb agreement circuits focus on relatively simple grammatical phenomena and this might not be an issue with scale but no mention of anything more complex in the paper.

questions:
- The paper shows strong results with linear approximations of indirect effects. Have you checked when/why these approximations break down? This seems especially relevant for early layers where you note integrated gradients improves accuracy.

- For the automated circuit discovery, how robust are the discovered clusters to different choices of projection dimension and clustering hyperparameters?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730747909516

thread_type:
official_review

<thread_object>
creator: reviewer_gvvk

<content>
summary:
This paper presents Sparse Feature Circuits, a method to discover interpretable causal graphs within language models (LMs) based on sparse, human-interpretable features. Leveraging sparse autoencoders (SAEs), the authors identify feature circuits that reveal causally implicated components in model behaviors, contrasting with prior work focused on more complex components like attention heads and neurons. The technique allows for targeted interventions to refine model predictions by eliminating task-irrelevant features, as demonstrated in a new method called SHIFT. SHIFT enables selective feature removal, improving classifier generalization without requiring disambiguating labels. The paper’s contributions include a scalable pipeline for automatic discovery of interpretable circuits, and applications in improving model behavior by editing these circuits.

strengths:
- By focusing on sparse feature circuits instead of larger, polysemantic components, this approach introduces a new level of granularity in LM interpretability. The use of SAEs is particularly innovative, as it allows for fine-grained, interpretable features that are scalable and can be applied across thousands of discovered behaviors, improving previous interpretability efforts that were limited by reliance on human-defined hypotheses. 

- The SHIFT technique, designed to improve model generalization by removing unintended signals, offers a practical, actionable method for handling spurious features. The fact that SHIFT can operate without disambiguating labels and apply human judgment to identify spurious features highlights its practical utility for real-world model refinement, especially in settings where unintended biases are a concern. 

- The unsupervised interpretability pipeline for discovering feature circuits across various model behaviors (including grammatical tasks like subject-verb agreement) is impressive. It enables scalable, automatic generation of feature circuits, addressing the labor-intensive nature of previous interpretability work. This unsupervised aspect expands the approach’s applicability across domains and data types, making it a versatile contribution. 

- The paper presents quantitative evaluations on interpretability metrics such as faithfulness and completeness for both feature and neuron circuits, providing objective assessments of the interpretability advantage of sparse features.

weaknesses:
- The method relies on human judgment to determine which features are spurious, which may lead to inconsistencies across applications or datasets. The framework lacks a clear evaluation on the repeatability and consistency of human-selected spurious features in SHIFT, especially when applied by different users or in different contexts. Including a quantitative study or qualitative analysis on the consistency of feature selection across different users or scenarios would strengthen the reliability of SHIFT’s results and enhance confidence in its practical application. For instance, assessing inter-annotator agreement or providing a standardized checklist could help mitigate inconsistencies. 

- Although SHIFT leverages human judgment for feature interpretation, this reliance could hinder scalability in complex or high-dimensional models, as manual inspection becomes increasingly challenging. The paper does not provide clear guidelines on how to efficiently scale this process for larger models or more diverse tasks. 

- The method’s success is heavily tied to the availability and quality of sparse autoencoders (SAEs). If the SAEs are suboptimal, the interpretability and causal circuit discovery might suffer, particularly if they fail to capture meaningful latent structures. This dependency on pretrained SAE quality raises questions about the method’s consistency and applicability across different model architectures or under domain shifts. 

- The faithfulness and completeness metrics for feature circuits are assessed primarily on grammatical tasks like subject-verb agreement, which may not generalize to more complex language tasks that involve nuanced semantic understanding or context-sensitive behaviors. This limited evaluation could affect confidence in the generalizability of the results. Expanding evaluations to include complex, semantically challenging tasks (e.g., multi-turn dialogue or reasoning tasks) would provide stronger evidence of the method’s robustness.

questions:
1. How do the authors ensure consistency in identifying spurious features across different annotators or applications? Are there any plans to standardize the interpretation criteria?

2. Could the authors clarify how the performance of SAEs affects the interpretability and effectiveness of feature circuits? Have they tested their approach on varying SAE architectures to assess robustness? 

3. For large-scale models with complex circuits, what optimizations or techniques do the authors recommend to maintain interpretability while managing the increased number of features?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731269228358

thread_type:
official_review

<thread_object>
creator: reviewer_sior

<content>
summary:
This paper introduces sparse feature circuits as a causal network built from human-interpretable features, rather than neurons. The author discuss how to quantify the Indirect Effects (IE) of steering or patching individual neurons on model behaviors and propose approximations to extend this approach to models with many components. 

Through experiments on the Gemme2-2B and Pythia-128M models, they demonstrate that a relatively low number of sparse features, compared to neurons, can interpret model behaviors with high fidelity. Building on these findings, the authors present the SHIFT framework, which enables the surgical removal of unintended features to prevent them from affecting model classification performance. While the framework involves some human judgment, it highlights that these Sparse Auto-Encoder (SAE) features are effective representations of concepts that can be manipulated to control model behaviors. Finally, the authors propose an unsupervised circuit discovery pipeline, moving from next-token clustering to zero-ablating features, to identify the most important feature circuits.

strengths:
* This paper is well-motivated and articulated;
* The studied subject is of good importance to the NN interpretability domain;
* The experiment setup is clear and the results are promising;
* The idea has sound novelty.

weaknesses:
* SAE features are vast, and identifying specific circuits among those formulated by these features can be challenging.
* SAE errors cannot be fully interpreted, while they play an important role in making sparse feature circuits. This is ongoing work to train more proficient SAEs, but it should be discussed more in the paper regarding how to perform surgical edits on such errors.
* I suspect that the circuits may only correlate with certain features. Simply removing gender-related features and observing a drop in classification performance is not universally convincing as a causal intervention approach; additional experiments on other features are needed to support this claim;
* Trivial concerns: 1) several typos: 283: fair; 201: produce; 492: succession; 2) citation formatting errorsat line 95-96; 3) line 454-457 needs more clarity.

questions:
* Can you add explanations regarding the efficiency of the sparse feature circuit finding with the vast amount of possible feature circuits with respect to neurons'?
* Can you help better understand how to interpret SAE errors at part of the sparse feature circuits?
* Can you share your thoughts on extended experiments on more feature ablating studies and how would the results be?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731629649753

thread_type:
public_comment

<thread_object>
creator: ~neel_nanda1

<content>
title:
Paper Thoughts

comment:
Speaking as an unrelated mechanistic interpretability researcher, I think this is **one of the most important mechanistic interpretability papers of the past year, and worth being highlighted at the conference**. I think this paper had two key contributions:

Demonstrating that sparse autoencoders were useful on the downstream task of SHIFT. Much mechanistic interpretability work purely focuses on showing that it produces understanding, and trying to directly argue that this understanding is true, by eg ablating model components predicted to be important, or finding patterns in where a neuron activates. But if we have achieved true understanding, it should enable greater control, and allow us to do things to models that we couldn't easily do before - and if we can't, perhaps our understanding is illusory. Through demonstrating that SAEs could enable useful disentanglement of two concepts on SHIFT, this work provided a new kind of evidence that SAEs were giving some form of real understanding.

On a more zoomed-out level, I think that looking more into applying mech interp to downstream tasks is a healthy direction for keeping the field grounded, and I've been focusing on this in my own research. This paper provided some useful inspiration for me to make that shift.

The second contribution was being the first serious attempt at circuit finding with SAEs. The goal in circuit finding is finding a sparse set of model components that matter for some task, and a key problem in prior work is that often the key "component" is distributed across many heads or neurons. The promise of SAEs was to find the "right" set of monosemantic components to analyse, but no one had properly tested whether it made circuit analysis substantially easier! I don't consider this paper to have conclusively answered that question, but it provided a lot of useful data, and laid useful conceptual and methodological groundwork, like thinking about error nodes and whether to include/exclude them, whether to use integrated gradients over activation patching (and providing a principled translation of integrated gradients into this context), etc. I also consider it impressive that they do circuit finding for Gemma 2 2B, involving dealing with about 6B parameters of SAE weights, which is a fair bit more technically involved than Pythia-70M. This provides valuable data that the method is scalable, though I do expect more efficient approximations to be required to scale it to frontier models. 

As empirical evidence of the impact, two of my current ongoing papers are building on this paper, and I often point people to the paper as a guide for the right methods to use when working with SAE circuit finding.

(Disclosure: I was not at all involved in this paper, but I do know the authors and likely have some positive bias due to that. No one asked me to write this)

(Note also: I don't feel confident in the norms here, so feel free to ignore this comment if these kinds of thoughts are not welcome! I don't see other researchers doing this, but I figure that as ICLR has made a deliberate choice to allow public comments during the rebuttal process, they likely want this kind of public feedback from uninvolved researchers, so long as it's constructive)
</content>
</thread_object>

</thread>


Metareview:
metareview:
Meta-Review

This submission presents a method for discovering and editing interpretable causal structures—referred to as sparse feature circuits—within large language models (LLMs). Previous work in mechanistic interpretability has often focused on more coarse units (e.g., entire attention heads or neurons) that tend to be highly polysemantic. In contrast, this paper leverages recent advances in sparse autoencoders (SAEs) to decompose models into monosemantic features and constructs causal circuits from these interpretable features. By applying causal interventions on these fine-grained circuits, the authors demonstrate applications such as SHIFT, a method for selectively ablating human-identified, task-irrelevant features to improve model generalization and mitigate biases (e.g., gender cues in a profession classification task). Furthermore, the authors scale their approach to discover thousands of feature circuits across diverse behaviors, highlighting the generality and potential impact of their method.

Key Strengths:
	1.	Conceptual Innovation & Importance:
The paper tackles a crucial challenge in mechanistic interpretability—finding a comprehensible “atomic” unit of analysis. SAEs produce monosemantic features, enabling clearer and more faithful circuit-level explanations. This sets the stage for more granular and robust causal analyses than previous neuron- or head-level methods.
	2.	Downstream Utility (SHIFT):
By demonstrating that interpretability improvements can yield practical interventions, the paper moves beyond purely descriptive analyses. The SHIFT experiment shows that once we identify features responsible for spurious correlations (e.g., gender bias in classification), we can remove them to improve generalization without requiring extra supervision. This strongly supports the claim that better understanding leads to actionable control.
	3.	Scalability and Automation:
The authors show that their approach extends to large models and can discover thousands of circuits without human-specified tasks, relying only on an unsupervised pipeline. This advances the field toward automated mechanistic interpretability that can systematically cover broad model behaviors rather than a small set of human-curated phenomena.
	4.	Strong Empirical and Qualitative Evidence:
Multiple reviewers praised the clarity and breadth of the experiments. The results include direct causal tests, human annotations of interpretability, and comparisons to neuron-based approaches. The paper’s extensive supplementary materials, along with a public demo, enhance reproducibility and transparency.