Scaling Sparse Feature Circuits For Studying In-Context Learning
Authors: Dmitrii Kharlapenko
Stepan Shabalin
Arthur Conmy
Neel Nanda
Venue: ICML 2025 poster (2025)
Keywords: SAE
ICL
SFC
Interpretability
Gemma
LLM
Mechanistic Interpretability
Sparse Autoencoders
Circuits
Primary area: deep_learning->large_language_models
TL;DR: Explaing In Context Learning by finding circuits in terms of sparse autoencoder latents, on much larger models than have been studied before

Abstract: Sparse autoencoders (SAEs) are a popular tool for interpreting large language model activations, but their utility in addressing open questions in interpretability remains unclear. In this work, we demonstrate their effectiveness by using SAEs
to deepen our understanding of the mechanism behind in-context learning (ICL). We identify abstract SAE features that (i) encode the model’s knowledge of which task to execute and (ii) whose latent vectors causally induce the task zero-shot.
This aligns with prior work showing that ICL is mediated by task vectors. We further demonstrate that these task vectors are well approximated by a sparse sum of SAE latents, including these task-execution features. To explore the ICL mechanism, we scale the sparse feature circuits methodology of Marks et al. (2024) to the Gemma 1 2B model for the more complex task of ICL. Through circuit finding, we discover task-detecting features with corresponding SAE latents that activate earlier in the prompt, that detect when tasks have been performed. They are causally linked with task-execution features through the attention and MLP sublayers.

Scaling Sparse Feature Circuits For Studying In-Context Learning
Dmitrii Kharlapenko
Stepan Shabalin
x Arthur Conmy? Neel Nanda
Abstract
caused by the interventions Templeton et al. (2024). In this work, we address these limitations by interpreting in-context learning (ICL), a widely studied phenomenon in LLMs. In summary, we show that SAEs a) enable the discovery of novel circuit components (task-detection features; Sec- tion 4.2) and b) refine existing interpretations of ICL, by e.g. decomposing task vectors Todd et al. (2024); Hendel et al. (2023) into fask-execution features (Section 3).
Sparse autoencoders (SAEs) are popular tool for interpreting large language model activations, but their utility in addressing open questions in interpretability remains unclear. In this work, we demonstrate their effectiveness by using SAEs to deepen our understanding of the mechanism behind in-context learning (ICL). We identify ab- stract SAE features that (i) encode the model's knowledge of which task to execute and (ii) whose latent vectors causally induce the task zero-shot. This aligns with prior work showing that ICL is mediated by task vectors. We further demonstrate that these task vectors are well approximated by a sparse sum of SAE latents, including these task- execution features.
In-context learning (ICL; Brown et al. (2020)) is fun- damental capability of large language models that allows them to adapt to new tasks without fine-tuning, ICL is a significantly more complex and important task than other behaviors commonly studied in circuit analysis (such as IOI in Wang et al. (2022) and Kissane et (2024), or subject- verb agreement and Bias-in- in Marks et al. (2024)). Recent work by (Todd et al., 2024) and Hendel al. (2023) has introduced the concept of task vectors to study ICL in simple setting, which we follow throughout this paper. In short, task vectors are internal representations of simple operations performed by language models during the pro- cessing of few-shot prompts, e.g. as "hot - cold, big + small, fast slow". We call these simple operations tasks. Task vectors be extracted and added into different LLM forward passes to induce 0-shot task behavior, e.g. making LLMs predict that "slow" follows "fast --" without explicit context. Section 2.3 provides a full introduction.
explore the ICI. mecha- nism, we scale the sparse feature circuits method- ology of Marks et al. (2024) the Gemma 2B model for the more complex task of ICL. Through circuit finding, we discover task-detecting fea- tures with corresponding SAE latents that activate earlier in the prompt, that detect when tasks have been performed. They are causally linked with task-execution features through the attention and MLP sublayers.
1. Introduction
Unfortunately, it is not possible to decompose task vec- tors by naively applying sparse autoencoders to reconstruct them. This because task vectors out-of-distribution for SAEs and empirically, their SAE decompositions tend to be cluttered with irrelevant features. To address this limitation, we developed the TASK VECTOR CLEANING (TVC) algorithm (Section 3. 1), which enabled us to extract task lask-execution features: SAE latents that preserve most of the task vector's 's effect on task loss while being more inter- pretable. These task-execution features demonstrated two key properties: they could partially substitute for complete task vectors in steering experiments, and they often exhib- ited clear task-related patterns in their maximum maximum-activating tokens. Through extensive testing across diverse tasks, we
Sparse autoencoders (SAEs; Ng (2011); Bricken et al. (2023); Cunningham et al. (2023)) are a promising method for interpreting large language model (LLM) activations. However, the full potential of SAEs in interpretability re- search remains be explored, since most recent SAE re- search either i) interprets single SAE's features rather than the model's 's computation as a whole Bricken et al. (2023), or ii) performs high-level interventions in the model, but does not interpret the effect on the downstream computation
'Equal contribution 'ETH Zurich, Switzerland Georgia Insti- tute of Technology. US 'Joint senior authors. Correspondence to: Dmmitii Kharlapenko <dkharlapenko@ ct?z.ch>>, Stepan Shabalin <sshabalin3 @ gatech.edu>>.
'Task vectors Hendel et al. (2023) are also called "function vectors" Todd et al. (2024). but use "task vectors" throughout this paper for consistency.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
validated that these features play causal role in the model's ICL capabilities (Section 3.2).
short
Artarym deleded
To understand how these features operate within the broader ICL mechanism, we extended the Sparse Feature Circuits (SFC) discovery algorithm introduced by Marks et al. (2024) (Section 4), Applying this algorithm to the Gemma 2B model Team et al. (2024) revealed a network of SAE latents crucial ICL processing. This analysis uncovered com- plementary set of features call task-detection features, which identify the specific task being performed based on earlier prompt information. Importantly, our experiments demonstrated that disabling task-detection feature directions also disabled task- execution directions, revealing their fun- damental interdependence in the ICI. circuit (Section 4.2).
Antoryn comry..ccmcccooo
cown
Arkoyy peR des93s
Antryyn nodd
12hH
Figure I. A diagram of the in-context learning circuit, showing task detection features (yellow) causing task execution features (blue) which cause the model to output the antonym (left + right). A more concrete circuit, along with texts these features activate on, can be seen in Figure 12.
Our findings not only advance our understanding of ICL mechanisms but also demonstrate the potential of SAEs powerful tool for interpretability research on larger language models. By unifying the task vectors view with SAEs and uncovering two of the most important causally implicated feature families behind ICL, we pave the way for future work to control and monitor ICL further, to improve either the safety or capabilities of models.
in the hidden layer activations Elad (2010). In the context of language model interpretability, SAEs are used to decom- pose the high-dimensional activations of language models into more interpretable features (Cunningham al., 2023; Bricken al., 2023). The basic idea behind SAEs is to train neural network to reconstruct its input while constraining the hidden layer to have sparse activations. This process typically involves an encoder that maps the input to a sparse hidden representation, a decoder that reconstructs the input from this sparse representation, and loss task that balances reconstruction accuracy with sparsity. 2 The encoding step is as follows, with f denoting the art-avcivation features and Wene and Dene the encoder weights and biases respectively:
Our main contributions are follows:
1. We demonstrate that SAEs can be effectively used to analyze a complex ICL mechanism in Gemma 1 2B, which is one of the largest models studied at this depth in comparable, end-to-end circuits-style mechanistic interpretability research Wang al. (2022): Marks et al. (2024) (Section 4.1.3).
2. We identify two core components of the ICL circuit in Gemma 2B: task- features that identify required tasks from the prompt, and task-execution features that implement those tasks during generation (Sections and 4.2).
f(x) a WeneX + benc)
For JumpReLU SAES Rajamanoharan et al. (2024b), the activation function and decoder are (with H being the Heav- iside step function, the threshold parameter and Wace/bas the decoder affine parameters):
3. We uncover how these components interact: atten- tion heads and MLPs process information from task- detection features to activate the appropriate task- execution features, revealing how the model inte- grates information across the prompt to perform ICL tasks (Section 4.2).
x(f)
Waec dec (fo H(f (T)
+ baec
4. We develop the Task Vector Cleaning (TVC) algorithm, novel transformer--pecifie sparse linear decom- position method that enables precise analysis of ICL mechanisms by decomposing task vectors into their most relevant features (Section 3.1).
In our work, we train SAEs residual stream activations and attention outputs, and also train transcoders on MLP layers, all of which use the improved Gated SAE architec- ture Rajamanoharan et al. (2024a).
2. Background
"Typically, the L1 penalty on activations is used Bricken et al. (2023) with some modifications Rajamanoharan et al. (2024a); Conerly et al. (2024), although there are alternatives: Rajamanoha- ran al., 2024b; Farrell, 2024; Riggs & Brinkman, 2024.
2.1. Sparse Autoencoders (SAEs)
Sparse autoencoders (SAEs) are neural networks designed to learn efficient representations of data by enforcing sparsity
TTranscoders (Dunefsky et al., 2024) are modification of SAEs that take MLP input and convert it into MI. output instead of trying to reconstruct the residual stream.
2

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
2.2. Sparse Feature Circuits
To illustrate the concept, consider the following simple prompt for an antonym task in the Example where boxes represent distinct tokens:
Sparse Feature Circuits (SFCs) are a methodology intro- duced by Marks et al. (2024) to identify and analyze causal subgraphs of sparse autoencoder features that explain spe- cific model behaviors. This approach combines the inter- pretability benefits of SAEs with causal analysis to uncover the mechanisms underlying language model behavior. The SFC methodology involves several key steps:
BOS
Follow
the
pattern :
\n
hot
+ cold \n
big
small
\n
1. Decomposing model activations into sparse features using SAEs
fast
slow
Example 1. All token types in an example input: prompt
2. Calculating the Indirect Effect (IE, (Pearl, 2001) of each feature on the target behavior
input arrow output newline
( (target tokens for cal-
culating loss on included).
3. Identifying a set of causally relevant features based on thresholds
In this case, the task vector would encode the abstract tion of "finding the antonym" rather than specific word pairs. Task vectors are typically collected by averaging the resid- ual stream of "-"" tokens at a specific layer across multiple ICL prompts for a given task. This averaged representation can then be used to study the model's internal task represen- tations and to manipulate its behavior zero-shot settings. We perform our analysis the datasets derived from Todd et al. (2024). Details can be found in Appendix A.
4. Constructing a circuit by analyzing the connections between these features
The IE of a model component is measured by intervening on that component and observing the change in the model's 's output. For component and metric m, the IE is defined using do-calculus Pearl (2009) in (Marks et al., 2024) as:
As noted in the task vector paper, the impact of task vectors on loss varies across model layers (Figure 2). Middle layers show the strongest effects, while earlier and later layers demonstrate reduced effectiveness. For our experiments, we selected a single optimal layer termed the target layer from these high-performing middle layers. In the Gemma I 2B experiments, layer 12 served this target layer.
IE(m; a) = m(x/do(a a')) - (x)
(3)
Where (a = a')) represents the value of the metric when we intervene to set the value of component to a', and m(x) is the original value of the metric. In practice, attribution patching Syed et al. (2023) used to approx- imate IE, allowing for efficient computation across many components simultaneously.
3. Discovering Task-Execution Features
SFC is described in detail in Marks et al. (2024). We de- scribe our modifications in Section 4 and Appendix E.
3.1. Decomposing task vectors
To gain deeper understanding of task vectors, we attempted to decompose them using sparse autoencoders (SAEs). How- ever, several of initial naive approaches faced significant challenges. Firstly, direct SAE reconstruction, i.e. passing the task vector as input to the SAE, produced noisy results with more than 10 non-zero SAE features on average on target layers (in Gemma 2B), most of which were irrele- vant to the task. Moreover, this reconstruction noticeably reduced the vector's effect on task loss. These issues arose partly because task vectors are out-of-distribution inputs for SAEs, as they aggregaaee information from different residual streams rather than representing single one.
2.3. Task Vectors
Continuing from the high-level description in Section 1, task vectors independently discovered by Hendel al. (2023) and Todd et al. (2024). The key idea behind task vectors is that they capture the essence of task demon- strated in a few-shot prompt, allowing the model to apply this learned task to new inputs without explicit fine-tuning. Task vectors have several important properties:
1. They can be extracted from the model's hidden states given ICL prompts inputs.
2. When added to the model's activations in zero-shot setting, they can induce task performance without ex- plicit context.
We then explored inference-time optimization (TTO; Smith (2024)) as an alternative. However, this method also failed to reconstruct task vectors using a low number of SAE features while maintaining high effect the loss.
3. They appear to encode abstract task information, inde- pendent of specific input-output examples.
3

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
Given these observations, we developed novel method called task vector cleaning (Figure 4). It produces opti- mized SAE decomposition weights 0 E idssa for task vector Ut.. At a high level, the method:
Average relative loss change
Reconstruction type
Cleaning
SAF reconstruction
-022
ITO (20)
1. Initializes 0 with weights from SAE decomposition of Ute
Original Task vector
-0.4
ITO (5)
2. Reconstructs a new task vector uo from f; steers the model with up on a batch of zero-shot prompts and computes negative Log-likelihood loss (N on them
5
10
Layer
Optimizes 0 to minimize & = (0) + A|00|1. where À is the sparsity coefficient
Figure 2. The effect on the Gemma 2B's 's task losses by steering with different kinds of reconstructed task vectors, each layer. We see that cleaning performs similarly to the original task vector until layer 14, Average relative loss change measured post-steering relative loss change compared to 0-shot, averaged across all tasks.
This approach allows us to maintain or even improve the effect on the loss of task vectors while reducing the amount of active SAE features to by 70% (Figure 3) for Gemma I 2B and other models. The algorithm overview can be found in Figure 4. More details are provided in Appendix D.
Rolative loss. decresse
0.4
We evaluated TVC against four baseline approaches: (1) original task vectors, (2) naive SAE reconstruction, (3) ITO with target Lo norm of 5, and (4) ITO with target To norm of 20. The evaluation methodology involved steering zero-shot prompts using reconstructed task vectors and measuring the relative improvement in log-likelibne likelihood loss, averaged across all tasks. The steering is done in the same manner as in the TVC algorithm, further details can be found in Appendix D. Figure Figure 2 presents the layer-wise comparison results. To validate robustness, we conducted extensive parameter sweeps of the L₁ regularization coefficient À across multiple model scales, architectures and SAE configurations, includ- ing various widths and target sparsities for both Gemma-2 2B and 9B models, with the aggregated results shown in Figure 3. As detailed in Appendix D. these experiments demonstrated that TVC consistently achieves a 50-80% re- duction in active SAE features while maintaining the task vectors' effect on loss. The results further indicated en- hanced performance when using SAEs with higher target Lo values.
LD Dractio
0.4
10p
100g
Ll coefficient
Figure 3. Evaluation task vectors after applying our TVC al- gorithm across different La coefficient À values. Top: relative decrease in loss after steering (higher - better); bottom: fraction of retained active features (lower - better). Transparent lines represent diffcrent model and SAE combinations; solid lincs show means across all of them; the results are averaged across tasks; (r-axis: L1 coefficient A). Further details in Appendix D.I.
nishot ICL
Task vectors
SAE fextures
10 20 features naliyy
cord...
loyer
This decomposition approach revealed interpretable features that clearly corresponded to their intended tasks. Most notably, we identified a class of features we termed "task- execution feecuutiofeaturrs" " (or "executor features"), characterized by three properties:
Reconstruct
Training weights
Cleaned toights
featurea
D D-shot ICL prompos
Task loss
Lf norm
1. They activate upon encountering tesk-relevant exam- ples in natural text.
bell
Optiniza
2. Their activation peaks on the token immediately pre- ceding task completion.
Figure 4. Overview of the task vector cleaning algorithm; TV stands for task vector.
3. They have a high causal task-specific effect, which we measure later.
4

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
To illustrate, consider an antonym task feature processing the phrase hot and cold "" The feature activates and "" indicating the model's anticipation of an upcom- ing antonym. This suggests the model recognizes antonym relationships before observing the complete pair. Figure 5 provides additional examples of such features. Further examples in Appendix I demonstrate these features' maxi- mum activations on SAE training data, consistently showing task-specific activation patterns.
We have manually examined the features with the highest effect and found that their maximum activating dataset ex- amples tend to align with their hypothesized role in the ICL circuit. Interestingly, we observed that tasks requir- ing translation from English all share a generic English- to-foreign language task execution feature. This shared feature suggests a common mechanism for translation tasks, with language-specific information encoded separately. Max activating examples of the most interpretable features present in Appendix I.
import and domestic, Ul
ecial joie de vivre [joy of life
We also observed that with high Lo constraints TVC may drop this common translation feature in favor of a combi- nation of other multi lauli--anguage features, that have weaker effect individually. This did not happen with English-to- Spanish, so we attribute this biases in SAE and Gemma training data, since Spanish is usually much better repre- sented than French or Italian. This is supported by to-French and to- Italian translation tasks having much higher loss on their ICL prompts than to-Spanish.
etwenn fresh and traditional, > Banderitas (little paper ban both local and remote event lddad dorada' (the golden city th tropical and temperate we jihad, or struggle, in (a) Antonyms executor feature (b) Translation to English execu- 1 11618.
tor feature 5579.
Figure 5. A subset of max activating examples for executor features from Appendix I.
Token Type
Mass (%)
To analyze the activation patterns of executor features, we split all ICL prompt tokens into several types (highlighted in Example 1 and discussed later in Section 4.1.1). For each executor feature, we calculate its token type activation masses: the sum of all its activations on tokens of a particu- lar type across a batch of ICL prompts. Table shows the percentages of total mass split among different token types for executor features. We can that executors activate largely on arrow tokens.
arrow
89.80
output
6.46
input
3.2
newline
0.54
prompt 0.00
Table I. Activation masses for executor features across different token types, averaged across all tasks. We can notice they activate largely on arrow tokens.
3.2. Steering Experiments
To validate the causal relevance of our decomposed task features, we conducted series of steering experiments, observing the features' impact on task loss across different contexts and model layers.
aled Meugh
The experiments were performed on the dataset of diverse tasks taken from Todd et al. (2024). We first extracted relevant task features using our cleaning algorithm. Then steered the zero-shot prompt using them and calculated relative loss improvement, normalizing and clipping it after that. In this section we present main results from the Gemma 1 2B model. Further details and additional experiments that include other models can be found in Appendix F.
pesert caack proed
prafesics
javve shople por
plant
Figure 6 shows a heatmap of steering results for each pair of tasks and task-relevant features. Higher values indicate greater improvement in the loss after steering. It can be seen that most features that have high effect on some task generally do not significantly affect unrelated tasks. Another notable detail is that features from related tasks (like the translation group) at least partially affect all tasks within the group
Figure 6. Hcatmap showing the effect of steering with individual task- features for each task. Most features boost exactly one task, with few exceptions for similar tasks like translating English. Full and unfiltered versions of the heatmap are available in Appendix F.
5

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
4. Applying SFC to ICL
Fokhfulues
0.8
After identifying task-execution features through our task vector analysis, we sought to expand our understanding of the in-context learning (ICL) circuit. To this end, we apply the Sparse Feature Circuits (SFC) methodology Marks et al. (2024) to the Gemma 1 2B model. However, due to the increased complexity of ICL tasks and the larger model size, the original SFC approach did not work out of the box. We had to implement several key modifications to address the challenges we encountered.
0.4
0.6
500
1008
1500
Number nodes
of
(a) Faithfulness of C'
(b) Faithfulness of M \ C
Figure 7. Faithfulness for task circuits (a) and their complements
M C' (b). The ideal score for C is I, and 0 for M \ C.
4.1. Our Modifications
circuit's 's effect on those pairs being lost to either diminishing gradients in backpropagation or because copying circuits were much more relevant to predicting the last pair. By considering all pairs except the first one, we amplified the effect of the task-solving circuit relative to the numerous cloning circuits that activate due to the repetitive nature of ICL prompts.
4.1.1. TOKEN POSITION CATEGORIZATION AND FEATURE AGGREGATION
We modified the SFC approach to better handle the structured nature of ICL prompts. Instead of treat- ing each SAE feature as separate node, cat- egorized token positions into the following groups: Prompt The initial instruction tokens (e.g.. "Follow the pattern:")
4.1.3. SFC EVALUATION
To evaluate our SFC modifications, we measured faithful- ness through ablation studies on our ICL task dataset. Fol- lowing Marks et al. (2024), we define faithfulness as:
Input
The last token before each arrow in an exam-
ple pair
Arrow
The arrow token itself ("-"")
m(C) m( Faithfulness(C) m(M) m(Ø)
(4)
Output
The last token before each newline in
where mn()) is the metric with circuit C, m(()) repre- sents the empty circuit baseline, and m m(M) is the complete model's performance. While Marks et al. (2024) uses mean ablation, we opt for zero ablation since better aligns with the sparse nature of SAE features.
example pair
Newline
The newline token
Extra: Any tokens not covered by the above categories (e.g., in multi-token inputs or outputs)
This metric quantifies how much of the model's original capabilities are preserved when isolating specific circuit components.
Each pair of an SAE feature and token type was assigned its own graph node. The effects of the feature were ag- gregated across all tokens of the corresponding type. This categorization allowed us to evaluate how features affect all tokens within the same category, separating features based on their role in the ICL circuit. It also enabled us to selec- tively disable parts of the circuit for one task while testing another, verifying the task specificity of the identified cir- cuits.
To measure faithfulness, we first approximated node Indirect Effects (IEs) using a batch of few-shot examples for a given task. We then evaluated ablation effects held-out batch from the same task to ensure our findings generalize beyond the examples used for IE estimation. Our ablation studies targeted both the discovered circuit C' and its complement M \ C, removing nodes according to their IE thresholds. Figure 7 shows that circuits of 500 nodes were enough to achieve an average faithfulness of 0.6 across tasks, matching the performance seen in complex cases from the SFC paper. While the original SFC paper required only tens of top features disabled (Faithfulness of M \ C) to impact task performance, our approach needed several hundred features to achieve similar degradation. Should we focus ablation just on layers 11-12, we need much less active nodes for faithfulness 0.6 around 10-60 on average.
4.1.2. LOss FUNCTION MODIFICATION
An ICI. prompt can be viewed as an (ær, y) pair, where represents the entire prompt except for the last pair's output, and y represents this output. The original SFC paper suggested using the log probabilities of conditioned on for such datasets. However, this approach often resulted in task-relevant features having high negative IEs on other example pairs in the prompt. This likely due to the
6

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
with task-detection features connected task execution fea- tures through attention output and transcoder nodes. We applied our task vector cleaning algorithm to extract task- detection features, identifying layer 1 as optimal for steer- ing, preceding the layer 12 task-execution features. The details can be found in Appendix G. As with executor fea- tures, we present the steering heatmap in Figure 10 and the activation mass percentages in Table 2. We again see the task and token-type specificity of these features.
Croatia. Th his item by giving it score iomeland of ties
Reuters) Bulgaria's s presiden about sendinn our help
Figure 8. We study how useful the most important nodes on task A are for performance on task B. Specifically, we ablate the most important features for task A (the ablated task on the y-axis) so that faithfulness reduces by 0.7, and measure how much faithful reduces on another task (the tested task on the r-axis).
The Dubline: Ireland's oldest ipped. By doing this the few
(a) Country detector feature (b) Gerund form detector fea- 11459.
ture 8446.
Figure 9. A subset of max activating examples for detector features from Appendix I.
We need to note that we constrained our circuit search to in- termediate layers 10-17 of the 18 total layers. This stemmed from two practical considerations. First, earlier layers pre- dominantly process token-level information rather than the task-specific, prompt-agnostic features we aimed to identify. Second, our analysis revealed lower quality in IE approxi- mations for these earlier layers, as detailed in Appendix F.2. Marks et al. (2024) similarly excludes the first third of the layers in their circuit analysis of Gemma 2B.
Token Type
Mass (%)
output
96.76
input
3.22
newline
0.01
arrow
0.0
prompt
0.0
Table 2. Activation masses for task-dctection features across dif- ferent token types, averaged across all tasks. We can notice that they activate almost exclusively on output tokens.
To evaluate the task specificity of discovered eircuits, we conducted pairwise ablation studies examining cross-task faithfulness impacts. For each task circuit, we ablated the nodes with highest IEs until the task's s faithfulness metric dropped to 0.3 - representing a substantial degradation of the model's original performance. We then measured how this ablation affected faithfulness scores across all other tasks. The results, presented in Figure 8, demonstrate that our extracted circuits exhibit strong task specificity, with performance degradation largely confined to their tar- get tasks. We observed expected exceptions only between closely related task pairs, such as different translation tasks or conceptually similar tasks like personJanguue and loca- tion language, where ablating one circuit naturally impacted performance on its counterpart.
Thet
ped prosed glo
4.2. fask-Detection Features
Figure 10. Hcatmap showing the effect of steering with the task- detection feature most relevant to each task, on every task. We see that task detection features are typically specific to the task, with exceptions for similar tasks.
Our modified SFC analysis revealed a second crucial compo- nent of the ICL mechanism: task-detection features. Unlike executor features that activate before task completion, these features specifically activate when task is completed in the training data- precisely on the output tokens from the Example 1. Figure contains two examples of such features. Both task-detection and task- execution features showed high Indirect Effects (IEs) in the extracted sparse feature circuits,
To evaluate the causal connection between task-detection features and task-execution features, we matched the strongest detection and execution features for each task based on their steering effects. We then ablated detection di- rections while fixing attention patterns and measured the de-
7

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
crease in execution features activations. Figure 11 presents the results.
area of research studying its applications Dong et al. (2024), high-level mechanisms Min et al. (2022) and limitations Peng et al. (2023). Elhage et al. (2021) and Olsson et al. (2022) find induction heads partly responsible for in-context learning. However, since these attention heads do more than just induction Goldowsky-Dill et al. (2023), and are not sufficient for complex task-following, induction heads alone cannot explain ICL. Anil et al. (2024, Appendix G) proposes a mechanistic hypothesis for an aspect of simple in-context task behavior. (Hendel et al., 2023) and (Todd et al., 2024) find that simple in-context learning tasks create strong directions in the residual stream adding which makes it possible for network to perform tasks zero-shot, but does not explain how task vectors form nor what interpretable components the task vectors are composed of. A more de- tailed discussion can be found in Appendix H. Of particular interest is (Wang al.), which investigates a simple ICL classification task and finds similar results with different terminology (information flow instead of circuits, "label words" instead of task-detection features). Recent work by (Park et al., 2024) demonstrates that language models reorganize existing representations of objects to adapt them to new tasks from the current context. This finding suggests that for the tasks they study, our task exccuting features may become more detached from their maximum activating ex- amples, presenting an interesting class of tasks for applying our methods.
Flwssegg
M
Figure/ If. Heatmap showing the fraction of the executor activation lost the top task-detection features of each task, after the detec- tors were ablated. Averaged across all initial non-zero activations in all tasks.
Our causal analysis demonstrated that ablating task- detection features substantially reduced the activation of their corresponding task-execution features, validating their hypothesized roles in the ICL. circuit. Translation tasks exhibited particularly high interconnectivity, point- ing to shared circuitry across this task family. Two tasks diverged from this pattern: prrsonnprofession and present simpllsssipeepepp gerund showed unusually weak detection- execution connections, suggesting need for deeper investi- gation.
Sparse Autoencoders Superposition, where interpretable neural network units don't align with basis directions, re- mains key challenge in mechanistic interpretability EI- hage et al. (2022). Sparse autoencoders (SAEs) Ng (2011); Bricken et al. (2023) address this, with recent works improv- ing their training Rajamanoharan et al. (2024b); Bussmann et al. (2024); Braun et al. (2024). Building on this, Cun- ningham et al. (2023) applies automated circuit discovery to small language models, while Marks et al. (2024) adapts attribution methods in the SAE basis. Dunefsky et al. (2024) introduces transcoders to simplify MIP circuit analysis, which we incorporate into Gemma SAE suite.
5. Related work
Mechanistic Interpretability Olah et al. (2020) defines a framing for mechanistic interpretability in terms of features and circuits. It claims that neural network latent spaces have directions in them called features that correspond to meaningful variables. These features interact through model components sparsely to form circuits: interpretable compu- tation subgraphs relevant to particular tasks. These circuits can be found through manual inspection in vision models Cammarata et al. (2020). In language models, they can be found through manual patching (Wang et al., 2022: Hanna et al., 2023; Lieberum et al., 2023; Chan et al., 2022) or automated circuit discovery (Conmy et al. (2023); Syed et al. (2023); Bhaskar et al. (2024). though see Miller et al. (2024)). Marks et al. (2024) extends this research area to use Sparse Autoencoders, as discussed below.
6. Conclusion
We use sparse autoencoders (SAEs) to explain in-context learning with unprecedented mechanistic detail. Our work demonstrates that SAES serve as valuable circuit analysis tools, with our key innovations including TVC (Section 3.1) and SFC adaption for ICL (Section 2.2). We will also plan to share SAE training codebase in JAX with a full suite of SAES for Gemma 1 2B after the paper publication. These advances lay the groundwork for analyzing more complex tasks and larger models.
In-Context Learning (ICL) ICL was first introduced in Brown et al. (2020) and refers to models learning to perform tasks from prompt information at test time. There a large
8

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
Limitations Our analysis focused on the simple task vec- tor setting study in-context learning (Section 2.3), which represents only a subset of ICL applications in practice. While our SFC analysis centered on Gemma 2B, we suc- cessfully identified task execution features across multiple model architectures and scales, supporting the broader appli- cability of our findings. This aligns with prior work showing that task vectors exist across models Todd al. (2024).
A., Majercak, D., Mazzola, M., Mendes, C. C. T., Mitra,
A., Modi, H., Nguyen, A., Norick, B., Patra, B., Perez- Becker, D., Portet, T., Pryzant, R., Qin, H., Radmilac, M., Ren, L., de Rosa, G., Rosset, C., Roy, S., Ruwase, O., Saarikivi, O., Saied, A., Salim, A., Santacroce, Shah, S., Shang, N., Sharma, H., Shen, Y., Shukla, S., Song, X., Tanaka, M., Tupini, A., Vaddamanu, P., Wang. C., Wang, G., Wang, L., Wang, S., Wang. X., Wang. Y., Ward, R. Wen, W., Witte, P., Wu, H., Wu, X., Wyatt, M., Xiao, B., Xu, C., Xu, J., Xu, W., Xue, J., Yadav, S., Yang, F., Yang. J., Yang, Y., Yang. 7., Yu, D., Yuan, I.,, Zhang, C., Zhang, C., Zhang, J., Zhang, L. L.,, Zhang, Y., Zhang, Y., Zhang. Y., and Zhou, X. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https: // //tp///avvv////////////////eaaaa org/ abs////////// 20t.//444..122999
In analysis, we often tion and detection features, though these features remained highly task in nature- a pattern that appears com- mon in LLM interpretability due to their regenerative capa- bilities.
multiple competing execu-
Acknowledgements
Anil, C., Durmus, E.,, Sharma, M., Benton, J., Kundu, S., Batson, J., Rimsky, N., Tong, M., Mu, J., Ford, D., et al. Many-shot jalbbrekin..... 2024.
This work conducted during the ML Alignment & The- ory Scholars (MATS) Program, which is sponsored by Open- Philanthropy. We are grateful to MATS for providing a collaborative and supportive research environment. We also thank the TPU Research Cloud (TRC) for providing com- putational resources. Special thanks to Matthew Wearden and McKenna Fitzgerald for their invaluable guidance and management throughout the project. We also extend our gratitude to the MATS and Lighthaven staff, as well as all contributors who provided insightful feedback and discus- sions that shaped this work.
Bai, Y., Chen, F., Wang. H., Xiong, C., and Mei, S. Trans- formers as statisticians: Provable in-context learning with in-context algorithm selection.
Bansal, H., Gopalakrishnan, K., Dingliwal, S., Bodapati, S., Kirchhoff, K., and Roth, D. Rethinking the role of scale for in-context learning: An interpretabillty-beee based case study at 66 billion scale. URL hnttp://arree org/ abs/ / 2212. 09095.
Impact Statement
Bhaskar, A., Wettig, A., Friedman, D., and Chen, D. Find- ing transformer circuits with edge pruning, 2024. URL https hatps://arxivv hrrg/abs/2406...
This paper presents work whose goal is advance the field of mechanistic interpretability, rather than machine learning in general. Whilst there are potential societal consequences to the advancement of the ML field, we feel that research that improves understanding of ML models is unlikely to additionally contribute to these consequences, rather giving the field more tools avert them.
Bloom, J. Open source sparse autoencoders for all residual stream layers of gpt2-small, 2024. URL https://www al fonment org/ /pttts/s/////// 19EgfLLurAiqRJJSSD
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations of Python+NumPPy programs, 2018. URL attp://githun wwt.:ocooaajaaaa http:///gtthub..om///amlll.l jax.
References
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan,
A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H. Benhaim, A., Bilenko, M., Bjorek, J., Bubeck, S., Cai, M., Cai, Q. Chaudhary, V., Chen, D., Chen, D., Chen, W., Chen, Y.-C., Chen, Y.-L., Cheng, H., Chopra, P., Dai, X., Dixon, M., Eldan, R., Fragoso, V., Gao, J., Gao, M., Gao, M., Garg, A., Giorno, A. D., Goswami, A., Gunasekar, S., Haider, E., Hao, J., Hewett, R. J., Hu, W., Huynh, J., Iter, D., Jacobs, S. A., Javaheripi, M., Jin, X., Karampatziakis, N., Kauffmann, P., Khademi, M., Kim, D., Kim, Y. J., Kurilenko, L., Lee, J. R., Lee, Y. T., Li, Y. Li, Y., Liang, C., Liden, L., Lin, X., Lin, Z., Liu, C., Liu, L., Liu, M., Liu, W., Liu, X., Luo, C., Madan, P., Mahmoudzadehh
Braun, D., Taylor, J., Goldowsky-Dill, N., and Sharkey, L. Identifying functionally important features with end- to-end sparse dictionary learning, 2024. https:
//arxiv. www//os////....... 2400 12241.
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T.. Joseph, N., Hatfield-Dodds, Z.. Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards monosemanticity: Decomposing language

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
models with dictionary learning. Transformer Cir- cuits Thread, 2023. https://trans
Dai, D., Sun, Y., Dong, L.., Hao, Y., Ma, S., Sui, Z., and Wei, F. Why can GPT learn in-context? language models implicitly perform gradient descent meta-optimizers. URL http: arxiv. org/ abs / 221222222222 2 10559.
peccuits.uub/.. http:////22/////////eeeeee monosemantic features/index htt.r html.
Brown, T., Mann, B., Ryder, N., Subbiah, M.. Kaplan, J. D.,
Dong, Q., Li, L.., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., Chang, B., Sun, X., Li, L.,, and Sui, 7. A survey on in-context learning, 2024, URL https: / //aaiiiiiii//.a. www. www.////..... 00234.
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-Voss. A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler. D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877- 1901, 2020. URL hitps: /ttp///eccccc....///////// neurips.cal htt.rpps.cc/papeer//2////heee//
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., al. The Ilama 3 herd of models, 2024. URL https: /arxiv.
org/ 0tt//2447..111
Dunefsky, J., Chlenski, P., and Nanda, N. Transcoders find interpretable IIm feature circuits, 2024. URL https: / //arxiv.org////// abs 2406. 199.4.1944...44
15570448foo099771118888811222222 Abstract. html.
Elad, M. Sparse and Redundant Representations: From
Theory to Applications in Signal and Image Processing.
Bussmann, B., Leask, P., and Nanda, N. Batchtopk:
Springer, New York, 2010. ISBN 978-1-4419-7010% doi: 10.1007/978-14449-01111
A simple improvement for topk-saes, 2024. URL
http:://wwww.lllgneennforrro.. http://mennnooooooooorosseeeeeee org/posts/ akkrsyilttossfticcco////tttttppp simpl
Elhage, N., Nanda, N., Olsson, C., Henighan, T.,, Joseph,
N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds. Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L.. Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. URL http://s///etaaaffoeeeeeieeeeeee
Cammarata, N., Carter, S., Goh, G., Olah, C., et Thread:
Circuits. Distill, 2020. doi: 1023995/distill.0002
https://distillee httpalll..ppp/////iiiiiii/
Chan, L., Garriga-Alonso, A., fooldwwsky-Dill, N., Green- blatt, R., Nitishinskaya. J., Radhakrishnan, A., Shlegeris, B., and Thomas, N. Causal scrubbing: A method for rigorously testing interpretability hypotheses,, 2022. URL
httb///2//Eraeewo////i/ htmll.
Elhage. N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds. Z., Lasenby, R., Drain, D., Chen, C., et al. Toy Models of Superposition. arXiv preprint arXiv: 10652, 2022.
bxzhaayccc222df77////aaaa----bb2 mettodd-oo-rilocrouul--ttett---e
Farrel,, E. Experiments with alternative method promote sparsity in sparse autoencoders, 2024. URL http:::/www.eesswrroo
Chen, S., Sheen, H., Wang, T., and Yang, Z. Unveiling induction heads: Provable training dynamics and feature learning in transformers. htt:tt htttp://axxiv.ort abs/ 2409/240... 10559.
<=BSeExyB0Q@jj0BBeexeeeeeett an an-alternate to sparsity.
Conerly, T., Templeton, A.. Bricken, T., Marcus, J., and Henighan, T. Update on how we train saes, 2024. URL circuits wwtpp///////rrraupooeoeeee/ html4traaiiiggssee
Garg, S., Tsipras, D., Liang, P., and Valiant, G. What can transformers learn in-context? a case study of simple function classes.
Goldowsky-Die Diil, N., MacLeod, C., Sato, L., and Arora, A. Localizing model behavior with path patching, 2023. 05969.
Conmy, A., Mavor-Parker, A. N., Lynch, A., et al. Towards automated circuit discovery for mechanistic interpretabil- ity. In Proceedings of NeurlPS, 2023.
URL https:tppss.. : / arxiv. .ea.iiioorrg. org.
Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Taj- dini, A., Steiner, B., Li, D., Durmus, E., Perez, E., Hub- inger, E., Lukošiūté, K., Nguyen, K., Joseph, N., Mc- Candlish, S., Kaplan, J., and Bowman, S. R. Study- ing large language model generalization with influence
Cunningham, H., Ewart, A., Riggs, L., et al. Sparse autoen- coders find highly interpretable features in language mod- els, 2023. URL https: // arxii.. org/ abs /tab//3.., 2309. 08600.
10

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
functions, 2023. URL https::/aaxxii..rrrt
Miller, J., Chughtai, B., and Saunders, W. Transformer
2308. 03296.
circuit faithfulness metrics not robust, 2024. URL
https:
08734.
Han, X., Simig, D., Mihaylov, T. Tsvetkov, Y., Celiky- ilmaz, A., and Wang, T. Understanding in-context learning via supportive pretraining data. URL http: //arxiv. www.xii.rrg/..... www.abaaaaa 1509..
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M.,
Hajishirzi, H.,, and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.18833 2022. URL https:
Hanna, M., Liu, O., and Variengien, A. How does gpt-2 compute greater-than" Interpreting mathematical abili- ties in pre-trained language model, 2023. URL https:
/ wawpww.wrggab///222222222..
Ng, A. Sparse autoencoder. C5294A Lecture Notes, 2011. Unpublished lecture notes.
//arxiv. orgg abs/ /2305. 00586.
Hendel, R., Geva, M., and Globerson, A. In-context learn- ing creates task vectors, 2023. URL https: //rxiv.. org/ abs/ /rg/abs// 2310. 6.
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov,
M., and Carter, S. Zoom in: An introduction circuits. Distill, 2020. doi: 00.23115/disiilll000224.. https:
/ //distill, http:///2////rrttt////o/oooo//
Johnson, D. D. Penzai treescope: toolkit for interpret- ing, visualizing, and editing models as data, 2024. URL https: / /arxivivivv. orggabgg//...oooooooo........ebe 2408 00211.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds,
Kidger, P. and Garcia, C. Equinox: neural networks in jax via callable pytrees and filtered transformations, 2021. URL https:::: www.v.w.oo.......oo.. abs/2111. 00254.
Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, I.., Ndousse, K., Amodei, D., Brown, T. Clark, J., Kaplan, J., McCandlish, S., and Olah, C. In-context learning and induction heads, 2022. URL https:// arxiv. org/ abs 2209 11895.
Kissane, C., Krzyzanowski. R., Conmy, A., and Nanda, N. Attention output saes improve circuit analysis, 2024.
URL https: http:///wwwwwllnmnnn.nnee..ooooa formm org/ poste//ErrkgpPcccffaxaa/t/ttttt output saes cippuie--ircuittt-e-..p wwrciitttaallsssssttiiee analysis.
Oswald, J. v., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent. URL http: http://xxivv..oo www..iiioi.ooooooooooooooooooere orgg abs 222222222222 07677.
Lieberum, T., Rahtz, M., Kramár, J., et al. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla, 2023, URL https: //arxiv. owgg///aaoo. abs 2tcb.///3/... 09458.
Pan, J., Gao, T., Chen, H., and Chen, D. What in- context learning "learns" in-context: Disentangling task recognition and task learning. In Rogers,
Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L., Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R., and Nanda, N. Gemma scope: Open sparse autoencoders everywhere at once on gemma 2, 2024. URL https: //arxiv. org/ abs/ 05147.
A., Boyd-Graber, J., and Okazaki, N. (eds.), Find- ings of the Association for Computational Linguistics: ACL 2023, pp. 8298-8319, Association for Computa- tional Linguistics. doi: 10. 10.18663/1120033.immmmm 527. URL https: /er///alaaann /aaalaannthlooyyo http://ooooorgg/222/22//2// 2023. E Findings-acl. 527.
Lin, J. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023. URL https: www.nnneuropeeia.. org. Software available from neuronpedia.org.
Park, C. F., Lee, A., Lubana, E. S., Yang, Y., Okawa, M., Nishi, K., Wattenberg, M., and Tanaka, H. Ielr: In- context learning of representations, 2024. URL https: //arxiv. www.ii.oog/aaaaaa..e...eooeeee.e www.oogg//bbb........o /0/0//000000000000000
Mahankali, A., Hashimoto, T. B., and Ma, T. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. URL http:// arxiv. org/ abs/ 22007 03576.
Pearl, J. Direct and indirect effects. In Proceedings of the Seventeenth Conference Uncertainty in Artificial Intelligence, UAI'01, pp. 411-420 San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001.
Marks, S., Rager, C., Michaud, E. J., et al. Sparse fea- ture circuits: Discovering and editing interpretable causal graphs in language models. Computing Research Reposi- tory, 1rX1v:2403.19647, 2024. URL https: : arxxiv. org/abs/ 240b.//2443.. 19647.
Pearl, J. Causality: Models, Reasoning and Inference. Cam- bridge University Press, USA, 2nd edition, 2009. ISBN 0521 192169X

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
Penedo, G., Kydliček, H., allal, B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, I.. V., and Wolf, T. The fineweh datasets: Decanting the web for the finest text data at scale, 2024. URL https: arxiv. org/o///aa abs/ 2406.10 10006175550 17557.
teanfFeeplaciigg.BBBBBBoooesr
Syed, A., Rager, C., and Conmy, A. Attribution patching outperforms automated circuit discovery, 2023. URL
Peng. H., Wang, X., Chen, Li, W. Qi, Y., Wang. Z., Wu, Z., Zeng, K., Xu, B., Hou, L., and Li, J. When does in-context learning fall short and why? a study on specification-heary tasks, 2023. URL https:// arxiv.org wwwp://rggggbb/aeeeee/ee wwt.aa/......... 08993.
https: http://waxi....rraa//22111// 10348.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju,
S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Tafti, P., et al. Gemma: Open models based gemini research and technology, 2024. URL https: //trxiaa.vvvv//////// org/ abs 2403. ...03..00 082.95.
Rajamanoharan, S. Improving ghost grads, 2024. URL https://www..lllgmmmeteeroom..mn
(554000bbbzzzppyycc/pgggesssss-b Troon-the-gghh--mec----tttrrr-re full- Lypatee-Inncoovnnggggootssssss_
Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones,
A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E.. Batson, J., Jermyn, A., Carter, S., Olah, C., and Henighan, T. Scaling monosemanticity: Ex- tracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https:
Rajamanoharan, S., Conmy, A., Smith, L., Lieberum, T., Varma, V., Kramár, J., Shah, R., and Nanda, N. Im- proving dictionary learning with gated sparse autoen- coders, 2024a. URL https: /ttp/aaavvv/./////a/eeeeeaaaa.la org/ abs/ 2404.16014
Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V. Kramár, J., and Nanda, N. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, 2024b. URL https:/ /ttp:///axvvvoooo abs/2407.144 14435.
scaling- htw.angnooooooommmiiiiitiiteeeed html.
Todd, E., Li, M. L., Sen Sharma, A., et al. Function vectors in large language models. Proceedings of the 2024 International Conference on Learning Representations, 2024.
Raventós, A., Paul, M., Chen, F., and Ganguli, S. Pretrain- ing task diversity and the emergence of non-bayesian in- context learning for regression. URL http:
Wang. K., Variengien, A., Commy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: A circuit for indirect object identification in GPT-2 small, 2022. URL https: /tte:://rxxiii /aa/aaiivvvve/.ie//eea//aaeaevve org abs 2211 00593.
org/abs/ 2306 15063.
Riggs, L. and Brinkman. J. Improving sae's by sqrt(--ing 11 and removing lowest activating features. 2024. URL
Wang. L., Li, L., Dai, D., Chen, D., Zhou, H., Meng, F. Zhou, J., and Sun, Label words are anchors: An information flow perspective for understanding in- context learning. In Bouamor, H., Pino, J., and Bali, K. Empiri-
hwtp://wwwwwwwoooo//eeeoommeeeee www.postss xis3@@ggaaasssee22/////oooooonnr sqrt sqrt-ing-11- 1atp---1-add--emooiieoe Lowerreoetooiin--oos.......
by-
Shen, L., Mishra, A., and Khashabi, D. Do pretrained transformers learn in-context by gradient descent? URL http://arxix org/ abs 2210 08540.
(eds.), Proceedings of the 2023 Conference cal Methods in Natural Language Processing, pp. 9840- 9855, Association for Computational Linguistics. doi: 10. 10.186633w1/2023.mmml/. m09n URL https:
Si, C., Friedman, D., Joshi, N., Feng. S., Chen, D., and He, H. Measuring inductive biases of in-context learn- ing with underspecified demonstrations. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11289-11310. Association for Computational Linguis- tics. doi: 10. 10.186653/112223.adcl 662.. URL https: //aclanthology 2tw. acl acl-llll0 -lnggn 632.
a sclanthology .rrg 2rrg/2023.2m emnlp wwwnp-mmainn..... main. 609.
Wang. X., Zhu, W., Saxon, M., Steyvers, M., and Wang, W. Y. Large language models are latent variable mod- els: Explaining and finding good demonstrations for in- context learning, 2024. URL https: //tp://aaxovvo//// abs / 2300.11911
Xic, S. M., Raghunathan, A., Liang, P., and Ma, T. An explanation of in-context learning as implicit bayesian inference. URL http:/ /arxiv. org/ org/abs// 2111. 02080.
Smith, L. Replacing sae encoders with inference- time optimisation, 2024.
URL https:
//www
12

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
Xu, Z.,, Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y. Magpie: Alignment data synthesis from scratch by prompting aligned Ilms with nothing, 2024. URL httht:::/// / arxiv.. org/ abs/ 2406. 08464.
Yadlowsky, S., Doshi, L., and Tripuraneni, N. Pretraining data mixtures enable narrow model selection capabilities in transformer models. URL http: / /aw.xaxxvvv.r/...//.a abs/ 2211.231.100 00871.
13

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
A. Model and dataset details
For our experiments, we utilized the Gemma 1 2B model, a member of the Gemma family open models based on Google's Gemini models Team et al. (2024). The model's architecture is largely the same as that of Llama Dubey et al. (2024) except for tied input and output embeddings and a different activation function for MLP layers, so we could reuse our infrastructure for loading Llama models. We train residual and attention output SAEs as well as transcoders for layers 1-18 of the model on Fine Penedo et al. (2024).
Our dataset for circuit finding is primarily derived from the function vectors paper Todd et al. (2024), which provides a diverse set of tasks for evaluating the existence and properties of function vectors in language models. We supplemented this dataset with three additional algorithmic tasks to broaden the scope of our analysis:
Extract the first element from an array of length 4
Extract the second element from an array of length 4
Extract the last element from array of length 4
The complete list of tasks used in our experiments with task descriptions is as follows:
Task ID
Description
Dncation_continen
Name the continent where the given landmark is located.
football_player position
Identify the position of given football player.
location religion
Name the predominant religion in given location.
location_ lattuuge_l
State the primary language spoken in a given location.
person crrssonproffesion
Identify the profession of given person.
location country
Name the country where given location is situated.
country -cantry
Provide the capital city of given country.
person person_langue language
Identify the primary language spoken by given person.
singular plural
Convert singular
its plural form.
siepent pappleepastssimpple
Change a verb from present simple to past simple tense.
antonyms
Provide the antonym of given word.
Blural-singular
Convert plural noun to its singular form.
presentssimple.patt.peee
Change a verb from present simple to past perfect tense.
present simple gerund
Convert verb from present simple to gerund form.
en_it
Translate a word from English to Italian.
it_en
Translate a word from Italian to English.
en fr
Translate a word from English to French.
en_es
Translate a word from English to Spanish.
fr_en
Translate a word from French to English.
es_en
Translate a word from Spanish to English.
algo_last
Extract the last element from an array of length 4.
alro-first
Extract the first element from an array of length 4.
algo_second
Extract the second element from an array of length 4.
This diverse set of tasks covers a wide range of linguistic and cognitive abilities, including geographic knowledge, language translation, grammatical transformations, and simple algorithmic operations. By using this comprehensive task set, we aimed to thoroughly investigate the in-context learning capabilities of the Gemma 2B model across various domains.
B. SAE Training
Our Gemma 1 2B SAEs are trained with learning rate of le-3 and Adam betas of 0.0 and 0.99 for 1 150M (+ 100) tokens of Fine Penedo et al. (2024). The methodology is overall similar to Bloom (2024). We initialize encoder weights orthogonally and set decoder weights to their transpose. We initialize decoder biases to O. We use (Rajamanoharan, 2024)'s ghost gradients variant (ghost gradients applied to dead features only, loss multiplied by the proportion of death features) with the additional modification of using softplus instead of exp for numerical stability. feature is considered dead when
14

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
its density (according to a 1000-batch buffer) is below 5e-6 when it has not fired in 2000 steps. We use Anthropic's input normalization and sparsity loss for Gemma 2B Conerly al. (2024). We found it to improve Gated SAE training stability. We modified it to work with transcoders by keeping track of input and output norms separately and predicting normed outputs.
We convert our Gated SAFs into JumpReLU SAEs after training, implementing algorithms like TVC and SFC in a unified manner for all SAEs in this format (including simple SAEs). The conversion procedure involves setting thresholds to replicate the effect of the gating branch. For further details, see (Rajamanoharan et al., 2024b).
We use 4 v4 TPU chips running Jax Bradbury et al. (2018) (Equinox Kidger & Garcia (2021)) train our SAEs. We found that training with Huggingface's 8 Flax LM implementations was very slow. We reimplemented LLaMA Dubey et al. (2024) and Gemma Team et al. (2024) in Penzai Johnson (2024) with a custom layer-scan transformation and quantized inference kernels as well as support for loading from GGUF compressed model files. We process an average of around 4400 tokens per second, which makes training SAEs and not caching LM activations the main bottleneck. For this and other reasons, we don't do SAE sparsity coefficient sweeps to increase TPU utilization.
For caching, we use a distributed ring buffer which contains separate pointers on each device allow for processing masked data. The (in-place) buffer update is in separate JIT context. Batches are sampled randomly from the buffer for each training step.
We train our SAEs in 6 precision. We found that keeping weights and scales in bíloat16 and biases in float32 performed best in terms of the number of dead features and led to a Pareto improvement over float32 SAES.
For training Phi 3 Abdin et al. (2024) SAEs, we use data generated by the model unconditionally, similarly to Xu et al. (2024)'. The resulting dataset we train the model on contains many math problems and is formatted as a natural-seeming interaction between the user and the model.
Each SAE training run takes us about 3 hours. We trained 3 models (a residual SAE, an attention output SAE, and a transcoder) for each of the 18 layers of the model. This is about 1 week of v4-8 TPU time.
Our SAEs and training code will be made public after paper publication.
C. Example circuits
An example output of our circuit cleaning algorithm can be found in Figure 12. single high-IE attention feature from a task-detection feature (activating on output tokens) to transcoder and residual execution features (activating on arrow tokens). The feature activates on antonyms on the detection feature #1 1050: one can assume the first sequence began as "Short Term Target", making the second half an antonym.
We the flow of information through
We will release a web interface for viewing maximum activating examples and task feature circuits.
D. Task Vector Cleaning Algorithm
The task vector cleaning algorithm is a novel approach we developed to isolate task-relevant features from task vectors. Figure 13 provides an overview of this algorithm.
Our process begins with collecting residuals for task vectors using a batch of 16 and 16-shot prompts. We then calculate the SAE features for these task vectors. We explored two methods: (1) calculating feature activation and then averaging across tokens, and (2) averaging across tokens first and then calculating the task vector. They had similar performances.
The cleaning process is performed on a training batch of 24 pairs, with evaluation conducted additional 24 pairs. All prompts are zero-shot. An example prompt is as follows:
*Phi-3 is trained primarily with instruction following data, making it an aligned chat model.
15

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
r:11:0utput111500
IE: 0.0187
Target: $0.10 Long Term Target: $0.45
...Souulle Fiber and grams of Insoluble Fiber.
a:11 :arrow:4080
0.092
t:11 :rroow:0:276
t:11 :arrow226009
IE: 0.0137
IE: 0.0638
...I think quantity and quality go hand in hand...
...nndss N at 5 to 10 mph. Chance of rain...
r:1 r:12:arrww:1168
0.083
alternating between lower and upper registers...
between northern and southern Italian cooking...
Figure 12, An example of circuit found using our SFC variant. We focused on subcircuit with high indirect effects. Maximum activating examples from the SAE training distribution included.
BOS Follow
the
pattern
I
tall
short
old young
\n
hot
cold
Example The stcered token is highlighted red. Loss is calculated the yellow token.
The algorithm is initialized with the SAE reconstruction starting point. It then iteratively steers the model on the reconstruction layer and calculates the loss on the training pairs. To promote sparsity, we add the L, norm of weights with coefficient À to the loss function. The algorithm implements early stopping when the Lo norm remains unchanged for iterations.
def tve Eve_algorithm (task_vector, model, sae): initial_weights sae. encode (task_vector)
def tvc_loss (weights, tokens)
task_vector sae decode (weights)
mask tokens self. separator model. reddllrreiiuulllttemmmaaalaa
mask] += testovector
# loss only the "output" " tokens,
# ignoring input and prompt tokens
loss = logprobs (oodel.logits, tokens, ...)
10
return loss cleff * _o_norm (weights)
l
weights initial weltiallweights..coo
16

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
n-shot ICL prompts
SAE features
10-20 features noisy
Task vectors
...hot cold...
on layer
Reconstruct TV
Training weights
Cleaned weights
2-4 features interpretable
0-shot ICL
Task loss
L1 norm
prompts
...tall
Optimize
Loss
Figure 13, An overview of our Task Vector Cleaning algorithm, TV stands for Task Vector.
optimizer adam (weights, 1r=0.15)
last 10, without Sathout_changeee 0, early stopping
14
for in range (1000):
15
grad = jax. (tve_loss) (weights, tokens)
16
weights = optimizer. step (grad)
17
if no_m (weights) !- last_10:
18
last_10, without_change
oo_normwweightt.
19
elif without_change >= 50:
break
return weights
Algorithm 1. Pscudocode for Task Vector Cleaning.
The hyperparameters X, n, and learning rate a can be fixed for a single model. We experimented with larger batch sizes but found that they did not significantly improve the quality of extracted features while substantially slowing down the algorithm due gradient accumulation.
The algorithm takes varying amounts of time to complete for different tasks and models. For Gemma 1, stops at 100-200 iterations, which is close to 40 seconds at 5 iterations per second.
It's worth noting that successfully applied this method to the recently released Gemma 2 2B and 9B models using the Gemma Scope SAE suite Lieberum et al. (2024). It was also successful with the Phi-3 3B model Abdin et al. (2024) and with our SAES, which were trained similarly to the Gemma 2B SAES.
D.I. Li Sweeps
To provide more details about the method's effectiveness across various models and SAE widths, we conducted L1 coefficient sweeps with our Phi-3 and Gemma 2B SAES, as well as Gemma Scope Gemma 2 SAEs. We chose two SAE widths for Gemma 2 2B and 9B: 16k and 65k. For Gemma 2B we also sweeped across several different target SAE Lo norms. We studied only the optimal task vector layer for each model: 12 for Gemma 1, 16 for Gemma 2, 18 for Phi-3, and 20 for
17

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
Gemma
Method
ITO
TVC
TV loss delta
0
-0.4
-0.6
0.5
1.5
LO fraction
Figure 14. Performance of ITO and TVC across different tasks and optimization parameters compared to task vectors for Gemma 2B. The Y-axis shows relative improvement over task vector loss, while the X-axis shows the fraction of active TV features used. Metric calculation details are available in D.1
Gemma 2 9B. We used a learning rate of 0.15 with the Gemma 2B, Phi-3, and Gemma 2 2B 65k models, 0.3 with Gemma 2 2B 16k, and 0.05 with 200 early stopping steps for Gemma 9B.
Figures 14, 15, 16 compare TVC and ITO against original task vectors. The X-axis displays the fraction of active task vector SAE features used. The Y-axis displays the TV loss delta, calculated as (Lrv LMtethod)/ L where Lrv is the loss from steering with the task vector, Method is the loss after it has been cleaned using the corresponding method, and Lzero the uninformed (no-steering) model loss. This metric shows improvement over the task vector relative to the loss of the uninformed model. Points were collected from all tasks using different L1 coefficient values.
We observe that our method often improves task vector loss and can reduce the number of active features to one-third of those in the original task vector while maintaining relatively intact performance. In contrast, ITO rarely improves the task vector loss and is almost always outperformed by TVC.
Figures 17, 18 and 19 show task-mean loss decrease (relative to no steering loss) and remaining TV features fraction plotted against L, sweep coefficients. We see that L coefficients between 0.001 and 0.025 result in relatively intact performance, while significantly reducing the amount active SAE features. From Figure we can notice that the method performs better with higher target SAEs, being able to affect the loss with just a fraction of active SAE features.
E. Details of our SFC implementation
E.1. Implementation details
Our implementation of circuit finding attribution patching is specialized for Jax and Penzai.
We first perform a forward-backwant pass on the set of prompts, collecting residuals and gradients from the metric to residuals. We collect gradients with jax. grad by introducing "dummy"" zero-valued inputs to the metric computation function that are added to the residuals of each layer. Note that we do not use SAES during this stage.
We then perform an SAE encoding step and find the nodes (residual, attention output, and transcoder SAE features and error nodes) with the highest indirect effects using manually computed gradients from the metric. After that, find the features with the top K indirect effects for each layer and position mask and treat them as candidates for circuit edge targets. We compute gradients with respect to the metric to the values of those nodes, propagate them to "source features" up to one layer above, and multiply by the values of the source features. This way, we can compute indirect effects for circuit edges and prune the initially fully connected circuit. However, like (Marks et al., 2024), we do not perform full ablation of circuit edges.
18

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
Phi-3
Method
ITO
TVC
022
TV delaa
-02
-0.4
0.5
1.5
fraction
Figure 15. Performance of ITO and TVC across different tasks and optimization parameters compared to task vectors for Phi-3. The Y-axis shows relative improvement over task vector loss, while the X-axis shows the fraction of active TV features used. Metric calculation details are available in D.1
We include a simplified implementation of node-only SFC in Algorithm 2.
resids_pred pre: L x N x D the pre-residual stream at layer # resids_mid: L x
the middle of the residual stream # (between attention and MLP) layer L
# grads_pre: L N D gradients from the metric to resids_pre # grads_mid: L N D gradients from the metric to resids_mid
# all of the above are computed with a forward and backward
# pass without SAEs
# saes_resid: L residual stream SAES
10 saes_attn: L attention output SÄEs transcoders_attn: transcoders predicting resids_pre [1+1]
# from resids_mid[I]
def inn 14
(layer):
15
sae_ sae_encoding saes_ gees_residd[layer] .ew.ode encode (
16
resids_pres (esids_pre(layerl)
17
grad_to to sae sae_latents tax.vipp saes_resid[layer]. decode, sae_encoding (grads_pre[1])
18
20
return (grad_to_sae__ateree sae_encoding). (um sum (-1)
22
23
def inatrett_efferttoooaatttttttoott (layer):
24
sae_encoding saes_ caes_attn (tg___attt(layyy]_eeeecco
25
resids_mid[layer] resids_pre [layer])
grad to paad_ttosae_latents = jax.vip(
saes_attn[layer] decode,
28
sae_encoding
29
) (grads_midd[])
30
return (grad_too_sel sae latents sae sae_encoding) sum(-1) (-1)
31
19

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
Gemma
Gemma 16k
Metool
Mebol
ddta
15
TV frction
(335 10)
Metrol
FIU TVC
ITo TVC
Genma 2 28 65k (21 10)
Gemmia 2 28 6Sk (244 10)
Telondds
-0.4
Lo
LD fraction
Gemma 2 98 65k
Gemma 2 08 16k
Metrol
Meisol
QA
-0.4
TV. LD Coatie
TV LDI franttee
Figure 16. Performance of ITO and TVC different tasks and optimization parameters compared task vectors for Gemma Gemma Scope SAEs. The Y-axis shows the relative improvement over the loss from steering with a task vector, while the X-axis shows the fraction of active TV features used. Metric calculation details available in Appendix D.I.
def indi effecteffect_.. wtweeeefeffffttrrrrccoooooo for_trranscoder_nod (layer)
33
sae sae_encoding transcoders l [layer] encode (
34
resids (
35
ste_stt___e________________eee__ grad_te to_ttt__oo sae latents = jax.vij vip
36
transcoders [layer] decode,
20

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
Gemma
Phi-3
Mean relative loss decrease
Mean relative loss decrease
0.8
0.4
0.6
0.4
1e-05
0.0001
0.001
0.01
0.025
0.1
1e-05
0.0001
0.001
0.025
L1 Coefficient
L1 Coefficient
Mean TV L0 Fraction
0.6
Mean TV LO Fraction
0.6
0.4
0.4
0.2
0.2
1e-05 0.0001 0.001
0.01
0.025
0.05
0.1
1e-05 0.0001 0.001
0.01
0.025
0.05
0.1
L1 Coefficient
L1 Coefficient
Gemma 2 2B 65k (128 10)
Gemma 2 2B 16k (78 10)
Mean relative loss decrease
Mean relative loss decrease
0.8
0.8
0.6
0.4
0.4
0.2
0.2
0
1e-05
0.0001
0.001
0.01
0.025
0.05
0.1
1e-05
0.0001
0.001
0.01
0.025
0.05
0.1
L1 Coefficient
L1 Coefficient
Mean TV LO Fraction
0.6
Mean TV LO Fraction
0.4
0.2
0.2
0
1c-05 0.0001 0.001 0.01 0.025
0.05
0.1
1e-05 0.0001 0.001
0.01
0.025
0.05
0.1
L1 Coefficient
L1 Coefficient
Figure L1 coefficient sweeps across different models SAEs. All metries are averuged across all tasks. Error bars show standard deviation of the average for each case. Metric calculation details available in D.1.
21

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
Gemma 2B 16k (23 10)
Gemma 2 2B 16k (335 10)
Mean relative loss decrease
Mean relative loss decrease
0.6
0.4
0.5
0.2
0
1e-05
0.0001
0.001
0.01
0.025
0.1
1e-05
0.0001 0.001
0.025
L1 Coefficient
L1 Coefficient
Mean TV L0 Fraction
0.6
Mean TV LO Fraction
0.4
0.4
0.2
0.2
I
1e-05 0.0001 0.001
0.01
0.025
0.05
0.1
1e-05 0.0001 0.001
0.01
0.025
0.05
0.1
L1 Coefficient
L1 Coefficient
Gemma 2 2B 65k (21 10)
Gemma 2 2B 65k (244 10)
Mean relative loss decrease
Mean relative loss decrease
0.6
0.4
0.5
0.2
1c-05
0.0001
0.001
0.01
0.025
0.05
0.1
1e-05
0.0001
0.001
0.01
0.025
0.05
0.1
L1 Coefficient
L1 Coefficient
Mean TV LO Fraction
Mean TV LO Fraction
0.4
0.4
0.2
0.2
I
0
1c-05 0.0001 0.001 0.01
0.025
0.05
0.1
1e-05 0.0001 0.001
0.01 0.025
0.05
0.1
L1 Coefficient
L1 Coefficient
Figure L1 coefficient sweeps across different target SAE spansities and widths for Gemma 2 2B. All metrics are averaged across all tasks, Error hars show the standard deviation of the average for each case. Metric calculation details are available in Appendix D.I.
22

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
Gemma 9B 65k
Gemma 2 9B 16k
Mean relative loss decrease
Mean relative loss decrease
0.4
0.4
0.2
0.2
-0.2
-0.2
1e-05
0.0001
0.001
0.01
0.025
0.05
0.1
1e-05
0.0001
0.001
0.01
0.025
0.05
0.1
L1 Coefficient
L1 Coefficient
Mean TV LO Fraction
Mean TV L.0 Fraction
0.6
0.4
0.4
0.2
0.2
T
1e-05 0.0001 0.001 0.01 0.025
0.05
0.1
1e-05 0.0001 0.001
0.01 0.025
0.05
0.1
L1 Coefficient
L1 Coefficient
Figure 19. coefficient sweeps two SAE widths for Gemma 2 9B, All metrics averaged across all tasks. Error bars show the standard deviation of the average for cach case. Metric calculation details are available in D.I.
23

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
sae_encoding (grads_pre return torad_to_sae_latee sae_encoding) sum (-1)
[1+1]] [1+1])
Algorithm Pseudocode for Sparse Feature Circuits indirect effect calculation.
E.2. IE approximation quality
Our IE calculation approach, which aggregates effects across all tokens of the same type, resulted in each layer having limited number of non-zero nodes. This allowed to directly examine the impact of disabling each of these nodes. We assessed the quality of the IE approximation by calculating correlation coefficients between the actual effects and their approximations. To further reduce computation time, we focused exclusively on nodes from the "input," "output," and "arrow" groups. Figure 20 displays the correlations averaged across all tasks for all SAE types combined, while Figure 21 presents the metric for each SAE type separately.
0.7
Correlation
0.6
0.5
0.4
10
15
Layer
Figure 20. Average correlation of predicted and actual IEs across tasks for "input", "output" and "arrow" non-zero nodes.
Overall, observe that the approximation quality remains relatively low before layer 6, which is much deeper in the model than layer 2, reported by the original SFC paper. Non-residual stream SAEs begin to show adequate performance only in the last third of the model. This may be due to the quality of our trained SAEs, the increased task complexity, or token sype-wise aggregation, and warrants further investigation. This is the primary reason our analysis focuses mainly on layers 10-15.
SAE Type
Attention Out
Correlation
Residual
Transcoder
0.2
10
Layer
Figure 21. Average correlation of predicted and actual IEs across tasks for "input", "output" and "arrow" non-zero nodes for diffcrent
SAE types.
F. Steering with task-execution features
To evaluate the causal relevance our identified ICL features, we conducted series steering experiments. Our methodology employed zero-shot prompts for task-execution features, measuring effects across batch 32 random pairs.
24

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
We the target layer 12 using Figure 2 and extracted all task-relevant features it using our cleaning algorithm. To determine the optimal steering scale, conducted preliminary experiments using manually identified task-execution features across all tasks. Through this process, we established an optimal steering scale of 15, which we then applied consistently across all subsequent experiments.
For each pair of tasks and features, we steered with the feature and measured the relative loss improvement compared to the model's task performance on prompt without steering. This relative improvement metric allowed us to quantify the impact of each feature on task performance. Let (t. f) L zero. be the effect of steering on task with feature f.
To normalize results and highlight the most significant effects, applied several post-processing steps:
M(t, f) (1,
(tt, f) [Mtt, min ((,
min (t. (t)
M(t, f) = if (t, f) 0.2
Finally, we removed features with low maximum effect across all tasks to reduce the size of the resulting diagram. The full version of this diagram is present in Figure 22.
Prompt example with the steered token highlighted in red. Loss is calculated on the yellow token:
BOS
Follow
the
pattern
hot
cold
Example 3. Task-execution steering setup. The steered token is highlighted in red and the loss is calculated the yellow token.
smongh
countyy capial
locaton religinn
location laggoge
footall, pwaysp position
singulac_puaal
piscorssippi,c gerund
Task
pinpetsiiple pet
professio
mimple simple
locatoos sountry
aigg
antoymm
plani stoprior
tt.
E1011
19102
058
1872
1969
1226
VILLE
0224
112
1988
EIP9
00:5
0098
Feoture tesk vector
Feature present after cleaning
Feature
Figure 22. Full version of the heatmap in Figure 6 showing the effect of steering with individual task-execution features for each task. The features present in the task vector of the corresponding task are marked with dots (i.e. from the naive SAE reconstruction baseline Section 3.1). Green dots show the features that were extracted by eleaning. Red dots features present in the original task vector. Not all original features from the task vectors are present.
We also share the version of Figure 22 without normalization and value clipping. It is present in Figure 24. We see that task vectors generally contain just a few task exs-uexe features that can boost the task themselves. The remaining features have much weaker and less specific effects.
25

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
F.I. Negative steering
To further explore the effects of the executor feature, we also conducted negative steering experiments. The setup involved batch of 16 ICL prompts, each containing 32 examples for each task. We collected all features from the cleaned task vectors for every task. Similar to positive stecring, steered with features arrow tokens, but this time multiplying the direction by -1. Prompts this time contained several arrow tokens, and we steered on all of them simultaneously.
An important distinction from positive steering is that performance degradation in negative steering may occur due to two factors: (I) our causal intervention on the ICL circuit and (2) the steering scale being too high. To address this, we measured accuracy across all pairs in the batch instead of loss, as aceuracy does not decrease indefinitely. We also observed that features longer share common optimal scale. Consequently, for each task pair, we iterated and 30. For each feature, we then selected scale that reduced accuracy by at least 0. for at least one task. Steering results at this scale were used for this feature across all tasks.
several scales between I
Figure 23 displays the resulting heatmap. While we observe some degree of task specificity - and even note that some executing features from Figure 22 have their expected effects - we also find that negative steering exhibits significantly lower task specificity. Additionally, we observe that non-task-specific features have substantial impact this experiment. This suggests that steering experiments alone may not suffice for comprehensive analysis of the ICL mechanism, thus reinforcing the importance of methods such modification of SFC.
Accenscy decres
jonce laagage
plas stopdar
cuskyy siptal
preeet proeet dirplo
alpu
diyyde ploca
fooseee peligio
seogle perker
profesiion
keatoe Jespune
Vig
1222
prement
civering
Feotre
Figure 23. Negative steering heatmap. Displays accuracy decrease after optimal scale negative steering full ICL prompts. Green circles show which features were present in the cleaned task vector of the corresponding task. More details in Appendix F.I.
F.2. Gemma 2B positive steering
Additionally, we conducted zero-shot steering experiments with Gemma 2B 16k and 65k SAEs. Contrary to Gemma 2B, task executors from Gemma 2 2B did not have single common optimal steering scale. Thus, we added an extra step to the experiment: for each feature and task pair, we performed steering with several scales from 30 to 300, and then selected the scale that had maximal loss decrease any of the tasks. We then used this scale for this feature in application to all other tasks. Figure 25a and Figure 25b contain steering heatmaps for Gemma 2 2B 16k SAEs and Gemma 2B 65k SAEs respectively.
We observe a relatively similar level of executor task-specificity compared to Gemma 1. One notable difference between 16k and 65k SAES that 65k cleaned task vectors appear to contain more features with strong effect on the task. However, this may be due to the regularization coefficient being too low.
26

---

Scaling Sparse Feature Circuits For Studying In-Contex. Learning
Relarive loss decreas
couner__capht
0.
loaaion vilaion
locatio, Langago
Jenson_lnngsee
fooball wwwpppyyopppoooooo positiin
singula__phaal
poopet.iipple. grund
Jas
somplt
peron profesiio
04
pasetssipleeeg cmmple
locaio contnen
secord
fir
antoyms
stipter
fuon
08931
033
PS551
1850
UINTE
2090
1985
$199
0075
00305
Feoture lask
Festure
after cleaning
Feature
Figure Unfiltered version of heatmap Figure showing effect steering with individual task-execution features for each task. The features present in the task vector of the corresponding task marked with dots. Green dots show the features that extracted by cleaning. Red dots are the features present in the original task vector. Since the chart only contains features from cleaned task vectors, not all features from the original task vectors present.
G. Task-Detection Features
For our investigation of task-detection features, we employed a methodology similar to that used for task execution features, with a key modification. We introduced fake pair to the prompt and focused our steering on its output. This approach allowed us to simulate the effect of the detection features the way happens on real prompts.
Our analysis revealed that layers 10 and 11 were optimal for task detection, with performance notably declining subsequent layers. We selected layer 11 for our primary analysis due its proximity to layer 12, where we had previously identified the task execution features. This choice potentially facilitates more direct examination of the interaction between detection and execution mechanisms.
The steering process for detection features followed the general methodology outlined Appendix F, including the use of a batch of 32 random pairs, extraction of task-relevant features, and application post-processing steps to normalize and highlight significant effects. The primary distinction lies in the application the steering the prompt.
This approach allowed to create comprehensive representation of the causal relationships between task-detection features and the model's ability to recognize specific tasks, visualized in Figure 10.
BOS
Follow
the
pattern
Y
n
hot
cold
Example Task-detection stcering setup. The stecred token is highlighted red and the loss is calculated on the yellow token.
H. ICL interpretability literature review
This section will cover work on understanding ICL not mentioned in Section 5.
(Raventós et al.) provides evidence for two different Bayesian algorithms being learned for linear regression ICL: one for limited task distributions and one that is similar to ridge regression. also intriguingly shows that the two solutions lie in
27

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
provet siplle par sirple
from
languge
prnse, buyage
dig sound
bobdll pastoo
beston, pontier
dp
199
1200
Festree vecuur Feoture proser ahe cleasing
Feaure
(a) Gemma 2 2B 16k
Relate los decrms
Fearue
Gemma 65k
Figure 25. Unfiltered positive steering heatmap for Gemma 2 2B SAEs showing the effect of steering with individual task-execution features for each task. Steering scales were optimized each feature. The features present task vector the corresponding task marked with dots. Green dots show the features that were extracted by cleaning. Red dots the features present in the original task vector. Since the chart only contains features from cleaned task vectors, not all features from the original task vectors are present.
different basins of the loss landscape, a phase transition necessary to go from one the other. While interesting, is not clear if the results apply real-world tasks.
The existence of discrete task detection and execution features hinges the assumption that in-context learning works by classifying the task to perform and not by learning task. (Pan et al.) aims to disentangle the two with black-box approach that mixes up outputs to force the model to learn the task from scratch. (Si et al.) look at biases in task recognition in ambiguous examples through a black-box lens. We find more clear task features for some tasks than others but do not consider whether this is linked to how common a task pretraining data.
(Xie al.) proposes that in-context learning happens because language models aim to model a latent topic variable predict text with long-range coherence. (Wang et al., 2024) show following the proposed steps rigorously improves results in real-world models. However, they do not endeavor to explain the behavior of non-finetuned models by looking at internal representations; instead, they aim to improve ICI. performance.
(Han et al.) use a weight-space method to find examples training data that promote in-context learning using a method akin to (Grosse et al., 2023), producing results similar to per-token loss analyses in (Olsson et al., 2022), and, similarly to the studies mentioned above, finds that those examples involve long-rau longraange coherence. Our method is also capable of finding
28

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
examples in data that similar to ICL, and find crisp examples for many tasks being performed Appendix I.
(Bansal et al.) offers a deeper look into induction heads, scaling up (Olsson et al., 2022) the way we scale up (Marks et al., 2024). Crucially, it finds that MLPs in later layers cannot be removed while preserving ICL performance, indirectly corroborating our findings from Section 4.2. (Chen et al.) come up with proof that states that gradient flow converges to a generalized version of the algorithm suggested by (Olsson et al., 2022) when trained on n-gram Markov chain data.
(Garg et al.) studies the performance of toy models trained on in-context regression various function classes. (Yadlowsky al.) find that Transformers trained on regression with multiple function classes have trouble combining solutions for learning those functions. (Oswald et al.) construct from gradient descent and find evidence for the algorithm being represented on real models trained on toy tasks. (Mahankali et al.) proves that this algorithm is optimal for single-layer transformers on noisy linear regression data. (Shen et al.) questions the applicability of this model to real-world transformers. (Bai et al.) finds that transformers can switch between multiple different learning algorithms for ICL. (Dai et al.) find multiple similarities between changes made to model predictions from in-context learning and weight finetuning.
of weights for linear attention Transformers that reproduce updates
While important, we do not consider this direction of interpreting transformers trained on regression for concrete function classes through primarily white-box techniques. Instead, we aim to focus clear discrete tasks which are likely to have individual features.
The results of (Wang et al.) are perhaps the most similar to our findings. The study finds "anchor tokens" responsible for aggregating semantic information, analogous to our "output tokens" (Section 2.3) and task desk-detection features. They tackle the full circuit responsible for ICL bottom-up and intervene on models using their understanding, improving accuracy. Like this paper, they do not deeply investigate later attention and MLP layers. Our study uses SAE features find strong linear directions on output and arrow tokens corresponding to task detection and execution respectively, offering a different perspective. Additionally, considered in (Wang et al.).
consider over 20 diverse token-to-toker tasks, as opposed to the text classification datasets
I. Max Activating Examples
This section contains max activating examples for some executor and detector features for Gemma 1 2B, as described in Bricken al. (2023). They are computed by iterating over the training data distribution (FineWeb) and sampling activations of SAE features that fall within disjoint buckets for the activation value of span 0.5. We can observe that the degree of intuitive interpretability depends on the amount of task misklar contexts in the training data and SAE width.
We also provide max activating examples for Gemma 2 2B executor features from Figures 25b and 25a. These max activating examples are taken from the Neuronpedia Lin (2023) and are available in Figures 29 and 28.
Here we can notice the main difference between executors and detectors: executors mainly activate before the task completion, while detectors activate on the token that completes the task. We also found that in Gemma 1 2B detector features for some tasks were split between several token-level features (like the journalism feature in Figure 270) and they did not create single feature before the task executing features activated. We attribute this to the limited expressivity of the SARs Es that we used.
29

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
Judgment Staff ( 0 Sabaki Tsue?), also known
Rift The Judgment Staff (š) 3 0ix. Sabaki no Tsue?), also
more commonly known as the Four-Tails (29 Yonbi), is
the Four-Tails (29 Yonbi), tailed beast sealed 1 meters dybde er vigtige for et-årige afgr te vinden (Inclusief een directe link naar de publicatie online als dez elders te vinden (inclusief een directe link naar de publicatie online een publicatie elders te vinden (Inclusief een
st by alternating between lower and upper registe erences between northern and southern Italian co models, both import and domestic, Ulmer's spec between fresh and traditional, casual and elegant ces and both local and remote event logging. That globally in both tropical and temperate waters. Blu e, light and darkness, life and death. This assembl
Oh (f A 16 > /
Samural Gattai Shinken 0?)
Ft 9TC- - 5 Mf
naar de publicatle online als deze beschikbaar is In een
atle online als deze beschikbaar is In een database op het Internet).
Goendagiri ( न |. 1al 91c THC न (-----1 -9-
did, upstage-bos-tsrael (or mnnrr) is a small yet diverse Agencia Española de Medicamentos y Productos Sanitarios, A authority (Agencia Española de Medicamentos y Productos Sanitar
(b) Max activating examples for the English to foreign lan- guage translation executor feature 26987.
(a) Max activating examples for the antonyms executor feature I 11618 cultural diversity and that special joie de vivre of life), that Mont of creating Papel Picado Banderitas (little paper banners). Populart t nicknames: "la cludad dorada" (the golden city). Salamanca also from the same root as jihad, struggle, in the sense that ijti conurbation "tsukin jigoku," or commuter hell. Images of rail worken he acted according to our Sunna (tradition), and whoever slaughten and, of course, your karma (good and bad). John brings wealth The "it-sa Sicherheitsmesse (security trade show). OWASP conferer Cerveseria Catalan along with a caña cdaaf beer) and rosé...yes s Apostle! slaughtered the Nusuk (before the prayer) but I<bos-Tru the living entities; vva-artha-interes;; vyatikramah -ob
working. connecting with diverse people and seeking out sustain croll, and 2) Isolating unifying elements that transcend the indivi nt like landing the moon or the discovery of DNA. The focus by using our search feature or by following the links above. Feel as Liking and Favoriting photos] but will expire after spends her free time traveling and visiting exotic locations arou enses tighten, grabbing offensive rebounds and making putback than participating in observing or<bos>l tend to specialise inother, rather than participating or observing or<bos>l tend to
passenger car with plastics sheets and inhaling toxic fumes fr
by her given name (Angelella little angel), but called her Columba
nilies when going a long distance or flying with them when we ca
(c) Max activating examples for the translation to English executor feature 5579.
(d) Max activating examples for the "next comes gerund form" executor feature 15554.
came all the way from Oslo Norway for the event. This has migrated to New York City from the Galicia area, in northwest Spain.
scientistb, Rury Holman, directorc behalf of the United Kingdom Stinton] NAR CEO Charlie Youngi President/CEO
nal. There were a few folks folks/from Canada (British Columbia, Ontario and Quebec immigrant from Bangladesh who was granted political asylum by the and world number six Li Na of China. Azarenka, who is
San Fernando Realty Dale Stinton,, NAR CEO Charlie Youngl President/ director (aa, Philip Clarke| research fellowa, aallowa,-Andrew Andrew Farmer
hood in Seattle my college years Boston,
Hummel| MD, Ezio Bonifacioj PHD, and Anette-G.
oweeer, batteries from rival manufacturers in the U.S. are exempt from USA and four in England. Of these, only three have
lives in Bombay. Faruq Hassan Poet and critic; teaches at Dawson Coll
<bos-llla Williams, President Randall Ramsay, Vice President Texas Cha
(e) Max activating examples for the prediction of citylcountry feature 850.
(I) Max activating examples for the person's 's occupation ex- ecutor feature 13458.
Figure 26. Max activating examples for executor features from Figure 6.
30

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
Target: $0.10 Long Term Target: $0.45
Soluble Fiber and 3 grams of Insoluble Fiber. Ground Flaxseeds are a ge 5 In. x Outer Dimensions: In.
a service: [Morning Services|] Evening Servicel Morning Worship at 8
other system that substantively usessbos-Wkkpeddaassoe sobre física de particulas Directed By Tom Grundy Es gibt noch keine Kommentare. Sei der erste .bos>
temp: 15"C min temp: C1"C
Upper Zone and 75 Bottles in Lower Zone Read More... The page and 30% viewing the right half" "apple's decision
006 213 halaman The book was selected as one of
Hour | Webcast- enregistré |Où et quand What is the Webcast About
integration is performed first, followed the quantitative combination.
[score hidden] 23 Minuten zuvor You just said you exposed
access to the content item. Returns FALSE if the current<bosxOraclee@ As alternate between defensive positions and offensive positions,
hazard [score hidden] 23 Minuten zuvor You just sald 'ff you
DA-LINKERbbied mogelljkheden om een publicatle elders te vinden (
(a) Max activating examples for the antonyms detector feature 1 11050.
(b) Max activating examples for the English to foreign language switch detector feature 7928.
Say You rate this item by giving score of one (poor), please let us know about it by sending our help desk an email. which your order will be shipped. By doing this the few products ppad=-pab-padb-bos-eeeee for music by typing a word or ph
Superficie Lunare (Composizione)" (Lunar surface- - composition), execut hardline group Tawhid wal Jihad (Monotheism and Holy War). hardline group Tawhid wal Jihad (Monotheism and Holy War). One hid wal Jihad (Monotheism and Holy War). One civilian was among line group Tawhid wal Jihad (Monotheism and Holy War). One civilian that reads "Arbeit Macht Frei" ("Work Brings Freedom") is seminal mor Tavola San Giuseppe (St. Joseph's 's Feast). You'll
one month non-recurring subscription by sending cashier's s... Learn more about Concordia by following the links below: C
this product deliver? Pay forward by sharing what you loved (a
Browse: Browse the database applying
more filters to
we are celebrating Valentine's Day by sharing some gorgeous a
page needs content. You can help by adding sentence ph NewsOK. He composed the ad by animating still photos taken
As part of the Tres Fronteras (Three Borders) area that includes Foz and
(c) Max activating examples for the gerund form detector feature 8446.
(d) Max activating examples for the translation English detector feature 31 123.
><bos>> have been technology journalist and consultant for near
donation will help independent Adventist journalism expand across an 50 journalists gathered at Klosters, a Swiss ski
tting junters.journalinnn football managers and players. We also
Modelo<bos> The award-winning journalist Robert Fisk gave the in
the homeland of ties- Croatia. There we found three local brands that (Reuters) BBllarra's president Thursday called for
Pulitzer Prize-winning journalist, formerly with The Washington Po: IW. If you are Journalist seeking comment on a story or more info <bos>Peripatetic journalist and translator Porter (Road to Heaven: will likely endanger the lives of journalists and aid workers in the houghtful post about the hazards of journalism following revelation
s>Welcome to The Dubline: oldest and newest discovery trail. httpr--padddbbbuuuuggaaaaaaee com com is owned and mi papadd-pad-pa---a---aa----
should know that "Deutschland" means Germany German. Germany is urban Budapest sketch... (Hungary) The old building standing on V seMssage Behind African Heaters For Norway Spoof An online video, Mrr of Switzerland: More Personal Ads from the London Review
about interviewing and journalism. Just like marketing person do
(e) Max activating examples for the country detector feature 1 11459.
(f) Max activating examples for the journalist feature 26436. (The strongest detector for the person ceasooo_rofession task).
Figure 27, Max activating examples for detector features from Figure 10.
31

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
anyway, dogs will dogs. My name
means lost and found in the Mandingo language
hook. Boys will be boys, I suppose
prob Leme speak eng lish
really hate exp shock
shock
because he could not speak Mandarin or Cantonese.
Extracttiondptionnssooddddeeeee
A. Heim. German French
Vertice3r Vertice3r
"market field" in Ma It was
Vertice3f Vertice3f
of her tean speak in English, Elena immediately
four ways, and only four ways, in
international projects. I speak eng Lish, spanish and XmL: Lang=
ProfiterJniMMthoo Sanp LingProfiternnn
particular device-and only for that device.
(a) Max activating examples for the language prediction ex- ecutor feature 13804.
(b) Max activating examples for repetition exccutor feature 12646. Extracted from the algo last TV.
st 1ssue of Norsk Entomo Tidsskrift now Norwegian Journal of Entoso appeared in May 1921.
the Swizz capital, Mbalanne
vision and adaintstration of the Dirección Provincial de Vialidad Provincial Dept. of transportation)
1n the densely ly popula lated Dapltal of Ronrovia,
north the capital Kabul crowd
-Insekt-Nytt Insect News is written in popu lar science style and is the society's
Poznari.co References
oja Universal do Reino Deus [Universal Church of the Kingdon of God dented invo Lvenent in the scandal.
the regional capital Bialy
play Dan Dronning (he Quear) 1843. Nany artist frequented the
West of the regional capital Mrocian eeReeerences
ero reighhourhood hood in the capital, Bujusbura
Lithuanian Zaliof1 rinktine Squad), ) longing
Algimantas litaryy distric
the seell 11s that filled downtown Greenevillé in December
rvals. -Norr rvals.-Norske ske Insektabeller Lter (Nooeegian Insect Tab series inexpensive Norwegian-
(c) Max activating examples for the capital pre- diction executor feature 16315.
(d) Max activating examples for the translation feature 493.
Figure 28. Max activating examples for Gemma 2 2B 16k executor features from Figure 25a.
32

---

Scaling Sparse Feature Circuits For Studying In-Context Learning
apparent use of comp Liant and non-compliant forn die soon, today tomorrow, :n
), Judd Ringer (right end , George Benson
various degrees ees luctant and un likely. There is matter how different or diverse these may be What reliable trusted websites? are clearly upsides and downsides for those companies
plays as winger or as left back
Casti lla as either central defender or a left
standing up falling back down
it S right tackle. Filling in
(a) Max activating examples for the antonyms executor feature 45288.
(b) Max activating examples for the foothall player position executor feature 18981.
Hai- Katanga
(Secede Katanga ") #etherr Mai-Mai groups large
Engtish rugby p SSee
Saxon, 1.e, they
together undivided)
) Christian ascension
014), American racing iverew
Zalioj1 rinktiné Green Squad), be belonging partisans' Alginantas mi military distri:
936 British composer, conductor and
square ca laledd Pasar Medan literally. "harket field in Malay. was here that the cit
was an Eng Lish spor tsman who played rugby
had plastic kagam1 moch1, which trans lates to mirror rice cake. Basically snowman
was an Italian footbal ler from Bastia
and Roberto Jefferson. The Ministério Público Federal ( (the Federal Prosecutor Office
(c) Max activating examples for the per-
son poo_prrfession executor feature 46729.
(d) Max activating examples for translation to English executor feature 62633.
Figure Max activating examples Gemma 2B 65k executor features from Figure Figure 25b.
33


Reviews:
[{"review_id": "PGa30HnMar", "reviewer": "Reviewer_K3Cy", "summary": "This paper explores how sparse autoencoders (SAEs) can enhance our understanding of in-context learning (ICL) mechanisms in LLMs. The paper's main contributions include:\n- Identifying two core components of ICL circuits: task-detection features that identify required tasks from the prompt, and task-execution features that implement those tasks during generation.\n- Developing a Task Vector Cleaning (TVC) algorithm that decomposes task vectors into their most relevant sparse features, enabling more precise analysis of ICL mechanisms.\n- Adapting the Sparse Feature Circuits (SFC) methodology to work with the much larger Gemma-1 2B model (30x larger than models in previous circuit analysis studies) and applying it to the complex task of ICL.\n- Uncovering the interaction between these components: attention heads and MLPs process information from task-detection features to activate appropriate task-execution features.\n- The authors provide evidence that these task vectors can be represented as sparse sums of SAE latents, and they demonstrate causal relationships between detection and execution features through circuit analysis.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hello reviewer K3Cy,\n\nThank you for your thoughtful and supportive review of our paper. We appreciate your detailed feedback and are grateful for your recommendation to accept the paper.\n\nRegarding your specific question about the weak detection-execution connections for the `person_profession` and `present_simple_gerund` tasks:\n\nFor `person_profession`, our analysis revealed the model's general performance of this task is relatively poor. As seen in Figure 27f, the strongest detector feature is a single token feature focused on journalism, indicating that SAE features did not pick up strong task-related directions alongside poor model's accuracy. This both contributes to weak connection of executing and detecting features.\n\nFor `present_simple_gerund`, the situation is more nuanced. One possible explanation is feature splitting. Since gerund forms are common in the training data, they have several corresponding detector and executor features. At the same time, executor and detector features with the strongest steering effect may correspond to different cases of gerund occurrences, and thus be weakly connected.      \n\nThank you again for your positive feedback on our work!", "claims_and_evidence": "The claim that task vectors can be decomposed into sparse, interpretable features is supported by the TVC algorithm results, showing the algorithm can reduce the number of active SAE features by 70% while maintaining or improving effect on loss.\nThe identification of task-detection and task-execution features is supported through thorough analysis of activation patterns, steering experiments, and ablation studies that demonstrate their causal roles.", "methods_and_evaluation_criteria": "The methods and evaluation criteria used in this paper are appropriate for the research questions:\n- The Task Vector Cleaning algorithm effectively isolates task-relevant features \n- The steering experiments provide measure of causal influence by testing how individual features affect performance on specific tasks.", "theoretical_claims": "There is no proofs or theoretical claims", "experimental_designs_or_analyses": "The experimental designs are sound:\n- The token position categorization approach for handling ICL prompts is well-motivated and effective for isolating different functional roles.\n- The steering experiments with both positive and negative steering provide complementary evidence for feature functions.\n- The ablation studies establish causal relationships between detected circuit components.\n\nOne minor issue is that the authors restricted their circuit search to intermediate layers 10-17 of the 18 total layers, justifying this based on IE approximation quality in earlier layers. While reasonable, this limitation means some potential early-layer mechanisms might have been missed.", "relation_to_broader_scientific_literature": "The paper effectively situates its contributions within SAEs and ICL literature:\n- It builds upon prior work on task vectors (Todd et al., 2024; Hendel et al., 2023) by providing a mechanistic explanation of how these vectors function through sparse features.\n- It extends the SFC methodology from Marks et al. (2024) to handle more complex tasks and larger models.\n- It connects to work on induction heads (Olsson et al., 2022) while demonstrating that ICL requires more complex mechanisms beyond just induction.", "essential_references_not_discussed": "The paper could have benefited from more discussion of how their findings relate to theoretical models of IC", "other_strengths_and_weaknesses": "Strengths:\n- The paper successfully scales mechanistic interpretability techniques for ICL.\n- The identification of two distinct mechanisms (detection and execution) provides a clear conceptual framework for understanding ICL.\n- The paper demonstrates how SAEs can be used for more than just interpreting individual features but for understanding complex model behaviors.\n- The TVC algorithm is a valuable contribution that could be applied to other research on task vectors.\n\nWeaknesses:\n- The analysis focuses on relatively simple ICL tasks (e.g., antonyms, translation). It's unclear how well the approach would extend to more complex reasoning tasks.\n- The authors note that there are often multiple competing execution and detection features, suggesting redundancy in the model's representations that could complicate interpretation.\n- The paper doesn't explore how these mechanisms might differ across model", "other_comments_or_suggestions": "It would be interesting to see how the detected ICL circuits relate to other mechanisms that have been studied in language models, such as factual recall or logical reasoning.", "questions_for_authors": "You mentioned person_profession and present_simple_gerund showed unusually weak detection-execution connections. Do you have hypotheses about why these specific tasks might be processed differently?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v2vdO1kWqF", "reviewer": "Reviewer_zx2e", "summary": "The paper applies sparse autoencoders (SAE) to better understand in-context learning (ICL).\n\nThe paper first learns SAE representations of task vectors. Task vectors are first constructed in a heuristic manner (averaging the residual streams of arrow tokens (between inputs and outputs). The paper proposes \"task vector cleaning,\" in which the SAE decomposition is fine-tuned on the ICL task with a sparsity constraint. This improves the zero-shot task performance as well as sparsity. Here, \"task execution features\" are discovered, which activate directly before an output. Steering experiments are conducted, which establish the causal effect of task latents on the task. Similar tasks have similar strong steering vectors.\n\nThen, a circuit learning approach is applied. The primary finding in this section is the identification of \"task detection features,\" which activate on output tokens in the training data. It is shown that detection features causally activate executor features.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hello reviewer `zx2e` – thank you for your thoughtful review. We appreciate your recognition that our methods are \"solid\" and that task vector cleaning is \"a simple approach that appears to work well to get strong SAE decompositions.\"\n\n**Regarding feature definitions**: we consider *task-specific causal influence to be the defining characteristic of both detector and executor features*, and apologize for not making this clearer in the main text. We will update the manuscript based on this feedback. To expand specifically on what we mean by task-detection and task-execution features:\n\n**Task-detection** features are defined as *intermediate circuit components between the raw prompt inputs and the executor features*. They function as a specialized probing mechanism that identifies and encodes which specific task is being performed. These features primarily activate on output tokens (96.76% as shown in Table 2), consistent with their role in monitoring completed examples to determine the pattern being demonstrated. Importantly, our steering experiments in Section 4.2 demonstrate that these detector features also have direct causal effects on task performance when steered on blank output tokens.\n\n**Task-execution** features are defined as features that *directly impact task completion*. They affect whether the specific task happens without requiring additional intermediate processing. Their strong activation on arrow tokens (89.8% as shown in Table 1) demonstrates their positioning at precisely the point where the model must apply the identified transformation.\n\nFor greater specificity, we classify these features using clear criteria:\n- **Detection features**: \n(1) consistent activation on output tokens in examples, (2) minimal activation during generation, (3) causal influence on executor features (quantified in Figure 11), and (4) direct causal effect on task performance when steered on output tokens\n- **Execution features**: (1) peak activation on arrow tokens immediately preceding task completion, (2) strong causal effect on task performance when steered with (Figure 6), and (3) task-specific activation patterns on raw data.\n\n**Regarding your suggestion about \"constructing task vectors on new tasks\"** - if you're referring to zero-shot generalization, our steering experiments in Section 3.2 and Figure 6 demonstrate the causal effect of our identified features on model behavior for the tasks studied. While running additional experiments on entirely new tasks is beyond our current rebuttal time frame, we believe the consistent patterns across our current task set suggest potential for generalization.\n\nThe circuit analysis in Section 4.2 and Figure 11 provides empirical validation of our proposed abstraction by showing that: (1) ablating detection features reduces execution feature activation, (2) this relationship is consistent across tasks. These quantitative measures establish that detection and execution features form distinct functional components in the ICL circuit.\n\nWe appreciate your feedback and will incorporate these clarifications to strengthen the paper.", "claims_and_evidence": "The primary claim is that sparse autoencoders can help understand in-context learning circuits. This is supported by the identification of single SAE features that causally effect performance in the in-context learning task (Figure 6). The claim that this helps with understanding is backed up by the identification of \"detection\" and \"execution\" features. The evidence here is somewhat less convincing, primarily because the definitions of these features are not clearly stated. For example, while Table 1 shows that executor features activate mostly on arrow tokens, it is unclear if this is remarkable given that executor features are \"characterized by\" the property that \"Their activation peaks on the token immediately preceding task completion.\" A more comprehensive analysis of what these features are for each task would be beneficial.", "methods_and_evaluation_criteria": "The methods seem solid. The task vector cleaning method in particular is a simple approach that appears to work well to get strong SAE decompositions.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Experiments generally seemed valid. However, as mentioned above, definitions of detection and execution features were underspecified.", "relation_to_broader_scientific_literature": "The paper contributes to a growing literature in mechanistic interpretability, in particular in using sparse autoencoders. While sparse autoencoders have been shown to identify interpretable and causal features in LLMs, there use to study more specific behaviors and capabilities is very nascent. This paper considers the task of in-context learning, which is a topic of significant interest. The existence of detection and execution features are an interesting finding.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Overall, the paper pursues a promising direction in applying sparse autoencoders to better understand in-context learning. The development of task vector cleaning appears to be a clean way to learn task vectors that are cleanly expressed by SAE latents.\n\nThe primary weakness of the paper is a lack of specificity when describing and analyzing the proposed detection and execution features. Without a more precise understanding of how these features are characterized, it is difficult to assess the validity of this abstraction. Here, the most convincing evidence of this abstraction would be a demonstration of (1) the ability to construct task vectors on new tasks based on learned features, similar to how it has been demonstrated that LLM behavior can be manipulated through SAE latents, and (2) a more precise connection between detection and execution features and behavior that distinguishes their specific roles.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bhGH10ZXHU", "reviewer": "Reviewer_fzas", "summary": "This paper uses sparse autoencoders (SAEs) to study in-context learning (ICL), specifically focusing on ICL tasks that can be abstracted into a task vector. The authors propose task vector cleaning (TVC), a methodology to identify task-execution features from the set of SAE features that implement the ICL tasks during generation. The authors adapt the sparse feature circuit (SFC) methodology starting from these features, demonstrating evidence of the faithfulness and task specificity of discovered circuits. This also reveals another set of features, termed as task-detection features, which are activated prior to the task-execution ones and have a downstream activation effect on them. Experiments are mainly launched on Gemma-1 2B.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hello Reviewer `fzas`. Thank you for your impressively extensive and thoughtful feedback, and for your careful attention to details even in the Appendix.\n\n**On our \"10-35x more parameters\" claim:** You note that \"[1] has already performed circuit-style analysis [...] using models up to 70B parameters.\" We should have been more precise in our claim. By \"comparable, circuits-style mechanistic interpretability research,\" we specifically meant work that provides end-to-end descriptions of task performance using SAEs. The Lieberum et al. paper focuses on attention heads' impact on final logits rather than providing a comprehensive explanation of MMLU processing. They state: \"Our main focus in this work was on the final parts ... the rest of the circuit is still undiscovered.\" We will edit this into our manuscript.\n\nExperimental design concerns:\n\n**On \"lack of quantitative description of experiments\"**: We moved many implementation details to the Appendix due to space constraints. We'll improve figure captions in our revision to better explain used metrics like the Average relative loss change you mentioned. The normalization process for the heatmaps (Fig. 6, 10, 11) follows these steps (Appendix F):\n1. Calculate raw metric: metric[task, feature] = steered_loss[task, feature] / loss[task]\n2. Clip: metric[task, feature] = clip(metric[task, feature], 1)\n3. Normalize per task: \n   metric[task, feature] = (metric[task, feature] - min_f(metric[task, f])) / \n                          (max_f(metric[task, f]) - min_f(metric[task, f]))\n4. Threshold: metric[task, feature] = 0 if metric[task, feature] < 0.2\n\n**On token mass calculation:** We use the same parameters as for TVC: a batch of 32 prompts each with 20 randomly sampled ICL examples.\n\n**On \"missing results though reported\":** You cite that we mentioned \"...we successfully applied ... to Gemma 2 2B and 9B models... it was also successful with the Phi-3 3B model.\" This statement refers specifically to the TVC algorithm's ability to reduce feature count while preserving task vector effects, not to the full suite of steering experiments. Our Appendix contains TVC results for all mentioned models (Figures 17-18), confirming this capability. Full steering experiments require more computational and human resources than just TVC. We decided to limit this type of analysis to Gemma 2B models, but expect that it will extrapolate to other models, judging by some of the max activating examples we analyzed. \n\n**On \"mismatched results between heatmap and text description\":** You noted that \"the feature that has the most significant effect on en_fr and en_it is eliminated through the TVC process according to Figure 22.\"\n\nFeature 26987 is a generic \"English to foreign language\" translation feature. Since Spanish tokens are prevalent in model training data, the model handles English-to-Spanish translation well, while English-to-French/Italian ICL performance is weaker. Task vectors without cleaning perform poorly for these language pairs.\n\nTask vectors contain multiple mixed-language features (26987, 26594, 6594), with 26987 having the cleanest translation examples. Since en_fr and en_it directions aren't naturally strong in the model, TVC reconstructs them using a different mix of features.\nIn better-trained models and SAEs (Figure 25), these tasks have separate execution features. This shows that TVC finds a decent performing sparse combination of features in the absence of strong task-specific features.\n\nQuestions:\n\n**1. On task vectors being ood:** Yes, we tried identifying common features across individual residual streams, but this highlighted many dense features alongside task-relevant ones. For Gemma-1 2B SAEs, individual residual streams activate ~40 features versus ~10 in averaged task vectors and 3-5 after cleaning.\n\n**2. On robustness to sample sizes:** We briefly explored different batch sizes and shot counts. Larger samples didn't significantly improve the TVC results. Smaller shot counts risk missing task-specific features whose presence is weaker early in the prompt. Smaller batch sizes may lead to extracting batch-specific features, especially for tasks with weaker vectors. Strong task-executing features are extracted even from small batches.\n\n**3. On ablation:** Our experiments showed no dramatic differences between zero and mean ablation. Zero ablation also doesn’t require us to store SAE activation statistics.\n\n**4. On the apparent contradiction in Section 4.1.3:** Figure 12 shows only a small subset of the circuit focusing on core task-related features. The ~500 features needed for 0.6 faithfulness include generic pattern repetition features, task executing features split across multiple features and layers, and features encoding answers in later model layers.\n\nOverall, we appreciate your feedback on the specificity of our claims, and will clearly state the takeaways from our research in our manuscript. We hope you will revise your score in light of this!", "claims_and_evidence": "Generally yes, however the contributions and claims should be clarified, and the experimental results should be approached with caution, as follows.\n- The authors claim in the first contribution that the studied **Gemma-1 2B** has “10 – 35x more parameters than prior models in comparable, circuits-style mechanistic interpretability research”. However, [1] has already performed circuit-style analysis on multiple-choice question-answering tasks using models up to **70B** parameters. Thus, the claim is invalid.\n- The paper lacks a detailed, quantitative description of the experimental setup throughout the manuscript, and main results are constrained to Gemma-1 2B. This affects the scope of the paper on whether the claim holds true for more general ICL tasks and larger models.\n\n[1] Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla", "methods_and_evaluation_criteria": "Yes. The task vector is a widely accepted methodology to understand and steer ICL behavior in LLMs, and the proposed decomposition of task vectors by SAE features and the subsequent study using SFC are well-inspired by current literature. However, there are concerns regarding writing and ablation studies, which affect the scope of evaluation. Please refer to **experimental designs or analyses.**", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "Yes. I have several concerns as follows.\n- **The scope and applicability of the proposed method.** The experiments presented in the main text used Gemma-1 2B SAE trained by the authors. Given the current open-sourced SAEs on larger models (Gemma Scope, Llama Scope etc) and the claimed simplicity (“For Gemma 1, it stops at 100-200 iterations, which is close to 40 seconds at 5 iterations per second.”), I assume it is plausible for the authors to launch the same series of experiments on larger scale LLMs to demonstrate the scalability.\n- **Lack of quantitative description of experiments.** For example, in Figure 2, the metric “Average relative loss change” is not described in the main text until Appendix D.1. This issue also applies to the same quantity in Figure 3, and the quantities displayed in heatmaps (Figure 6, 10, 11) among others. Numerical results should be described, or at least captioned, precisely to accurately reflect the objective of interest, facilitating the interpretation and understanding of the outcome. This statement also applies to SFC-related experiments. Could you explain the normalizing and clipping process when producing the heatmap in more detail? Further, when calculating the token type activation masses to produce Table 1, what dataset are you using, and how large is the batch?\n- **Missing results though reported.** This connects to point 1. The authors mentioned “...we successfully applied … to Gemma 2 2B and 9B models… it was also successful with the Phi-3 3B model.” However, I can only find the Gemma 2 2B results in the appendix, and the results with all other models are **partial** (only the TVC comparison to inference-time optimization, ITO). How do the steering results appear for these models? Will the pattern shown in the heatmaps persist?\n- **Mismatched results between heatmap and text description.** In Section 3.1, task-execution features are extracted via the proposed TVC to filter out the originally noisy features. However, the features displayed in heatmaps, such as in Figure 6, are **not** the task-execution set. The feature that has the most significant effect on en_fr and en_it is eliminated through the TVC process according to Figure 22. Thus, the description of the heatmap is improper - these non-task-execution features should be highlighted in the main text instead of in the appendix. Do you have an explanation for why certain important features are discarded by TVC? Furthermore, how does this impact the trustworthiness of TVC for task-execution feature extraction?\n\nFor other concerns, see Questions For Authors.", "relation_to_broader_scientific_literature": "This paper studies ICL via SAE, proposes to decompose a task vector through SAE features to locate task-related crucial features that allow for steering.", "essential_references_not_discussed": "I have not found essential missing references, however I have one concern as in Claims and Evidence, point 1. This reference was cited but not discussed in the manner described.", "other_strengths_and_weaknesses": "The writing of the paper can be improved. In addition to the issue of imprecise descriptions in the evaluation, there are other points that can be addressed. For example, some notations (e.g., d_SAE in Section 3.1) and important reference methodologies (e.g., inference-time optimization, ITO, in Section 3.1) are directly used without any definition or introduction. Further, there are also mixing usage of \\citet and \\citep throughout the paper.", "other_comments_or_suggestions": "- **Please use the correct template with line numbers clearly shown on the sidebar.**\n- Section 2.1: \"...with f denoting the pre-activation features...\" (This should be post-activation since the activation \\sigma is included.)\n- Section 3.1: \"Figure Figure 2 presents...\" (Duplicate text)", "questions_for_authors": "My questions are as follows.\n- In Section 3.1, you found that task vectors are out-of-distribution to the SAE. Since a task vector is effectively an average over in-distribution vectors, have you tried to establish a baseline by feeding these iid inputs to the SAE, and check if there are commonly-activated features? Can the task-execution and task-detection features be identified through this method? If not, what is the ratio that these two sets of features are activated respectively, when these original inputs are fed into the model?\n- As described, when launching the proposed TVC, you stated it “begins with collecting residuals for task vectors using a batch of 16 and 16-shot prompts”, and then optimize the coefficients “on a training batch of 24 pairs, with evaluation conducted on an additional 24 pairs.” Have you tested how robust TVC is to these hyperparameters? Can TVC recover the task-execution features with fewer prompts when constructing the task vector? How does the quality of identified features vary when the size of the training batch changes?\n- In Section 4.1.3, you mentioned, “we opt for zero ablation since it better aligns with the sparse nature of SAE features.” How do different ablation choices here affect your experimental results? Will it lead to a detrimental effect?\n- The result in Section 4.1.3 appears contradictory to the claims made in the main text. For example, the circuit in Figure 12 only consists of **5 features**, while SFC requires **~500 features** to achieve a faithfulness level of 0.6. Do you have an explanation for this phenomenon? What is the functionality of the other features, given that the discovered task-related features only occupy a tiny fraction of this circuit?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ik7GwkPkye", "reviewer": "Reviewer_1tPd", "summary": "This paper  aim to understand the mechanism of ICL by leveraging SAE, along with other techniques such as ITO and SFC, to analyze the properties of ICL task vectors. And how these techniques can be combined and improved to enhance the understanding underlying mechanisms.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hello Reviewer `1tPd`, thank you for your review. We appreciate your time and feedback, but it appears there may be some fundamental misunderstandings about our paper's focus and contributions that we'd like to clarify. **You stated: “1. What is the main contribution of this paper? The paper attempts to introduce a new TVC” but this is not the main contribution of our paper.** Instead, the main contribution of the paper is the use of SAEs to understand the circuitry behind in-context learning (ICL), as is clear from the title of our paper. This is important because our contributions are considerably more exciting for mechanistic interpretability researchers. TVC is a tool we developed to help identify which existing SAE features are most relevant to ICL tasks. Critically, TVC **does not** retrain or modify the pretrained SAE in any way - it only identifies which features in the already-trained SAE are causally relevant to ICL. Therefore, the comparison to AxBench (a paper that was released two days before the final submission deadline for ICML 2025 and therefore does not need to be addressed by us since it is contemporary work) is much less relevant to our work because our research is principally about circuits rather than steering.\n\n**On the motivation for using SAE to analyze task vectors.** Our motivation is to understand the circuit mechanisms behind in-context learning. While previous works like Todd et al. (2024) and Hendel et al. (2023) discovered task vectors, they didn't explain how these vectors are mechanistically implemented in the model. SAEs help us decompose these task vectors into interpretable features that form part of the model's computation.\n\n**On separating detection and execution features:** This separation was an empirical discovery, not an assumption, supported by multiple lines of evidence:\n\nDifferent activation patterns: execution features activate on arrow tokens (89.8%), detection features on output tokens (96.76%) - Tables 1 and 2\n\nLayer positioning: detection features work best in earlier layers (11), execution features in later layers (12)\n\nCausal structure: ablating detection features reduces execution feature activation (Figure 11)\n\nTask-specific steering effects: both feature types affect different tasks differently (Figures 6 and 10)\n\nOur answer to reviewer `zx2e` also contains a more thorough definition of both feature types.\n\n**On results with more models and sparsity levels:** Our Appendix includes experiments across multiple models (Gemma 1 2B, Gemma 2 2B, Gemma 2 9B, Phi-3 3B) and SAE configurations. Figures 16-19 show TVC performance across these settings, demonstrating that our approach generalizes well. These figures include different sparsities, widths and models, including Gemma Scope SAEs.", "claims_and_evidence": "I have few questions and concerns:\n1. What is the main contribution of this paper? \n\nThe paper attempts to introduce a new TVC. My understanding is that this approach primarily involves continued training of a pretrained SAE specifically for ICL. If my understanding is correct, this adaptation may compromise the generalizability of SAE. If the goal is to extract features for specific tasks, why not use supervised methods instead? The authors might consider referring to benchmark techniques such as DiffMean and ReFT, as outlined in AxBench [1], which may offer more effective approaches for task feature extraction.\n\n2. What is the motivation of using SAE to analyze task vector?\n\n3. What empirical observations or justifications led to the decision to separate task detection and task execution vectors? \n\nThe distinction between these two components should be further elaborated, with supporting evidence demonstrating why such a separation is meaningful and necessary.\n\n[1] AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders.", "methods_and_evaluation_criteria": "Evaluation is good to me, but results on more models and sparsity level are needed. The current experiments are primarily conducted on Gemma-1, but recent models such as LLaMA and Gemma-2 also feature pretrained SAEs with varying sparsity levels. How would different levels of sparsity in pretrained SAEs impact the conclusions drawn in this study? Additional experiments on multiple models with different sparsity settings would strengthen the validity of the findings and offer a broader perspective on the generalizability of the proposed approach.", "theoretical_claims": "N/A (This paper do not have much theoretical analysis.)", "experimental_designs_or_analyses": "1. observations that motivate the separation of task-detection and execution vector are needed\n2.  results on more models and sparsity level are needed.", "relation_to_broader_scientific_literature": "This paper  investigates the task vector of in-context learning (ICL) using sparse autoencoders (SAE), and design new TVC algorithms to better assist the analysis, but the main contribution need more elaboration.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "see summary", "other_comments_or_suggestions": "see summary", "questions_for_authors": "see summary", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper feels truly borderline. While the reviewers find merit in many of the ideas, they also identify some weaknesses that should be addressed.

As I have read the paper and considered the reviews, I feel that the strengths outweigh the weaknesses. I think this paper is thoughtful, well-written and contributes interesting insights; I believe that it represents a positive contribution to the field. The TVC algorithm is particularly interesting.

I strongly encourage the authors to incorporate the reviewer feedback. In particular, I caution the authors against overclaiming, given the limited nature of the tasks and models considered.
Decision: Accept (poster)