Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Authors: Daniil Laptev
Nikita Balagansky
Yaroslav Aksenov
Daniil Gavrilov
Venue: ICML 2025 poster (2025)
Keywords: Mechansitic Interpretability
Sparse Autoencoders
Steering
Large Language Models
Primary area: deep_learning->large_language_models

Abstract: We introduce a new approach to systematically map features discovered by sparse autoencoder across consecutive layers of large language models, extending earlier work that examined inter-layer feature links. By using a data-free cosine similarity technique, we trace how specific features persist, transform, or first appear at each stage. This method yields granular flow graphs of feature evolution, enabling fine-grained interpretability and mechanistic insights into model computations. Crucially, we demonstrate how these cross-layer feature maps facilitate direct steering of model behavior by amplifying or suppressing chosen features, achieving targeted thematic control in text generation. Together, our findings highlight the utility of a causal, cross-layer interpretability framework that not only clarifies how features develop through forward passes but also provides new means for transparent manipulation of large language models.

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Daniil Laptev Nikita Balagansky 12
Yaroslav Aksenov Daniil Gavrilov
Abstract
cosine similarity, that aligns SAE features across multiple modules (MLP, attention, and residual) at each layer, captur- ing how features originate, propagate, or vanish throughout the model in form of "flow graphs".
We introduce approach to systematically map features discovered by sparse autoencoder across consecutive layers of large language mod- els, extending earlier work that examined inter- layer feature links. By using a data-free cosine similarity technique, we trace how specific fea- tures persist, transform, or first appear at each stage. This method yields granular flow graphs of feature evolution, enabling fine-grain fine-grained inter- pretability and mechanistic insights into model computations. Crucially, we demonstrate how these cross-layer feature maps facilitate direct steering of model behavior by amplifying or sup- pressing chosen features, achieving targeted the- matic control in text generation. Together, findings highlight the utility of causal, cross- layer interpretability framework that not only clar- ifies how features develop through forward passes but also provides new means for transparent ma- nipulation of large language models.
1. Cross-Layer Feature Evolution. Using the pretrained SAEs that can isolate interpretable monosemantic di- rections, we utilize information obtained from cosine similarity between their decoder weights to track how these directions evolve or appear across layers. This reveals distinct patterns of feature birth and refinement not seen in single-layer analyses.
Mechanistic Properties of Flow Graph. By building flow graph, uncover evolutionary pathway, which is also an internal circuit-like computational pathway, where MLP and attention modules introduce new features to already existing ones or change them.
3. Multi-Layer Model Steering, We show that flow graphs can improve the quality of model steering by targeting multiple SAE features at once, and also offer a better understanding of the steering outcome. This framework provides the first demonstration of such multi-layer steering via SAE features.
1. Introduction
Large language models (LLMs) excel at generating coher- ent text but remain largely opaque in how they store and transform semantic information. Previous research has re- vealed that neural networks often encode concepts as linear directions within hidden representations (Mikolov et al., 2013), and that sparse autoencoders (SAEs) can disentangle these directions into monosemantic features in the case of LLMSs (Bricken et al., 2023; Cunningham et al., 2023). Yet, most methods analyze a single layer or focus solely on the residual stream, leaving the multi-layer nature of feature emergence and transformation underexplored (Balagansky al., 2024; Balcells et al., 2024).
Our method helps to discover the lifespan of SAE features, understand their evolution across layers, and shed light on how they might form computational circuits, thereby abling more precise control over model behavior.
2. Preliminaries
2.1. Linear representation hypothesis
understand how models encode and process the infor- mation they learn, one can examine the geometric struc- ture of their hidden representations and weights. Research has shown (Mikolov et al., 2013; Marks & Tegmark, 2024; Gurnee & Tegmark, 2024; Engels et al., 2025) that linear directions carry semantically meaningful information and may be used by models to represent learned concepts. Ob- servations of this kind led to the development of the linear representation hypothesis, which can be stated as follows.
In this paper, we propose a data-free approach, based
TT-Tech "Moscow Institute Physies and Technology. Corre- spondence to: Nikita Balagansky <n.n. balanaalggnskki @ tbank.ru>..
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Hidden states h Rd can be represented as sparse linear combinations of features f E IR that lie in linear subspaces I' C Rd The impact of each feature is encoded by its mag- nitude || The total number of these linear subspaces with unique semantics greatly exceeds d, forcing the model to build an overcomplete basis in the feature space embedded in Id. During a forward pass, the model typically uses only small fraction of them. These subspaces are usually one- dimensional lines, but more complex structures appear (Engels et al., 2025).
Transcoders (Jermyn al., 2024) are very similar to SAES, but they reconstruct different target. Typically, they trained as interpretable approximations of MLPs:
h post TC hpost ||?
Lec
where hpc is the pre-MLP hidden state, hpsst is the post- MLP hidden state, and hpos is the transcoder's prediction.
2.3. Features On Different Layers
2.2. SAE and Transcoders
Interconnections among SAE features trained on different layers of the same model have been reported and studied (Balagansky et al., 2024; Balcells al., 2024; Ghilardi et al., 2024). Features in earlier layers tend be low-level, often indicating word characteristics (e.g., words starting with certain letters), while features in later layers are typically more high-level and guide model behavior.
To retrieve such linear directions, Sparse Autoencoders (SAEs) (Bricken et al., 2023; Cunningham et al., 2023) were introduced. They decompose the model's hidden state into a sparse weighted sum of interpretable features.
Let (PP) (F(P) | E {1..... D)), where D > d is the dictionary size, be collection of one-dimensse iieediiensional features learned by an SAE at position P in the model (e.g., after the MLP block). Then the SAE can be represented as
Sparse autoencoders are typically trained at three points in each layer: the output of the attention mechanism, the output of the MLP, and the residual stream. The latter is the main conduit of information within a transformer, MLP and attention modules read from it, process the data, and write their outputs back into it. According to Balagansky et al. (2024), most features in the residual stream remain relatively unchanged across layers. To identify similar fea- tures between different layers, one can define a permutation matrix p(A+B) that maps feature indices from layer A to layer B, both having the same number of features [F]:
z 0 W enc h + bene);
h = W + baec:
where Wene; bene, W dec, Ddec are SAE parameters, (-)) is a nonlinear activation function, h E I2 is a model's hidden state, z E IR FFl is the feature activation, and h is the SAE's reconstruction of the hidden state.
Sparse autoencoders are usually trained to reconstruct model hidden states while enforcing sparse feature activations:
d
P(A-B) arg min PEP
ww!a) doe,,;
(A(A) ((A+B) dor,.
= Ty (h, h) + [xeg(z))
Typically, Liee | 1 while [zeg(2) is an la proxy.
where W( E [dx[FF is a parameter of the SAE trained on the residual stream after layer A, and Pizj is the set of permutation matrices of size |F x [F]-
(c)
The choice of activation function (-)) is crucial for achiev- ing the desired representation properties. JumpReLU (Ra- jamanoharan et al., 2024) introduces a threshold parameter E TR. IFl that controls how large each pre-activation must be for the feature become active:
Dunefsky et al. (2024) finds a computational graph through the MLP layers by training transcoders:
(2) (zz 0).
z ((bacc)::
W (A)T doe
enc
where H is the Heaviside function.
Here, dec W(A) ((B) E R x|=i serves as a transition oper-
Top-K K (Makhzani & Frey, 2014; Gao et al., 2025), allows one to control the desired sparsity level by fixing
ator between the feature spaces of layers and B, revealing which features in B are ancestors for the ith feature in A.
ohh) Lopx (Wh + b).
Matrices P(A-B) and W (B()T similar. We explore this further in Appendix F.
ene are in some sense
Instead of taking the top-k per sample, BatchTopK selects the top l x b activations over all samples in the batch (Buss- mann et al., 2024).
3. Method
'There is distinction between the weak and strong LRH. The strong version posits that there are only linear representations, while the weak version says that representations are mostly linear and one-dimensi.nal.
3.1. Motivation
Although SAES provide human intterrrattt features, they do not explain how these features interact or how the
2

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
W
W
Rodeitall SAB
Roektoal SAB
Was
de
maat W
Suun (M Wil
Figure 1. Schematic illustration of inner-layer matching. We select feature with index the SAE trained at the layer output. Its embedding f, which the ith column of this SAE's decoder weight, compared to every column of other SAEs on the same layer (after the MLP and attention blocks, well as with SAE the residual stream before some layer). These comparisons indicate the feature's source. See Section 3.3 for more details.
model's computation is carried out. Understanding this crucial for more precise model manipulation.
at position A. This vector is the ith column W+(A) let w E gdx|F1 be the decoder weights an SAE trained position B. We find the matched feature index as
dec Also
A key principle is that such understanding can be obtained by linking features different levels of model (Balagansky et al., 2024; Dunefsky et al., 2024). If we want to find features shared by two SAEs trained at positions and B, we need to discover mapping
j = arg max ((BB) doe..
Then we say that (((A) corresponds to F. We assume that both f and the columns of w have unit norm.
TA-B F(A) + FBB)
dee
This drives methods (Balagansky al., 2024; Balcells al., 2024) for finding these shared features and architectures (Lindsey et al., 2024) that ensure persistent collections of features by design.
More generally, we define
(A-BB) To>oottpp (W())
where lles an indicator function and topx zeroes out values below the kth order statistic. When k 1, this many-to-one matching extends the one-to-one approach in Balagansky et al. (2024). Although top-k handles many- to-many cases, we focus on many-to-one as a substantial extension of previous work.
By grouping similar features, we can find those that remain the same across different positions (by repeatedly applying mapping rules) uncover those unique specific points in the model. This helps us understand how semantic structure and computational modes evolve, while SAE features serve as an interpretable proxy.
This technique assumes SAEs are trained on hidden states whose structure is aligned. For instance, Gemma Scope (Lieberum al., 2024) attention SAEs trained before nonlinear transformation at dimension 2048, whereas MI. and residual SAES are trained on dimension 2304, so our method cannot be applied there. As shown in Section 5.1, the data distribution can also affect these results.
3.2. Feature matching
Several methods exist for matching features between lay- ers and modules. One approach uses correlations between activations (Wang et al., 2025; Balcells er al., 2024), but requires considerable data to compute activation statis- ties. Another is data-free approach based on SAE weights (Dunefsky et al., 2024; Balagansky al., 2024). We found that cosine similarity between decoder weights is valuable similarity metric, and focus this approach.
3.3. Tracking the evolution of feature
There are four main computational points in a standard transformer layer: the layer output Ry. the MLP output M, the attention output and the previous layer output R,-1
Let f E Rdd be the embedding of some feature F trained
3

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Scientific Concepts and Entities Graph
eilerds object member access tokens
Alteniion
allends to reletional tokens from conesponding
attemids the token "paint" flom
soertific toeminolgy relared to stuty resutts causees
terms particle physiss
Residual Stream
terms and reflerences related to particle physics and standard model parameteers inchnical tominologics rolated to particle physics measurements
phrases or structures indicating action otter
MLP
references gauge symmetiies in theoretical ...
Layer
Figure 2.
An illustration of the resulting flow graph, which we also use in the deactivation experiment (section 5.2). As a starting point, we select the feature on the 24th-layer residual with index 14548. For detailed explanation of this graph, see Appendix E.
(the layer input). MLP and attention modules read from Ry-1 and their outputs produce RL.
Long-range feature flows. As we progress through the model, semantics undergo substantial changes, making di- rect long-range matching challenging. We address this by performing short-range matching consecutive layers and composing the resulting transformations. For given fea- ture, we construct flow graph from the initial layer to the final layer. This flow graph traces a path that reveals how the feature's semantic properties evolve. example of such graph is presented in Figure 2.
We pick feature from the SAE trained on RL with embed- ding f E Rdd Let w dec E ædx|F| for P {M, A, R
R be the corresponding decoder weights. We com- pute the similarity between the target feature and P as the maximum cosine similarity over the columns of w(!), dec
max (f w(() doc..))
Currently, individual SAE features or their groups (Engels et al., 2025) are treated units for study. However, believe that these flow graphs may also become a compelling area for future research.
as illustrated in Figure 1. From these scores, we can infer how the feature relates to the previous layer or modules:
A) High (R) and low s(M),s((); isted in Ry-1 and was translated to Rg.
s(A): The feature likely ex-
3.4. Identification of linear feature circuits
Model behavior can be decomposed into computational subnetworks, called circuits, which perform task-specific operations (Elhage et al., 2021; Marks et al., 2025). Our method helps identify potential circuits where MI. and mostly linear way. High values of ((M) or ((A) are strong indicators of these circuits. We validate this in experiments, focusing how feature's meaning evolves. Examples appear in Appendix E.
B) High R and high (MM A(A): The feature likely processed by the MLP or attention.
C) Low (l) but high (M) or (AA); The feature may be newborn, created by the MLP or attention.
attention modules add or remove features
D) Low ( and low (MM) ((A); The feature cannot be easily explained by maximum cosine similarity alone.
3.5. Model steering
Thresholds for "high" and "low" are specific for each layer.
Flow graphs can also help steer the model toward desired behaviors by identifying feature sets we want to manipulate. By carefully selecting them, one can preserve both align- ment and core model capabilitics, and our method facilitates
We use a maacward-matching approach because it naturally answers, "Where did this feature come from?" Forward- matching answers, "Where does this feature go?" but less helpful for finding novel or transformed features.

---

Analyze Feature Flow to Enhance Interpretation and Stecring in Language Models
discovery of such feature groups. By examining flow graphs built from those features, one can better understand and predict the behavior of the model after steering. Section 5.3 and Appendix B illustrate this process.
0.8
0.6
0.4
02
4. Experimental Setup
4.1. Models and SAEs
Tromineffer From RES
From
on
From AFT
We conduct our main experiments with the Gemma 2BB model (Gemma Team, 2024) and the Gemma Scope SAE pack (Lieberum et al., 2024) with a JumpReLU activation function and dictionary size of 16k features. We also test our approach on LLama Scope (He et al., 2024) (see Ap- pendix D), which was trained with Topk activation function and was converted to a JumpReLU after training. addi- tion, we train our own JumpReL.U SAEs for the attention output (before it is added back the residual stream) on every layer of the Gemma model, following the Gemma Scope training pipeline.
0.6
64
02
0A
0.8
0.4
Figure 3. Example of cosine similarity vs. simultancous activa- tion with predecessor (350 features were sampled per layer). "From MLP" and "From RES" groups are notably different: high (MM) and low a (8) suggest simultaneous activation with an MLP- module match. Cosine similarity serves a gond proxy for shared semantic and mechanistic properties.
We obtain interpretations from Neuronpedia", which also serves as an additional evaluation tool. Interpretations for newly trained attention features were not available, and none were provided for I.Lama Scope.
h + h h- 1)v, where a is the predecessor's activation strength, its embedding, and r a rescaling coefficient (r for deactivation). We expect this remove the feature from the hidden state, preventing further propagation.
4.2. Overview of experiments
We design our experiments to analyze how residual features emerge, propagate, and can be manipulated across model layers. Specifically, we aim to: (i) determine how features originate in different model components, (ii) assess whether deactivating a predecessor feature truly deactivates its de- scendant, and (iii) use these insights to steer the model's generation toward or away from specific topics.
We evaluate four matching strategies: (1) random sam- pling from top-5 cosine-similarity matches, (2) permutation- based (Balagansky et al., 2024), (3) top, cosine similarity (our method), and (4) tops cosine similarity where all five matched features must be inactive to treat this predeces- sor as inactive. Effectiveness is quantified via successful deactivation rate and activation change (higher when new strength approaches 0).
Below is concise summary of each experiment. See Ap- pendices A and B for detailed setup.
Identification of feature predecessors. We first verify that cosine similarity relations used for single-layer analysis align with actual activation correlations. A target feature in the residual stream RG, is matched with the previous residual RI-11 the MLP output M, or the attention output A features. none are active, we label it "From nowhere." " By applying this process on four diverse datasets, we confirm the above- stated relation, and also analyze how these groups are distributed across layers.
Model Steering. We test whether multi-layer feature ac- divation/deacciivation can control theme generation. For target topic, we intervene relevant features across layers and assess text quality.
As a baseline we use initial features from which we build flow graphs. We compare single-layer (layer only) and cumulative (layers to l) interventions, applying the same rescaling for deactivation. For activation, we add scaled embeddings. Multi-layer strategies include linear and ex- ponential decay of steering coefficients with respect to the layer index, and constant scale for all layers (Appendix B).
Feature Deactivation. We measure causal relationships by intervening on hidden states: if deactivating a predeces- sor also deactivates target feature, we infer a causal link.
Given hidden states the predecessor's 's position (previous residual, MLP, or attention output), we apply transformation
We measure Behavioral (topic presence) and Coherence (language quality) scores, and use their product as final metric (for deactivation (1 Behavioral) x Coherence).
Pttpps: ///ww//wwww neuronpedia wwt.rroopndaa.oooooooooooo gemma- 2-2b
5

---

Analyze Feature Flow to Enhance Interpretation and Stecring in Language Models
RES medd
MUP nonis
AFT modie
Toeilee
PorviEs
1.02
0.3
0.03
0.83
0.75
AD A0 AA 40 18
Figure 4. Percentage of statistically significant differences between groups for each modulc's similarity scores. AO means module P is active in only one group, AB means active in both, and IB means inactive in both. For MLP, two groups differ in (() only 87% of the time when MLP is active in both groups.
Figure 5. Percentages of each group at each layer of Gemma 2 2BB, illustrating how feature formation proceeds in the model.
5. Results
5.1. Identification of feature predecessors
In this experiment, we validate the single-layer analysis patterns from Section 3.3 by checking when target residual features and their predecessors activate simultancously. For each activated residual feature, we assign it to group based on which predecessors are also active. For example, if both the previous residual and MI.P predecessors are active, the feature is categorized as "From RES & MLP." We then examine the distributions of scores within these groups.
P
Figure 6. Deactivation methods compared. Group labels show which active predecessors were deactivated. The random approach underperforms, suggesting that choosing the top, feature is al- ready meaningful for causal analysis.
Figure 3 reveals visually distinct score distributions across different groups. We quantify these differences with a Mann- Whitney U test on every pair of groups, for each dataset and layer, and then compute the fraction of tests with < 0.001.
We observe that two groups may differ with respect to (P)) if module P is active only in one group (and indis- tinguishable if P is active inactive in both groups). For example, "From MLP" and "From MLP & ATT" differ by g(R), s(M) s(A) in 67%, 72%, and 100% of tests, respec- tively. Figure 4 shows the total percentage of passed tests.
language, and TinyStoories has the most natural and simple language structure. The rarity of groups with activated attention could stem from our SAE training rather than an inherent property of Gemma. However, in the LLama Scope case (Figure 19(b)), we observe slightly similar pattern, which indicates that this is indeed the property they share.
Figure shows how these groups spread across layers, sug- gesting conceptual formation in earlier layers. From layers 0-5, "From nowhere" and "From RES" may reflect high- entropy, early-stage process that stabilizes by about layer 5. After layer 18, where we see bump for "From MLP", fewer new features emerge, and most features propagate from preceding layers.
We have observed that group identification performance is on par with Pearson correlation based matching methodol- ogy. The latter reduced the "From nowhere" group pres- ence, but did not consistently outperform our method and performed worse out-of-distributin Python code. See more details in Appendix C.4.
There also a three-part partition in the distribution of groups: approximately [0, 5] where uncertainty dominates, [6, 15] with somewhat stable dynamics, and [16, 25] where "From RES" group presence starts to rise and "From MLP" group diminishes after layer 18, implying that fewer new features appear in later layers.
5.2. Deactivation of features
We compare the top, approach (choosing the most similar predecessor by cosine similarity) with randomly picking one of the tops candidates. Figure 6 shows that the random method sharply reduces deactivation success, confirming that top, is informative for causal analysis.
We observe differences between datasets in the latter layers.
The Python code dataset contains the least amount of natural
For MLP and attention predecessors, top, and top; per-
6

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Change
Selrtee and crange
Perceerngy of teacthanss
Mean AC
Success rate
Top-1 Cosine
0.75
65%
Top-1 Pearson
0.74
65%
Exhaustive Search
0.83
73%
33304
-33
23-
PED
1353
Table I. Comparison of deactivation methods. The exhaustive search evaluates all activated predecessor features individually and reports maximum performance. The similar results between correlation-haser and data-free methad validate our approach.
Figure 7. Impact of different r values on deactivation success, with rescaling of all available predecessors. When r < 1, the activation change grows nonlincarly, indicating alternative causal pathways still convey information. Relative loss change measured as (Lnew / / Les is proxy for forward pass impact.
Per-layer
0.4
0.1
Conerente
0.6
0.4
0.2
RESA MEPLATT
Figure 8. Mean activation changes when deactivating one prede- cessor at a time. Deactivation of some predecessor causes less impact if this predecessor is activated alone, which leads to the conclusion that combined groups exhibit circuit-like behavior.
Figure 9. Deactivating the "Seientific concepts and cntities" theme. The dashed black line shows the default generation score. Red points mark best layer for each the single-layer method. Larger boosts performance but shifts the optimal layer earlier.
form similarly. Differences arise mainly when a residual predecessor combines with another module, indicating that we might miss other types of causal relations.
features identified by either cosine or Pearson matching as "From RES", "From MLP",, or "From ATT" to ensure fair comparison and computational feasibility. Testing 1,894 features across two layers (each deactivated via all three methods) yields the results in Table 1, showing comparable performance between cosine and Pearson methods, addi- tionally validating our data-free approach.
Finally, vary the rescaling coefficient to how it affects deactivation results (Figure 7). Different groups react differently to rescaling. Positive rescaling (boosting active features) matters most when residual features mix with MLP or attention. Negative rescaling most strongly affects "From RES." Reducing "From RES & MLP" "From RES & MIP & ATT" increases the loss change more than reducing "From RES" alone, highlighting MLP's critical role in these circuit-like like interactions.
5.3. Model steering
To evaluate interventions based flow graphs, we them to suppress activate topics in generation. Figure 10 demonstrates that our method identifies more effective steer- ing features across layers compared to single-feature inter- ventions on the initial feature set. The cumulative approach additionally provides two key advantages: (1) reduced sen- sitivity hyperparameter choices, and (2) improved perfor- mance with smaller hidden state perturbations.
Figure 8 further shows that deactivating a single predecessor causes a greater activation strength drop if it is a group with a single predecessor, which may indicate circuit-like behavior in combined groups.
To compare method with optimal performance, we test three approaches: (1) top-1 cosine similarity matching, (2) top-1 Pearson correlation matching, and (3) an exhaustive search for maximum achievable performance. The search procedure deactivates each active predecessor feature indi- vidually, with activation change computed only for target
Figure 9 analyzes the impact of rescaling coefficient on deactivation effectiveness. We observe that larger values shift the optimal intervention point toward earlier layers, while smaller r values distribute the intervention effect more evenly across the network depth.

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
residual stream with a module tend to hold processed fea- tures. The decline of "From MLP" and the rise of "From RES" groups shown in Figure 5 may indicate that later layers form fewer new features than intermediate layers.
0.70
0.63
0.60
0.55
One layer Oummlstive
0.50
32 .44
10
15
20
25
Layer
Deactivation of features. We (i) confirm that top; sim- ilarity provides valuable information about causal depen- dencies, and (ii) conclude that groups respond differently to the deactivation of certain predecessors, indicating that they have distinct mechanistic properties and may exhibit circuit-like behavior. The fact that residual predecessors are the most influential could explained by the nature of the residual stream as the main communication channel, removing the feature module will not prevent from further propagation if it already exists in the residual stream.
Figure 10. Comparison of best dcactivation scores. The green line indicates deactivation using only the initial feature set. Interven- tions layers detected by our method (orange, blue) perform better across different values, suggesting additional discovered features reduce hyperparameter sensitivity.
baselire
stngee
con5-19
exponertat
intar
Model steering. we want to reduce a particular feature at inference, we typically adjust its magnitude. However, achieving significant reduction may require large adjust- ment scales, which can alter the distribution of hidden states. Because we know which features contribute to the appear- ance of the feature we want to reduce, we can also adjust those. From this perspective, it is possible to make multi- ple smaller adjustments rather than one large one, avoiding dramatic changes to the overall distribution of hidden states.
0.4
Anger and fucrittion
Hertal heshhh ingues
Weoding and mariagg
Religion And fodt
Figure 11. Activation of specific topics. We compare single-layer stecring and cumulative approaches with three rescaling strategics (Appendix B). Activating multiple similar features amplifies topic's presence but may degrade overall text coherence.
Flow graphs may help to understand the effect of steer- ing and identify related features, but the downstream result depends on the properties of the specific graph. Overall, we conclude that they allow to find impactful features for intervention. We hypothesize that removing topic-related information early allows later layers to recover general lin- guistic information, aligning with the ability of LLMs "self-repair" after "damaging" the information processing flow by pruning or, in case, intervention into the struc- ture of hidden states (McGrath et al., 2023).
Figure 10 shows that cumulative intervention outperforms the single-layer approach in low r regime, suggesting that small interventions distributed over multiple layers may be more effective for controllable generation.
For activation tasks, we boost topic presence by activating multiple similar directions. Figure 11 shows that cumulative methods typically strengthen the topic signal but can reduce text quality. In some cases, the effect clear: steering feature tied to "Religion and God" can shift outputs toward biblical text, and if we examine the flow graph for that feature, we see that earlier layers are indeed linked to it.
Overall, our method provides straightforward way to iden- tify and interpret the computational graph of the model without relying on additional data, achieving performance similar to Pearson correlation matching; the resulting graph can then be used for precise control over the model's in- ternal structure. To the best of our knowledge, we are the first to use SAE features from different layers to control I.IM generation. We believe that this work opens a new perspective for zero-shot steering.
6. Discussion
Identification of feature predecessors. Our results indi- cate that (i) similarity of linear directions is indeed a good proxy for activation correlation, and (ii) the structure of these groups differs across layers, possibly reflecting the properties of information processing within the model.
7. Related Work
Multiple works have investigated feature circuits in lan- guage models. Conmy et al. (2023) proposed pruning con- nections between modules that do not affect the output. Ge et al. (2024) suggested using gradients to decide whether to prune connections between modules; they also demonstrated that their method can be used to find circuits on the feature
We suspect that "From MLP," "From ATT," and "From MLP & ATT" primarily contain newborn features introduced at their respective layers, whereas groups that combine the
8

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
level with skip SAE, which is equivalent transcoders. Dunefsky al. (2024) showed that circuits can be found without a backward pass, relying solely on activations and transcoders' weights. Balagansky et al. (2024); Balcells et al. (2024) studied feature dynamics in the residual stream during the forward pass; however, these works focus exclu- sively on residual stream features and do not investigate the properties of the resulting computational graph or its appli- cation to steering. Additionally, SAE features steering vectors were explored in Chainev et al. (2024), but their ap- proach is data-dependent and does not involve multi-layer steering procedure. In contrast, our work advances these findings by introducing a straightforward and interpretable data-free method for multi-layer steering, which also en- ables the tracking of concept evolution across layers and the identification of computational circuits through targeting the weights of pretrained SAEs.
velop robust norms, policies, and technical safeguards that promote beneficial applications while mitigating the risks of misuse.
References
Balagansky, N., Maksimov, I., and Gavrilov, D. Mecha- nistic permutability: Match features across layers. In The Thirteenth International Conference on Learning Representations, 2024. URL
Balcells, D., Lerner, B., Oesterle, M., Ucar, E., and Heimersheim, S. Evolution of sae features across lay- ers in Ilms, 2024. URL https: http//aaxvvvi..a..aa...a.... abs/ 2410 08869.
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravee, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield Doddss, Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards monosemanticity: Decomposing language models with dictionary learning, Transformer Circuits Thread, 2023, https:/transfoommr- html.
8. Conclusion
In our work, we propose using SAEs trained on different modules and layers of the base model to find computa- tional graph consisting of SAE features. Through our ex- periments, we validate that these graphs can describe most of the feature dynamics. Finally, we show that such graphs can be used for steering model behavior, thereby improving steering of LLMs with SAFS.
Bussmann, B., Leask, P., and Nanda, N. Batchtopk sparse autoencoders. arXiv preprint arXiv: 2412.06410, 2024.
Advancements in model steering suggest focusing on more sophisticated steering approaches. For example, while we can reconstruct feature predecessors from multiple blocks in previous layers, it is evident that features are somewhat tangled across layers (when reducing the magnitude of predecessor feature, all subsequent computations change). Thus, it may be helpful to concentrate on disentangling these connections across different layers. Other directions for bet- ter steering could also exist, thus opening new possibilities on further enhancing LLM controllable generation.
Chalnev, S., Siu, M., and Conmy, A. Improving steering vectors by targeting sparse autoencoder features, 2024. URL https::t //tp:v/aaiivv////a///// org/ h/as///b//// /2411 02193.........
Conmy, A., Mavor-Parker, A. N., Lynch, A., Heimersheim, S., and Garriga-Alonso, A. Towards automated circuit dis- covery for mechanistic interpretability. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Cunningham, H., Ewart, A., Riggs, L.. Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models, 2023. URL https: a arxiv. urxiv.org/et /ass///////// 22099. 08600.
Impact statement
Our work offers a method to systematically identify and ma- nipulate latent features in large language models, thereby ad- vancing the field of controllable generation. This improved controllability has positive implications for alignment, in- terpretability, and safe deployment of Al systems, as it can allow developers to steer models away from harmful or bi- ased outputs. At the same time, similar techniques could be repurposed for unsafe or malicious behavior by those aiming to bypass safeguards or exploit hidden model path- ways. These dual-use concerns highlight the importance of continued research and open discussion on controllable generation, rather than cessation of study. By deepening collective understanding, we better equipped to de-
Dunefsky, J., Chlenski, P., and Nanda, N. Transcoders find interpretable Ilm feature circuits. arXiv preprint arXiv: 2406. 19444, 2024.
Eldan, R. and Li, Y. Tinystories: How small can language models be and still speak coherent english?, 2023. URL https: httpaaiivvv..oo .rgg abs / 22305. 07759.
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield- Dodds, Z.,, Hernandez, D., Jones, A., Kernion, J.,

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J. Kaplan, J., McCandlish, S., and Olah, C. A mathematical framework for transformer circuits, 2021. URL pub htt///raawwwrrr/oo inwe...nde/e.////// html.
Jermyn, A., Batson, J., and Olah, C. Random open prob- lems. Transformer Circuits Thread, 2024, URL https: cewennssromeeerrruutee pub//224/ jan-update/ Suuuppattiindex.
Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L., Sonnerat, N., Varma, V., Kram's ar, J., Dragan, A., Shah, R., and Nanda, N. Gemma scope: Open sparse autoen- coders everywhere all at once gemma 2. BlackboxNLP Workshop Analyzing and Interpreting Neural Networks for NLP, 2024. doi: 00448550/r2xv.24088.00 00740447
Engels, J., Michaud, E. J., Liao, I., Gurnee, W., and Tegmark, M. Not all language model features are one-dimennt mmediiensiionally linear. In The Thirteenth In- ternational Conference on Learning Representations, n forum?
2025. URL htttp:://opereeeeeeeeee id=d63a (add43444MM4. hMM4hb.
Lindsey, J., Templeton, A., Marcus, J., Conerly, T. Batson, J.. and Olah, C. Sparse crosscoders for cross-layer features and model diffing, 2024. URL 2024 / htta//cosscooeers//// wwwpeerrriiddie....ae.
Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations, 2025. URL btttps://openreeee httpi//wwwwe//oourtnteee id=tcs2t 9ZNKD.
Makhzani, A. and Frey, B. k-sparse autoencoders, 2014. URL wwtpts:/ttttttteeeee www.riiww......o.ooooooooooooooe /atrr//aa//31...55500
Ge, X., Zhu, F., Shu, W., Wang. J., He, Z., and Qiu, X. Auto- matically identifying local and global circuits with linear computation graphs. arXiv preprint arXiv: 2405. 13868, 2024.
Marks, S. and Tegmark, M.
The geometry of truth: Emer- gent linear structure in large language model represen- tations of true/false datasets, 2024. URL https:// // arxiv. axxiv.org/att owtoorgg//aooo ht/////s/230 /2310 06824.
Gemma Team. Gemma 2: Improving open language models at a practical size, 2024. http:: /tappp:///arxxviy arxiv.org/ abs/ abs/2408 00118.
Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., and Mueller, A. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /twpn//vee.....eee/e/oeeee net
Ghilardi, D., Belotti, F., Molinari, M., and Lim, J. Ac- celerating sparse autoencoder training via layer-wise transfer learning in large language models. In Be- linkov, Y., Kim, N., Jumelet, J., Mohebbi, H., Mueller, A., and Chen, H. (eds.), Proceedings of the 7th Black- boxNLP Workshop: Analyzing and Interpreting Neu- ral Networks for NLP. pp. 530-550, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10. 10.86555w110022bbcckbxxl/ 1.
McGrath, T., Rahtz, M., Kramar, J., Mikulik, V., and Legg, S. The hydra effect: Emergent self-repair in language model computations, 2023. URL https: //trx//aaavvv/// org/ ablaaba/222 /200..2300222 /2077.1777.. 15771.
Mikolov, T., Yih, W.-t,, and Zweig, G. Linguistic regular- ities in continuous space word representations. In Pro- ceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, pp. 746-751, At- lanta, Georgia, June 2013. Association for Computational Linguistics. URL https: haa//aaaaaaatalccacaeaaaa org/ N 3
32. URL https:/ //accllaaaa oow.lannnoooooy.orrrr/ blackboxnip-1.33/
bettps://aclanthtn
Gurnee, W. and Tegmark, M. Language models represent space and time, 2024. URL abbs/3100 02207.
Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troit- skii, D., and Bertsimas, D. Finding neurons in haystack: Case studies with sparse probing. Transac- tions on Machine Learning Research, 2023. ISSN 2835- 8856. URL h http://pppeeeeeeeeeeeeeeeee forum? id=JYs R9MM...
Penedo, G., Kydlícek, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track, 2024. URL https: /ttp:::////ppeeeeeeeeee net/ forum? i id=n6sCkn2gaG.
He, Z., Shu, W., Ge, X., Chen, L., Wang, J., Zhou, Y., Liu, F. Guo, Q. Huang, X., Wu, Z., Jiang, Y.-G., and Qiu, X. Llama scope: Extracting millions of features from 1lama-3,1-8b with sparse autoencoders. arXiv preprint arXiv: 2410.20526, 2024.
Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V., Kramár, J., and Nanda, N. Jumping ahead:
10

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv: 2407. /4435, 2024.
Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and Henighan, T. Scaling monosemanticity: Ex- tracting interpretable features from claude sonnet. Transformer Circuits Thread, 2024. URL https: cww.aaaeooorrerrcctatttt.ee..p/s ccaling-nooneeeattiiitt//..ee... html.
Wang, J., Ge, X., Shu, W., Tang, Q. Zhou, Y., He, Z., and Qiu, X. Towards universality: Studying mechanis- tic similarity across language model architectures. The Thirteenth International Conference on Learning Representations, 2025. URL https:tttt/... // /pwpreeveeeeeeeeeeeeeeeeemmee.e. net/fose Eorumiid----oo 2it.iidd22211o00-
Zhang, Y., Luo, Y., Yuan, Y., and Yao, A. C.-C Au- tonomous data selection with language models for math- ematical texts, 2024. URL https: /aaxxaxx.vvrrg..e org/ Noo/2440.07622.

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
A. Detailed Experimental Setup
A.1. Identification feature predecessors
This experiment aims to validate our proposed approach for determining the origin of a feature. Specifically, we verity whether the cosine similarity relations described for single-layer analysis align with the correlation between the features' activations. For target feature from RL, we consider it to originate from Ry-1 if the matched feature on Rz-1 1 is active while the matched features M and A are inactive. There seven possible combinations of activated predecessors; if none of these is active, the feature is assigned to eighth group. "From nowhere."
We use four datasets for this analysis: Fen (Penedo et al.. 2024) (general-pun purpose texts), TinyStories (Eldan & Li, 2023) (short synthetic stories), TutoMatt'Text (Zhang et al., 2024) (math-related texts), and PythonGithubCooee" (pure Python code). From each dataset, we select 250 random samples; for each sample, we pick 5 random tokens (excluding the BOS token). We then iterate over every activated feature on every layer and determine its group (i.e., which predecessor combination leads to that feature's activation).
A.2. Deactivation of features
To further validate the proposed method, we measure the causal relationship between a feature and its predecessor by intervening directly in the model's hidden state. Specifically, we deactivate the predecessor by removing its corresponding decoder column from the relevant hidden state (i.e., at the MLP output, attention output, previous layer output). We expect that deactivating the matched predecessor feature will also deactivate the target feature (at the end of the layer).
Feature rescaling. Consider hidden state h Rd for a specific token f. Suppose we want to modify the strength features within this hidden state. Let V E Rdx be the embeddings of these / features, and let a, R/ be their activation strengths for token t. We define rescaling as:
h + h, + ( () VT),
where r is the rescaling coefficient. This method also allows to rescale feature to desired strength for steering. We refer to rescaling as positive when r 2 1, and negative otherwise.*
In the context of SAEs, we approximate hidden states with a linear combination of feature decoder columns (plus a bias term that does not depend on activation strength, and is therefore omitted). Setting r removes the selected features from the existing linear combination, which is (up SAE reconstruction error) the same setting those features' activations zero.
Experimental protocol. In this experiment, we apply the above transformation only to the specific token where we detect residual feature. We select 35 texts from Fibe choose random tokens per text, and focus on layers 6, 12, and 18. For each layer-token pair, we randomly sample up to 25 features and deactivate them if they do not belong to the "From nowhere" group
To assess the effectiveness of deactivation, we compare four matching methods:
permutation: Deactivate the predecessor feature identified by permutation,
top;: Deactivate the most similar predecessor feature (based on cosine similarity of decoder embeddings),
top; (k = 5): Deactivate the five most similar predecessor features,
random: Randomly choose one from the top five most similar features.
The top; method our main focus. For each method, we first identify the group of the target feature and then perform the deactivation. For the top; method, we consider the predecessor active at least one of the selected features is active.
We evaluate two main metrics:
code *This essentially equivalent to method discussed Templeton et (2024), see section "Methodological details."
pattts://hugeee agfacce...... co/ /ataaattssesstee www.onkkkkrbbappytttooee
12

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Feature index
Interpretation from Neuronpedia
3/res/9811
terms related gravity and its influences
18/res/14053
terms related theoretical frameworks and conceptual models
18/res/1336
references to Dark Matter and astronomical phenomena
20/res/4506
terms related to physical laws and scientific principles
21/res/13226
references to quantum concepts and theories
22/res/9002
terms related to models and their specifications,
22/res/15105
terms related to force and energy dynamics
23/res/4086
terms related to forces and dynamics in physical systems
2A/res/7017
terms related electromagnetic interactions and properties
24/res/14540
terms and references related to particle physics and standard model parameters
Table 2. Features initially chosen for deactivation of "Scientific concepts and entities" theme.
Successful deactivations: The number of times feature was deactivated, divided by the number of times it had an active predecessor.
Activation change:
/z("d) for target feature i. This metric equals when the feature is fully deactivated, and can be interpreted as measure of causal dependency between predecessor and target features.
Defined as 1 - (z!T0// (zf"s/22"0
B. Steering details
To further test whether our feature- matching approach enables effective model steering, we design a procedure to either suppress promote particular themes in the generated text. We begin by identifying small set of features for each theme, guided by Neuronpedia entries. We then build flow graphs (from layer 0 to layer 25) trace how theme-related features evolve across the network. If the semantic meaning of feature remains consistent and relevant, we add it to our target collection; otherwise, we continue searching until we have satisfactory set of features.
We compare a single-layer steering strategy (affecting only the features one layer) to a cumulative strategy (affecting features from the current and all previous layers) to see whether distributing interventions across the network can produce more robust thematic shifts. We also measure any corresponding changes in text coherence and the overall success in promoting or suppressing the target themes.
B.1. Deactivation of theme
In the deactivation task, we target a theme we call "Scientific concepts and entities," which is related black holes, quantum mechanics, and other advanced physics topies. The full list of related features is presented Table
Generation and evaluation setup. We use the prompt, "I think that the biggest problem of contemporary theoretical physics ", and generate text with a maximum length 96 tokens, lop,- = 0.7, and temperature whether each theme is present in the generated text, we query gpt4o-mini language model for score from to 5 on each theme, following approach similar to Chalnev al. (2024). We use the following system prompt for validation:
1.27. To determine
Y"nYou will receive a text that was generated by some language model.
You need to evaluate this text based on the following criteria:
1. Coherence score: text is gramatically correct, language is clear and coherent.
2. Behavioral score: text is directly related to some predefined subject.
The subjects are:
a) ** Personalities, their influence and achievements **. Text may contain references to individual or group achievements or lack of results, discussion about Nobel prizes etc.
13

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Theme
Feature index
Interpretation
Anger and frustration
12/res/4111 111
expressions of anger and frustration
Mental health issues
12/res/1622)
ref. to mental health issues and their connections other health conditions terms related weddings and marriage ceremonies
Wedding and marriage
12/res/4230
Religion and God
12/res/5483
spiritual themes related to faith and divine authority
Table Initial choice feature for activation task.
b) Scientific concepts and entities **. Text relates some specific scientific concepts, such particles, black holes, particular theories, Standard Model, quantum wavefunctions, gravitation etc.
c) ** Research methodology and experimentation **. Text relates to methodology and philosophy of science, scientific research techniques and procedures, especially method of experiment, empirical basis of theories etc.
Rate the behavioral score in this order, i.e. [a, cl.
Give an integer score from 0 to 5 for each criteria.
Do not judge text based on ethical considerations, preferences etc., it is important for that evaluation to be objective.
Format your output as JSON:
"coherence"; 5, "behavioral": [1, 1, 1]
And the following input formatting:
Text: \"\"""
(text) {"("""
mun
B.2. Activation of theme
We select the features used in Chalnev et al. (2024) (Table 3), so we do not significantly alter the initial feature choices.
Flow graph building. Starting from the target feature, we build a flow graph forward and backward, computing similarity scores g(R) , s(M) ((A) for each residual feature, referencing its predecessors. We then cut our graph on layers where (R)) below a threshold value {(R) 0.5, forming the similarity span from 1tta to land- We also remove features from modules using a threshold value of 0.15.
Feature activation transtormation. For steering, we add scaled decoder columns of selected features:
h, h, + s. VT,
where s E R is a vector scaling coefficients for f features whose embeddings are in Rdxf, We apply this transformation all tokens to globally promote or suppress certain features.
14

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Distribution of steering coefficient. To steer multiple related features, can distribute single steering coefficient across them, rather than manually tuning each feature. We consider main strategies.
Let be the initial scaling coefficient and I the layer index. We define exponential scaling of a related feature as:
=88x x
and linear scaling as:
5' = x
where
k
and
b s Luarts
lend
start
where lstart lend the first and last layers the obtained graph.
Our intuition is that the more features we steer across different layers, the less strength each subsequent feature should have. Exponential scaling with a < O causes s' to decrease monotonically. Linear scaling is essentially an interpolation between (Isant: s) and (lend: s*). Constant scaling a special case of both methods where s' = =.
We also employ folding (Balagansky et al., 2024) to incorporate information about typical activation levels of different features, helping us distribute the coefficient more effectively. We set -0.05 and 1, based generating a small batch of test completions and manually checking the trade-off between coherence and theme intensity.
The single method reported in 5.3 employs a per-layer activation strategy, and other methods are cumulative.
Generation and evaluation setup. We use the prompt, "I think ", maximum generation length of 36 tokens, top, 0.7, and temperature T 1.27. We validate the result with the following system prompt:
You will receive a text that was generated by some language model.
You need to evaluate this text based on the following criteria:
1. Coherence score: text is gramatically correct, language is clear and coherent.
2. Behavioral score: text has explainable relation to some predefined subject.
Give an integer score from to 5 for each criteria.
Do not judge text based on ethical considerations, preferences etc., it is important for that evaluation to be objective.
To evaluate behavioral score, also consider related terminology or entities which may be not directly discuss the subject, but associated with it. The name of the subject is just an orienting point for further search of ** explainable ** connection to that theme in text. For example, if subject the History of Rome, then strong presence of latin or italian language, or discussion about not history, but architecture of ancient Rome should also lead high behavioral score, because it has strong explainable connection.
Format your output as JSON:
"coherence": 5, "pehavioral";
And the following input formatting:
15

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Lewer Soyl
Lowe Rid
Froe
MLP
Fror M.P 6-ATT
Layer
Laye
Farttin
Trodbeex
Pyhvon Cole
Aratertal
(a)
(b)
Figure (a) Percentage feature groups obtained for each datasct. (b) Distribution of scores for layers and 18. observe clear distinction between groups, which additionally indicates the validity of the proposed method.
Aroe rosber
6-44
0:32
3.14
8:26
R12
6-1
0.11
Fiors
6:39
U224
9.75
M2
0:82
0.5
9:25
6:45
014
9.1
6:16
Flax ATT
0:83
576
0:42
8:30
045
5.1
8:20
Fox RGS MMP
616
96
0222
1:18
1.22
Fri 485 6XT
0:34
6.72
0.32
0.44
8:56
0.51
8:30
Fran M.P EXTT
0:83
8:51
0/67
3.57
8-46
0.37
8.42
Fax RCB M&P SKIT
0.74
8.42
25
347
2:53
3.5
0.27
From suffch
HumAAt HyP
From
Pous
LTT
325 AATT
Figure 13. Probability of group A (row) to appear in group B (column), aggregated over all layers. For example, if we take the "From ATT" group, then with probability 0.45, features from this group would appear in the "From RES ATT" group. High scores for the "From nowhere" group represent its stochasticity.
"""Subject: (theme)
Text:
""""""
(text)
""("""
mun
C. Additional results for experiments
C.1. Identification of feature predecessors
The "From nowhere" group is the most present among all other groups (Figure 12(a)). This may be the consequence of sporadic activation of some features matching error. The absence of groups with attention module is probably the consequence of our training procedure, which clearly contrasts with the distribution for Llama Scope (Figure 19(a)).
In Figure 12(b), we see that certain groups are indeed distinct with respect to corresponding similarity scores, which we describe Section 5.1. Figure 14 shows the percentage of tests passed with a p-value threshold 0.001 for each pair of groups. aggregated for each layer and dataset.
However, observe that features may fall into different groups depending the context and chosen token (Figure 13), which indicates that we need to estimate, for every feature, the most probable groups they fall into.
A three-part partition of the group distributions for both Gemma Scope (Figure 5) and Llama Scope (Figure 19(b)) aligns with earlier observations on monosemanticity of neurons across layers (Gurnee et al., 2023). Partitioning the model into the first 20%, the next 40% and the final 40% layers reveals varying degrees of monosemanticity. which may have a connection with the three-part partition in the distribution groups across layers for Gemma Scope, we have mentioned
16

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Pervoutess
890
COS
0
E10
2M
GMI
CMI
GMR
XXT
EXT
XYO
60R
AN
EXT
OST
CNN
CM
CIN
DUT
6.
NEA
Git AAM
Figure 14. Percentage of statistically significant differences between groups with respect certain score.
Percent of features per category
MS
PrDuATT Laf
60
From nowhere
From RES & MLP
200
From RES
From RES & ATT
50
From MLP
From MLP ATT
From ATT
From RES & MLP & ATT
40
30
20
Fon LAT
10
0
random
permutation
top-1
top-k
(a)
(b)
Figure 15. (a) Percentage of features per each method. There was a total of 13106 activated features, and for every feature, four matching strategies were applied. We see that top, method detects many more combined groups than other methods, especially "From RES MLP". (b) Probability for a feature from some group A (labeled as the subplot title) to become from group B (shown in legend) after deactivation of some predecessor. Each bar shows the percentage of times the feature falls into new category.
parts [0, 5], [6, 15], and [16, 25], and in the case of Llama Scope we observe segments [0,8], [9,16], and [17,31].
C.2. Deactivation of features
We observe that the top; method happens to detect many more activated predecessors than other methods, and detects more combined groups as depicted in Figure 17(a).
Deactivation of residual predecessor in the case "From RES & MLP" and "From RES & ATT" with almost equal chance also deactivates the predecessor on the corresponding module or deactivates the target feature entirely, as depicted in Figure 15()). This suggests that in those cases, the residual predecessor is indeed blocked from further propagation. However, in most cases, full deactivation (of all predecessors) is required to deactivate the target feature. In many cases, "Deactivated" and "From nowhere" are equally probable, which indicates remaining causal dependencies that we miss with our method.
We also observe the appearance of new groups, i.e., a feature might initially be "From MLP", but after deactivation of the MLP feature (which is actually a re-calculation of the full forward pass with intervention on the MLP module), we observe that sometimes the feature might have new predecessors, for example, on the attention module. This is unexpected since the MLP module actually comes after the attention module, but the presence of such groups is not so strong, so we think of it as sporadic behavior of internal computations.
We must take into account that there may be other causal relations for some feature to appear, for instance, interaction between different tokens on the attention module, different features, different modules, or even different layers. Furthermore, the appearance of feature means a certain structure of the hidden state, and this structure was built by many previous layers where information was somehow encoded in a complex way by the interaction of many different components and
17

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Anger trustration
Mentai health Issues
Wedding marrigee
Rellgion and God
12
21
0.5
0.3
3.0
0.4
02-
0.0
single
corsiant exponentis
Single
exponential
single
constant exponental
single
exponental
Figure 16. From each flow graph, we select fcatures on a particular layer and perform steering with the four different strategies. Bars represent result each layer among scores some cases, steering on layer other than 12 may improve results.
features. This like an optimization process a non-convex scenario, which may converge to some local minima with certain properties, and information processing inside the model may converge certain structure hidden states with certain semanties contained in it.
Therefore, in ideal situation, really deactivate some feature, must somehow influence the hidden state to behave as if there never had been such feature, its evolutionary ancestors, or any other causal predecessors, and they had never been involved in information processing. Our current steering procedure works in "neighborhood" of some local hidden state "minima", but efficient deactivation consists of changing the convergence direction toward another hidden state "minima" at an early stage computations. This most likely also applies to the activation of some feature.
C.3. Model steering
We also measure the effect of steering on different layers. The best result among all available is shown in Figure 16. Note that single a per-layer method, while others are cumulative. We see that different layers perform differently, and while the initial features were located at the 12th layer, sometimes the best layer located elsewhere.
We also have performed small experiment test the activation of another theme with many flow graphs using the same prompt as in the deactivation case. We start with the features described in Table 4 and build flow graphs from them. Then we manually choose some of the subgraphs based on semantic considerations and threshold values. The total amount of features selected on different layers is presented in Figure 15(a). After that, we steer the resulting features with manually obtained s = 8 and a -0.05 for the single-layer case, and = 3 and a = 0.25 with the exponential decrease method for the cumulative setting.
By using our method, we found influential features on the 5th layer that gave us the best result among all other layers (Figure 17(b)), while none of the initially found features placed that layer. However, did not tune the hyperparameters properly, there may be room for another conclusion.
C.4. Comparison with Pearson Correlation Baseline
While data-driven methods provide useful insights, they face challenges with sparse SAEs and low-frequency features. Our data-free approach overcomes these limitations through adjustable top-k matching, particularly advantageous in sparse activation regimes.
We evaluated Pearson correlations on 100K non-special tokens from Fine Web's "default" subset for features in Gemma Scope's 's even layers and all layers of Pythia-70k, 70M-Dedupp and GPT-2. Using an expanded sample size of 500 (Appendix A), we identified feature groups. Figure 18 presents results for the Gemma model.
18

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Feature index
Interpretation from Neuronpedia
12/res/6778
references to testing and experimentation processes
16/res/1 3806
references to experimental studies and methodologies
18/res/1056
references to experiments and experimental protocols
18/res/7505
terms and phrases related to research activities and methodologies
10746
terms related to modeling and model- moodel-buildin in scientific contexts
24/res/1 1794
terms and phrases related to scientific reasoning and methodology
24/res/1027
concerns related to study validity and bias in research methodologies
24/res/7391
phrases related to inquiry and questioning
2A/res/1714
references to academic studies and their outcomes
25/res/6821
terms related experimental methods and results in scientific research
Table 4. Features initially chosen for activation of "Research methodology and experimentation" theme.
Single layer
Cumullative
0.0
Cchorence Brhororal
5
25
15
20
layer
Layer
Layer
(a)
(b)
Figure 17. (a) Amount of features selected for activation of "Research methodology and experimentation" theme. Vertical lines represent the placement of the initially selected features. (b) Results for steering of selected fcatures. Score is a total metric measured as Behavioral x Cumulative. We can see that despite none of the initial fcatures being placed on the 5th layer, it gives us the best result.
Fimetleb
Tiny Stories
Python Code
Fror
10
1ayer
Layer
13ye7
Figure 18. Feature group identification comparison (Section 5.1) between top, cosine similarity and Pearson correlation. While correlation better captures predecessors with trden-trained embeddings, it exhibits stronger dataset dependence and sparsity sensitivity.
Correlation-basee matching reduced the "From nowhere" group presence and improves predecessor identification on attention module, though potential misalignment between attention SAEs and Gemma Scope's residual/MLPP SAEs may affect quality. Results aligned closely with Llama Scope for Pythia (showing clearer attention features), while GPT-2 displayed increasing "From nowhere" presence in later layers.
The correlation method failed to consistently outperform top, cosine similarity, particularly on out-of distriibitiiito Python code. Strong agreement emerged between methods for Gemma Scope and GPT-2 residual SAEs, but weaker alignment occurred for module-basee based SAEs, consistent with prior feature propagation studies.
19

---

Analyze Feature Flow to Enhance Interpretation and Stecring in Language Models
Fraiho
hew LVfT
From RES
From RES
From RES & ATT
From MLP
From MLP & ATT
From
From
20
10
FinWWeb
TimyStories
Pytno Code
AutoMa
(a)
(b)
(d)
Figure 19. Distribution of groups for Llama Scope. We observe clear distinction from Gemma Scope results (Figure 12(a)) due much smoother distribution. This may be consequence of various factors: the architecture of the models or SAEs, the training procedure, differences in data distribution, etc. (b) Distribution of groups across multiple layers. We observe approximately the same pattern as for Gemma Scope (Figure 5), indicating shared properties between the models. (c) Distribution of scores for different groups. We see that the groups are slightly less distinct from each other compared the case Gemma Scope (Figure 12(b)), but they are still present. This also reflected in (d) the separability of different groups based on their cosine similarity relations.
D. Experiments with Llama Scope
We have also used the Llama Scope SAE pack (He al., 2024) evaluate our proposed approach and have found that it is well aligned with the results observe for Gemma Scope. However, did not perform steering experiment graph building, and we consider it one of the future study directions. For these SAEs, the main picture remains the same.
First, they have a more uniform distribution of feature groups, with a clear prevalence of attention features (Figure 19(a)). This indicates that our attention SAES for Gemma were perhaps not trained well enough. We suspect that experiments with other models will show that Llama Scope results are more accurate with respect to predecessors distribution.
Second, despite the uniformity, we observe that Llama Scope groups are slightly harder to separate from each other in terms of similarity scores (Figures 19(c) and 19(d),, which may also be the consequence of the different architecture of the model itself or because these SAEs initially were trained with the TopK activation function.
Third, the dynamics of the group distribution is slightly different (Figure 19(b)), but the overall pattern (with a three-pare separation and an increase "From RES" the latter layers) and overall percentage still approximately the same. Perhaps we may interpret this similarity between Llama Scope and Gemma Scope as an argument for the validity of our analysis; however, it still requires experimentation with other architectures and sizes.
20

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
London Graph
Neywords and concests relased fastion
refie contienences and related wvents references fron and reluted demoris various
reterenoess to fistion and misted opecepts
Resdual Stream
references to the city of London
mentions London assodiated
sers to organi paaaizationall spucaur eo and
words and phrases refened to fastron
10
15
20
25
Layer
Figure 20. Flow graph for the 12/res/1 feature. As reported in Chalnev et al. (2024), steering of that feature might produce themes related to fashion, and we clearly observe that our flow graph captures this semantics in the earlier layers.
E. Examples of flow graphs
In this section, we describe some of the interesting flow graphs we have found. For simplicity, we denote each feature as "layer index / module feature index".
Particle physies graph. We start with the graph presented Figure that was built from feature 24/res/14548 Once we obtain it, we might explore how its semantics evolved across different layers. The full list of features with interpretations that belong to this graph is presented in Table 5.
From the first to the sixth layers, we have semantics mainly related to experiments and abstract particle physics. Then we have feature 7/res/1 16335 with the following scores: g(M) = 0.82 for 7/mlp/61 10, with semantics related to datasets and measurements, and s! (R) = 0.3 for 6/res/2452 with semantics about Dark Matter. After this, the semantic flow has a tighter connection to measurement and observation-related themes, while maintaining the quantum physics semanties.
We hypothesize the following relation: initially, the flow graph was related science and experiment, and on the 7th layer it was transformed in a way that 7/mlp/61 10 introduced slightly new semantics to the already existing one, perhaps also replacing the vague "experimentation theme. Thus, we think of this interaction as example of linear circuit, and feature 7/r5 falls into the processed category.
After the 7th layer, we observe a slight strengthening of particle physics semantics, perhaps because of some other interactions, while also introducing the bosons theme. From this layer, g(f)) is large and s(M) is small. On the 17th layer, we encounter feature 17/res/8130 with RR) 0.48 and (M) 0.79 for 16/res/10649 and 17/mlp/8454, respectively. The MLP feature relates to gauge theories and theoretical matters, and the feature 17/res/8 1/3/es//130 drifts toward gauge bosons and their interaction theme. We also hypothesize that at this particular point, the feature from MLP added new information the already existing one; therefore, 17/res/8 130 is also a processed feature.
After this, the semantic meaning sticks more to the Standard Model and particle interaction, but with less practical (such as measurements and data) and more theoretical aspects. We can also see that MI.P features on layers from 20 to 24 are more related to theoretical aspects than their residual matches.
London graph. An interesting observation was made Chalnev et al. (2024): steering feature 12/res/14451 with interpretation "mentions of London and its associated contexts" with a large steering coefficient led to the generation of theme related to fashion, design, and exhibition. If we build the flow graph from this feature, we observe that in the first half it clearly has fashion-related semantics (Figure 20). This indicates that feature 12/res/14455 contains the semantics of its evolutionary predecessors. We also see feature 17/res/9260 with references to conferences (followed by feature 18/res/2010 with the same semantics), which relates to shows and exhibitions mentioned in Chalnev et al. (2024). Perhaps we might interpret this particular flow graph as "references fashion and design exhibitions performed in London"..
21

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Wedding Graph
othends ens dekrziptie
Atention
events EvoMing openings inauguratins
ands Diiin "baaf" from...
straments and confimations rogarding evartss or actions taken by officisss or fgures in posiiiass dhaubee
Resdua Stremm
roferences to sedciings, anmiverstrics and relued ceremonies
weddings mgrnee antiversanes diptengtic pegtooooooo woddings ga and reluted everss refor ivvants "Mlated opeinigg ninning snttthinnn refarinness wedding ceremonies
sezements made inghtduats, particutaly those inwoiing auocations or direct speech
referencce widdinggs mariiague
10
15
20
25
Layer
Figure 21. Flow graph for the 12/res/4230 feature. In this case, observe that the second half of the model is closely related wedding and marriage ceremonies. We helieve that the "official" aspect in the interpretation of fcatures in earlier layers is closely related to the fact that wedding ceremonies and marriage are themselves official procedures the registration of specific type interpersonal relationship.
SAE Layer T
Transition
SAE Layer T+1
Transcoder from T to T+1
Figure 22. Two SAEs with a learned transition matrix T can be seen transcoder from layer / to layer & + 1.
Wedding and marriage graph. We have observed in experiments that steering feature 12/res/4230 with interpretation "terms related to weddings and marriage ceremonies" indeed increases the presence of ceremony-related tokens in wedding context. If we obtain a flow graph for that feature, we see that it begins with themes related to official meetings and agreements, suggesting that the "ceremony" part of the flow graph interpretation may arise from this official context.
conclude that these flow graphs may be used not only for interpretation and understanding of feature evolution, but they can also explain the outcomes of certain steering procedures.
F. Similarity between Matching and Transcoders
Dunefsky et al. (2024) proposed using transcoders to study computational graphs, and Balagansky et al. (2024) proposed using a permutation to find matching features across layers. In this section, we study the similarity between these two methods. Similarly to Balagansky et al. (2024), we chose explained variance as a metric measure the quality of the translayer transcoder. See Figure 22 for the schematic overview of the transcoders obtained by transition mapping T
Setup. We SAE trained the residual stream after layers 14 and 15. We vary the methodology to find and apply the transition T. In cases, consider only linear map that T E R [
First, we study how folding proposed in Balagansky et al. (2024) affects final transition performance. Results are presented in Figure 23. From these results, we conclude that folding is useful in the inference case to match different scales of activations across layers; in contrast, it has almost no effect on finding permutations, with the exceptional case of incorporating bene to find permutations. Notably, the baseline with a simple approach of finding cosine similarity outperforms permutations.
Second, compare cosine similarity with other methods obtain the transition map T. Instead of relying on matrix containing and 1, we use the topa operator. Results are presented in Figure 24. Interestingly, folded top, dec dec outperforms the permutation bascline; however, cosine similarity (Iaz0 top W-(4)) (14)T w-(15) performs best.
(14))
doc
22

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
0. 0.786
0.784 706
0.775
0.750
6.751
0.249
0.749
Explained Variance
0.700
0.675
0.650
stop; Woll was))
Nat Faided
Folded Permute Finding
Folded inferwnce Nat Folded Finding
0.625
Will W115
Wi34 W131
Wy wiss W15 + Wil with 61386/133
Figure 23. Explained variance of the various permutation variants. (LasD top W((4)T ((( performs best. See Appendix for more details. dee
Cosine
similarity between decoders' vectors
Layer 15-1 16 Residual
0.80
0.75
Explained Variance
top, (atp//AAWWBB11111
0.70
top,(W top,(w/(TTwiB)
top:(W/4) w/ WTAlwiB))
0.65
Folded (Wp((Wi)
Folded (Wp! wit)))
Folded top, will)
Folded top, (WW)) (iae Wene Win)
wTllwile)
0.55
1x-ttop, (/34) WiTs
Permutations
0.50
2
3
Figure 24, Comparison of various & in top, operator and different weights of the SAE, Cosine similarity (Iss0 top (14)) (11))) performs best. See Appendix for more details.
23

---

Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Layer
Feature index
Interpretation
0/mlp/12987 Ofres/1 4403
punctuation, particularly quotation marks and dialogue indicators
elements that indicate neglect or care familial relationships
1/mlp/16168
mentions of astronomical phenomena and their characteristies
1/res/13755
metaphorical language and scientific terminologies related to variables and coefficients
2/res/ 1299
numerical data or metries related surveys and observations
3/res/1615
scientific terminology related to study results and causes
4/res/1 1935
terms related to particle physics and their interactions numeric or symbolic representations related to mathematical notation or scientific data
5/res/1455
6/res/2452
key terms related to Dark Matter detection and experimental setups
7/mlp/6110
terms related datasets and classification in statistical machine learning contexts
7
7/35
technical terminologies related particle physics measurements
8/res/9666
scientific measurements and data related to particle physics
9/res/8318
references to particle physics concepts and measurements
10/res/13754
technical terms and measurements related to particle physics
11res//614
terms related particle physics and specifically the properties of W and Z bosons
12/res/2812
statistical terms and measurements associated with quark interactions
13/res/4955
terms and concepts related to particle physics experiments and measurements...
14/res/5262
keywords related to particle physics, specifically concerning quarks and their properties
15/res/9388
concepts related to particle physics measurements and events
16/res/10649
complex scientific terms and metrics related to particle physics experiments
17/mlp/8454
theoretical concepts and key terms related to physics and gauge theories
17
17/res/8133 130
terms related to gauge bosons and their interactions within the context of particle physics
18/res/1199 1987
technical and scientific terminology related particle physics
19/ess/1699
references to scientific measurements and results related particle physics...
20/mlp/601
terms associated with quantum mechanics and transformations
20
20/res/12523
terms and concepts related to particle physics and the Standard Model
21/mlp/594
technical terminology and classifications related to data performance metrics
21
21/res/145)
scientific terminology and concepts related to fundamental physics...
22/mlp/14728
references to gauge symmetries theoretical physies
22
22/res/1 1460
terms and concepts related to particle physics and theoretical frameworks
23/mlp/6936
terms related to theoretical physics and particle interactions
23
23/res/9592
terms related to particle physics and their interactions
34/mlp/11342
terms and concepts related to theoretical physics and particle interactions
24
Aa/res/14544
terms and references related to particle physics and standard model parameters
25/res/1646
technical terms and measurements related to particle physics and the Standard Model
Table 5. Graph built from 24/res/ 14548 feature with MLP features dropped by threshold {(M) 0.25.
24


Reviews:
[{"review_id": "dRapLZYDiS", "reviewer": "Reviewer_jyW6", "summary": "The paper investigates whether features in LLM activations identified by SAEs trained on individual activation locations - namely residual streams, attention outputs and MLP outputs - can be linked to one another, so that we can, broadly speaking, try to answer questions like \"where did a feature come from?\" and \"how did a feature evolve through the layers?\". \n\nFeatures across SAEs are linked to each other via the cosine similarity of their decoder vectors. Specifically, given:\n- a residual stream SAE feature $f$ in layer $L$;\n- SAEs trained on the previous residual stream of layer $L-1$, as well as the MLP and attention outputs in-between;\n\nwe define $s^{(R)}, s^{(M)}, s^{(A)}$ to be the top cosine similarities of $f$ with any decoder vector in the previous resid (R), MLP (M) and attention (A) SAE decoder vectors. The relative magnitude of these values is used to classify a feature as being one of:\n- translated from the previous residual stream without MLP/attn involvement;\n- processed by the MLP or attn when all values are high\n- \"newborn\", created by the MLP or attn\n- unexplained when all values are low.\n\nThe main body experiments aim to answer the following questions:\n- **feature predecessors**: this checks that feature predecessors identified using the paper's method correlate with SAE feature activation, i.e. if the predecessor of $f$ activates on a datapoint, does $f$ activate too? \n- **deactivation of features**: if we deactivate a predecessor feature of $f$ by subtracting it from the activation, will this lead to $f$ not activating?\n- **model steering**: can we improve upon naive single-layer steering by steering multiple features across layers jointly when they're identified as connected via predecessor relations?\n\nThe results suggest the following:\n- we generally observe a clear cluster of features that have a strong predecessor in the preceding MLP layer but not in the previous residual stream layer, and for these features it's also generally the case that if the feature is active, its MLP predecessor is also active. By contrast, for attention we see no such pattern.\n- feature deactivation guided by cosine similarity performs significantly better than a baseline using random choice. Deactivation seems to have higher impact on activations when done using residual stream predecessors vs MLP/attn predecessors.\n- for steering to activate a certain topic in the generation, results don't clearly show that the proposed methods outperform baselines or single-layer steering. For steering to de-activate a topic, i.e. suppress mentions of it, there seems to be some benefit to multi-layer steering versus single-layer.\n\n## Update after rebuttal\nThe rebuttal has not meaningfully changed my assessment of the paper. Despite other issues, I hesitate to recommend acceptance mostly because I don't see a strong motivation behind the problem being studied and don't see the usefulness of the results to the broader field of interpretability.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and extensive review. We aim to address your concerns below.\n\n**Claims And Evidence**\n\nOur work examines SAEs at various points in the model and linking their features allows us explore its computational structure—such as the technique described in Appendix F, which mimics a transcoder approach. Thus, we can shed more light on the model’s behavior than just geometry. Permutations [2], top-1 correlation [3], and top-1 cosine similarity (our work) are all special cases of this feature mapping approach.\n\nWe respectfully disagree with labeling our approach as non-\"data-free\", since we use this term following the prior work [2]. While SAEs could be trained on the LLM’s data, we solely analyze model weights, not external data, justifying the \"data-free\" description.\n\n**Methods And Evaluation Criteria**\n\nWe appreciate your suggestion for a data-driven baseline. While useful, such methods struggle with sparse SAEs, demonstrating the advantage of our data-free approach where adjusting k in top-k matching can better address these limitations.\n\nWe tested Pearson correlations on 100K non-special tokens from the FineWeb “default” subset for each feature in Gemma Scope’s even layers and all layers of Pythia-70M-Deduped and GPT-2. Using 500 samples (instead of 250) as described in Appendix A.1, we identified feature groups. \n\nhttps://anonymous.4open.science/r/icml_rebuttal-5064/gemma_corr_cos.png\n\nhttps://anonymous.4open.science/r/icml_rebuttal-5064/pythia_corr_cos.png\n\nhttps://anonymous.4open.science/r/icml_rebuttal-5064/gpt_corr_cos.png\n\nCorrelation-based matching reduced the \"From nowhere\" group and better identified attention module predecessors, although the mismatch with Gemma Scope SAEs still lowered quality. For Pythia, results aligned closely with Llama Scope, reflecting clearer attention features.\n\nHowever, correlation-based matching did not consistently outperform top-1 cosine similarity and performed worse on out-of-distribution Python code (further from FineWeb). Top-1 cosine and top-k correlation predecessors showed strong agreement for Gemma Scope and GPT-2 residual SAEs but weaker alignment for module-based SAEs, consistent with prior feature propagation findings.\n\nThese results broadly characterize correlation-based performance. We plan to include these comparisons but welcome requests for specific details to enhance our discussion.\n\nBehavioral and Coherence scores follow the setup from [1], with details and the system prompt described in Appendix B. In brief, we ask a model to assess whether a specific theme is present (Behavioral) and to rate the text’s language quality (Coherence), assigning each an integer score from 0 to 5. These scores are then normalized to the [0, 1] range. We acknowledge that this explanation belongs in Section 4.2 and will revise it accordingly.\n\nAs you noted, multiplying the scores ensures that moderate Coherence and high Behavioral (or vice versa) results in a moderate overall score as expected. Both scores are illustrated in Figure 9.\n\n**Experimental Designs Or Analyses**\n\nWe define the activation change as $1-\\mathbf{z}^{new}/\\mathbf{z}^{old}$, where $\\mathbf{z}$ is a feature activation after applying JumpReLU.\n\nEach feature may have up to three predecessors (residual, MLP, attention). To identify active predecessors, we use four methods (Appendix A.2).\n\nFor features with active predecessors in both residual (R) and MLP (M), we label them “From RES & MLP” and run forward passes deactivating R, M, or both (“deactivated one at a time”). In top-k method, up to five features may be deactivated per predecessor (though few are typically active).\n\nFigure 7 subplots group results by predecessor type; bars show deactivation effects (e.g., deactivating “mlp” in “From RES & MLP” yields ~0.25 mean activation change).\n\n**Other Strengths And Weaknesses**\n\n\nWhile we did not explicitly justify cosine similarity as a metric early on, Section 5.2 and Appendix F compare it with a permutation-based method.\n\nBy “removal” or “suppression,” we refer to specific linear combinations of features (e.g., “king – man”), similar to combinations that create new semantics (e.g., “woman + power”). We suspect these combinations are common.\n\n**Questions For Authors**\n\nWe use the L2 norm, treating vectors as sliced decoder columns. \n\nWhile our presentation assumes B < A,  this is not strictly necessary.\n\nThank you again for your valuable feedback. We will reorganize our paper, improve clarity, and add the missing reference. We hope these clarifications sufficiently resolve your concerns and encourage you to reconsider your evaluation. Thank you for your time, and please let us know if you have any further questions or requests.\n\nReferences:\n\n[1] https://arxiv.org/pdf/2411.02193\n\n[2] https://arxiv.org/pdf/2410.07656\n\n[3] https://arxiv.org/abs/2410.08869", "claims_and_evidence": "In general, the claims made in the paper are supported with evidence reasonably well, however there are some issues with the methodology and presentation of results (described below and in subsequent sections) that make it difficult to evaluate the scope and novelty of the findings.\n\n- a key property of the methods in this paper is that they can only tell us about how the same feature persists (or not) through layers, or when it \"appears\".\n\t- This limits the conclusions that can be drawn from the results; by contrast, a much more interesting (and more difficult) question would be to understand how different features combine through attention and MLP blocks to create *new* features.\n\t- Furthermore, the approach in this paper fails to fully account for the possibility that features \"rotate\" across layers, as described e.g. in https://transformer-circuits.pub/2024/crosscoders/index.html; to some extent, this is mitigated by linking features through successive layers, but it is unclear if this overcomes the problem. I think that a careful study using a tool similar to cross-coders, as well as comparing feature activations over datasets of texts, could help answer this question.\n- in general, the methods and claims may be shedding more light on the geometric structure of the SAE matrices as opposed to the workings of the model. For instance, it has been observed that SAE encoder and decoder vectors for the same feature have substantial though not perfect cosine similarity (see e.g. https://transformer-circuits.pub/2023/monosemantic-features#comment-nanda). This alone *may be* enough to explain the results on predecessors and deactivation!\n- in general, the paper would have benefited from more methods and experiments that establish the relevance of the phenomena explored to the end-to-end behavior of the model, as opposed to internal geometric structure that may be an artifact of SAE optimization. The steering experiments are one promising step in this direction.\n- it should be noted that while the cosine similarity method is advertised as \"data-free\", the features we're taking the cosine similarity of themselves came from SAEs, which are trained on vast amounts of data. So strictly speaking this is not a data-free interpretability method (as opposed to a method that is purely based on the weights of the model). In general, \"data-free\" interpretability methods have the promise of explaining the out-of-distribution behavior of a model precisely because they are independent of any properties of the data apart from what is encoded in the model weights, which is a key motivation for considering such methods. However, this motivation does not really apply to the methods in this paper. However, it should be noted that truly data-free methods have so far struggled to provide useful interpretability insights.", "methods_and_evaluation_criteria": "Some shortcomings:\n- there is a lack of a baseline approach for linking features to one another across layers that the approach of the paper can be compared to. The top candidate that comes to mind is an activation-based metric that correlates feature activations over a large set of examples. \n- in general, there is often not enough detail to get a full picture of how a given method is implemented.\n- a potential source of noise for the given methodology is that it is known that SAEs trained on the same activations can arrive at different features with different random seeds. See the paper \"Sparse Autoencoders Trained on the Same Data Learn Different Features\" by Paulo and Belrose. In the absence of any \"anchor\" between different layer SAEs, linking features in such a naive way may be suboptimal; a strong activation-correlation baseline is desirable.\n- the way in which behavioral and coherence scores are assigned in the steering experiments is not explained. \n- combining behavioral and coherence scores in a single metric is a potential source of misconceptions and illusory results. Furthermore, doing so via multiplication is not very principled: what's the interpretation of a unit in the resulting scale? It feels overly confusing to think about that. A very high behavioral score can be achieved at moderate coherence if the model repeats a phrase related to the topic. In general it's more insightful to look at both behavioral & coherence metrics together on a 2D plot.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- In general, for many of the experiments and figures there is insufficient detail describing what is being done/visualized. \n- I don't quite follow Figure 7. Is \"activation change\" measuring the drop in the SAE feature's pre-activation? Also, if we're deactivating 1 predecessor at a time, where is it shown how the effect evolves with the number of predecessors deactivated? Also, how do we get multiple predecessors for a feature? Do we just look at the top k highest cosine similarities?", "relation_to_broader_scientific_literature": "It's hard to situate the contributions of the paper in the broader literature because of the lack of baselines and the possibility that many of the results follow from known facts about SAE encoder/decoder geometry. The exception is the steering experiments, which may be a useful addition to the literature, especially if presented in more detail / with easier to interpret analyses (see previous points).", "essential_references_not_discussed": "- in the discussion of the topk activation function, a foundational reference that for the first time proposed the use of this activation in SAEs and established its improved metrics and scaling properties is missing: Gao, L., la Tour, T.D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J. and Wu, J., 2024. Scaling and evaluating sparse autoencoders. _arXiv preprint arXiv:2406.04093_.", "other_strengths_and_weaknesses": "Weaknesses:\n- The presentation of the methods is confusing (also see comments/suggestions section below). In particular, the \"Results\" section is actually the one that describes the methods in sufficient detail, whereas previous methods sections only vaguely point at what's going to be done. \n- You say \"We found that cosine similarity between decoder weights is a valuable similarity metric, and we focus on this approach.\" (line 160), but this is never justified in terms of alternative methods and metrics. It would be a valuable addition to the paper. \n- You say in 3.4. that your method can detect when MLP or attn *remove* features from the residual stream, but there's no evidence for what removal would look like in the paper?\n- The 4 subplots of Figure 5 are basically identical - maybe just include 1 in the main body, and say that for other datasets it looks basically the same. Furthermore, this suggests that these lines are more a geometric property of the ensemble of SAEs as opposed to a data-specific property.", "other_comments_or_suggestions": "- Section 3.1. feels somewhat redundant, given the introduction which already establishes the motivation? Consider adding a brief sentence to the introduction citing the main relevant works describing the motivation, and cutting this section. \n- I think the exposition would flow better if it is stated upfront that we assume that the decoder vectors have unit norm (lines 139 and 140, right column)\n\t- Sections 3.4 and 3.5 are quite vague in terms of methodology; are they really necessary? I don't feel like I got much out of them beyond \"we do some cool stuff\". Maybe cut & incorporate in later sections that flesh out these ideas?\n\t- Same applies to some extent for 4.2.; I wonder if presentation would be better if there was a single place in the paper that has a 1-sentence brief intuition for each experiment + a very concrete description?", "questions_for_authors": "- it is customary to subtract the decoder bias $b_{dec}$ from $h$ before applying $W_{enc}$ (cf line 77, left column). Omitting this changes the architecture of the SAE, though in principle the bias can be absorbed in the encoder bias by changing $b_{enc}$ to $b_{enc} -W_{enc}b_{dec}$. \n- what is the matrix norm in line 87 (right column)? I assume Frobenius\n- in 2.3., is the intention that $B<A$?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8FKQaJInyP", "reviewer": "Reviewer_z5bz", "summary": "The authors introduce a method to allow for cross layer and mulit-module (mlp, residual stream, attention block) level mapping of SAE based features, creating a flow graph using a data-free cosine similarity technique (between feature embeddings and encoder blocks), that allows a person to interpret and trace how specific features persist, transform, or first appear across layers. The paper analyzes such feature evolutions on GemmaScope and LlamaScope SAEs and models and also demonstrates how their method facilitates cumulative layer wise direct steering of model behavior by amplifying or suppressing chosen features for targeted control in text generation and potential causal circuit discoverability.\n\n## update after rebuttal\nWhile the paper provides an interesting method by which to do cross layer, multi-module analysis, and the authors have said they will address the concerns I stated ( which are largely organizational and for clarity),  without seeing the revisions and organization changes ( which will be substantial ) its hard to move the current overall recommendation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading and insightful questions.\n\n---\n\n**Q1.** The purpose of Figure 4 is indeed to show that these groups differ significantly. Below are the raw counts of elements in each group:\n\n| Nowhere | RES | MLP | ATT | RES & MLP  | RES & ATT | MLP & ATT | RES&MLP&ATT |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| 3396295 | 2338643 | 972662 | 574635 | 769450 | 606080 | 261068 | 308578 |\n\nSee Figure 13 for an evaluation of how frequently groups intersect. Those values are calculated as $|G_1\\cap G_2|/|G_1|$, where $G_1$ is a row group and $G_2$ is a column group.\n\n**Q2.** Figure 5 alone cannot confirm feature emergence. We instead analyze group proportions across layers. Features are classified as *emergent* if closer to module (MLP/ATT) features than residual stream features (Figures 12b, 18c), placing \"From MLP\", \"From ATT\", and \"From MLP & ATT\" in this category. *Translated* features are those closer to the previous residual stream (\"From RES\"). See also our answer to Q6.\n\n**Q3.** Comparing FineWeb and Python Code in the later layers reveals a higher presence of “From nowhere” features than “From RES,” distinguishing this dataset from others. The same trend appears for Llama (see Figure 18b) and Pythia-70M (mentioned in our response to reviewer jyW6).\n\n**Q4.** We may have misparsed your comment—did you mean \"I did notice\"? Model differences exist but stem from varying architectures, parameter counts, and SAE training procedures. A full explanation requires deeper study (e.g., identically architected SAEs on shared data). We will provide additional discussion and experiments on Pythia and GPT-2 in the revised manuscript to address this further.\n\n**Q5.** We believe there are two primary causes for features labeled “From nowhere.” First, there may be a matching error—artifacts from SAE training or a situation where the true predecessor is not the top-1 but perhaps the top-2 or top-3 similar feature. Second, some features could be combinations of two or more features, making them harder to detect via a simple top-1 match; they could also transform (e.g., rotate) as they pass through the layer, as reviewer jyW6 pointed out. Using top-5 cosine similarity, as in our deactivation experiment, reduces the “From nowhere” group substantially, but it is unclear if it can ever be made negligible. Expanding the SAE dictionary might also risk increasing false positives under the top-1 strategy, so this trade-off requires more investigation. We will extend our discussion of the “From nowhere” group to clarify these points in the revised version.\n\n**Q6.** We currently think of two general approaches for validating these shorthands.\n\nThe first involves analyzing their semantics and activation patterns. For instance, if a target feature activates on tokens like *“the, a”*, while its predecessor in the residual stream fires on *“the, The”* and the corresponding MLP feature activates on *“a, A, an”*, this suggests the target feature emerges from their interaction—we categorize it as *“processed.”* Features with nearly identical descriptions and activation patterns across layers likely belong to the *“translated”* group, indicated by high $s^{(R)}$ values. Conversely, features without semantic ties to the residual stream but aligned with MLP or attention counterparts may be *“newborn”*—new linear directions introduced by those modules. Expanding this analysis beyond top-1 matches (e.g., to top-5) could further enhance its reliability.\n\nFigures 3 and 18b illustrate clusters in the top-left corner, where features lack $s^{(R)}$-similar predecessors in the residual stream. This could imply either new feature introduction by the MLP (e.g., stored knowledge) or transformation of an initially dissimilar residual feature via MLP processing. In contrast, Figure 5 shows a clear predominance of the ”From RES*”* group, suggesting most features propagate across layers with minimal modification.\n\nWhile this analysis could be automated, it hinges on either reliable feature explanations or raw activation data.\n\nThe second approach tests feature roles through steering experiments. For example:\n\n1. Steering only the target feature,\n2. Steering its residual predecessor, or\n3. Steering the residual predecessor while deactivating active MLP/attention predecessors.\n\nComparing these setups could reveal mechanistic differences, though we have not yet implemented this systematically. Preliminary versions of (a) and (b) appear in Section 5.3, though (b) in our current work involves steering the entire flow graph, not just the residual stream.", "claims_and_evidence": "The main methodological proposals/claims made by the paper involve:\n1. cosine similarity based feature matching (section 3.2), \n2. tracking the evolution of features across modules (mlp, res, att) (section 3.3), \n3. 4 heuristics for understanding patterns seen over similarity scores for features/modules (translating, processing, new born and not explained by method) (3.3), \n4. discovering long range feature flows by composition of short range matching over consecutive layers (section 3.3)\n5. Identification of linear feature circuits /  flow graphs ( section 3.4)\n6. Model steering at the flow graph level\n\nThe points overall are well motivated and backed up, however points 1, 3, and 5 lack some supporting evidence in the main body of the paper that could help with clarity.  \n\nPoint 1, section 3.2 is quite clear and easy to follow, however the corresponding results section 5.1 “Identification of feature predecessors” was less so.  \n\n(Q1) For Fig 4, what are the raw counts of these groups before looking at differences?  its hard for me to understand 4 in general and that may help, but from this figure i mostly see that most groups similarity scores are stats sig different (with the slight exception of ATT where AB group has lower s(A) , but still above 75%).  \n\n(Q2) In Lines 316-19 how can you tell if features are emergent from Fig 5 alone? In terms of “propagating from preceding layers” do you mean because “From Res” goes up?  it might help to have a guide to reading this graph to make it more clear.\n\n(Q3) Lines 326-329 make references to differences in dataset in latter layers, but its hard to tell from graph alone of any differences as they look relatively the same to me?  \n\n(Q4) LlamaScope is mentioned in the models section 4.1 and mentioned in one line in section 5.1 (which itself points to Figure 18b in the appendix).   Because its relegated to the appendix, i didn’t notice that the  Gemma and Llama are quite different over all ( see From Res relationship to other lines in both for instance).. How is this difference explained ?   Discussion comparing findings on these two models and their SAEs should be expanded as they seem fairly central to any generalization claims for point 1\n\n(Q5) Throughout, its appreciated you include a “not explained” heuristic ( heuristic D in section 3.3), but discussion on what might be causing “from nowhere” to be the highest scoring pattern observed throughout in Fig 5 and Fig 18b?\n\n\n\nPoint 3, in 3.3 in describing the heuristics section for phenomena that can be explained, the phrases used are “the feature likely exists …”, “the feature was likely processed ..”, and “the feature may be newborn ….” , and the experiment results assume these heuristic short hands to be true in a way that is difficult to assess.  (Q6) Is there anyway to validate these short hands?   \n\nPoint 5 ( section 3.4 ) is said to be validated in experiments and Appendix E while the appropriate experiment setup section 4.2 “deactivation of features” which I think allows for casual circuit claims says to look at Appendix A for experimental matching strategies and metrics quantifying effectiveness.  The appropriate results section ( 5.2 deactivation of features ) makes reference to rescaling coefficient r ( not mention in the main body of the paper ).  (Q7) For clarity the appropriate context/setup needs to be introduced in the main body of the work and towards that end the findings related to Figure 7 (which while interesting don’t seem central to the papers main claims) seem like they could be moved to the appendix and you could use the space gained to give more setup for deactivation in 4.2?   \n\n(Q8) Its unclear how Figure 10 shows that cumulative intervention outperforms the single layer approach?  It seem reasonable to say that for layers 18 on this true, but not for early ones?  Is my understanding correct?  If not, this should be expanded/clarified.  \n\n(Q9) Also in the caption of Figure 10, the orange and blue lines are said to be multi-layer interventions while in the legend of the Fig 10 (RHS) blue is said to be One Layer?  \n\n(Q10)  In Figure 11  ( single, constant, exponential, linear ) steering approaches are shown without being explained in the main body of the paper?  Precise details are fine for the appendix, but setup and metrics needed to understand figures in the main body of the paper should be explained in the main section so that the paper is self contained.\n\n(Q11) In reference to lines 420 and 421, “We conclude that multi-layer interventions indeed affect the model more than single-layer approaches “, given my questions on Fig 10 and Fig 11 this is not very clear to me.\n\nNit:\nLine 191, R == R_L-1 was a little confusing as a convention to me. its explained clearly later so maybe just use R_L-1 alone instead?\n\nOverall the method is reasonable and the findings are interesting, but it seems selecting the more important findings to highlight and focus on and moving some of the others out to the appendix for space could benefit clarity.", "methods_and_evaluation_criteria": "Proposed and evaluation criteria are mostly reasonable.  See Claims And Evidence section on questions and suggestions for how to improve clarity.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Proposed and evaluation criteria are mostly reasonable.  See Claims And Evidence section on questions and suggestions for how to improve clarity.", "relation_to_broader_scientific_literature": "This work introduces a novel interpretable data-free method for multi-layer steering, which enables the tracking of concept evolution across layers and the identification of computational circuits through targeting the weights of pretrained SAEs.", "essential_references_not_discussed": "None outside of the 4 month window for concurrent works that I know of.", "other_strengths_and_weaknesses": "This work introduces a novel interpretable data-free method for multi-layer steering, which enables the tracking of concept evolution across layers and the identification of computational circuits through targeting the weights of pretrained SAEs.  While there are some issues with clarity, overall the method could allow for expanded analysis of models leveraging SAEs at th moment.\n\nSee Claims And Evidence section on some improvements needed for clarity and setup of some experiments.  Future work would need to address newer work (https://arxiv.org/abs/2501.17727)  showing SAEs can interpret randomized vectors and the need to tie them to downstream performance in order for them to be grounded.", "other_comments_or_suggestions": "See Claims And Evidence section for comments/suggestions.", "questions_for_authors": "See Claims And Evidence section on questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ROP0mXYnpw", "reviewer": "Reviewer_vBgp", "summary": "The paper introduces a new approach to systematically map features discovered by SAEs across consecutive layers of LLMs. By using a data-free cosine similarity technique, the authors trace how specific features persist, transform, or first appear at each stage. This method yields granular flow graphs of feature evolution, enabling fine-grained interpretability and mechanistic insights into model computations.   \n\nThe authors demonstrate how these cross-layer feature maps facilitate direct steering of model behavior by amplifying or suppressing chosen features, achieving targeted thematic control in text generation. Key contributions are threefold:\n- Cross-Layer Feature Evolution: Using pretrained SAEs that can isolate interpretable monosemantic directions, the authors utilize information obtained from cosine similarity between their decoder weights to track how these directions evolve or appear across layers. \n- Mechanistic Properties of Flow Graph: By building a flow graph, the authors uncover an evolutionary pathway, which is also an internal circuit-like computational pathway, where MLP and attention modules introduce new features to existing ones or change them.   \n- Multi-Layer Model Steering: The authors show that flow graphs can improve the quality of model steering by targeting multiple SAE features at once, and also offer a better understanding of the steering outcome.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and questions.\n\n> 1. Some layers share less cosine similarity with others, e.g., the first layer; does this affect the mapping/steering?\n\nWe observe that layers indeed vary in feature-matching quality, though we have not yet quantified this systematically (e.g., via similarity analysis of feature descriptions). Your suggestion to assess this further is valuable, and we will incorporate such analysis to strengthen our paper.\n\nNotably, we find that certain layers (e.g., the 4th, 9th, and 18th in Gemma) exhibit differences in how well features match their top-1 similar counterparts in the MLP versus the previous residual stream. This aligns with feature clustering patterns, as shown in Figures 3 and 18b: early layers lack strong cluster separation, leading to poorer matching, while later layers converge toward a single dominant cluster with less distinct groupings.\n\nhttps://anonymous.4open.science/r/icml_rebuttal-5064/out_mlp_scatters.png\n\nhttps://anonymous.4open.science/r/icml_rebuttal-5064/out_attn_scatters.png\n\nWe have not identified generalizable differences in steering effectiveness across layers, as our analysis so far has focused on a limited set of features and graphs. However, we note that steering MLP or attention features often produces negligible effects, regardless of the target layer or steering coefficient. We hypothesize that this could stem from imperfect matching or the possibility that these features operate differently (e.g., requiring other active features in the residual stream to function). Further investigation is needed here.\n\n> 2. More charts/graphs about the dynamics of cosine similarity?\n> \n\nWe have measured the mean cosine similarity between hidden states at layer outputs and three SAE-related positions, plotted against relative layer distance:\n\nhttps://anonymous.4open.science/r/icml_rebuttal-5064/cossim_models.png\n\nThe divergence in Pythia’s results may arise from its parallel residual stream architecture: $\\text{Layer}(x)=x+\\text{MLP}(\\text{norm}_1(x))+\\text{Attn}(\\text{norm}_2(x))$. We hypothesize that higher cosine similarity between hidden states correlates with better matching quality, though this depends on SAE and model architectures—an aspect we plan to explore further.\n\nThese findings may also explain why steering MLP and attention features is less impactful than steering residual features. For additional results on correlation matching and group dynamics in Pythia and GPT-2, please see our response to Reviewer jyW6.\n\n> 3. Can the co-existence of activation values or the transference be further evidence of the evolution claims?\n> \n\nYes. If you refer to transference across forward layers, we find that matching via the residual stream generally works well, whereas matching with MLP or attention features in the next layer is often poor. We have prepared example graphs of activations for specific texts, available at.\n\nhttps://anonymous.4open.science/r/icml_rebuttal-5064/g_3_t_0.5.png  (g_3_t_0.8.png/g_3_t_0.85.png)\n\n> 4. How about the sensitivity of the steering results to different threshold values?\n\nThe steering effect varies significantly depending on the graph and target feature. For instance, steering the \"London\" feature barely influences London-related tokens unless the coefficient is very high, at which point it introduces fashion-related terms (see [1] and Appendix E). Some features even show minimal response to steering.\n\nThus, different graphs react differently, and a systematic study of threshold sensitivity would require extensive experimentation across many features and setups. While we did not aim to optimize thresholds (given their dependence on model/SAE architecture, SAE quality, etc.), we acknowledge this as an important direction for future work.\n\nFor our steering experiments:\n\n- Deactivation did not use thresholds, i.e. full graphs from 0th to 25th layers were used.\n- Activation thresholds were chosen empirically: higher thresholds sometimes diluted the theme by including weakly related features, while lower thresholds reduced theme prominence.\n\n> 5. How was the threshold selected?\n\nInitially, we derived thresholds from feature group analysis (Section 5.1) and correlation studies. For example, in Gemma’s 18th layer, we colored features with >0.65 correlation to MLP (orange) or residual (blue) features, then applied linear separation:\n\nhttps://anonymous.4open.science/r/icml_rebuttal-5064/thresholds.png\n\nFor experiments, we determined thresholds $s^{(R)}=0.5$ and $s^{(M, A)}=0.15$ by computing feature activations across texts, building graphs with active features highlighted, and by  manually examining those graphs for threshold values that preserved semantic consistency and co-activation of features in graphs. While expert input remains helpful, we believe this process could be automated in future work.\n\n[1] https://arxiv.org/pdf/2411.02193", "claims_and_evidence": "From my perspective, the following are claims made by the authors (and their evidence), with my personal reasons to mark them as convinced or not:\n\n1. **The authors believe that using a data-free cosine similarity technique can trace how specific features persist, transform, or first appear at each stage.**\n    \n    Evidence: some previous (Not really convinced)_\n    \n    From a **theoretical** perspective, this seems reasonable, as previous research has claimed high cosine similarity in nearby layers [1] [2]. Since how polysemantic hidden states are inherited with high similarity, it is reasonable to claim that the linear feature, as its components, are inherited to the next layers. However, from an experimental perspective, _I think the authors should give more illustrations of the cosine similarity results and analysis of features that are mapped to be the same. \n    \n2. **The four evolutionary pathways with cosine similarity.**\n    \n    (Relatively convinced) The clarification of four evolution paths, i.e., translated, processed, newborn, and not related, does in some way make sense. _However, I am curious about the activation patterns. I think the co-existence of activation values or the transference can be further evidence of these claims. e.g., if the feature of \"Paris\" is activated in two layers with the position predicted with the cosine similarity, this will verify the conclusion of this evolution pathway._ \n    \n3. **Identification of linear feature circuits**\n    \n    (Convinced)\n    \n4. **The authors observe that two groups may differ with respect to sP if module P is active only in one group (and indistinguishable if P is active or inactive in both groups).**\n    \n    (Convinced)\n    \n5. **Differences arise mainly when a residual predecessor combines with another module, indicating that we might miss other types of causal relations.**\n    \n    (Convinced?)\n    \n    _I think this may also come from the normalizations and the relatively low cosine similarity between Res - MLP/Attn._\n    \n6. **Deactivating a single predecessor causes a greater activation strength drop if it is a group with a single predecessor, which may indicate circuit-like behavior of combined groups.**\n    \n    (Convinced)\n    \n7. **Different groups react differently to rescaling. Positive rescaling (boosting active features) matters most when residual features mix with MLP or attention. Negative rescaling most strongly affects \"From RES.\"**\n    \n    (Convinced)\n    \n8. **Multi-layer intervention outperforms single-layer steering of the initial feature set, reducing hyperparameter sensitivity.**\n    \n    (Convinced)\n    \n9. **Removing topic-related information early allows later layers to recover general linguistic information, aligning with the ability of LLMs to \"self-repair\" after \"damaging\" the information processing flow by pruning or intervention into the structure of hidden states.**\n    \n    (Relatively Convinced)\n    \n    I think the conclusion may be right, _but I think a more controllable experiment is needed, following the contrastive construction method from CAA.\n\n[1] https://arxiv.org/abs/2403.17887\n[2] https://openreview.net/forum?id=XAjfjizaKs", "methods_and_evaluation_criteria": "Based on my understanding, the methods and evaluation criteria proposed in the paper seem to make sense for the problem and application at hand. Here's a breakdown:\n\n**Problem:** The paper addresses the challenge of interpreting and controlling the behavior of LLMs. Specifically, it aims to: (1)**Improve interpretability** (2) **Enable steering**.   \n \n**Proposed Methods:**\n- **Data-free cosine similarity:** This method is used to track the evolution of features across layers. As discussed in Claims And Evidence 1, I think the authors should give more illustrations of the cosine similarity results and analysis of features that are mapped to be the same. \n- **Feature flow graphs:** These graphs visually represent how features evolve across layers. This is a helpful tool for providing a clear and intuitive way to understand the complex interactions between features.\n- **Multi-layer steering:** This technique leverages the feature flow graphs to control model behavior by intervening on specific features across multiple layers.", "theoretical_claims": "I have checked the correctness of proofs about the feature matching, evolution of feature, and activation of theme. I believe they are theoretically correct.", "experimental_designs_or_analyses": "Based on my review, there are some points that could be strengthened or clarified:\n1. The paper relies heavily on cosine similarity to map features across layers. The choice of threshold for determining whether two features are related could be further justified. I suggest the authors could explore the sensitivity of their results to different threshold values and provide more analysis on how the threshold was selected.\n2. The paper demonstrates successful steering in specific scenarios. However, it would be helpful to explore the generalizability of the steering method to a wider range of tasks and topics.", "relation_to_broader_scientific_literature": "1. The method provides a straightforward way to identify and interpret the computational graph of the model without relying on additional data.\n2. To my acknowledgement, the authors are the first to use SAE features from different layers to control LLM generation.\n3. The improved controllability has positive implications for alignment, interpretability, and safe deployment of AI systems, as it can allow developers to steer models away from harmful or biased outputs", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "I have no further suggestions to this paper.", "questions_for_authors": "1. Some layers share less cosine similarity with others, e.g., the first layer; does this affect the mapping/steering? \n2. More charts/graphs about the dynamics of cosine similarity?\n3. Can the co-existence of activation values or the transference be further evidence of the evolution claims?\n4. How about the sensitivity of the steering results to different threshold values? And\n5. How was the threshold selected?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "slWWYzF1zh", "reviewer": "Reviewer_t2gC", "summary": "This paper studies sparse autoencoders (SAEs) trained on different layers and modules (residual stream, MLP and attention), proposes using cosine similarity to locate predecessor features in the previous layer given any target feature represented by SAE decoder weights. Through this approach, this paper traces how SAE features evolve throughout the model, and constructs a flow graph that connects features in different layers that allows for effective multi-layer cumulative steering for text generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for valuable questions.\n\n> To further validate the goal in Section 3.3 of “tracking the evolution of feature”, it would be nice to improve this step by ensuring the features considered are rarely activated on other datasets so they are tailored to the examined distribution.\n> \n\nWe appreciate this suggestion. To address feature specificity, we estimated activation frequencies across 100k non-special tokens from FineWeb and categorized features into quantiles based on their activation rates. As expected, the results vary significantly depending on the quantile selected, supporting the need for careful feature selection.\n\nhttps://anonymous.4open.science/r/icml_rebuttal-5064/quantiles.png\n\n> It is unclear whether there are preceding features that are co-activated with the target feature with high probability, **but cannot be identified via cosine similarity search.**\n> \n\nTo evaluate this, we calculated the following score: the fraction of features for which the top-1 match by cosine similarity also appears in the top-*k* matches by Pearson correlation.\n\nhttps://anonymous.4open.science/r/icml_rebuttal-5064/top_k_corr.png\n\nhttps://anonymous.4open.science/r/icml_rebuttal-5064/top_k_gpt.png\n\nhttps://anonymous.4open.science/r/icml_rebuttal-5064/top_k_pythia.png\n\nWe observe strong agreement for residual features in Gemma and GPT-2, while MLP and attention features show lower consistency. However, we have not yet validated whether Pearson-based matches reliably predict causal relationships or steering outcomes—a valuable direction for future work, and we thank you for highlighting this.\n\n> is it possible that a similar feature is reactivated again in later layers, even though it appears to be eliminated when examining the current target feature?\n> \n\nYes, this is very common. Since deactivation typically applies only to a small subset of features, and rescaling coefficients are often modest, hidden states undergo only minor perturbations, allowing for later reactivation. We hypothesize this stems from the model’s self-repair mechanisms, which can recover information even after layer pruning (cf. [1–3]). To mitigate this, effective deactivation may require not just zeroing target features but also adding another steering vectors to the hidden state to prevent such recovery (discussed in Appendix C.2). \n\nTo test the occurrence of reactivation, we used three strategies: deactivating only the first layer in a graph, deactivating a random layer in a graph, deactivating the first half of layers in a graph. Subsequent deactivations were measured on residual nodes that comes after the deactivated layer, confirming that reactivation is present, but it depends on intervention method and strength.\n\nhttps://anonymous.4open.science/r/icml_rebuttal-5064/reactivation.png\n\n> From Figure 10, how could the conclusion that “cumulative intervention outperforms the single-layer approach” be drawn, given that the blue curve shows a better best score than the orange curve?\n> \n\nThank you for highlighting this. Cumulative steering generally achieves comparable effects at *lower rescaling coefficients* (Figure 10), improving stability. While single-layer steering can yield higher total scores, it requires either precise selection of highly impactful layers, or spanning multiple layers with a well-constructed feature graph.\n\nThis introduces additional optimization challenges not faced by cumulative steering. We will revise this conclusion in the updated manuscript to reflect this nuance.\n\nReferences:\n\n[1] [Your Transformer is Secretly Linear] (https://arxiv.org/abs/2405.12250)\n\n[2] [What Matters in Transformers? Not All Attention is Needed] (https://arxiv.org/abs/2406.15786)\n\n[3] [The Hydra Effect: Emergent Self-repair in Language Model Computations] (https://arxiv.org/abs/2307.15771)", "claims_and_evidence": "Yes. However, I have the following concerns regarding certain parts of the paper:\n- **Section 5.1, Identification of feature predecessors.** The authors illustrate the (statistical) difference between each group of features, categorized by how each feature is co-activated with its predecessors. From Figure 5, it is noted that among the target features examined, **about 60-70%** can be explained as originated from RES or being “created” via MLP/Att. However, I have concerns about this estimand as follows.\n    - (1) The target feature set is identified through random sampling (Appendix A.1), which might include features that are generally activated but not specific to the dataset considered. To further validate the goal in Section 3.3 of “tracking the evolution of feature”, it would be nice to improve this step by ensuring the features considered are rarely activated on other datasets so they are tailored to the examined distribution. This might also explain why the curves look similar to each other albeit obtained from drastically different datasets.\n    - (2) It is unclear whether there are preceding features that are co-activated with the target feature with high probability, **but cannot be identified via cosine similarity search.** Thus, it is necessary to examine all preceding features to identify candidates with the highest frequency of co-occurrence with the target feature, and verify if these features truly correspond to the ones determined by the cosine similarity search.\n- **Section 5.2, Deactivation of features.** My concern is whether a cascading effect of deactivation could exist, given the discussion in Section 2.3 that “most features in the residual stream remain relatively unchanged across layers”. Specifically, is it possible that a similar feature is reactivated again in later layers, even though it appears to be eliminated when examining the current target feature? My suggestion is to check the downstream features in the constructed flow graph, to determine if such an event occurs.\n- **Section 5.3, Model steering.** From Figure 10, how could the conclusion that “cumulative intervention outperforms the single-layer approach” be drawn, given that the blue curve shows a better best score than the orange curve?", "methods_and_evaluation_criteria": "Yes. The proposed use of cosine similarity has already been adopted for clustering SAE features (e.g., [1]), and in this paper it is taken to examine feature evolution across different layers. The evaluation of deactivation and steering follows the intervention standard of SAEs in current literature.\n\n[1] Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes. For questions, see claims and evidence.", "relation_to_broader_scientific_literature": "This paper provides evidence on how SAE features evolve through different layers. The enabled automatic similar feature search (flow graph) and the following intervention proposal could enhance the quality of model steering and contribute to a better understanding of these models.", "essential_references_not_discussed": "I have not found essential missing references.", "other_strengths_and_weaknesses": "The evaluation is comprehensive, and the idea of tracing the evolution of features through SAEs is novel to date.\n\nThough cumulative intervention is claimed to be beneficial, I don’t see the effect significantly, see claims and evidence.", "other_comments_or_suggestions": "- Section 2.2 line 055: wrong reference to transcoder\n- Section 3.2 line 144: should the topK operation be executed per row, instead of been applied globally?\n- Caption in Figure 10: the blue curve is not a multi-layer steering method.\n\nIn addition, I strongly recommend that the authors move some essential information from Appendices A and B into the main text. The authors emphasize from the beginning that the technique is “data-free”, without mentioning how the features are selected given a dataset of interest in the main text. Additionally, there are technical details omitted in the main text that are required to understand the experimental results. For example, the labels of “random” and “permutations” and the $r$ parameter in the deactivation experiment and the “cumulative” terminology. For my other concerns, please refer to claims and evidence.", "questions_for_authors": "Please see claims and evidence.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors introduces a method for analyzing and manipulating the evolution of SAE-derived features across layers in large language models. By aligning sparse autoencoder features using cosine similarity and constructing flow graphs, the authors provide a framework for tracking feature propagation, identifying circuit-like structures, and performing multi-layer model steering. The experiments show that interventions guided by these flow graphs can enhance or suppress specific themes in generated text while preserving coherence.

The motivation is well grounded, and the use of data-free methods for inter-layer feature matching is a strength. The paper also makes a solid empirical case that steering based on multi-layer feature tracking can improve control over model generations, with applications in interpretability and alignment. That said, some reviewers raised concerns about novelty and generality. While the results are promising, the feature matching and steering procedures extend known ideas (e.g., cosine similarity and SAE-based control) rather than introducing fundamentally new techniques. Evaluation is primarily focused on a single model family, and it’s unclear how broadly the results generalize.

After considering the reviews and paper, I believe the positives outweigh the negatives for this paper and believe the work advances our understanding of cross-layer interpretability and model control.  I recommend the authors take the reviewer comments and my notes above into account in their revisions.
Decision: Accept (poster)