Evolution of Concepts in Language Model Pre-Training
Authors: Xuyang Ge
Wentao Shu
Jiaxing Wu
Yunhua Zhou
Zhengfu He
Xipeng Qiu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model; Pre-Training; Mechanistic Interpretability; Training Dynamics; Crosscoder
Primary area: interpretability and explainable AI
TL;DR: We track linear interpretable feature evolution across pre-training snapshots using a sparse dictionary learning method called crosscoders.

Abstract: Language models obtain extensive capabilities through pre-training. However, the pre-training dynamics remains a black box. In this work, we track linear interpretable feature evolution across pre-training snapshots using a sparse dictionary learning method called crosscoders. We find that most features begin to form around a specific point, while more complex patterns emerge in later training stages. Feature attribution analyses reveal causal connections between feature evolution and downstream performance. Our feature-level observations are highly consistent with previous findings on Transformer's two-stage learning process, which we term a statistical learning phase and a feature learning phase. Our work opens up the possibility to track fine-grained representation progress during language model learning dynamics. Our code is available at https://github.com/OpenMOSS/Language-Model-SAEs.

Published as a conference paper at ICLR 2026
EVOLUTION OF CONCEPTS IN LANGUAGE MODEL TRAINING
Xuyang Gel, Wentao Shu Jiaxing Wu', Yunhua Zhou', Zhengfu Hel, Xipeng Qiu
OpenMOSS Team, Shanghai Innovation Institute; Fudan University
$ Shanghai AI Laboratory
xyge 24 24 @m. fudan. edu. en, zfhel 90 wedef.daeeeee eel.dannedu.ccm
Initialization Features
Statistical Learning Phase
Feature Learning Phase
Features activating spacific tokens exist nntializatonn
Single-toker Festure
Praylyk said socalat Dam
ring drums stall. The seeal processing an
centre stage vittuosity of
Learning
Provious Taken Feature
During training, regularibies
RDER
odes that "TTon Legocy"
Wrap Tool Window mi Kit Wrap Stick
5507 Dueueu
Feature e Emergence
Concept Magazine
Most features begin to emerge at this point
oolment startup, told Education Dive, HHH
Iginall published by Techrellog/sseee
founding editor Watch,
Complex Features
Temporal Relationships
Complex features (e.g, sentence structures) form later training stages.
L..SON-- like stringg The resulting astinng is then directie
packages, produce compost product
Training Snapshots
ABSTRACT
Language models obtain extensive capabilities through pre-training. However, the pre-training dynamics remains black box. In this work, we track linear interpretable feature evolution across pre-training snapshots using a sparse dic- tionary learning method called crosscoders. We find that most features begin to form around specific point, while more complex patterns emerge later training stages. Feature attribution analyses reveal causal connections between feature evolution and downstream performance. Our feature-level observations are highly consistent with previous findings Transformer's two-stage learn- ing process, which we term a statistical learning phase and a feature learning phase. Our work opens up the possibility track fine-grained representation progress during language model learning dynamics. Our code is available at htttss///iiihubb..cm//eeeeeoooo
1oriealageMMde1----- Model-SAa SAEs
INTRODUCTION
Pre-training RRaffrdd et al. 2018 Devlin et al. 2019) has emerged as the dominant paradigm for de- veloping frontier large language models (LLM) (OpenAT 2023 Grattafiori et al., 2024 Yang al.)
2025). Despite its remarkable success, the pre-training process remains largely black box. While scaling laws (Hestness et al.] 2017]] Kaplan et al.) 2020 Bahri et al.] (022)) reveal the predictable relationships between compute, data, and loss, they offer limited insight into the internal reorgani- zation occurring within the model parameters. Other theoretical frameworks on learning dynamics, including neural tangent kernel (Jacot al. 2018). information bottleneck theory (Tishby & Za-
slavsky 2015 Shwartz-Ziv & Tishby 2017) and singular learning theory (Watanabe) 2009 Lau
"Corresponding author.

---

Published as a conference paper at ICLR 2026
provide high-level explanations why observe gener- alization and grokking PPwerr et al. 2022, Nanda et al., (023). However, a fundamental question how do LLMs actually develop their capabilities internally during pre-training has remained largely opaque.
al. 2024 Wang al. 2025a)
Recent advances in mechanistic interpretabilitty particularly through dictionary learning methods based on sparse autoencoders (SAEs), have begun to illuminate the internal representation of neural networks (Bricken et al., 2023 Templeton et al., 2024 Gao et al. 2025). By disentangling the phenomenon of superposition (Elhage et al. 2022), sparse autoencoders show their capabilities in extracting millions of features from LLM activations, demonstrating that LLMs encode human- interpretable concepts as linear directions in their activation spaces. However, these analyses have predominantly focused on studying fully-trained models. Consequently. the process of how features initially emerge and evolve throughout the training process remains largely unexplored.
In this paper, we propose to track feature evolution across pre-training snapshots using cross- coders (Lindsey et al. 2024), variant of sparse autoencoders designed to simultaneously identify and align features from a family of correlated model activations. While originally introduced to resolve cross-layer superposition and track features distributed across layers, we adapt this approach to analyze activations from different training checkpoints. By applying cross-snapshot crosscoders, observe where features emerge, rotate, and degenerate, thereby providing deeper insight into the internal dynamics of model pre-training. Our main contributions are as follows:
1. To the best of our knowledge, this work is the first to adapt crosscoders to study training dy- namics (Section B) We evaluate completeness and faithfulness of interpretations provided by crosscoders in Appendix
2. We perform in-depth analyses on cross-snapshot features in Section 4 the decoder norms can serve as proxies of feature evolution status, and showcase both general and per-feature evolutionary properties.
We empirically show that
3. Our method successfully connects the microscopic features to the macroscopic downstream task metries using attribution-based circuit tracing techniques (Section 5
4. We show evidence for the phase transition from statistical learning phase feature learning phase in Section
2 RELATED WORKS
Learning dynamics and phase transitions. Multiple theoretical frameworks explain neural net- work training dynamics. Neural Tangent Kernel (NTK) theory (Jacot et al., 2018) establishes that infinite-width networks evolve as kernel machines with fixed kernels during gradient descent, with extensions finite-width corrections (Dyer & Gur-Ari 2020) and modern architectures (Yang) 2020 Yang & Littwin| 2021). Recent work identifies phase transitions and lazy regimes in training dynamics Kummar al. [024] Zhou 2025). Information Bottleneck (IB) theory (Tishby & Za-
slavsky 2015) formulat lates deep learning optimizing compression- prriiictioneeiiiionno tradeoffs. [Swwartz- Ziv & Tishby (017) empirically demonstrates two distinet training phases-inite fitting followed aligns with our findings in Section 6 Singular I Learning Theory (SLT) treats neural networks as singular statistical models. WWatnaabe (1999: 2009) introduces the Real Log Canonical Threshold to provide geometric complexity measures predicting phase transitions, with recent advances enabling practical estimation at scale through Local Learning Coefficients (Lau et al. 2024 Furman & Lau 2024 Wang al.) 2025a).. Our work complements these theoretical frameworks by providing detailed mechanistic accounts of feature evolution during transformer pre- training, bridging high-level theory with empirical observations.
by compression-wiiee
Sparse dictionary learning. The superposition hypothesis posits that models use linear repre- sentations (Bengio et al.) 2014 4 Alain & Bengio 2017, Vargas Cotterell 2020) to embed more features than neurons (Olah et al. 2020 |020] et al. 2022). Sparse Autoencoders (SAEs) (Bricken) et al., 2023 Huben et al., 2024| He et al.) (024) extract monosemantic features from superposi- tion using sparse dictionary learning, with subsequent improvements and scaling efforts (Gao al. 2025 Rajamanoharan et al. 2024). Ge et al. (2024) and Dunefsky et al. (222)) propose transcoders, SAE variant that predicts future activations improve eircuit tracability. Recently, [indsey al.
2

---

Published as a conference paper at ICLR 2026
2024 introduces crosscoders to simultaneuusly read and write to multiple layers. Mishra-Shama et al. (2024); Minder et al. (2025) leverages crosscoders for capturing the difference between pre- traned models and their chat-tuned versions.
3 TRACKING THE EVOLUTION OF FEATURES
Collecting Activations
Decoder Norms
The Crosscoder trained to simultaneously decompose the activations at a given position and given input - of multiple pre-training snapshots.
Crosscoder decoder norms reflect the evolution of features throughout pre-training.
Form initialization
Emerge at mid-training
Training Progress
Layer i+1
DOOO
"Jump" Feature
Singular Verb
Layer
ah, R''s nice to juun ind
symphory bulcs as she Guy Peerce spiils from wile versity. who works
Snapshot 0
1ot
hot
3
4
hot
the trick too, juumpstarting
Interpretation
Input: The fox jumps over the lazy dog
Features are interpreted by their top activations.
Figure 1: Overview of method. The crosscoder is trained to decompose activations of multiple pre-training snapshots (left) into sparse features (right).
Crosscoder architecture. For a given text corpus C and a family of training snapshots O saved during I.IM pre-training, let A {a C
+) denote corresponding family of parameterized functions mapping input datapoints to model activations at a specific layer. Each datapoint I (c,j) C x N is training token in indexing the j-th token sequence
TR dRoie
A cross-snapshot crosscoder (Lindsey et al. 2024) operates on the parameterized function family A over corpus C (Figure D. The crosscoder architecture is defined by:
f(x) Σ w. a + bene DEE
(I)
à (x) W Weee (x) + 1g
with W E [ Xdmmon benc R Ritattmm
W E R. D and Ug Ridme The parameters dec
w W eac- dec. and b% correspond to snapshot-specific encoder and decoder weights for parameter 0. The activation function o(-) produces sparse feature activations f(x shared for all snapshots, and â" (z) denotes the reconstructed term of model activation a (z)).
Training objectives.
The crosscoder is trained to minimize the loss:
((x)=
Σ ||a (α) à (x)1 860
Asparsity
Σ
Σ i=1
( dec
DE
Reconstruction loss
Sparsity loss
where Asparsity is a hyperparameter to control the trade-off between reconstruction fidelity and feature sparsity. The regularization function ?(-)) serves as differentiable substitute for LO regularization, penalizing non-sparse feature activations. We include the decoder norm | the regulariza- tion term to prevent the crosscoder from trivially reducing the feature activation f. while inflat- ing the decoder norms under imperfect L.0 approximations such as LI regularization. Appendix A.

---

Published as a conference paper at ICLR 2026
COADAA 5A0
0.023
1,0 Namm
90
0.052
0.022
20-
10
1e000 62000 127030
1995 6200
18005 mnico 12700
Faining Step
Traising Stop
(a)
(b)
(c)
Figure 3: Comparison between crosscoders and per-snapshot SAEs. (a) The explained variance of crosscoders versus SAEs at each snapshot. (b) The 1.0 norm of crosscoders versus SAES at each snapshot. (c) The Pareto frontier comparison of crosscoders and SAEs trained the final snapshot.
shows the details for selecting the proper activation function a(-) and regularization function 12(-) to optimize crosscoder feature sparsity.
Experimental setup. We use Pythia-160M and Pythia-6.9B (Biderman et al. 2023) for our ex- periments throughout this work. Pythia is Transformer language model suite with well-controlled training settings and accessibility to training snapshots. We select the middle layers (Layer 6 of Pythia-160M, and Layer 16 of Pythia-6.9B) for training crosscoders. To balance training cost with the granularity of feature evolution analysís, we select 32 snapshots out of 154 open-source snap- shots ranging from step 0 143,000, with stratified sampling approach: (1) all 20 snapshots before step 10,000 to capture early feature evolution with maximum temporal resolution, and (2) 12 evenly spaced snapshots from later training stages, containing snapshots from steps 14,000-34,00 and 8 snapshots from steps 47,000-1433 143,000. We use SlimPajama (Shen et al. 2023). comprehensive text dataset covering variety of data sources to sample activations.
Results. Figure demonstrattss crosscoder performance decomposing model activations into sparse feature rep- resentations. We evaluate reconstruction quality (mea- sured by activation variance explained) and sparsity (mea- sured by LO norm averaged across snapshots). Increasing dictionary size Neeuuures yields significant Pareto improve- ments across both metrics.
0.85
Pycra
0.84
To examine whether crosscoders can extract cross- snapshot features and align identical features in consistent directions within the feature space, we compare cross- coder performance against corresponding SAEs trained individually on each Pythia snapshot. We train SAEs us- ing identical settings and hyperparameters as crosscoders on each snapshot. Figures and 9b demonstrate that crosscoders achieve comparable performance in LO norm and explained variance each snapshot, even when SAEs are optimized for individual snapshots where they should have theoretical advantage.
50
50
70
80
50
100
11.
120
Norm
Figure 2: Explained variances versus LO norms our crosscoders.
We further compare the Pareto frontiers (explained variance versus L.0 norm) between crosscoders and SAFs trained on the final snapshot. Figure shows that crosscoders exhibit slightly superior Pareto frontier, demonstrating their effectiveness in sparse dictionary learning beyond their primary capability of tracking feature evolution across snapshots.
Feature evolution revealed by crosscoders. An important advantage of crosscoders is the unified feature space (or sparse codes) they reveal. The crosscoder encoder aggregates cross- coassst infor- mation to produce shared feature activations. Then these activations are translated back to recover the original activations by group of independent decoders (or dictionaries).
4

---

Published as a conference paper at ICLR 2026
If feature activates but "exists" only subset of snapshots, the sparse penalty will suppress the decoder norms of this feature at irrelevant snapshots to near-zero so they won't interfere with reconstruction these snapshots and also reduce sparsity loss.
This design principle leads to a crucial observation: the decoder norm ||Wac,;l directly reflects the strength and presence of feature at snapshot Therefore, tracking decoder norm changes across snapshots provides direct window into feature evolution dynamics. Appendix provides a sanity check of whether [Waee,il can indeed serve as a proxy of feature strength using linear probes.
Potential failure modes crosscoder feature alignment. Before assessing crosscoder features, further discuss two theoretical concerns about cross-snapshot feature alignment:
Will crosscoders misalign unrelated features? Such misalignment would strongly contradict the optimization objective of crosscoders. The distinct activation patterns of unrelated features would conflict when forced into the same dimension. Once a feature activates, would cause others to activate simultaneously, introducing noise in the reconstruction. In terms of results, misalignment would lead to the polysemanticity of crosscoder features, damaging the consistency of their inter- pretation, which our interpretability evaluation demonstrates does not occur.
Will crosscoders split shared features? Suppose features from different snapshots represent the same underlying concept with identical activation patterns, splitting them across multiple dimen- sions would be suboptimal: it wastes representational capacity in feature space, as both previous works (Gao et al.] 2025; Templeton et al.] (2024) and Appendix A.4 prove that feature space di- mensionality is crucial for better reconstruction fidelity. Nevertheless, in rare cases, we do observe feature splitting among highly active features. These features share semantic and directional simi- larity but exhibit subtly different activation patterns that justify separate dimensions. We detail this phenomenon in Appendix
These considerations suggest that crosscoders should naturally achieve effective feature alignment, mapping semantically equivalent features from different snapshots to consistent positions in unified feature space.
ASSESSING CROSS-SNAPSHOT FEATURES
4.1
OVERVIEW OF FEATURE EVOLUTION
Pythia
1.0
0.8
0.6
Docoder Norm
0.4
0.2
Pack PaakiSSazee Snapphoot teooottnaa Indox
0.0
1.0
Occoser Norm (Nommalioo
0.8
0.8
0.4
0.2
0.0
1000
6000
14300
60009
127000
Traniigg Step
Figure 4: Overview of cross-snapshot feature decoder norm evolution. Features extracted by a 98, 18.304-fcature crosscoder on Pythia-16000 160M (top) and a 71-768ffeatur crosscoder on Pythia-6.9B (bottom).

---

Published as a conference paper at ICLR 2026
Figure shows 50 randomly sampled features and their decoder norm evolution snapshots. Each feature's s decoder norms are linearly rescaled to a maximum of 1. Most cross- cross-snapshot features exhibit two distinct developmental patterns:
1. Initialization features that exist from random initialization, exhibit sudden drop and recovery around step 128, then gradually decay. The existence these features has been established by Bricken al. 2023) and Heap et al. (2025).
2. Emergent features that begin forming primarily around step 1000, reaching peak intensity at various subsequent training steps. There also exists emergent features only appearing in late training, which we discuss in later sections.
4000
10002
6002
1000
2000
Peak Snapshot Index
Lifetime
(a)
(b)
(c)
Figure 5: Statistics of emergent features in a 08,304-featurre crosscoder Pythia-160M, (a) Distri- bution of peak emergence times. (b) Distribution of feature lifetime. (c) Mean projection of each feature's decoder vector of snapshot 0; onto its decoder vector of snapshot 0z-
Feature emergence steepness. Previous studies suggest that loss curves comprise discrete phase changes, each contributing to different circuits at distinct training stages, with evidence from toy model features (Elhage al. (022), 5-digit addition (Nanda et al., 2023). and in-context learn- ing (Olsson et al. (022). From a feature evolution perspective, we investigate whether features emerge abruptly enough constitute the fundamental units of these phase transitions. By evalu- ating the time spent by each feature from initial emergence to peakness, we find that emergence steepness exhibits considerable variation, demonstrating the coexistence of both gradually-formed and abruptly-appearing features.
Feature persistence after emergence. We next examine whether emergent features persist after formation. We define the lifetime of a crosscoder feature as Kfj
dee &eelll > 0.3} where a threshold is introduced to block out near-zero decoder norms. Figure 5b shows the lifetimes of all emergent features. We find that most features persist for extended periods (above 60% of snapshots) after formation, indicating that: (1) LLMs retain learned features robustly, and (2) our crosscoders successfully align and track these features across snapshots.
[
universal directional turning point. To further investigate the geometry of feature evolution, compute the projections between feature directions across snapshots. The directional evolution patterns prove remarkably consistent: most features undergo drastic directional shifts around step 1,000, rendering pre- and post-step 1,000 directions nearly orthogonal (Figure 5cc. Subsequently, features continue to rotate more gradually, with directions at the final snapshot maintaining notable cosine similarities to their initial post-step 1,000 orientations-a significant finding given the high- dimensional activation space.
4.2 CORRELATION BETWEEN EMERGENCE STEP AND COMPLEXITY
To gain deeper insights into features emerging at different training stages, we leverage LLMs to automatically assess their complexity based on top activation patterns.
We follow Cunningham & Conerly (2024) to score feature complexity for 100 randomly sampled emergent features based their top activations, ranging from (simple) 5 (complex). The

---

Published as a conference paper at ICLR 2026
result in Figure 6a reveals non-trivial correlation with moderate strength between peak timing and complexity scores, suggesting that more complex features tend to emerge later in training. The scoring rubrics and complete prompt for automated complexity scoring can be found in Appendix
16
10
3.5
Decode Nerrn
Deccodde Norrm
2.0
0.6
2.5
0.4
2.0
02
1gr sliee Tel de coeal
1.5
edutte (5400)
Co dCPm apaie 1910
08
0.
1900
6000
14000
127800
1000
5000
16307
50000
127800
6:00 o0des 129036 Training Step
Training Stop
Traning
(b)
(c)
Figure 6: (a) Complexity scores (evaluated by Claude Sonnet 4) versus peak emergence times, showing a moderate positive correlation (Pearson 0.309, p = 0.002). (b) Decoder norm evolution trajectories for all previous token, induction, and context sensstivv features across training. (c) More cases feature decoder norm evolution trajectories.
4.3 CASE STUDY ON TYPICAL CROSS- -OOSS-SNAPSHOT FEATURES
We employ simple rule-based approaches find several well-studied feature types in a crosscoder with 32,768 features trained on Pythia-6.9B, including previous token features (which activate based on the preceding token), induction features (which activate on the second [A] in patterns [A][B]..[A][[]] and help predict [B]). and context-sensitive features. These feature categories are
extensively documented prior research on neurons and SAE features (Gurnee et al. 2024 Huben Ge et al. 2024p.
et al. 2024
Figure 6b demonstrates the distinct temporal pattern in feature emergence: previous token features arise early (around steps (,0005,000), while induction and context-sensitive features appear later and over a wider range (mostly steps 10,000-100,000), This suggests a general emergence hierarchy, from previous token induction to context-sensitive features, which aligns with their increasing complexity. This finding is also consistent with the causal dependency between induction heads and previous token heads (Olsson al. 2022).
For the majority of features that cannot into specific rule-based patterns, we further demonstrate additional cases of random emergent features with different evolutionary patterns (Figure 6c) La- bels are annotated by summarizing the top activation samples.
5 CONNECTING MICROSCOPIC EVOLUTION TO MACROSCOPIC BEHAVIORS
One of the ambitious missions of mechanistic interpretability research is connect feature-level findings with the model's downstream performance. To this end, we employ attribution-based circuit tracing techniques (Syed et al.) 22023] Ge et al. 2024 Marks al.) (025) investigate the causal effects of crosscoder features formation on downstream task metrics improvements.
Method. Let metric m Rrin + R be an arbitrary valuer function of model activations a (z) To quantify the causal effect of each feature activation fi(z) on metric m, we first decom- pose model activations into per-feature representations:
Pilantrres
(x)x) = (l(x) + (x) =
f((x) Wie. + b + te (2)
'The original notation in Marks et al. 2025) uses m as function of input where the computation graph flows through nodes of interest. Since we focus feature effects within a single layer (c.g., layer of Pythia- 160M), simplify function of model activations avoid complex intervention notation.

---

Published as a conference paper at ICLR 2026
where cx) Rd represents the crosscoder reconstruction error. This decomposition incorpo- rates crosscoder features into the computation graph, enabling gradient computation with respect to features. We then estimate each feature's causal effect using its attribution score:
0m(a((x)) (x) = fi))- (f((z)
(4)
where the gradient dm flows through the decomposition in Eq. This attribution score employs first-order Taylor expansion linear approximation of model computation.
5f.
For structured downstream tasks with clean/corrupted input pairs, such as subject-verb agree- ment (Finlayson al. 2021), , we can employ the full framework of attribution patching (Syed al.
2023
Marks et al. 2025 by emphasizing differences between clean and corrupted inputs:
attr, (x, (f((z)) - f(())) dm(a
(5)
(f((z)
where is the corrupted version of input Attribution patching isolates components explain- ing the transition from corrupted to clean performance, excluding the majority of components that contribute to baseline model function. This focused approach improves the validity of the linear approximation.
In practice, we employ the integrated gradient (IG) version of the attribution scores defined Eq. and Eq. [5 which compute gradients at evenly-spaced points from to ã (r to zero vector in the pure attribution case). Details of IG computation can be found in Appendix
0.6
1.1
D97
0.4-
0.9
2013
0.5
0.8
J0 fo
0.1
0.5
1000 6000
14002
60300 127000
32
1000
5000
14000 DOoOp 127000
32
1000
6000
14000
60000 127000
Training Stop
Thiining Shep
Training
(c)
brosen concrite banid
1905 bosk8
lanee that part.81 thedseratop
Follower Voldemiil lamoo dealt
(d)
Figure 7: Crosscoder feature attribution on the Across-PP variant of the subject-verb agreement task, e.g. "The teachers near the desk are". We use crosscoder train at layer 6 of Pythia- Mitia-160MM 160M with 98,304 features. (a) The attribution scores of top contributing features over time. (b) The metric recovery when ablating all features except the top & contributing features. (c) The metric recovery when ablating the top contributing features. (d) Top activation samples of key features contribut- ing to this task. Features recognizing plural nouns appear before features recognizing postposition attributives.
Experimental setup. evaluate our method on subject-verb agreement (SVA) (Finlayson al., 2021), induction, and indirect object identification (IOI) (Wang et al., 2023) tasks, using 1000 sam- ples each to identify critical features.
For SVA and IOI, we create corrupted controls by swapping singular/plural forms (SVA) altering the second subject (IOI), using logit differences between clean and corrupted answers as metrics. For induction, which lacks natural corruptions, attribution scores for all features on a 98,304-feature crosscoder trained on Pythia- 160M, and rank crosscoder feature contribution mean attribution scores across all snapshots. We then perform
target answer log probabilities. We compute IG
8

---

Published as a conference paper at ICLR 2026
complementary ablations: (1) removing top-ranked features, and (2) removing all except top-ranked features.
Results. Figure 7 shows results for the Across-PP variant of SVA, where postpositional attribu- tives separate subjects and verbs, e.g. "The teachers near the desk are". : We identify key contributing features ordered by emergence time: (1) Features 18341 and 47045 capture plural nouns, with 47045 specialized for plural subjects; (2) Feature 68813 marks compound subjects and postpositional at- tributives; (3) Features 50159 and 69636 identify endings of postpositional attributives, with 69636 showing higher accuracy. Additional features include subject-specific and context-specialize plu- ral markers with lower task contributions. Notably, Features 68813, 50159, and 69636 alternately dominate the metric, revealing circuit-level model evolution through component iteration.
Ablation experiments across all tasks demonstrate that within tens of features, we can consistently disrupt or recover model performance on downstream tasks across training snapshots, confirming our method identifies necessary and sufficient task components.
OBSERVATIONS OF A MAATITIIALLTOOSSUPPPPOTTTTT TRANSITION
Logges Omios
7W100-39 941
wur.uraaaaoooo Inutss
10
Locs (nwit
0.5
15"
10"
15
107
1P
15a
10'
12F
J0'
1P
32
1000 Mids 16300 82002 127000 Taliing Step
Trahing Stops
Trihing Steps
(a)
(c)
Figure 8: Observations supporting toaiistical-o-ssueepppsiiioo transition in Pythia-160M and Pythia-6.9B. 6. (a) Unigram and bigram KL divergence evolution in wythia-160M. The convergence timing of KL divergences coincides with when training loss approaches the theoretical minimum (unigram/bigrann entropy). (b) Unigram and bigram KL divergence evolution in Pythia-6.9B. (c) Total feature dimensionality ratio (relative to activation space dimension) over training time.
What is the model learning at the beginning of training while the training loss rapidly decreases, if no features are formed at this period? We hypothesize that the rapid loss decrease in early train- ing driven by learning coarse statistical patterns, rather than forming distinct features. After this initial optimization nears completion, sparse features emerge in superposition for further loss reduction.
This two-stage structure is deeply consistent with the fitting-t--ooppressin phase transition pre- dicted by the information bottleneck theory (Shwartz-Ziv & Tishby, 2017).
Early training almost exclusively learns uni- and bi-gram distributions. Following previous work on language model learning statistical patterns TTkahassi & Tanaka-Ishii 2017 Xu et al.
2019 Choshen et al., 2022 Belrose et al., 2024| Svete & Cotterell 2024| Nguyen 2024| Chen al. 2024), we compute the KL divergence between the true token distribution Q and the model's predicted distribution P. We randomly sample 10M tokens from SlimPajama to approximate both distributions. We then evaluate unigram KI. divergence DKL (P(x) || Q(x)) and bigram KI. diver- gence DKL (P(æ, x.-1) || Q(x;; | x=).))
The results show that both unigram and bigram KL divergences rapidly converge to low values dur- ing early training (Figure 8al and 8b). Furthermore, the training losses during this period approach the theoretical minimum achievable if the model perfeetly matched the true token distributions i.e., the entropy of these distributions. This suggests that the model primarily learns to fit statistical reg- ularities (Zipf's 's law (Zipi 1935; Piantadosi] 2014)) during the early training stage, which explains the dense nature of internal representations this phase.

---

Published as a conference paper at ICLR 2026
Total feature dimensionality undergoes compression and expansion. To directly measure perposition status at each snapshot, we adapt the approach from Elhage et 2022) and compute the dimensionality of each feature at snapshot 0 as:
I Wede..ll dll
Dj
Wi70 dec.s
Wo
==1
where W is the normalized version of decoder vector W dec.i- In contrast to its original application toy models with ground-truth features, we apply this metric to crosscoder features due to their consistent alignment across training snapshots, providing insight into superposition dynamics.
dec.i
We compute total feature dimensionalities for crosscoders trained on Pythia-1600 160M and Pythia-6.9B (Figure (8c). Under ideal superposition, features should form symmetric arrangements with total dimensionalities summing to the activation space dimension. Feature dimensionalities should go down if large interference exists among features. We observe that total feature dimensionality first decreases then increases around the turning point, eventually reaching approximately 70% of avail- able dimensions in the crosscoder with 98,30k features for Pythia-160) Features from Pythia-6.99 account for smaller proportion of activation space dimensions, likely due to the limited representa- tional capacity of the 32.768-feature crosscoder. Nevertheless, both models exhibit the same trend, suggesting that initialization features initially form weak superposition, then become compressed accommodate emergent features. This indicates that the model develops into feature learning phase.
7 LIMITATIONS
Scope and generalizability. Superposition has been proven to be general phenomenon in deep neural networks (Elhage et et., al. 2022). However, our analysis focuses on feature evolution in Pythia suite models using their open-source training snapshots. While previous research on feature univer- sality (Wang et al. 2025b) suggest our method might be able to generalize to different architectures, datasets, post training dynamics, and tasks beyond language modeling, the extent to which feature evolution patterns are consistent across diverse settings remains to be established. We leave broader generalization studies future work.
Limited downstream task complexity. Section the connection between feature evo- lution and model behavior. However, the downstream tasks we examine are relatively simple, con- strained by multiple factors including Pythia model capabilities, sparse dictionary quality. and the current state of circuit tracing methodologies. Scaling to more complex downstream tasks represents a natural direction for future work.
Discrete snapshot requirement. Crosscoder training requires activations from discrete training snapshots, with memory and computational costs scaling linearly with snapshot count, which limits observational granularity. Potential solutions include architectural modifications for online multi- snapshot processing incorporating gradient information to capture continuous training dynamics.
CONCLUSION
We introduce crosscoders to study feature evolution in LLM pre-training. Our analysis reveals two patterns: initaaliaaiion-dpeedeet features and emergent features, with complex patterns emerging later. We establish causal connections between feature evolution and downstream performance. Sup- ported by uni- and bi-gram distribution analysis and feature dimensionality dynamics, we propose that model pre-training can be roughly divided into a statistical learning phase and feature learning phase. This work bridges mechanistic interpretability with training dynamics.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
All our crosscoders are based on open-source models (Pythia suite) and datasets (SlimPajama) with public accessibility. We provide source code for (1) generating model activations each snap- shot, (2) training crosscoders, and (3) analyzing trained crosscoders at hhtts:::// / com/ OpenMOSS LaaiDD/S//Lagggggg-ooe---------- SAEs Detailed instructions for replicating our results are pro- vided in the examples/zeprodue wtw.ooodueeeeeeeuueeeeeee
evolut ion ioututiooooooiioo of_.ccccetttt. s README. md file. We note that training large crosscoders requires substantial computational resources and disk space, as also listed in the README, md file.
ACKNOWLEDDEMENT
This work was supported by the National Natural Science Foundation of China (No. 62525602).
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. URLhttps //openrevi ew.
d=HJ4 d rAVt..
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neu- ral scaling laws. CoRR, abs/2102.00601, 2021. URL
06701
Nora Belrose, Quintin Pope, Lucia Quirke, Alex Mallen, and Xiaoli 7. Fern. Neural networks learn statistics of increasing complexity. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https:
7/ //opereeiiww.ocmmm wopenreeieww.nt//oorrreeeeeeeeee
net httt://fummmmimo IGCpKP ON6W
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013. URL
httt:://axxv..ozg// wtapzg//bs/11000 3432
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives, 2014.
http:::s///xxxi.oggg//o/o 5536
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Atlah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolull, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 2397-2430. PMLR, 2023. URL https:// http:////22///dddeaa22222222 html
proceedings mlr.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https:/transformmrr circuits.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005.1 14165, 2020. URL https: //tax//axxii .org abbs///b//// 2005. 14165
11

---

Published as a conference paper at ICLR 2026
Yihong Chen, Xiangxiang Xu, Yao Lu, Pontus Stenetorp, and Luca Franceschi. Jet expansions of residual computation. CoRR, abs/2410.06024,, 2024. doi: 10.4850/ARXXIV2410000222 URL
htttps://di...o wtwp://...r....... 48540 wtt.:0088855/222222....0000 06024
Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend. The grammar-learning trajec- tories of neural language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 828 3281-8297. As- sociation for Computational Linguistics, 2022. doi: 10. httpps://do. woo. /633/vllll11 wtt//2//2222222222 acl.-222.cclll.. .68
URL.
Hoagy Cunningham and Tom Conerly. Circuits updates june 2024. Transformer Circuits Thread, 2024. URL
html #hurdles
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- doi: 10.18633V1/N19--144 URL
4186. Association for Computational Linguistics, 2019.
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits. In Amir Globersons, Lester Mackey, Danielle Belgrave, An- gela Fan, Uirich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.). Advances in Neural Information Processing Systems 38: Annual Conference Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http::/papers...e.
wttpppper.fillsspppeeeee/eeee/ee html,
Ethan Dyer and Guy Gur-Ari. Asymptotics of wide networks from feynman diagrams. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
new wwt.nnrufffddidd
BlgFvANKDS
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfeld-Doddss, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. hutps://transformer- http:///2/ttyooooeeeeeeeee
html.
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 1828-1843, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.1886/11/2222 hon.lonng..44.. URL https: http://aaaattthhh org. 2021 acl-longg 1447]
Zach Furman and Edmund Lau. Estimating the local learning coefficient scale, 2024. URL
https:: /tt::///xxxiv orgg abs /4022 22022202222293
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth Inter- national Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL httpt:tt///// wwwp//vin.ree...eeeeeeeee /ttpps:///peereeieee...
net. forum? d=tcs 9ZNKD
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume I: Long
12

---

Published as a conference paper at ICLR 2026
Papers), Virtual Event, August 1-6, 2021, pp. 3816-8886688 tics, 2021. doi: v1
3830. Association for Computational Linguis- URL /tl.ittpp///o...o///111111
295
Xuyang Ge, Fukang Zhu, Wentao Shu, Junxuan Wang, Zhengfu He, and Xipeng Qiu. Automatically identifying local and global circuits with linear computation graphs, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak. Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes. Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arricta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- soo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Ku- mar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoy- chev, Niladri Chatterji, Ning Zhang. Olivier Duchenne. Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic. Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He. Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ra- mon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Ro- hit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mi- haylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong. Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaclle Gold- schlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, An- drew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, An- nie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Fisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leon- hardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Mon- talvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsian Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily
13

---

Published as a conference paper at ICLR 2026
Hahn, Emily Wood, Eric-Tuan Le. Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harri- son Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean- San-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jen- nifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Jun- jie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A. Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal. Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar. Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang. Rachad Alao, Rachel Ro- driguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ra- maswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar. Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyighe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk. Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiao- jian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li. Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vetoto, Zef Rosnbrick, Zhao- duo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL
Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons GPT2 language models. Trans. Mach. Learn. Res., 2024, 2024. URL httptts://ppeeeeiiiiie
net/ fort:/oormm22m wtw.:rmmmddeiii00000000
Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In First Conference on Language Modeling, 2024. URL
Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. Llama scope: Ex- tracting millions of features from 11.ma-3.1 1-8b with sparse autoencoders. CoRR, abs/2410.20526, 2024. doi: (044550/AXXI.22440205222 URL https: /ci org http://a....rrr.....5///5/////// 2410.20522
Thomas Heap. Tim Lawson, Lucy Famik, and Laurence Aitchison. Sparse autoencoders in- terpret randomly initialized transformers, 2025. URL /tttp:://axxxv.or//bbb..
T7727
14

---

Published as a conference paper at ICLR 2026
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianine- jad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. CoRR, abs/1712.00400 2017. URL http:://aaaaa
org/ abs 1 1712. 00409
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024. URL
net
6bwRSLeK
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle. Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8580-85588 2018. URL httpss:/ neu.eeee
cc/papprr/208//aaeee 2ebbo.ffleee22aaaeeee22222222222 at html
Abstract
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.0883 100011, 2020. httthttps://aaa
org/ abs 2001 08361
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
htt httt/a///arxi. wwwp.ar////////o///et...oe..eese 414.1.666 6980
Tanishq Kumar, Blake Bordelon, Samuel J. Gershman, and Cengiz Pehlevan. Grokking as the tran- sition from lazy to rich training dynamics. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL httpps://openeevlew net forumrid- Smn
Edmund Lau, Zach Furman, George Wang, Daniel Murfet, and Susan Wei. The local learning coefficient: A singularity-wware complexity measure, 2024. URL httts::/arxiv.orrt
foss/308... 12108
Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread, 2024.
Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. BtLhttt:: /wpp///oeeeeeoo.e//e/ooeeeee..ep /oppps/////perreeee.. I4e82CIDXV
net/ wttp:nett/fff.o.oeeo forum?id=
Julian Minder, Clément Dumas, Caden Juang, Bilal Chugtai, and Neel Nanda, Overcoming sparsity artifacts crosscoders to interpret chat-tuning, 2025. URL httpss//erxxvvoor//bb 2504. 02922
Siddharth Mishra-Sharmal Trenton Bricken, Jack Lindsey, Adam Jermyn, Jonathan Marcus, Kelley Rivoire, Christopher Olah, and Thomas Henighan. Sparse crosscoders for cross- layer features and model diffing. Transformer Circuits Thread, 2024. https:/ttansformmr-
htmm.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures abs/://///2/ http::///....i..ooo/aa//200.111
for grokking via mechanistic interpretability, 2023.
05217
Timothy Nguyen. Understanding transformers via n-gram statistics. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL URL /ptp:tt:ppppppappppppppppppp nips. wwwppppppccppppppeeeeeeeppeppppe hatpppppprrflee___eeeellell www.s/ http://par...llee/ppppr////////. b1c4 htg.://eeeoaaaaaooooo..ooaooooor
15

---

Published as a conference paper at ICLR 2026
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi:
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bha- gia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Py- atkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hanmanch Hajishirzi. olmo 2 furious, 2024. uRL.ttts:: http://axxvv.oo wttporrrabbb.aaaeeeeee 00656
Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: strat- egy employed by vI? Vision Research, 37(33):311-3325, 1997. ISSN 0042-6989. doi: https:/dc.oorr/11000011600222222
URL https: /www:///wwwwwww www.scienned.rrec.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. https:/transformercciccct wwrcuiiss.ub///22222..
OpenAl. GPT-4 technical report, 2023.
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data scale, 2024. URLttttss://azat
htt.gg/aab///000/ 17557
Steven T. Piantadosi. Zipf's word frequency law in natural language: critical review and future directions. Psychonomic Bulletin & Review, 21:1112 1130, 2014. URL https://api.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gener- alization beyond overfitting on small algorithmic datasets. CoRR, abs/2201.02177,, 2022. URL
https::/aaxa //tpiv/.aiivvvv/aa www.oogg//soo.. abs/ 2201. 02177
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. Technical Report, 2018
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, CoRR, abs/2407. 2024. doi: 10.48550AARXII2220.4444433 URL https:
//doi.org/10, 48550//a5X3.
Peter Sanders, Kurt Mehlhorn, Martin Dietzfelbinger, and Roman Dementiev. Sequential and Par- allel Algorithms and Data Structures: The Basic Toolbox. Springer Publishing Company, Incor- porated, 1st edition, 2019. ISBN 3030252086.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné. Alexandra Sasha Luccioni. François Yvon, Matthias Gallé, Jonathan Tow, Alexan- der M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Baw- den, Stas Bekman, Angelina McMillan-Major Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. BLOOM: A 176b- parameter apen-access multilingual language model. CoRR, abs/2211.05100 2022. doi: 10. 11-22211001100 105100. URL U /dooi www.orgg1..... 2tt.:///aa.wv..1........ 2tw.il..22.1.110
16

---

Published as a conference paper at ICLR 2026
Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric P. Xing. Slimpajama-de: Un- derstanding data combinations for LLM training. CoRR, abs/2309. 10818, 2023. doi: 10.48550/ ARXIVV2309.1088 10818. URL WtRlhttss: pattps://do:: /tw.//o.ooooooo
www.do...ooooooooooooooooooooo.. worg/o..g...... 0088/550001111 w.w..xii.v......... 1081 (00088))
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism. CoRR, abs/ http//999.0080 08053, 2019. URL
wtwp:///////////...../// 08053
Ravid Shwartz-Zii Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. CoRR, 0bss/103300811 10, 2017. URLhttt:://axxxt wttp:/www..gg//////3333333
Lewis Smith, Sen Rajamanoharan, Arthur Conmy, Callum McDougall, Janos Kramar, Tom Lieberum, Rohin Shah, and Neel Nanda. Negative results for SAES on down- stream tasks and deprioritising SAE research (GDM mech interp team progress update #2), 2025. URL https:://// /www:..://wwww.... l wessssssssss wwtp://gg.ooooooooooeoooo Sogaaive-reeullee for saes wtwp---ooooottttttttetttttte http://wrr.....oooooooo////////a
4uXCAJNUPREKBs122
LessWrong.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Teh (eds.), Pruceedings of the 34th International Conference on Machine Learning. ICML 2017. Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 3319-332. PMLR, 2017. URL ml
aandndaaaaaaaa1111 jan17a hhtml....ml
Anej Svete and Ryan Cotterell. Transformers can represent n-gram language models. In Kevin Duh, Helena Gómez-Adomo, and Steven Bethard (eds.). Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 6845-6881. Association for Computational Linguistics, 2024. doi:
10.18653/V Www.:044A.AAALLLLNNN.... URL
381
Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. CoRR, 10342300.10348, 2023. doi: 10445500ARKX11231000044 URL https:
//dei. 48550 (8850//aXiii0 www.iv.iiii..//... 10344
Shuntaro Takahashi and Kumiko Tanaka-Ishi.. Do neural nets learn statistical laws behind natural /as///bs///// 04848
language? CoRR, abs/ 7777704444 2017. URL http:://txxiii....////
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Trans- 202/
former Circuits Thread, 2024. URL Lcal htt.li/g.moooommmmmmmccccc//eeee html
Naftali Tishby and Noga Zaslavsky, Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop, ITW 2015, Jerusalem, Israel, April 26 May 1, 2015, pp. 1-5, IEEE, 2015. doi: 10.1109/TW.2015.713316. URL https::/ /dcps:://dc.... org 10.1g/90..019///
TTW.2013 5. 7133169 69
Oskar van der Wal, Pietro Lesci, Max Muller-Eberstein, Naomi Saphra, Hailey Schoelkopf, Willem Zuidema, and Stella Biderman. Polypythias: Stability and outliers across fifty language model pre-training runs, 2025. Utthsttss: pettttss//axxiv.oott wwwpr/xww.rrg..////r//eee
09543
Francisco Vargas and Ryan Cotterell. Exploring the linear subspace hypothesis in gender bias mit- igation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020. pp. 9902-2913 Association for Computational Linguistics, 2020. doi: 12118653/V120020. EMW.00200........... -AL.PP-AAI.223 232. URL
emn 1 pp-maaaa- i n. 232
17

---

Published as a conference paper at ICLR 2026
George Wang. Jesse Hoogland, Stan van Wingerden, Zach Furman, and Daniel Murfet. Dif- ferentiation and specialization of attention heads via the refined local learning coefficient. In The Thirteenth International Conference Learning Representations. ICLR 2025, Singapore, April 24-28, 2025. OpenReview.ner, 2025a. URL https: htttps://openrevieww wet/
id=SUcl UOWndp)
Junxuan Wang, Xuyang Ge, Wentao Shu, Qiong Tang, Yunhua Zhou, Zhengfu He, and Xipeng Qiu. Towards universality: Studying mechanistic similarity across language model architectures. In The Thirteenth International Conference on Learning Representations, 2025b. URL https:
//openzeview. httt//ooa0omm0000020001111100111
Kevin Ro Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. In- terpretability in the wild: a circuit for indirect object identification GPT-2 small. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023, URL https:: htttp:://openevveew net/ forum?ic
NpsVSN6o4ul,
Sumio Watanabe. Algebraic analysis for non-regular learning machines. In Sara A. Solla, Todd K. Leen, and Klaus-Robert Müller (eds.), Advances in Neural Information Process- ing Systems 12, INIPS Conference, Denver, Colorado, USA, November 29 December 4, 1999/. pp. 356-362. The MIT Press, 1999. URL http: http://paapess nips..iipp...... cc/paper/ 1739- algebraic analysis for non regular learning machii-schhnsseeeee
Sumio Watanabe. Algebraic Geometry and Statistical Learning Theory. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2009.
Zhi-Qu -in John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. In Tom Gedeon, Kok Wai Wong, and Minho Lee (eds.), Neural Information Processing- 26th International Conference, ICONIP 2019, Sydney, NSW, Australia, December 12-15, 2019, Proceedings, Part I, volume 11953 of Lecture Notes in Computer Science, pp. 264- 274. Springer, 2019, doi: 1007/9788 0000--330000000000000 36708-444444 22
22.. URL https: //doi. org/ 10.
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhennu Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL. htt.https:/// http::///axxiiii org/ wtw.////5000 09388
Greg Yang. Tensor programs III: neural matrix laws. CoRR, abs/2009. 2020. URL https:
7/ //arxiv. org/ abs/ /2009.0009... 10685
Greg Yang and Etai Littwin. Tensor programs iib: Architectural universality of neural tangent kernel training dynamics. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11762-11777 1772. PMLR, 18-24 Jul 2021. URL https::tpp http///oooedgggeeeeeeeee ngs ml p...ss/./////
v139/yang21E. html
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 12697-12706. PMLR, 2021. URL BuRK/tttpp:/prroeednnnnn www.mmrr.rress.lll..mm http:///1////aa21...c..
Zhanpeng Zhou, Yongyi Yang, Mahito Sugiyama, and Junchi Yan. New evidence of the two- phase learning dynamics of neural networks, 2025. uttpsstt////axi wtt.aaiii...rga/////0/0000 13900
George Kingsley Zipf. The Psychobiology of Language. Mbughton-Mifffh New York, NY, USA, 1935.
18

---

Published as a conference paper at ICLR 2026
A CROSSCODER TRAINING DETAILS
Section presents the mathematical definition of our cross-snapshot crosscoder. However, in practi- cal applications, additional architectural design and training techniques are required to advance the Pareto frontier of sparsity versus reconstruction fidelity. In this section, we detail our selection of activation function (.)) and regularization function 92()), and present our training hyperparameters and their results. We also compare crosscoder performance standard SAEs to evaluate how well crosscoders perform at sparse dictionary learning.
A.1 SELECTION OF ACTIVATION FUNCTION AND REGULARIZATION FUNCTION
The sparsity natural features the fundamental hypothesis underlying superposition (Olshausen) & Field 1997: Elhage al. 2022 Huben et al., 2024). To obtain crosscoder features with optimal sparsity for ideal interpretability and monosemanticity while maintaining reconstruction fidelity, we carefully select the activation function a(-) and the regularization function (2(-).
Previous SAE studies predominantly use ReLU activation with LI regularization (Bricken et al,, 2023 Huben al., 2024). However, this configuration produces weak feature activations that are largely noise, compromising both interpretability and sparsity. We address this by adopting JumpReLU (Rajamanoharan et al. 2024) as the activation function, which eliminates activations below learned thresholds (trained via
estimation (Bengio et al., 2013)).
To prevent features from becoming permanently inactive at certain snapshots, incorporate de- coder norms into the activation decision. Given pre-activation 2(x):
=(x)
Σ PEE
( W onc (x) + bene
The i-th feature activation at snapshot 0 defined as:
(x) (x) (2((x)) I|W dec, t2)
where H() the Heaviside step function and 4 E R is the JumpRel.U threshold for feature i.
This design ensures that features with small decoder norms require stronger pre-activations to acti- vate, preventing complete feature death while maintaining sparsity. Although this means truly in- active features retain small positive decoder norms rather than zero values, this architectural choice significantly improves crosscoder performance and enables better feature tracking across snapshots.
For regularization, employ combination of tanh and quadratic frequency penalties (Smith al. 2025): the tanh component provides a superior LO approximation by reducing penalties on strong activations, while the quadratic frequency penalty suppresses high-frequency features. This yields the following batched sparsity loss:
(B)
Σ B
tanh (fi(z) I| dec,sll)
Cspariity (B) Aaparsity
Σ OE0
Atoors
(B) (1+ wb
wf(BB)
1=1
where B C C is an input batch, and (B( is the single-batch differentiable estimate for the activation frequency on snapshot 0 of i-th crosscoder feature, using tanh as LO approximation. A new hyperparameter wh is introduced to quadratically penalize feature activation when wi (B) >> wa-
A.2 SELECTION OF PRE-TRAINING SNAPSHOTS
To balance training cost with the granularity of feature evolution analysis, train crosscoders using 32 source snapshots from the 154 soun-source snapshots in the Pythia suite. Figure I0 shows the training steps of the selected snapshots.
19

---

Published as a conference paper at ICLR 2026
0.023
1,0 Narm
90
6.05
0.022
20-
10
14000 63000 127030
1992 6200 18005 mnico 12700
Faining Step
Traising Stop
(a)
(b)
(c)
Figure 9: Comparison between crosscoders and per-snapshot SAEs. (a) The explained variance of
crosscoders versus SAES at each snapshot. (b) The 1.0 norm of crosscoders versus SAES at each snapshot. (c) The Pareto frontier comparison of crosscoders and SAEs trained the final snapshot.
2
24
230
Trairing
10
20
25
30
Snapshot Index
Figure 10: Selected snapshots from Pythia suite.
A.3 DISTRIBUTED TRAINING STRATEGY FOR CROSSCODERS
Crosscoders require parameters that scale with the number of source snapshots sasppsiioss resulting in significantly higher memory and computational requirements compared SAEs with equivalent feature counts, particularly when using many source snapshots.
To efficiently train crosscoders, we employ head parallelism distributed training strategy, a variant of tensor parallelism (Shoeybi et al. 2019). With k processes where k divides Rsnapshoxs each process
handles encoding and decoding for Tommmlas
source snapshots. Pre-activations are computed via All-
Reduce operations (Sanders et al. 2019). Unlike standard tensor parallelism, our approach processes
activations from each snapshot separately individual processes, reducing I/O overhead for reading activations from disk.
A.4 EXPERIMENTS
We train crosscoders on Pythia- 160M and Pythia-6.99 6.9B snapshots at various scales (6,144 144 98,304 features on fythia-160M, and 32,768 features Pythia-6.9B3 9..B). We primarily focus middle lay- ers (Layer in Pythia- 160M, and Layer 16 Pythia-6.9D but also train crosscoders of 24,576 features on all layers of Pythia- 160M for comprehensive analysis.
All crosscoders are trained on 800M tokens from the SlimPajama corpus using the Adam opti- mizer [Kingma & Ba 2017) with 3 values of (0.9, 0.999). To prevent straight-through estimation from causing rapid threshold increases, we apply reduced learning rate to JumpReLU threshold the global learning rate. JumpReLU thresholds
initialized 0.1 for all features. The learning rate schedule includes 10% warm-up steps followed by 20% decay steps. We initialize encoders as transposes of their corresponding decoders, with identical initialization matrices across all snapshots. We employ initialization search to identify optimal decoder norms that minimize loss at initialization. Additional hyperparameters are listed in Table
updates via multiplier
By "Layer a", we refer the activations at the output of the i-th transformer layer.
20

---

Published as a conference paper at ICLR 2026
Table I: Hyperparameters for crosscoder training
Parameter
Pythia-160M
Pythia-6.9B
Learning Rate
5e-5
le-5
Batch Size
2048
2048
Feature Expansion Ratio
8x, 16x, 32x, 64x, 128x
8x
Sparsity Coefficient (Aspansity)
0.3
0.3
JumpReLU Threshold LR Multiplier
0.1
0.3
Layero
0.96
Lager3
Layer5
Explained Variance
0.94
Lago
Laper?
Laper
0.92
0.86
50
70
80
90
Lo Norm
Figure 11: Explained variance versus L.0 norm of crosscoders trained on all layers (0-11) of Pythia 160M
A.5 RESULTS ON OTHER LAYERS
We train additional crosscoders with 24,576 features all layers (0-11) of Pythia-160M using the same hyperparameters as Layer 6 (Table Figure [1 shows the explained variance and LO norm for these crosscoders, which exhibit similar performance but different trade-off's between sparsity and reconstruction quality.
To demonstrate feature evolution across all layers, apply the same visualization strategy from Section 4 plotting decoder norms for 50 randomly selected features per layer (Figures) [2] and [3).
Most middle layers exhibit the same evolutionary patterns Layer 6, supporting our findings in Section
4 We would like to note that the majority of features in Layers 0 and I exist from initial- ization, which aligns with the observation that early layers implements more low-level, especially single-token features (He et al. 2024).
B AUTOMATED SCORING OF COMPLEXITY
We leverage Claude Sonnet 4 for automated complexity score evaluation. For each feature, we select the top 10 activating samples with 100 surrounding tokens around the strongest activating tokens and apply the following prompt:
Neural Network Feature Analysis Instructions
We' re analyzing features in a neural network. Each feature activates on specific words, substrings, or concepts within short documents. Activating words are marked as '<ttett, (activation)>>* where ((ativatiin) indicates the strength of activation (higher values stronger activation). You'11 receive documents containing highest activations tokens tokens surrounding then.
# Your task:
### 1. Summarize the Activation (<20 words)
21

---

Published as a conference paper at ICLR 2026
Pythia-160M Layer
0.8
0.4
0.2
Pythia-1600 160M Layer
0.8
0.6
0.0
Pythia-160M Layer
1.0
0.8
0.2
0.0
Pythis-160M Layer
1.0
0.4
0.2
0.0
Pythia-16aM Layer
1.9
0.6-
Norm
0.4
0.2
Pythia-160M Layer
0.8
0.4
0.2
1000
5000
14000
50000
127000
Training Step
Figure 12: Feature decoder norm evolution of Layer 0 to Layer 5 in a 24.576-fcature crosscoder trained on 160M.
22

---

Published as a conference paper at ICLR 2026
Pythie-160M Layer
0.6
Nor
0.4
0.
Pythia-1600 160M Layer
0.8
0.6
Pythia-160M Layer
1.0
0.8
0.0
Pythis-160M Layer
1.0
Decoder
0.2
0.0
Pythia-160M Layer 10
0.6-
Nom
0.4
Pythia-160M Layer
0.8
1000
5000
14000
50000
127000
Training Step
Figure 13: Feature decoder norm evolution of Layer 6 to Layer 11 in a 24.576-featur crosscoder trained on Pythia- 160M.
23

---

Published as a conference paper at ICLR 2026
Examine the marked activations and summarize what the feature detects in one sentence.
Avoid being overly specificly your explanation should cover most/all act activating words
If all words in sentence activate, focus the sentence's concept rather than individual words
Note relevant patterns in capitalization, punctuation, or formatting Prioritize strongly activated tckens
Keep explanations simple and concise
Avoic leng word lists
#*# 2. Assess Feature Complexity (1-5 scale, with decimal precision allowed)
Rate the feature's complexity:
- Rich feature with diverse contexts unified by interesting theme
==*/==: High-level semantic structure with potentially dense activation categories, or sentence structures ===2*=: Synonyms or words at same class
=**3==: Moderate camplexxty/phraseee
*+1+*: Single specific word or tcken
You may use decimal values (c.g., 3.7) for precise assessment.
### Output Format
Your output should be in JSON fermat, with two fields: surmarizatio and complexity. You should directly output the JSON object, without any other text.
C ASSESSING CROSSCODER DECODER NORM
0.6
0.5
0.4
Smarp not Indec
0.3
15
Praking
6.2
100
6.1
6.0
0.0
0.2
0.8
2.0
Decoder Norm
Figure 14: Linear probe training emrors versus feature decoder norms of 20 randomly sampled fea- tures from crosscoder with 6, 144 features on 160M
Section takes advantage of the crosscoder decoder norms to study feature evolution. However, are the crosscoder decoder norms well quantitative indicators of the extent to which features evolved? Aside from theoretical statements, we conduct experiments to examine whether crosscoder decoder norms reflect the intensity of features. We leverage linear probing to test the correlation between the feature decoder norm and the linear separability of the feature activations on each crosscoder feature.
24

---

Published as a conference paper at ICLR 2026
We train linear probing classifiers (referred probes) separately on activations of each model snapshot to classify whether each crosscoder feature activates. For each feature and each snapshot 8, our probes map model activations to the predicted feature activating probability p. (z) as:
(x) sigmoid( probe.s a (x) - 6 ppooee.))
(10)
where w probe. E IR chonell and bp pp R is the weight and bias of the probe w.r.t. feature i and snapshot 0. Each probe is trained to minimize the binary cross- cross-entropy loss yi log (2) + (1 v:) (log(1 - p (I))), given the label y/ = sgn (f)()) The training loss of each probe should be direct measure of the linear separability of the feature activations.
We train probes for 100M tokens for each feature in a 6, 6.144-feature crosscoder on Pythia- 160M. The probe errors of each feature show mean Pearson correlation with the corresponding decoder norms by -0.867, with a standard deviation of 0.153. We demonstrate example probe errors versus feature decoder norms 20 randomly sampled features in Figure [4] This indicates strong negative linear relationship between probe errors and crosscoder decoder norms, demonstrating the effectiveness of the crosscoder decoder norms as indicators of feature evolution.
D RULES FOR FINDING TYPICAL CROSS-SNAPSHEE FEATURES
We define the rules used to identify previous token features, induction features, and context-sensitive features as follows:
1. Previous Token Features:
We collect the directly preceding tokens of all activating tokens in the top 20 activating samples and assess their consistency. Token consistency is defined as the proportion of the largest single group when all tokens are normalized by stemming (removing leading and trailing spaces and ignoring case). To exclude bigram or multigram features, we also evaluate the consistency of the activating tokens themselves. A feature is classified previous token feature if exhibits high consistency in previous tokens (above 0.8) and low consistency in activating tokens (below 0.3).
2. Induction Features: Induction features should activate on the second [A] in patterns [A][B]..[A][BB. For each activating token [A], we collect its following token [B] and search for previous occurrences of the bigram [A] To distinguish induction features from sim- pler features that merely activate on any bigram [A][B], we require that the feature does not activate on the first appearance of [A][]] A feature exhibiting this behavior in at least 20 instances within the top 20 activating samples is classified as an induction feature.
3. Context-sensitive Features: We identify context-sensitive features using a simpler rule based on activation density within specific contexts. Context-sensitive features should tivate frequently in highly specific contexts, so we require features to have high activation counts within the top 20 activating samples (exceeding 4,000 activations). To exclude fea- tures that activate ubiquitously (such as positional or bias features), we filter out features with excessive total activations (below 2M total activations across 100M analyzed tokens).
E DETAILS IN DOWNSTREAM TASK ATTRIBUTION
E. FORMALIZATION OF IG ATTRIBUTION SCORE
To more accurately estimate the causal effect features, we employ the integrated gradient (IG) version of the attribution score (Sundararajan et al. 2017 Hanna et al. 20024] Marks et al.) 2025). IG attribution computes gradients along an interpolation path between baseline and target activations, providing more robust estimates than single-poing gradients.
For the standard attribution score without clean/corrupted input pairs, we compute the IG version as:
1 "N
@m(a (x))
attr (x) ( =
Σ M(xx)
(1I)
Maf((z)
25

---

Published as a conference paper at ICLR 2026
0.0
Motic
Mobri:
0.4
6000
14203
(a)
(b)
(c)
Figure 15: Crosscoder feature attribution on the induction task, using a crosscoder with 98,304
features trained on Pythia-1600 160M.
1.1
483.0 1345
J60 15035
14h.0-200 2014
160 15045
0.80
0.0
0.5
14003
66000 127000
1000
14003
66000 127000
Training Step
Training Step
Training Step
(a)
(b)
(c)
Figure 16: Crosscoder feature attribution on the Simple variant of the SVA task, using a crosscoder with 98,304 features trained on Pythia-160m 160M.
For attribution patching with clean/corrupted input pairs, the IG version is:
dm(a (x))
attr (x, (x.
= N Σ (f((
(12)
(1-
where E {0, ...... N-1) linearly interpolates between the baseline and target feature activa- tions. Consistent with Marks et al. (205)). we use N = 10 interpolation steps for the IG attribution score.
E.2 INDUCTION TASK
Transformers exhibit in-context learning capabilities (Brown et al. 2020 Zhao et al. 2021 Gao
al. (02T) through induction heads (Olsson al. cireuits look back over the sequence for previous instances of the current token (A), identify the subsequent token (B), and predict the same completion will occur again (forming sequences [A][B].. ... [A] + [B]).
To evaluate models' induction abilities and trace them the feature level, we construct samples with random tokens where identical patterns appear the middle and at the end of sequences. We test whether models can correctly copy previous text the next token. For precise feature-level analysis, we restrict next tokens to single capital letters with leading spaces. Since the induction task lacks natural corrupted counterparts, evaluation metric. Results are shown in Figure
the log probability of the correct token
E.3 SUBJECT- VERB AGREEMENT TASKS
Subject-verb agreement tasks evaluate whether models can predict verbs in the appropriate gram- matical form based on their subjects. We test four variants: (1) Simple-the verb directly follows the subject, e.g., "The parents are"; (2) Across-RC RC relative clause intervenes between subject and verb, e.g., "The athlete that the managers like does"; (3) Within-RC both subject and verb appear within the relative clause, e.g., "The athlete that the managers like": (4) Across-Pp prepositional
26

---

Published as a conference paper at ICLR 2026
0.6
1.1
1.1
J600 1004
0.9
0.8
1600 RooRS
0.80
0.2
0.7
0.5
0.0
0.4
0000
127000
6000
34503
0000
127000
(a)
(b)
(c)
Figure 17: Crosscoder feature attribution on the Across-RC variant of the SVA task, using a cross- coder with 98,304 features trained on Pythia-160M.
0.4
0.0
0.8
0.8
0.2
0.6
0.6
1000
0000
34b02
50000 127000
32
1000 6000
24502
50000 127003
32
1000
50000 127000 Training Stop
0000
24002
Training Stop
Training Stop
(a)
(b)
(c)
Figure 18: Crosscoder feature attribution on the Within-RC variant of the SVA task, using a cross- coder with 98,304 features trained on Pythia-160M.
phrase separates the subject and verb, e.g., The secretaries near the cars have". We use data pro- vided by Marks al. (2025), sampling 1000 examples from each variant. We the verb in the wrong form as the counterpart in attribution patching. Results are shown in Figure 16 17
and 18
E.4 INDIRECT OBJECT IDENTIFICATION TASK
Indirect Object Identification (IOI) evaluates whether models can correctly predict the indirect object (IO) based on repeated occurrence subjects (Wang et al. (023). In IOI tasks, sentences such as "When Mary and John went to the store, John gave drink to should be completed with "Mary." We generate IOI samples following the same strategy as Wang et al. (2023). using template sentences with random person names. We the subject name the corrupted counterpart in attribution patching. Results are shown in Figure 19
Note that while ablating top features significantly degrades performance, cannot recover the original metric using only these top features. This likely occurs because IOI is a complex task requiring multiple feature interactions. The features that distinguish between indirect objects and subjects represent only part of the full computational requirements, making isolated feature sets insufficient for complete task execution.
E.5 RANDOM BASELINE OF ABLATION
In Figure 20 we add basclines to Figure by ablating/prescrving random features. The results show that our feature selected by attribution patching is far more effective.
CROSSCODER FEATURE SPLITTING
Feature splitting is a well-known phenomenon in SAEs where increasing the dictionary size Neeatures causes features from smaller SAEs to fragment into multiple distinct features. Under feature split- ting, a single concept may not be represented by one feature, but rather by multiple specialized features that activate the same concept in different contexts.
27

---

Published as a conference paper at ICLR 2026
0.14
Poder
0.9
4680 1924
0.8
0.10
0.0
0.6
Modic
Meddc
0.7
0.01
0.92
0.00
14000
6000
Step
(a)
(b)
(c)
Figure 19: Crosscoder feature attribution on the IOI task, using a crosscoder with 98,304 features trained on 1ythia-160M..
Orig
1.0
10-
0.9
Moot
0.
60250g0000
1000
"tiiring Step
Thiining
(a)
(b)
Figure 20: Crosscoder feature attribution on the Across-PP variant the SVA task, with random baselines.
We observe similar feature splitting phenomenon in crosscoders, but across the temporal dimension. We find in rare cases, features with similar semanticity and direction in different snapshots may not be encoded in the same latent of crosscoder feature space, but result in separate features.
For example, we identify feature 66688, 53542, and 42307 that all activate on long sequences con- taining repeated text patterns, contributing to induction task (Figures 21 22 and 23 Each feature exists only during non-overlapping continuous periods across snapshots, with drastic emer- gence and disappearance.
To examine whether feature splitting across snapshots relates to dictionary size, search for fea- ture decoder vectors (across all snapshots) with cosine similarity above 0.7-a high threshold in such high-dimensional space- in crosscoders with varying Flteatures Across dictionary sizes ranging from 6, 6.144 to 98,304 features, we observe that while each snapshot after step 2,000 activates al- most exactly one feature representing this concept, the total number of distinct crosscoder features increases with dictionary size (Figure 24). This confirms that larger dictionaries lead to temporal feature splitting, where single underlying concept splits into multiple features active different training stages.
We also observe that temporal feature splitting predominantly occurs among densely activating fea- tures, i.c., features that activate frequently across many contexts. This crosscoder feature splitting likely arises from subtly different activation patterns that emerge over training. These findings sug- gest that language model features may evolve by refining their activation patterns, leading to more specialized representations that warrant separate feature assignments at different training stages.
G GENERALIZABILITY STUDIES
In this section, include more experiments training and analyzing crosscoders different base model/inniiaiizationddaaaa and different numbers of snapshot selection, to examine the general- izablity of methods.
28

---

Published as a conference paper at ICLR 2026
methodis is not threadsate, An enapoint should be contigured when the cient is created and before any sorvice requests are made. Changing atterwards creates
inevitable race conditions for any service requests inel "trrsiit retyyig ahoo *el
*Bperam endpontet
The endpoint (ex: "cogritc lio-dd us-easl-
famazcnwwsscom" con" II or URL, Inclusing the protosal lecc "hilps Doignttee www.annlll-pu-------eee ue--------s--eee waw.-oaaaooooosso bom'f] the region speciicc endpoint this dlent
communicate with.- - "dderecctedd use (glin wttpkknBBubbreettetdoooooooo wttp://ooooooooaaaaaaaaaiaaaaaee www.rr//eeeeeeetttaaaaaaaaeaeeee hwtp://iw.nndaammnnnnaartttooo for-
examplennt
(Bcode sellder.et mtContguratton//ee Enteonntaniiguaabbrlinngee #igringfegon)] 7/ @Ceprocatod= vad serEndpoimt Sering
seto the regional enopoint for this cllents service calls. Callers can method to which region they want to work with -ggoul default, service endpoints regions
endpoint pet An altemaaivve to (fiin
hilps protocol. htpp sppae <bo methed is not threadsals, A region should be gurieg whin
instaad, specily in theie "(il ClamiConiiguniiii) supp ied at conwirustion of
the client
sarvica-
trequusts
tranall hall The region this client will communicate with. See (Sink Reggoongeefeegggmmmmma www.://ammmmama.ooooooooooooomoo Regions))
Changing attanward
tetryings/boo
@parrm region
Figure 21: Top activation of Feature 66688 in the 98.304-fe crosscoder on Pythia- 160M.
Angry Binds Space Med APK 2022 v22.15 15 jattstt Version, Unlimited Moie game is the uitimate in family fun. it's easy to leam, it's challonging enough for adults and your shone
slke, hours replay vaue best part? play Angry Birde Space love this game as much we do and who doeen't? -don't wait another moment before downloeding Angry Urds Space Med APK today. 'Ravio Entertainment Corporationlipdstedd
for computer downloed! you
ago Size- Android 4.1Get
Birds
Unlimted Everything= Amaring Features
Bids
APK+
Red Bird-22 22 Yellow 2.. Blue Brd- White Bidd225 Black S/dd-2. Boomerang ang Brd+127 Big Sold 2.8 Bidd Sling-1288 Slow Moxion-12.00 Hitling Obstacies Singshot='2:1t Obstacles Space-lS FACs Angry to Play--34 Can play ottinn without an internet connection on my laptop, PC, Mac, of 35 Do need to mot my Android davicet= Anggy Birds Space Mod APK Unlimited Everything- kingry Birds Mod APK uttmmate replyy value. The best zort? You can play Angry Birds Space APK on your amone or labbee with need aicomputer downtsad. So you love this as much as we da- and doesn't? don't wait another moment before downloading Angry Birds Space Mod APK today DDownisdd Angry Birds Rio Mod APK-DDowddodd Adventure
Mod APK-1.1 Angry Binds Space Mad Works Offine?-1322 Ads-Freeh
it's
challenging enough adults
of sike, and
Figure 22: Top activation of Feature 53542 in the 98,304-feature crosscoder on Pythia-160M
G.1 GENERALIZATION ACROSS MODEL FAMILIES/SEEDS
We conduct experiments on two additional Pythia 160M models with different random seeds (Pythia 160M Seedl and Seed2) (van der Wal et al. 2025) and the Alias run of Stanford CRFM's GPT-2 model All of these models have activation space of 768 dimensions. We train crosscoders of 24,576 features (32x) on Layer 6 of them, and inspect their feature evolution.
Different initializations (Pythia 160M variants). We observe highly consistent results across differently initialized versions of Pythia 160M. Decoder norms of initialization features follow iden- tical trajectories, and emergent features begin rising around step 1000, exhibiting clear two-phase pattern with the same transition point (Figure 26 and 27). Attribution experiments confirm that we successfully identify sparse crosscoder features with significant contributions to downstream tasks (Figure 29 and 30
Different base model (GPT-2). For Stanford CRFM's GPT-2 model, we observe a clear two- phase pattern, with emergent features rising around step 1,000. Upon manual inspection of these features, we still find that initialization features exhibit token-level information, while more complex patterns emerge in emergent features.
However, we find notable difference: due to absolute positional embeddings, initialization fea- ture norms do not peak at the beginning but rather at later steps (with above-threshold norm at beginning). This difference arises because the absolute positional embeddings initially exhibit large norms (larger than the word embeddings) and dominate the activation norm. Since we normalize all activation sources the same average activation norm /dmodel, this large positional contribution substantially reduces the observed norms of initialization features. (Figure 28). phase transition still exists across models, despite small
To sum up, the aaaiistcal-to-ffatuueeeeeee differences. Attribution experiments (Figure(3] again confirm the effectiveness of sparse crosscoder features, with generally similar feature evolution patterns for specific downstream tasks.
Other models with accessible checkpoints. We also investigated other model families, including BLOOM (Scao et al. (2022) and OLMo (OMM et al. (024). Unfortunately, these models either lack sufficient intermediate checkpoints for fine-grained analysis of pretraining dynamics (BLOOM), or lack sufficiently early checkpoints capture the two-stage transition (BLOOM and OL.o).
29

---

Published as a conference paper at ICLR 2026
catrongacce Mitea-ll="mmme message-vide href="
http:p/w.w...../aeiieeooooo Cocpaoagaoaaararrrr rete "radooene norelerne nnfollow SPowel Rangers (2017 Moxie) Official
Teaser Tralle- "Discovr The Pawer <jo-<strong> no [Yo_Tube] sbyl <bn)=<ammSABAANS POWER RANGERS to lows five ordinary teena who must bacome something Angel
extroordinary when they Ioann
verge being oolterated by
destiny,
quicity discover they are the only ones who can save the planet. But so, they wil have overoome their real-ife issues and before it's too late, band together the Power Rangers, <bys ebrf-Pretty typical post-Nolon
Lionegate money. really
Rangers any emricing
in pressive. Future tralers night excite more +1. Power Rangers (2017 Moxie Official Teasel Tralle- "Bisooer The Power (rauTube))-SAAa POWER RANGERS follosel
ordinary
an allen threat. Choson by destin,, ouriheraes quilly discaver they are the only ones Who save the planet. But do they will have oversome their Issues before too band logether
Deesnkrealt reall tap into Power Rangersinn artiing way, Notvery moressive Future tralers might eschte more. TTralerr looks swesome,, like the humor.) lkke the tone. It's breakfast club meets Chronice, with an ion
Rangers. Pretty post-Hälan franchise
Figure 23: Top activation of Feature 42307 in the 98.3urfe crosscoder on Pythia-160MM
G.2 GENERALIZABILITY ACROSS DATASETS
We conduct experiments the original Pythia 160M snapshots, but train and analyze 24,576-fe crosscoders on the FineWeb-Eae dataset (Penedo et al. 2024), which consists of educational web- pages. Our results (Figure 32) show nearly identical patterns those observed with Pythia 160M crosscoders trained on SlimPajama, exhibiting very similar feature evolution dynamics.
G.3 SENSITIVITY TO SNAPSHOT SELECTION
All of our above experiments train crosscoders on 32 pre-training snapshots of the language model. To further examine whether snapshot selection affects the trained crosscoder and the captured fea- ture evolution, we train a 14.576-featurs crosscoder on 16 pre-training snapshots of Pythia 160M, downsampled from the original set by selecting every other snapshot. The result (Figure (33) shows that the overall feature evolution exhibits nearly identical trends, with only a reduction in temporal resolution.
H MORE EXAMPLES OF FEATURE
We additionally list emergent features and demonstrate their top activating samples and de- coder norm evolution in Figure 34
I THE USE OF LARGE LANGUAGE MODELS
The use of Large Language Models as an assistive tool in this paper is limited to the following two aspects:
1. The automated generation of feature complexity scores, as detailed in Section 4.2 and Ap- pendix B
2. Grammar correction and stylistic refinement of the manuscript.
OOtaineed from httpt///w///ggggoooooo//////ottr
30

---

Published as a conference paper at ICLR 2026
Pythia 1 160M (61 44 features)
1.0
#2851
Cosine Similarity
0.9
0.7
0.6
0.5
Pythla-160M (12288 features)
1.0
Cosine Similarity
0.9
0.8
0.5
Pythia-160M (24576 features)
1.0
Cosine Similarity
0.9
0.8
0.7
0.5
Pythia-1600 (49152 features)
#20072
#35594
Cosine Similarity
#35105
0.9
#154404
#2655
0.8
#3735)
#2606
0.7
0.6
0.5
Pythia-160M (98304 features)
1.0
#53511
+-22307
Cosine Similarity
#66685
0.9
#50712
#46001
#61395
#5222
0.7
#11.39
0.6
0.5
2000
6000
10000
34000
87000
143000
Training Step
Figure 24: Crosscoder features similar Feature 53542 of 98.304-feature crosscoder. The number of features split increases with dictionary size.
31

---

Published as a conference paper at ICLR 2026
adce 3023e intural
Patts tort 20332 Intural aaiee rnturel
00
23
Snapshot Indox
UMAPP
19
15
10
12
14
16
UMAP
Figure 25: UMAP visualization of features shown in Figure
24
Osccer Nom Oveltiie Ove Tranigg
Figure 26: Overview of cross-snapshot feature decoder norm evolution in Pythia- Pythia-160M 160M Seed
32

---

Published as a conference paper at ICLR 2026
Figure 27: Overview of cross-snapshot feature decoder norm evolution in Pythia- 160M Sced2.
Desocir Norm Dunition Thaniing
Figure 28: Overview of cross-snapshot feature decoder norm evolution Alias run of Stanford CRFM GPT-2.
1.1
Adcribukonn 500t
ACN 15035
Fetor 2032
0.9
168 150-5
0.8
Peatrr 7180
Testrr 11.55
0.7
0.7
0.6
9.05
0.5
0.4
32
1000
6000
14000 60000 123000
1000 6000
14500 60500 127000
14005 66000 127000
Training Step
Training Step
Training Step
(a)
(b)
(c)
Figure 29: Crosscoder feature attribution on the Across-PP variant the SVA task, using a cross- coder with 24,576 features trained on Pythia- 160M Seedl.
33

---

Published as a conference paper at ICLR 2026
1.1
0.0
0.0
0.8
0.7
3982
0.5
0.4
Training Stop
Training Stop
Training Step
(a)
(b)
(c)
Figure 30: Crosscoder feature attribution on the Across-PP variant the SVA task, using a cross-
coder with 24,576 features trained on Pythia-160M Seed2.
Poder 10o
M
0.25
1.0
1.0
0.35
Peder 2110
3
9.8
0.30
Peder 1112
0.7
0.7
0.6
0.00
0.5
50
400
1900
5000
50000 283000
35
420
1900
PROD
30000 203000
1900 pado 30000 263000 Training
Traiiing Stop
Training Stop
(c)
Figure 31: Crosscoder feature attribution on the Across-PP PP variant of the SVA task, using a cross- coder with 24,576 features trained on the Alias run of Stanford CRFM GPT-2.
Drodeetior bussutce Trurine
Snapero ader
Figure 32: Overview of cross-snapshor feature decoder norm evolution in Aythia-160M trained on FineWeb- Web-Fdu Edu dataset.
34

---

Published as a conference paper at ICLR 2026
Decider Norm Unsiudice Ova traviig
Smapshot
Figure 33: Overview of cross-snapshot feature decoder norm evolution in Pythia- 160M, trained on 16 snapshots.
Opinion
28664
with college degrees need bailoun." "It'shard forime stomach the idea of biling the massas-aboutt ing party and the people are sick starving. Peersonilly am aghast that the Liberals are chasing Green vot-
penalty. MMyy Personal Opinion the Pubjant
inclined to think (always recagnizing the Pope
ne: How come they never get any flack ite way figure the more other side howls the more KDE rodes="""pprson
trollers listen ovents from
thoughtl agree the first statement don't
Emphasis Sentence
29335
with Nabama flag. lit's refreshing that young people are learning about the nuances of our state's 'branding
thn Country community embracing trying her
oocaton CEO Steve Bates said: 'I's great shelthe UK gevernment and NHS committed to exploring a new pa
going agle follow "e""ts "e''''s really cool to me that from this small town went and like totally cha
disappointing
mislesding pa
st incredible, unbelievable.
Say Register
20887
"ycevelopers==Deeeepprrrr D= <a href= /ree==//egisserr Register<ss- <a href= "juuthenticate" -sgin---/
ORDER - = - public Iit getRagister)) -(-) rauurn register - ------------ - a public vold Eattnab
reyhaunds); eancel registration or refuse register a greyhound racing club or greyhound trial ussday, Jan. 3 Friday, Jan, 6: Cold Blonded Creatures (aatttrrrsRRatttttt heree Camp Black Bear at AD Barnes Park var registration require" (supportieed sngorrtegissrationn. "s" llver chall require ""o"""""
28,30
strictnass tradition and "Califormia *+lit
overlooked
himself reputation
on began se selling its products online. The store 'couldn't here" if they not begin sell selling products on the
the industry standard bucke passing. cannot danied that visually, clothes better slimmer
was to get local stories out to the public. could notbee more up date since the principal iconography can generated
series cannot baught
stick ball guys
URL
2900
sestan-hkes--etrrl-eee paarral-riceebb-r---2 1oay----2----pe--itteee 2--9h-perliire880055:---- China's natural gas outout to surpass ol around 2
resents:
Bzy "BANG"= 'd=177784178 width= height= 350"] http:/hhht/////w..///..
2et-0--bee-----00---00 3c330088// al-php- spl)- [[SSyleCC http::/www......eeeeo//o/e222222 (aste /ttt///eelo//eee//3333333
here-can-i-o can-i-downlaad-aee wwwpwladdwwidoowwwoeeo winddaddddddd-od-----e--eeeeeeee witp:i-havee////deaa-aae 0ttp:///888860022-2----000-----0
Saanking-o-buuybacle htt.://ww.dddiiddddaarrrrooo/eed cmm UK court hearing an urgent application
Serial Numbers
12295
Astronnmm and estrophysics "Bloooggi stations [i Cafeterias [1] Cataloo of Federal Domestic Assistal
sation for develoopment (1) Education emergencies (II HHI/AIDS (II+ Humanitrrran action and emergencie
vt Fair (I)+AAIPAD ARCC Madrid *Art Austion (I]- Besel ([]+Arr Besel Hong Kong (Alt+Art Fairs (4
+Management, Organization, Corporate Governance) (" [[msssin key: serch-facet.tree. pewe.open-sectonn. Intro
(-) Young people and femilles Heltth and welfare (B) 445 percent young people see pollution as prob
Figure 34: More features in the 2e.auu-fe crosscoder on Pythia-6.9B
35


Reviews:
[{"review_id": "8Eu9ctl0eY", "reviewer": "Reviewer_zNuD", "summary": "The paper claims to explore how meaningful features emerge as language models learn.  Using crosscoders to align representations across training snapshots, the authors trace training dynamics over time. They find a clear turning point early in pretraining, where models move from simple statistical patterns to more structured, feature-based understanding. Through attribution and patching analyses, these evolving features are linked to linguistic behaviors such as agreement and induction.", "questions": "1. Could the authors clarify the rationale behind choosing SlimPajama as the pretraining dataset? Were other datasets considered, and if so, why were they excluded?\n2. In Eq. (1), how were activations aggregated across snapshots? Why was summing chosen, and how might alternative strategies (e.g., averaging, normalization, or selective sampling) affect the results?\n3. Can “initial features” and “emergent features” be defined more formally within the paper’s notation? A clearer definition would make the results more interpretable and reproducible.\n4. Have the authors compared their findings with recent analyses of early-stage n-gram learning (e.g., Svete & Cotterell, 2024; Nguyen, 2024, Chen et al, 2024)? What new insights, if any, extend beyond those studies?\n5. How stable are the observed feature dynamics across different base models or seeds? Would larger or differently initialized models exhibit the same “turning point” behavior?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "- The paper touches on an interesting and relevant question, how internal features evolve during language model pretraining. \n\n- It attempts to provide a structured view using crosscoders. \n\n- The motivation to study model dynamics rather than static checkpoints is reasonable and aligns with growing interest in understanding training trajectories.\n\n- The work could serve as a preliminary exploration toward more rigorous analyses of concept formation in neural networks.", "weaknesses": "- Novelty. The paper offers little substantive novelty beyond prior work on crosscoders (Lindsay et al., 2024). The method follows existing frameworks closely, with minimal theoretical or heuristic grounding for its design choices. For instance, \n    - 1) The selection of the SlimPajama dataset is not clearly justified, nor is the limited set of base models sufficient to generalize the findings.\n    - 2) An ablation study on activation aggregation in Eq. (1) would help clarify whether summing activations is the optimal or even appropriate choice.\n\n- Contribution. The contribution remains narrow, more a descriptive report than a step toward formal interpretability. The reliance on visualizations such as heatmaps and curves, without clear quantitative or linguistic interpretation, makes it hard to connect the observed patterns to interpretable model behaviors. \n\n- Rigorousness. Several findings, particularly those regarding early n-gram learning, largely lack of contextualization with known results from prior work (e.g., Svete & Cotterell, 2024; Nguyen, 2024, Chen, 2024). \n\n- Writing. The writing quality is also uneven. The paper reads more like a lab report than a cohesive scientific paper. For instance\n    - 1) Lack of grounded formalization for key concepts. For example, can we define \"initial features\" and \"emergent features\" (line 202-line207) formally using the notations in the paper? \n    - 2) Some phrasing is imprecise. \"the pre-training process remains largely a black box.\" The sentence is a bit unclear. The process of pretraining is generally gradient descent on large-scale of datasets. I guess what the authors mean here is \"the pretraining dynamic\" instead of \"process\".\n\nOverall, the work lacks the theoretical grounding, novelty, and clarity needed to make a strong contribution. That said, it is definitely worthwhile for presenting at a workshop after revision.\n\n\n``References``\n- Anej Svete and Ryan Cotterell. Transformers can represent n-gram language models. arXiv preprint\narXiv:2404.14994, 2024.\n- Timothy Nguyen. Understanding transformers via n-gram statistics. arXiv preprint\narXiv:2407.12034, 2024.\n- Chen et al. Jet expansions of residual computation  arXiv preprint 2410.06024, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lnRBQikkA4", "reviewer": "Reviewer_uCjx", "summary": "The authors study the evolution of features during LLM training using crosscoder SAEs. They first study the emergence, persistence of features in LLMs, showing that many features persist for long periods and that their emergence time varies. They then consider feature complexity, showing a moderate correlation between feature complexity and the time at which the feature's emergence peaks. After analyzing feature dynamics, they study the causal relationship between features and task performance, using attribution patching to assess causal relationships for subject-verb agreement, indirect object identification, and induction tasks. Finally, they show that the initial part of training learns statistical features of token distributions and study the changes in feature dimensionality during training.", "questions": "How do the results in 6b and 6c compare with a similar plot produced by ablating all but/ablating $k$ random features?\n\nCould you produce similar analysis with additional models with available checkpoints, such as Stanford CRFM GPT-2 or Olmo?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The authors introduce a novel crosscoder method for studying feature dynamics in LLMs. They analysis of feature emergence and causal study is clear and thorough.", "weaknesses": "Figure 3c is a bit difficult to interpret, since the analysis references the number of training steps but the figure is labeled in terms of snapshot number. Since there is a stratified sampling approach to selecting snapshots (and since the Pythia models use log-spaced snapshots + snapshots every 1000 training steps), it's not obvious which training step corresponds to which snapshot in the figure and hard to verify the accuracy of the claims. \n\nThe argument about feature dimensionality and compression is a bit unclear with an unclear takeaway.\n\nOnly relying on one family of models reduces the potential impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yroK4K5tSc", "reviewer": "Reviewer_QXzd", "summary": "This paper investigates how internal features in large language models emerge and evolve during pre-training. Using cross-snapshot crosscoders—a variant of sparse autoencoders adapted to align activations across training checkpoints—the authors track interpretable features throughout the training process of Pythia models. They identify initialization and emergent features, quantify their persistence, and show that feature complexity increases with training progress. Attribution-based analyses link these microscopic feature dynamics to downstream task improvements. The study further reveals a clear two-phase transition in pre-training, from an early statistical learning phase dominated by token-level regularities to a later feature learning phase characterized by sparse, semantically rich representations.", "questions": "How sensitive are the results to the number or spacing of snapshots?\n\nCould decoder-norm scaling artifacts or feature-splitting affect the interpretation of “feature emergence”?\n\nDoes attribution patching preserve causal validity when features are highly correlated?\n\nHave the authors verified that the same cross-coder features generalize across layers or different datasets?", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper presents a technically novel adaptation of crosscoders for temporal analysis, offering the first fine-grained view of feature evolution across pre-training snapshots. The experiments are extensive and carefully controlled, combining mechanistic interpretability with learning dynamics. The empirical discovery of a statistical-to-feature-learning transition provides a coherent mechanistic complement to theoretical frameworks such as the information bottleneck and singular learning theory. The work stands out for its methodological clarity, strong empirical validation, and relevance to both interpretability and representation learning research.", "weaknesses": "The study provides interesting descriptive evidence, but the causal claims remain largely correlational—attribution patching is still a heuristic rather than proof of necessity.\nThe analysis is limited to Pythia checkpoints and relatively simple syntactic tasks, so generalization to larger or more complex models is unclear.\nSome methodological choices (e.g., decoder-norm as feature-strength proxy, selection of snapshots) lack formal justification or ablation.\nThe paper is also quite engineering-heavy, and clarity could improve in conveying what new conceptual insight the cross-coder adds beyond existing SAE frameworks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8kciQ5mzLi", "reviewer": "Reviewer_effr", "summary": "The paper separates the pretraining process of large language models (LLMs) into two distinct phases: a statistical learning phase and a feature learning phase. The authors argue that downstream performance improvements are closely tied to the emergence of features during the latter phase. To quantify this, they introduce a “crosscoder” network that detects features in activations with respect to a so-called decoder norm and track the development of this norm during pretraining. The paper presents empirical results linking these phases to downstream task performance.", "questions": "1) Why did you choose a log-scale for presenting all results?\n\n2) As all experiments use Pythia, I would like to see a discussion of the training algorithm used there. How many warm-up steps were used? When did the learning rate change?   There is a (small) chance that the seen effects may be trivial if they can be related to, for example, a learning rate change.\n\n3) The y-axis states a \"normalized decoder norm\". What is normalized exactly? It appears to me that each feature reaches one at some point, which may again make the results less surprising, as each feature must either be present at the start or emerge at some point if one is eventually reached.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) The idea of analyzing the emergence of features during pretraining is interesting and aligns with the current interest in the mechanistic interpretability of LLMs.\n\n2) The paper attempts to link internal feature emergence in the model to downstream generalization, which is a valuable direction.", "weaknesses": "1) The decoder norm is part of the training objective, and the weights in the norm are (obviously) part of the model that constructs the feature.\nI find this quantity borderline to abstract to serve as a metric that should verify the central concepts of this paper.\n\n2) Figures 4–6 are central to the main claims but are difficult to interpret, as the x-axis is on a logarithmic scale representing training steps.\nThe authors claim that two distinct learning phases appear, but on a linear scale, this might resemble a single continuous (albeit noisy) learning process. The choice of log-scale seems to visually exaggerate the phase transition; this requires at least discussion and justification. As a remark: on a linear scale, the \"statistical phase\" is less than 1% of training time.\n\n3) Figure 5 introduces an additional LLM for the interpretation of features, which adds another layer of complexity and potential confounding effects. For me, the interpretation of LLMs should ideally come without LLMs to avoid a circle.\n\n4) Minor remark: Including a figure before the abstract is unconventional.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)