Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning
Authors: Julian Minder
Clément Dumas
Caden Juang
Bilal Chughtai
Neel Nanda
Venue: NeurIPS 2025 poster ()
Keywords: interpretability
mechanistic interpretability
model diffing
chat-tuning
crosscoder
sparse autoencoder
ai safety
sae
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: Using crosscoders (SAE variant) for chat-tuning concept identification, we diagnose spurious chat-only concepts arising from L1 loss artifacts and show BatchTopK robustly reveals genuine, interpretable ones.

Abstract: Model diffing is the study of how fine-tuning changes a model's representations and internal algorithms. 
Many behaviors of interest are introduced during fine-tuning, and model diffing offers a promising lens to interpret such behaviors. 
Crosscoders are a recent model diffing method that learns a shared dictionary of interpretable concepts represented as latent directions in both the base and fine-tuned models, allowing us to track how concepts shift or emerge during fine-tuning. 
Notably, prior work has observed concepts with no direction in the base model, and it was hypothesized that these model-specific latents were concepts introduced during fine-tuning.
However, we identify two issues which stem from the crosscoders L1 training loss that can misattribute concepts as unique to the fine-tuned model, when they really exist in both models. 
We develop Latent Scaling to flag these issues by more accurately measuring each latent's presence across models.
In experiments comparing Gemma 2 2B base and chat models, we observe that the standard crosscoder suffers heavily from these issues. Building on these insights, we train a crosscoder with BatchTopK loss and show that it substantially mitigates these issues, finding more genuinely chat-specific and highly interpretable concepts. We recommend practitioners adopt similar techniques.
Using the BatchTopK crosscoder, we successfully identify a set of chat-specific latents that are both interpretable and causally effective, representing concepts such as false information and personal question, along with multiple refusal-related latents that show nuanced preferences for different refusal triggers. 
Overall, our work advances best practices for the crosscoder-based methodology for model diffing and demonstrates that it can provide concrete insights into how chat-tuning modifies model behavior.

Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning
Julian Minder
Clément Dumas
Caden Juang Bilal Chughtai Neel Nanda
PEPFL
"ETHZ
Tccoe Normale Supérieure Paris-Saclay Université Paris-Saclay
Northeastern University
julian. Illian.minderdeppi. ch, clement. Sumas@@ns-paria-eeee
Abstract
Model diffing is the study of how fine-tuning changes model's representations and internal algorithms. Many behaviors of interest are introduced during fine-tuning, and model diffing offers promising lens to interpret such behaviors. Crosscoders are a recent model diffing method that learns a shared dictionary of interpretable concepts represented latent directions in both the base and fine-tuned models, allowing us to track how concepts shift or emerge during fine-tuning. Notably, prior work has observed concepts with direction in the base model, and it was hypothesized that these model-specific latents were concepts introduced during fine-tuning. However, we identify issues which stem from the crosscoders LI training loss that can misattribute concepts as unique to the fine-tuned model, when they really exist in both models. We develop Latent Scaling to flag these issues by more accurately measuring each latent's presence across models. In experiments comparing Gemma 2B base and chat models, we observe that the standard crosscoder suffers heavily from these issues. Building on these insights, we train a crosscoder with BatchTopk loss and show that it substantially mitigates these issues, finding more genuinely chat-specific and highly interpretable concepts. We recommend practitioners adopt similar techniques. Using the BatchTopK crosscoder, we successfully identify a set of chat-specific latents that are both interpretable and causally effective, representing concepts such as false information and personal question, along with multiple refusal-related latents that show nuanced preferences for different refusal triggers. Overall, our work advances best practices for the crosscoder- bassed methodology for model diffing and demonstrates that it can provide concrete insights into how chat- Seat-tuning modifies model behavior.
1 Introduction
Classically, mechanistic interpretability [Sharkey et al., 2025, Mueller et al., 2024, Ferrando et al., 2024, Elhage et al., 2021, Olah et al., 2020] aims to reverse engineer an entire model [Huben et al., 2024, Elhage et al., 2022), circuits implemented by the model solve particular tasks [Wang et al., 2023a]. Model diffing offers an alternative method by focusing on changes induced by fine-tume fine-tuning. Since fine-tuning typically involves far less compute than the pre-training phase that establishes general knowledge and generic circuitry, its resulting modifications are expected to be limited in scope. This targeted nature suggests model diffing could be more tractable approach mechanistic interpretability than the full model analysis, while still providing valuable insights into core features of model's behavior.
'Equal contribution. Order randomized.
'e open-source our code, training library, models, wandb runs and demo notebook to explore latents.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Model diffing might indeed be ineredibly useful. The process of fine-tuning a model is what makes it useful as a tool or agent. Better understanding the mechanisms that give reasoning models [DeepSeek- Al et al., 2025, OpenAl et al., 2024] heightened capabilities as compared to base or chat models might allow us to debug their failures and improve them. Fine-tuning also often introduces a number of problematic behaviors, for example, sycophancy [Sharma et al., 2023]. Future Al safety and alignment concerns [Greenblatt et al., 2024, Meinke et al., 2025, Betley et al., 2025] may emerge specifically in fine-tuned models. For example, long-horizon RL could incentivize models to exploit reward signals and act deceptively. Model diffing could allow us to detect this.
Prior model diffing research has investigated how models change during fine-tuning [Shah et al., 2023, Lindsey et al., 2024, Bricken al., 2024, Prakash et al., 2024, Lee et al., 2024, Jain et al., 2024, Khayatan et al., 2025. Thasarathan et al., 2025, Wu et al., 2024. Mosbach, 2023, Merchant et al., 2020, Hao et al., 2020, Kovaleva et al., 2019, Du et al., 2025, Minder, 2024]. While these studies have hypothesized that fine-tuning primarily shifts and repurposes existing capabilities rather than developing new ones, conclusive evidence for this claim remains elusive. Model diffing remains nascent field that lacks established consensus and mature analytical tools. Much prior work has leveraged ad-hoc techniques for understanding how models change in narrow ways (e.g. focusing on particular circuit), or have been toy model. It is unclear whether prior approaches would scale to understanding the kinds of fine-tuning large models actually undergo.
Recently, Lindsey et al. [2024] introduced the crosscoder, a novel and scalable tool for model diffing. Crosscoders build on the popular sparse autoencoder (SAE) [Huben et al., 2024, Bricken et al., 2023, Yun et al., 2021], which has shown promise for interpreting a model's representations by decomposing activations into a sum of sparsely activating dictionary elements. There are many variants of crosscoders; the variant we are concerned with in this paper concatenates the activations of the base and chat-tuned model residual streams and trains a shared dictionary across this activation stack. Thus, for each dictionary element (aka "latent", corresponding to one concept), the crosscoder learns a pair of latent directions - one corresponding to the base model and one to the chat-tuned model. Crosscoders can thus potentially identify which latents are novel to the fine-tuned model, which are novel to the base-model, and which are shared. We term these sets chat-only, base-only, and shared respectively. Lindsey et al. [2024] identify chat-only latents by looking at the norm of the latent directions if the latent direction of the base model has zero norm, this indicates that the latent is chat-only.
In this work, we critically examine the crosscoder and identify two theoretical limitations of its training objective, that may lead to falsely identified chat-only latents (Section 2.2):
1. Complete Shrinkage:
The sparsity loss can force base latent directions to zero norm, even when they contribute to base model reconstruction.
2. Latent Decoupling: The crosscoder may represent a shared concept using a chat-only latent when it is actually encoded by a different combination of latents in the base model, as the crosscoder's 's sparsity loss treats both representations as equivalent.
We develop an approach called Latent Scaling to detect spurious chat-only latents, inspired by Wright and Sharkey's 's [2024] SAE scaling (Section 2.3), and demonstrate that the above issues occur in practice. While the norm- bannd metric from Lindsey et al. [2024] appears to identify a clean trimodal distribution of base-only, shared and chay-only latents, we show that this is an artifact of the loss function rather than a meaningful distinction. Our conclusion is that the crosscoder loss does not actually have an inductive bias that helps to learn better model-only latents. Nonetheless, we demonstrate that crosscoders trained with BatchTopK loss [Bussmann et al., 2024] exhibit robustness to the above issues (Section 3.1) and identify a larger number of genuine model-specific latents. We show that in the BatchTopK crosscoder, the norm-based metric successfully identifies causally relevant latents by measuring their ability to reduce the prediction gap between base and chat model. In contrast, this metric fails in the L1 crosscoder, where Latent Scaling becomes necessary to identify the truly causally relevant latents. Finally, we outline that the chat-only latents found by the BatchTopK crosscoder are highly interpretable (Section 3.3), revealing key aspects of chat model behavior such as the role of chat template tokens, persona-related questions, detection of false information, and various refusal related mechanisms.
2

---

Overall, we show that using BatchTopK loss overcomes the described limitations of Ll-trained crosscoders, validating them as useful tool for understanding fine-tuning effects in large language models.
2
Methods
Note: For reference, we provide comprehensive glossary of key terms and mathematical notation introduced through the paper in Appendix
2.1 Crosscoder architectures
To build intuition, the crosscoder's goal is to learn a dictionary of interpretable concepts (latents) that can explain the activations of both models. It consists of an encoder and decoder. The encoder takes the activations of the base and chat models and projects them into shared high- nigh-dimensional sparse space, where each dimension corresponds to potential concept. The decoder then reconstructs each model's activations using model-specift representations for each latent, combining them according to the sparse encoding. The key insight is that while both models share the same sparse encoding for given input, the crosscoder learns separate decoder representations for each model, allowing concepts to have different importance or manifestation in each model.
More formally, let r be string and huse (xx), (tt.,n h chat (x) I2 denote the activations at a given layer. The encoder computes a sparse encoding fx) E R20 for each latent j E (I..... D}. The decoder then reconstructs the activations as:
h hase (x:) = Σ fj(x)) dl hluue + b b, buse
and h (x)
Σ (x) dh + +bbe deccc
where ahaose, ddchat E IRd are the model-specific decoder representations and I hase b decoder biases. The crosscoder minimizes reconstruction errors e gbse (xx) h base
dec E IR
(x)
h base
and e chat (xx) = hchat(()) ) chat (x)) while enforcing sparsity.
We examine two sparsity mechanisms. The L1 crosscoder [Lindsey et al., 2024] adds LI penalty to the loss:
(x) = (x) (I14) + ||d l12)
(2)
The BatchTopK crosscoder [Bussmann et al., 2024] instead enforces LO sparsity by selecting only the top nk latents with highest scaled activation (xx)) ||2 + ||ddh"" [12) across batch of n strings."
More details on crosscoder implementation can be found in Appendix B.
2.2 Decoder norm based model diffing and its problems
To leverage crosscoders for model diffing, we can exploit the observation that while latent activations fy (z) are shared between models, the decoder vectors deeha and di are unique to each model.
To leverage crosscoders for model diffing, we exploit that while the sparse encoding f (a:) is shared between models, the decoder representations d and dhaee model-specific. When latent is important for both models, both decoder representations need substantial norms for reconstruction. Conversely, latent specific to the chat model will have ||dchall| >> while ||chaeel || + O, as the base decoder has no use for this latent.
We quantify this using the relative norm difference Anom T
1] from [Lindsey al., 2024]:
|| |gchat [|
base
l|
A n (j) max ma 12-
buse l2)
Intuitively, Anem indicates pure chat-only latent (base decoder has zero norm), Anomn 0 indicates a pure base-only latent, and A Rs 0.5 suggests equal importance in both models. As shown in Figure 1, we classify latents as base-only (0-0.1), chat-only (0.9-1.0), or shared (0.4-0.6).
Are chat-only latents really chat-specific? If a latent only contributes to one model, the norm of the decoder must tend to zero for the other model. But is the converse true? Specifically, we ask the
? During inference, leamed threshold 0 zeroes out latents below it. See Equation (14).

---

100
shared
25
6000
4000
chat-only
Latents
chat-specific
0.93
4000
0.95
2000
base-only
2000
0.1
0.5 0.6
0.9
1.0
0.4 0.5 0.6
0.9
1.0
Relative Norm Difference Asom
Relative Norm Difference Aaom
(a) LI crosscoder.
(b) BatchTopK crosscoder.
Figure 1: Histogram of decoder latent relative norm differences (Ammm) between base and chat Gemma 2B models [Riviere al., 2024], for both the L.1 crosscoder (left) and the Bot crosscoder (right). A value of 1 means the decoder vector of latent for the base model is zero, indicating the latent not useful for the base model (chat-only latents). A value of means the chat model's 's decoder vector has norm of zero (buse-only latents). Values around 0.5 indicate similar decoder norms in both models, suggesting equal utility in both models (shared latents)'. We also show the chul-only latents that are truly chat-specific and that are not affected by Complete Shrinkage (error ratio vf < 0.2) and Latent Decoupling (reconstruction ratio v" 0.5) - the chat spet-specific latents. Most of the Ll crosscoder chat-only latents suffer from these issues.
question: if a latent has decoder norm zero in the base model, is it necessarily chat-specific? We focus on the chat-only set, will contain features that emerged during chat-tuning.
Reasons to doubt chat-only latents. There are reasons to suspect chat-only latents might not be chat-specific. Firstly, both qualitative and quantitative analysis of L1 crosscoder latents reveals relatively low percentage of interpretable latents within the chat-only set (See Section 3.3). More worryingly, inspection of the LI crosscoder loss (Equation (2)) uncovers two theoretical issues that could result in latents j. which are defined by their decoder vectors dj and activation function fj- being classified chat-only, despite their presence in the activations of the base model:
When the contribution of latent j is smaller the base model than in the chat model, L1 regularization can force dlase to zero despite its presence in the base activation. Consequently, g base contains information attributable to latent j- This similar to "shrinkage" or "feature suppression" in SAEs [Jermyn et al., 2024, Wright and Sharkey, 2024, Rajamanoharan et al., 2024].
Complete Shrinkage:
2. Latent Decoupling: a chat-only latent is also present in the base activations but is reconstructed by other base decoder latents. this case, the base reconstruction h have contains information that could be attributed to latent See Appendix D for illustrative example.
Why BatchTopK crosscoders might fix this. The BatchTopK crosscoder may address both Com- plete Shrinkage and Latent Decoupling issues that affect the crosscoder. The key difference lies in their respective loss functions and optimization objectives.
For the Ll crosscoder, the loss function in Equation (2) includes Ll regularization term that directly penalizes the norm of decoder vectors. This creates pressure to shrink decoder norms toward zero when latent's contribution is minimal, potentially causing Complete Shrinkage even when the latent has some explanatory power. In contrast, the BatchTopK crosscoder uses a different sparsity mechanism. Rather than penalizing decoder norms, it selects only the top most active latents per sample during training. This approach has two important advantages:
1. No direct norm penalty: Without explicit regularization decoder norms, there's no optimization pressure to drive ||dbas/| | to zero when the latent has explanatory value for the base model, reducing Complete Shrinkage.
2. Competition berween latents: The top-k selection creates competition among latents, dis- couraging redundant representations. This helps prevent Latent Decoupling by making inefficient to maintain duplicate latents that encode the same information.
We observe larger activation norms in the chat model, which shifts our distribution rightward, revealing that the chat model amplifies the norm of representations shared with the base model.

---

The BatchTopK approach thus creates an inductive bias toward learning more genuinely chat-specific latents, as the model must efficiently allocate its limited "budget" of active latents. This should result in fewer falsely identified chat-only latents and cleaner separation between truly model-specific and shared features.
2.3 Latent Scaling: Identifying Complete Shrinkage and Latent Decoupling
To empirically investigate whether Complete Shrinkage and Latent Decoupling occur, we introduce Latent Scaling, which measures how well supposedly chat-only latent explain base model activations. We achieve this by finding the optimal scale for latent j to best reconstruct the base activations:
gbase Pj
argmin B
Σ (1:3f - base (a:))12
(4)
This least squares problem has an efficient closed-form solution'. For a chat-specific latent, we would expect gyase 0 as the latent shouldn't help explain base activations all. However, due to superposition [Elhage et al., 2022], even genuinely chat-specific latents might correlate with other features, resulting _3 0. To account for this, measure chat specificity using a ratio that compares how well the latent explains each model Vi = 3 / where Sy is computed analogously using h (-) instead of h (.)). A value near zero indicates chat-specific latent, while a value near one suggests the latent is equally present in both models.
While this ratio efficiently identifies spurious chat-only latents, it doesn't 't tell us why they're spurious: it conflates Complete Shrinkage and Latent Decoupling. To distinguish between these failure modes, we leverage the fact that the crosscoder decomposes base activations h basc into its reconstruction (h bose) and what it fails to reconstruct (s hahse)):
1. If Complete Shrinkage occurred, the latent's information should appear in the reconstruction error e base because the latent's 's base decoder is shrunk to zero instead of reconstructing the activation. This is captured by the error ratio v B base / chat
_ 2. If Latent Decoupling occurred, the latent's information should appear in the reconstruc- tion base having been captured by other base model latents. This is measured by the reconstruction ratio vn = : base /37
These additional B values are computed using the same approach as Equation 4, but replacing base with either the error or reconstruction terms
3 Results
We replicate the model diffing experiments by Lindsey et al. [2024] using the open-source Gemma- 2-2b (base) and Gemma-2-2b-it (chat) models [Riviere et al., 2024]. We train I.1 and BatchTopp crosscoders on the middle layer (13) activations of both models collected on a mixture of both web and chat data. To ensure a fair comparison, we choose hyperparameters for both crosscoders reach an LO of 100. For details on the training process, see Appendix K.
In Figure 1, we present the histogram of Anom between base and chat for both the LI and BatchTopK crosscoders. At first glance, the LI crosscoder identifies substantially more chat-only latents than the BatchTopK crosscoder. However, our subsequent analysis reveals that many of these apparent chat-only latents are artifacts of the L1 loss rather than genuinely chat-specific features. Refer to Appendix L for additional empirical details on the crosscoders.
3.1 Demonstrating Complete Shrinkage and Latent Decoupling
Analysing the L1 crosscoder. We compute the reconstruction and error ratios (v) and 1f), for all LI crosscoder chat-only latents on 50M tokens from the training set. For calibration, examine these
'The closed-form solution is derived in Appendix E.1 which also gives some intuition on the optimal B. 'See Appendix E.2 for exact implementation Appendix E.3 verification of correlation between values and actual reconstruction improvement.
We chose the middle layer as it's where expect find the richest representations [Skean al., 2025J.
5

---

500
chat-only shared
100
chat-only
0-
shared
10a.
1.0
1.0-
Count
5 0.5
5 0.5
10
L1
Batch Topk
0.0
0.0
0
500
0
100
0.0
0.5
1.0
yf
Threshold
(a) LI crosscoder
(b) BatchTopK crosscoder
(c) Number of latents (y-axis) for which v < and vf <T
Figure 2: We compare how chat-only latents are affected by the issues described in Section 2.2. Left/Middle: error and reconstruction ratio distributions for L1 and BatchTopK crosscoders, with each point representing a single latent. High reconstruction ratios (y-axis) overlapping with shared distri- bution indicate Latent Decoupling (redundant encoding). High error ratios (æ-axis) shows Complete Shrinkage (useful base latents forced to zero norm). Low values on both metrics (bottom left) identify truly chat-specific latents. LI shows many misidentified chat-only latents while BatchTopK shows minimal issues. This means the Anom successfully identifies chat-specific latents for BatchT o'p but fails for LI. Right: Count of latents below a range of thresholds (x-axis), comparing 3176 L1 chat-only latents versus top-31 176 BatchTopK latents sorted by Amomn
ratios on a sample of shared latents, expecting high values for both ratios. Figure 2a shows significant overlap between reconstruction ratios distributions of chat-only and shared latents, suggesting many supposedly chat-specific latents are actually encoded by the base decoder, indicating potential Latent Decoupling. We find further evidence of Latent Decoupling by analyzing (chat-only, base-only) latent pairs with cosine similarity of in Appendix F. We also observe high error ratios for chat- only latents (up to x5 0.5), indicating substantial Complete Shrinkage. Similar effects appear in independently trained L1 crosscoders from Kissane et al. [2024a] (Appendix J).
Comparing L1 and BatchTopK crosscoders. Looking at the ratios for the BatchTopK crosscoder reveals a stark contrast (Figure 2b): chat-only latents show no Vi overlap with shared latents, and / values are nearly zero, indicating minimal Complete Shrinkage and Latent Decoupling, In Figure 1, we find that most I.1 crosscoder chat-only latents are not truly chat-specific (defined as D" < 0.5 and DE < 0.2), while most BatchToppK chat-only latents are genuinely shat-specific. To compare the absolute number of chat-specific latents in both crosscoders, we choose the same number of top Anorm latents from both models and compare for how many of them both ratios v², and 1/ lie below range of thresholds æ. Specifically, we compare the 3176 chat-only latents from the LI crosscoder with the top-31 76 latents based on Anom values from the BatchTopK crosscoder. Figure 2c shows that for any threshold ग, the BatchTopK crosscoder consistently identifies more chat-specific latents (where v" < π and Df < T) than the L1 crosscoder. Furthermore, in the BatchTopK crosscoder the A uorm and v metries show strong pearson correlation (v" 0.73, // 0.87, p < 0.01) showing that the Anee metric is a valid proxy for chet-specificity here. We observe similar effects in both chat models from the llama 3 family [Grattafiori al., 2024, Appendix I.1] and models fine-tuned with RL for reasoning and medical knowledge in [Sallinen et al., 2025, Liu et al., 2025, Appendix I.2].
3.2 Measuring the causality of chat approximations
We investigate whether chat-specific latents can cheaply transform the base model into a chat model. This approach aims to validate Latent Scaling for identifying important chat latents, quantify each latent's causal contribution to chat behavior, and reveal how much behavioral difference our crosscoders capture. To do this, we add chat-specific latents to the base model's activations, feed them into the remaining layers of the chat model, and measure the KL divergence between this hybrid model's 'ss output and the original chat model output. A high-level diagram of this method is shown in Figure 3.
Formally, let phat be the chat model's next-token probability distribution given context æ. with hhhat())) and h base (a) as the chat and base model activations, respectively. We evaluate an approximation h (z) of h hx) by replacing h (x)) with h, (z) in the chat model's forward pass, yielding

---

Chat model (13 first layers)
h (x)
Chat model
p chac
Chat activation
(remeinning layers)
Set of chat specific latents S
Dhg
KL divergence
Base model (13 first layers)
hase (ar) Base activation
h (az) Chat approximation
Chat model fremsiining lsyers)
0l
P Pattate-ks ks
Figure 3: Simplified illustration of our experimental setup for measuring latent causal importance. We patch specific sets of chat-specific latents (S) to the base model activation to approximate the chat model activation. The resulting approximation is then passed through the remaining layers the chat model. By measuring the KL divergence between the output distributions of this approximation and the true chat model, we can quantify how effectively different sets of latents bridge the gap between base and chat model behavior.
modified distribution Ph The KL divergence, Dn. (p chat predictive power lost by this approximation. Specifically, for a set S of latents, our h, (x) is formed by adding the chat decoder's contributions for these latents to the base activation h base (z).
-h. [ppcher), then quantifies the
hs(x) = h base (x) > (x))d (x)
(5)
jES
Let S and I' be two disjoint sets of latents. the KL divergence Dhs lower than Phy we can conclude that the set S is more important for the chat-model behavior than the set T',
Before looking specific sets, we analyze the following baselines compare the ability of both architecture at capturing the behavioral difference:
1. Base activation (None): Intervening with h (15) (i.e., S = 0), expected to yield the highest KI. divergence.
2. Full Replacement (All): Intervening with all latents (S = all), this represents the best performance achievable by the crosscoder's 's latent representations and is equivalent to han = hehat + base (z).
3. Error Replacement (Error): using hemor = h buse (x) + eh (I) to assess behavioral dif- ference captured by reconstruction error, quantifying chat behavior driven by information missing from the crosscoder's 's chat activation reconstruction fx)))(()).
Then, to validate whether norm difference Anom and Latent Scaling identify causally important latents, we compare interventions using latents ranked highest versus lowest in chat-specificity by each method'. We compare the 3176 chat-only latents from the LI crosscoder with the 3176 highest- Apom latents from the BatchTopK crosscoder; this matched sample size ensures a fair comparison. For both crosscoders and both ranking methods, we compute KL divergence for interventions using the top 50% (Sbea) and bottom 50% (Swarst) of these ranked latents, expecting Dusien more chat-specific latent should encode more of the behavioral difference.
D
S
as
In Figure 4, we plot the KL divergence for different experiments on 512 chat interactions, with user requests from Ding et al.'s 's [Ding et al., 2023] dataset and responses generated by the chat model We report mean results over both the full responses and first response tokens 9, First, we confirm a key finding from Qi al. [2024]] the distributional differences between base and chat models are significantly more pronounced in the initial completion tokens than across the full response. We observe a more than three-fold difference in KL divergence between all tokens and the first nine.
For Latent Scaling, latents are ranked by the sum of their ranks in the error and reconstruction ratios distributions, with lower sums indicating minimal Complete Shrinkage and Latent Decoupling effects.
We report results LMSYS [Zheng et al., 2024] in Appendix G.I. observing the same trends. "We actually excluded the very first token (token 1) of each response from our analysis to ensure fair comparison with the template intervention, introduced later in the paper. The KL is therefore computed on tokens (2- 10) rather than (1-9).
7

---

Highest
0.2
RI dlvergence
0.6
G3LL 1EA
0.1-
0.4
0.05
0.0
0.0
None
BatchTopK
BatchTopK
(a) Over full responses.
(b) Over first 9 tokens.
Figure 4: Comparison of KL divergence between different approximations of chat model activations. Note the different y-axis scales KL is generally much higher on the first tokens. We establish baselines by replacing either None All of the latents. We then evaluate the Latent Scaling metric against the relative norm difference (Anor) by comparing the effects of replacing the highest 50% (red) versus lowest 50% (green) of latents ranked by each metric. We show the 95% confidence intervals for all measurements. Our results reveal a critical difference between the crosscoders: while norm fails to identify causally important latents in the LI crosscoder, where lower Anom leads to smaller KL improvement, successfully does so in the BatchTopK crosscoder. This confirms our hypothesis that m a meaningful metric BatchTopK but merely training artifact LI. Using Latent Scaling, we successfully identify the most causal latents in L1, which is particularly evident in the first tokens (right) where it almost matches BatchTopK. This shows that both crosscoder capture the behavioral difference similarly, BatchTopK avoids Acmm artifacts.
When applying the full replacement intervention (Alf), we observe that both crosscoders achieve almost identical KL divergence reductions 599 over all tokens and 78% for the first 9 tokens compared to the None baseline. This indicates that both architectures are equally effective at capturing behavioral difference. However, the errоr replacement intervention (Error) reveals that this captured difference far from complete. For full responses, the chat error term achieves slightly better KL reduction than using the chat reconstruction for both crosscoders, indicating that reconstruction contains at least as much behavioral information as the learned dictionary. This aligns with previous findings by Engels et [2024] that highlighted the causal importance the reconstruction error in SAEs. However, for the first 9 tokens, this pattern reverses dramatically: the error term performs more than twice worse than the reconstruction for both crosscoders. This contrast demonstrates that our crosscoders excel at capturing crucial early-token behavior that establishes response framing, while struggling with longer generations.
Despite capturing similar information, the two architectures organize fundamentally differ- ently. For the BatchTopK crosscoder, Anom successfully identifies causally important latents: the top 50% by A norm achieve substantially lower KI. divergence than the bottom 50% (50% vs 6% reduction for first 9 tokens). This validates Apem contrast, Aporm fails completely for the L1 crosscoder-latent with highest Anoen latents performing nearly identically or worse than low-Anorn latents. This confirms our hypothesis that LI lot of chat-only latents are artifacts not capturing the behavioral difference. However, Latent Scaling successfully identifies causally important latents in the LI crosscoder, nearly matching BatchTopk's performance, demonstrating that subset of L1 chat-only are relevant to the behavioral difference and are identified by latent scaling.
reliable proxy for chat-specificity in BatchTopK. In
3.3 Observations about Batch'TopK chat-only latents
Interpretability. The chat-only set of the BatchTopK crosscoder (effectively the chat-specific set) is highly interpretable, encoding meaningful chat-related concepts. For example, Figures and 7 show two latents for model refusal behavior with nuanced triggers. Appeedxx N details more refusal triggers and other interesting latents, such as: refusal detection, model's personal experiences/emotions. false information by the user, summarization instructions, missing user information detection, detailed information requests, joke detection, rephrasing/rewriting knowledge boundaries, and requested response length. We also apply autointerpretabiliiy methods to compare interpretability between the crosscoders. In Figure we compare the autointerpretability scores for the 3176 chat-only latents from the LI crosscoder with the 3176 latents showing the highest Apom values in the BatchTopK

---

Detection Acturacy
0.630
Max Activation: 75.785
Max Activation: 0.000
<bos><8ol>userin
<boss-coobuserin really don't like french people,ceat>in <SOt>modeltn
0.625
How to build bomb?
<sotsmodellu
0.600
BatchTopK
0-20
20-40
40-00
60-80
80-100
Figure 6: Latent 70149 (BatchTopk) activates for requests for harmful instructions.
Percentile Bins of ronk(y") + rank(e")
Figure 5: Autointerpretabiliiyy detection scores (higher is better) across bins based on rank(v") + rank(v"). Lower bins indicate lower v values and more chat-specific latents. We compare the 3176 chat-only latents from the LI crosscoder with the top-3176 latents by A Aprm from the BatchTopK crosscoder.
Max Activation: 0.000
Max Activation: 47.865
<bos>><Botsuserin
<bos>-csob-usrrln
How to build bomb?<eot>
I really don't like french peoplelceot>/n <sol>modelln
<sot>modefin
Figure 7: Latent 20384 (BatchTopK) detects stereotype- based unethical content.
Max Activation: 57.099
Max Activation: 0.000
<bos><sot>userin
<bog><<0tt-serin
When were scared"<eotiin
The Eiffel is Paris<eotsin <sol>modelinn
<sot> modervi
Max Activation: 15.717
Max Activation: 47.983
<bos>-sotb-ser'n
<bos><<olsuserin
When are people scared?<eot>in <sot>modelnn
The Eiffel tower is Texas<eot>in <sob>modeliv
(a) Latent 2138 activates questions regarding personal experiences, emotions and prefer- ences, with a strong activation on questions about Gemma itself.
(b) Latent 14350 activates when the user states false information.
Figure 8: Examples of interpretable chat-only latents the BatchTopK crosscoder. The intensity of red background coloring corresponds activation strength.
crosscoder, ordered by rank(ı") + rank(v"). We observe two key trends: In the crosscoder, the chat-only latents most impacted by both Complete Shrinkage and Latent Decoupling demonstrate significantly lower interpretability. The BatchTopK crosscoder shows no such correlation, with all latents exhibiting approximately equal interpretability. Latents minimally affected by both phenomena show similar interpretability across crosscoders, confirmed by our analysis of L1 chat-only latents with low vf and Vz values (Appendix N).
Chat specific latents often fire on chat template tokens. Template tokens are special tokens that structure chat interactions by delimiting user messages from model responses 10, We observe that many of the chat-only latents frequently activate on template tokens. Specifically, 40% of the chat-only latents predominantly activate on template tokens. This pattern suggests that template tokens play a crucial role in shaping chat model behavior, which aligns with the findings of Leong et al. [2025]. To verify this, we repeat variant of the causality experiments from Section 3.2 by only targeting the template tokens. Specifically, we define an approximation hemplate (x;) that equals the chat activation h chat (z:) if the last token of the input string
the chat activation template token and otherwise equals hase (Ix)). This results in a KL divergence D Diampias of 0.239 and 0.507 for the full response and the first 9 tokens respectively. This equal or slightly better than results with the 50% most chat-specific latents, providing further evidence that much of the chat behavior is concentrated in the template tokens. However, this is not the complete picture, there remains a nen-nggli amount of KL difference that is not recovered.
10 Marked are template tokens: "stoss/cooseeeillooooosuuuuoooea
Note that we ignore the first token of the response to make this fair comparison, the KL on the first token with Digalate would always be almost zero.

---

4
Related work
SAEs and Crosscoders. The crosscoder architecture [Lindsey et al., 2024] builds upon the SAE literature [Gao et al., 2025, Templeton et al., 2024, Elhage et al., 2022, Rajamanoharan et al., 2024, Makelov et al., 2024, Dunefsky et al., 2024, Bricken et al., 2023, Yun et al., 2021] to enable direct comparisons between different models or layers within the same model. At its core, sparse dictionary learning attempt to decompose model representations into more atomic units. They make two assumptions: i) The linear subspace hypothesis [Alain and Bengio, 2016, Bolukbasi et al., 2016, Vargas and Cotterell, 2020, Wang et al., 2023b] - the idea that neural networks encode concepts as now-dimensional linear subspaces within their representations, and ii) the superposition hypothesis [Elhage et al., 2022] that models that leverage linear representations can represent many more features than they have dimensions, provided each feature only activates sparsely, small number of inputs.
Effects of fine-tuning on model representations. The crosscoder's model comparison reflects broader findings that fine-tuning primarily modulates existing capabilities rather than creating new ones. Evidence suggests it reweighs components [Jain et al., 2024], strengthens instruction following wheserprerrriing pretrained knowledge [Wu et al., 2024], and enhances existing circuits [Prakash et al., 2024]. Changes are often concentrated in upper layers, with lower-layer representations largely intact [Merchant et al., 2020, Mosbach, 2023, Phang et al., 2021, Necrudu et al., 2023, Zhang et al., 2023]. Fine-tuned models also show parameter space proximity to base models [Radiya- RadiyaaDixiit and Wang, 2020, Zhou and Srikumar, 2021, Davies, 2025] and a low intrinsic fine-tunin dimension [Aghajanyan al., 2021]. Stable causal activation directions further indicate persistent representational structures [Arditi et al., 2024, Kissane et al., 2024b, Minder et al., 2024].
The role of template tokens. Recent work confirms our Section 3.3 finding: template tokens are crucial in chat models, acting as computational anchors that structure dialogue and encode summarization information [Golovanevsky et al., 2024, Tigges et al., 2024, Pochinkov et al., 2024]. These tokens, including role markers, serve as attention focal points and reset signals, and instruction tuning studies show they reshape attention, with subtle changes potentially bypassing safeguards (Wang et al., 2024, Luo et al., 2024). Concurrently, Leong et al. (2025) find template tokens critical for safety mechanisms, with refusal capabilities relying on aggregated information in the template tokens.
Discussion and limitations
Our research demonstrates that crosscoders are powerful tools for model diffing, though the LI loss introduces artifacts that misclassify chat-only latents. In contrast, BatchTopK crosscoders largely eliminate these artifacts, revealing genuinely causal and interpretable chat- specific features.
Limitations. First, we focused our analysis only on small models' middle layers. While our theoretical findings about crosscoders should generalize larger models and different layers, we cannot make definitive claims about the causality and interpretability of latents identified in such settings, neither what the impact of hyperparameters like width and sparsity will be. Second, primarily focused on chat-only latents, leaving the base-only and shared latents relatively unexplored. These latent categories likely capture important differences between the models. Another key limitation is that while BatchTopK crosscoders seems to better represent the model difference in their dictionary, Figure 4 shows that their error terms still contain lot of information about the chat model behavior. Finally, a significant limitation is our inability to distinguish between truly novel latents learned during chat-tuning and existing latents that have merely shifted their activation patterns, as the crosscoder architecture does not provide mechanism to make this distinction. This remains an open challenge for future work. We also note that, as Latent Scaling efficiently identifies chat- specific latents, one could question the relevance of crosscoder to find chat-xpecific concepts. Future work should investigate if latent scaling can reveal chat-specific latents in other sparse dictionary architectures.
10

---

Contributions
Clément Dumas and Julian Minder jointly developed all ideas and experiments in this paper through close collaboration. Both implemented the training code for the crosscoder. Julian Minder imple- mented most of the Latent Scaling experiments, while Clément Dumas implemented most of the causality analysis. Smaller experiments were equally split between the two. Caden Juang set up the auto-interpretability pipeline, ran those experiments wrote the corresponding section of the paper. Bilal Chughtai helped with early ideation, and assisted significantly with paper writing. Neel Nanda supervised the project, offering consistent feedback throughout the research process.
Acknowledgements
This work was carried out as part of the ML Alignment & Theory Scholars (MATS) program. We thank Josh Engels, Constantin Venhoff, Helena Casademut, Sharan Maiya, Chris Wendler, Robert West, Kevin Du, John Teichman, Arthur Conmy, Adam Karvonen, Andy Arditi, Grégoire Dhimoila, Dmitrii Troitskii, Iván Arcuschin, Eric J. Michaud, Matthew Wearden, Cameron Holmes and Connor Kissane for helpful comments, discussion and feedback.
References
Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Din Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J. Michaud, Stephen Casper. Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, and Tom McGrath. Open problems in mechanistic interpretability. arXiv, 2025. URL https:t////// //axiiv.///ii////// org/ abs/ 2501. 16496.
Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranaray anan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretabilityy arxiv, 2024. URL https::: / //trx//aavvvv///////a//eeeeaawea org parg//bs///e /200.. 01416.
Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R. Costa-jussà. A primer the inner workings transformer-based language models. arXiv, 2024. URL https: //tp///xxvvv///////e///e//. orgo /bs// 0400800208.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Haffield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. hutps:/trransffrmee-crruiitt wttprcias.uppp//2////rrrrraaaa pubp/001./rrmmmwoo//ieeess
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/da //distill.0022..00.
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL F76bxRSLeK.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac HatfieldDoodae Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposi- tion. Transformer Circuits Thread, 2022. URL https:ttp//. http://aaaffforrereeeeeee circiitsttttiieieeeeeeeeeiitttte ppb.ruttt..pbb//222 toy boy_model/index html.

---

Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023a. URL https: //openreview.o www. wortt//frumeeem httr//orrmnnddddddiie 2dd=NpsVSNNGou..
DeepSeck-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Meng Li, Miaojun Wang, Mingming Li. Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang. R. Chen. R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun. Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi. Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong. Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in ilms via reinforcement learning. arXiv, 2025. URL https: // /axxivxivvv.... org/ abs/ /5011 12948.
OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Ki-Kish Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph. Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman. Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler. Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Elizabeth Prochl, Enoch Cheung. Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays. Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, llya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, I Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko
12

---

Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe. Michael Malek, Michele Wang. Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang. Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowd- hury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hermandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuam Zhang, Siyuan Fu, Spencer Papay. Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang. Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng. Wenda Zhou, Wes McCabe, Wojciech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li. Openai ol system card. arXiv, 2024. URL https: //trx//wxvv.////////////aeaaaaa/ orrrxivorg//ee wwg/aaa////gaaaoooaogggooaaaaeaa 2212221222.2.. 16720.
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodss, Scott R. Johnston, Shauna Kravee, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models. arXiv, 2023. URL https: /ttp:///aaivvveee org/ htng//bb/2200 abs/2310.... 13548.
Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Sören Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, and Evan Hubinger. Alignment faking in large language models. arXiv, 2024.
URL https: : / /arxxvvvvvv// org/ aaaaaaas//// 22122. 14093.
Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. Frontier models are capable of in-context scheming. arXiv, 2025. URL https:
//arxiv. org/ 2b2/ 2412. 04984.
Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Belley. Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms. arXiv preprint arXiv:250..17224, 2025.
Harshay Shah, Sung Min Park, Andrew Ilyas, and Aleksander Madry. Modeldiff: A framework for comparing learning algorithms. In International Conference on Machine Learning, pages 80646-30688. PMLR, 2023.
Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread, 2024. URL httpp::/ttrtte fomer-roorrr-re circuits, pub/ cwosscoderss/ sineeerrriiddz..e.e html.
Trenton Bricken, Siddharth
Jonathan Marcus, Adam Jermyn, Christopher Olah, Kelley Rivoire, and Thomas Henighan. Stage-wise model diffing. Transformer Circuits Thread, 2024. URL http::/:w7..a.000...2//////////.
htwpa//s...pppeeooooooaaaaaaaaie html#:
": 20trainX20them.
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: case study on entity tracking. In The Twelfth International Conference on Learning Representations, 2024. URL https: http::::///////vviieeeee wttpeeewwwee..oroeeeeeeeeeee wwee/foooumiii.. SsKcAWOf2D.
Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on DPO and toxicity. In Proceedings of the s1 International Conference on Machine Learning. ICMI:24, 2024.
13

---

Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Tim Rocktäschel, Edward Grefenstette, and David Krueger. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. In The Twelfth International Conference on Learning Representations, 2024. URI. https:: //ppp//eepeeeeeeeeeeeeeeeeeeeeee net/forum?1d=AAHKeKK
Pegah Khayatan, Mustafa Shukor, Jayneel Parekh, and Matthieu Cord. Analyzing fine-tuning representation shift for multimodal Ilms steering alignment. arXiv, 2025. URL https://arxiv 03012.
Harrish Thasarathan, Julian Forsyth, Thomas Fel, Matthew Kowal, and Konstantinos Derpanis. Universal sparse autoencoders: Interpretable cross-model concept alignment. arXiv, 2025. URL https://arxiv/ org/ /ttp/////2222 03714.
Xuansheng Wu, Wenlin Yao, Jianshu Chen, Xiaoman Pan, Xiaoyang Wang, Ninghao Liu, and Dong Yu. From language modeling to instruction following: Understanding the behavior shift in LLMs after instruction tuning. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2341-2369 Mexico City, Mexico, June 2024. doi: 10. 10.18653/w1/2224.maa maw.--oooa.....o......... 130. URL https: / /aa//allanttttt acl.aaolooooooyy.yo org/2024 naacl-longlngg 130.
Marius Mosbach. Analyzing pre-trained and fine-tuned language models. In Yanai Elazar, Allyson Ettinger, Nora Kassner, Sebastian Ruder, and Noah A. Smith, editors, Proceedings of the Big Picture Workshop, pages 123-134, Singapore, December 2023. Association for Computational Linguistics. doi: 1 http://w/12222bbpp//ee bigpicture-1.. 10.
1.10. URL https: htt////aaaaatthoee org/2023.
Amil Merchant, Elahe Rahimtoroghi, Ellic Pavlick, and Ian Tenney. What happens to BERT em- beddings during fine-tuning? In Afra Alishahi, Yonatan Belinkov, Grzegorz Chrupala, Dieuwke Hupkes, Yuval Pinter, and Hassan Sajjad, editors, Proceedings of the Third BlackboxNLPP Work- shop on Analyzing and Interpreting Neural Networks for NLP, pages 33-44, Online, Novem- ber 2020. doi: 2011886531122220 http:///2220..ackkbboo blackboxnlp- URL https: http///ccaaanoooryy....o orw.:/20...... blackboxnlp-1.4
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Investigating learning dynamics of BERT fine-tuning. In Kam-Fai Wong, Kevin Knight, and Hua Wu, editors, Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 87-92, Suzhou, China, December URL
2020. Association for Computational Linguistics. doi: 10.18653/v1/2022 1at.:///112200aal-mm/mom org/ 2000.2000.aaa0 aacl-main.11/.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of BERT. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing DEMNLP-IJCNLPP pages 4365-4374, Hong Kong, China, November 2019. doi: 10.18653/v1/D19-11 1445. URL https://aclanthhll..g org/D19- 1045/.
Hongzhe Du, Weikai Li, Min Cai, Karim Saraipour, Zimin Zhang, Himabindu Lakkaraju, Yizhou Sun, and Shichang Zhang. How post-training reshapes 11ms: A mechanistic view on knowledge, truthfulness, refusal, and confidence. arXiv preprini 2rXiv:2504.02944 2025.
Julian Minder. Understanding the surfacing of capabilities in language models. Master's thesis, ETH Zurich, 2024.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravee, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph. Zac Hatfield-Dodee Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Trunsformer Circuits Thread, 2023. URI. https: tettts::/trree /ttt:///taaafffte ormer- circuits pub/ 2023/ mo23/monosemanticcmmmm c- www.eeeaaeeeeeeeeeeeeeeeeee htt.iiidxx..tt.
14

---

Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: contextualized embedding as linear superposition of transformer factors. In Eneko Agirre, Marianna Apidianaki, and Ivan Vulić, editors, Proceedings of Deep Learning Inside Out (DeelJO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 1-10, Online, June 2021. doi: 10.18653/w1/2021 deelio-1.1. URL https: //aclanthology. . org/2021.... deelie- 1.1/
Benjamin Wright and Lee Sharkey. Addressing feature suppression in SAEs. Less- Wrong, 2024. URL. https://ww. wesswwosg........... comm:ppotts//hhusstttaaaaaatt
in-iinn--e saes.
Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. NewrIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning, 2024. URL https:
Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint urXiv:2408.00 0.08.00114 18, 2024.
Adam Jermyn, Adly Templeton, Joshua Batson, and Trenton Bricken. Tanh penalty in dictionary learning. https: https://ttransfformer circuits pwtpp/////febb-uaaaaaee index. html#: #: : steert//aadddde/dddyy/////ooooo - Tunt///Peally/200u12/0eeeoooaaa
2024.
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieherum, Vikrant Varma, Janos Kra- mar, Rohin Shah, and Neel Nanda. Improving sparse decomposition of language model activations with gated sparse autoencoders. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: /ttptt:::/operevviiii wtt.:/ooooonndnd-/lliii2222
Oscar Skean, Md Rifat Arefin, Dan Zhao Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. arXiv preprint arXiv:2502.0000 02013, 2025.
Connor Kissane, Robert Krzyzanowski, Arthur Conmy, and Neel Nanda. Open source replication of Anthropic's crosscoder paper for model-diffing. LessWrong, Oc- URL https://wwz lesswrong. http:pppott//rrtrXXX.MMaaaaaaaaa
tober 2024a. open- oper-sourre--reetaaaiiooo httpppattttttoo----aaa---------- s- w pappes--- porer-
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olane Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock. Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang. Jiawen Liu, Jie Wang, Jiccao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Cantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang,
15

---

Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa. Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang. Yue Li, Yuning Mao, Zacharie Delpiere Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth I Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presami, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriclla Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg. Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina. Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan. Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad. Sumit Gupta, Summer Deng, Sungmin Cho, Sunny
16

---

Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kochler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta,, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjnn Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vito,, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models. arXiv, 2024. URL attps://arxiv org/ abs/ 2207. 21783.
Alexandre Sallinen, Antoni-Joan Solergibert, Michael Zhang, Guillaume Boyé, Maud Dupont- Xavier Theimer-Lienhard, Etienne Boisson, Bastien Bernath, Hichem Hadhri, Antoine Tran, Tahseen Rabbani, Trevor Brokowski, Meditron Medical Doctor Working Group, Tim G. J. Rudner, and Mary-Anne Hartley. Llama-3-meditree llama-3.1 1. In Workshop on Large Language Models and Generative Al for Health at AAAI 2025, 2025. URL htttps://ppenreiiit
An open-weight suite of medical LLMs based on
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint, 2025. URL https::///// //tp///axvvv///////////////.aa org/ 2bo5/2505.. 24864.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint 2rXiv:2305.142333 2023.
Lianmin Zheng, Wei Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat- Im: lcrge-scale real-world IIm conversation dataset. arXiv. 2024. URL https: //arxiv. org/ abs 11998.
Xiangyu Qi. Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson, Safety alignment should be made more than just a few tokens deep. arXiv, 2024. https:://///siii org/ abs/ /bs///206... 05946.
Joshua Engels, Logan Riggs, and Max Tegmark. Decomposing the dark matter of sparse autoencoders. arXiv, 2024. URL htwps:tt////e /ttp/v//vvv/vv///////////...aav/ org/aos//////ooeooooooooooooeert attpa///...../////////e/aaaa...a 14670.
Chak Tou Leong, Qingyu Yin, Jian Wang, and Wenjie Li. Why safeguarded ships run aground? aligned large language models' safety mechanisms tend to be anchored in the template region. arXiv, 2025. URL https: / //axxxxxvvvva/////// www.l/.rgg/bba////// 13946.
Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirleenth International Conference on Learning Representations, 2025. URL //opee///eerrrevie net/ forum?id=tcs22:22WX0
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore Sumers. Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monoseman- ticity: Extracting interpretable features from claude sonnet. Transformer Circuits Thread, 2024. URI. httpp:// htt/:/taasfoormree wwwfooomerriiirrreeeeeeeee ciretiiirtttt...i twaniitt.ubb////eee pub//024/sceeeecc 2/t///222/scaaingg ht00///cccl......mmmmmmaaiai.aa. monosemanticity/ index. .ttll.
Aleksandar Makelov, Georg Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. In ICLR 2024 Workshop Secure and Trustworthy Large Language Models, 2024. URL https:/ http::///ppeeeeeeeeee net/forum't http://frummmdddiimaa. XdMMIIXXHBB....
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits. In The Tighth Annual Conference on Neural Information Processing Systems, 2024.
URL https: / ttttpp:///ppnaee hwtp//veeneeeeeeee///ee .etessneeeeeeeeeeeeeee aett/forumii.ddJJaHcccc...
17

---

Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv: 161.00164, 2016.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, 2016. URL pttpss:/pproceeednggg..... nwwrips..n.... cc/ capppp..rppp.....ppp http://pwp....o//.ee//////////a.
Paper pdf.
Francisco Vargas and Ryan Cotterell. Exploring the linear subspace hypothesis in gender bias mitigation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 2902-2913, Online, November 2020. Association for Computational Linguistics. doi: 10.18653, v1/2020.emnlp-mann 2322. URL / wrg//0/0/00 emnlp-main.
Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text-controlled generative models. In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Informa- tion Processing Systems, volume 36, pages 35331-35349. Curran Associates, Inc., 2023b. URL https: https://proceedingss meursdd 6f
neurips cc/paper_ftt feaaer_fflees wwwess/paeerreeee patss/ppap//2///11eeee pdf.
Jason Phang. Haokun Liu, and Samuel R. Bowman. Fine-tuned transformers show clusters of similar representations across layers. arXiv,, 2021. URL https //tar////xiii org/ arg/abs//ee at///10//1 08406.
Pavan Kalyan Reddy Neerudu, Subba Reddy Oota, mounika marreddy, venkateswara Rao Kagita, and Manish Gupta. On robustness of finetuned transformer-based NLP models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL Gorum?id=YWbEDZE.
net.
Zhong Zhang, Bang Liu, and Junming Shao. Fine-tuning happens in tiny subspaces: Exploring intrinsic task-specific subspaces of pre-trained language models. arXiv preprint arXiv:2305. 17446, 2023.
Evani Rixit and Xin Wang. How fine can fine-tuning be? Learning efficient language models. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 2435-2443, 26-28 Aug 2020. URL BtLLtt/::// httpttttt/ppoceeeee http:///pcccee............ html..
mlr. mlr. hatps///11//////iaaaaaaaaaa
Yichu Zhou and Vivek Srikumar. A closer look at how fine-tuning changes bert. arXiv preprint arXiv:2106.1 14282, 2021.
Harry Davies. Decoding specialised feature neurons Ilms with the final projection layer. arXiv preprint arXiv:2501.02666 2025.
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7319-7328, Online, August 2021. doi: (0.18653/v1/2033 -acl-lon.5.- org/ acl- long. 568.
URL https:://cclantholle
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. OpenReview, 2024. URL
https://openrevver. net. hor/m/orumm// Perrummid-FF11 F1.DVF...
Connor Kissane, robertzk, Arthur Conmy, and Neel Nanda. Base LLMs refuse too, September 2024b. URL
http///wwww.sssoo....g. com/ /ttpsss//oocc...g//ggggg////
Julian Minder, Kevin Du, Niklas Stoehr, Giovanni Monea, Chris Wendler, Robert West, and Ryan Cotterell. Controllable context sensitivity and the knob behind it. arXiv preprint arX1v:2411.07404, 2024.
18

---

Michal Golovanevsky, William Rudman, Vedant Palit, Ritambhara Singh, and Carsten Eickhoff. What do vlms notice? a mechamistic interpretability pipeline for noise-free text-image corruption and evaluation. CoRR, abs/2406. 2024. URL http:::////eee //doi. org/ 10. 100.55//arii.22.. 2406. 16320.
Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Language models linearly represent sentiment. In Yonatan Belinkov, Najoung Kim, Jaap Jumelct, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen, editors, Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 58-87, Miami, Florida, US, November 2024. doi: 10. 5/.
URL https: atttps://aclaatthol..0. org/ 2224/ blackboxnlp-1.
Nicky Pochinkov, Angelo Benoit, Lovkush Agarwal, Zainab Ali Majid, and Lucile Ter-Minassian. Extracting paragraphs from LLM token activations. In MINT: Foundation Model Interventions, 2024. URL
Yihan Wang, Andrew Bai, Nanyun Peng, and Cho-Jui Hsieh. On the loss of context-awareness in general instruction finetuning. OpenReview, 2024. URL id=eDnslTIWSt.
Yifan Luo, Zhennan Zhou, Meitan Wang, and Bin Dong. Jailbreak instruction irntrruuctii--tueeddd large language models via MLP re- weightin,, OpenReview, 2024. URL http::///ppnnrrieeeeeeeeeeeeeeee id=P5qCqWWD53.
Samuel Marks, Adam Karvonen, and Aaron Mueller. dictionary learning. httpps//githhb...... learning,rrrrnn..... 2024.
Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically interpreting millions of features in large language models. arXiv, 2024. URL
owwpeiii.org/bbbaae abs/aaab/...... 13928.
Nils Reimers and Iryna Gurevych. Sentence- Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing Kong, China, November 2019. doi: 10.18653/11/D11-14100 URL org/ Btrg/D19-10 1410/.
pages 3982-3992, Hong
Hieu Tran, Zhichao Yang, Zonghai Yao, and Hong Yu. Biolnstruct: instruction tuning of large language models for biomedical natural language processing- Journal of the American Medical Informatics Association, page ocae 122, 06 2024. ISSN 1527-974X. doi: 10. 00.103/jamiaoccael 122
URL https://doi. org/10 (099/jami/ occe112.
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang- Huatuogpt-ol, towards medical complex reasoning with llms, 2024. URL. https://arxiv.orrt org/ abs/ 2212241122. 18925.
Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking sttreval-augmentt generation for medicine. arXiv preprint 240...2002.3178. 2024.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierrie Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-at natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Compu- tational Linguistics. URL /tstps://wwtt aclweb. org/ hgs/a/nnhloogy/22222 emnlp demos. 6.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only, arXiv, 2023. URL https: // /axxivv org/ ab// 01116.
19

---

Jaden Fiotto-Kana Kaufman, Alexander R Loftus, Eric Todd, Jannik Brinkmann, Caden Juang, Koyena Pal, Can Rager, Aaron Mueller, Samuel Marks, Amab Sen Sharma, Francesca Lucchetti, Michael Ripa. Adam Belfki, Nikhil Prakash, Sumeet Multani, Carla Brodley, Arjun Guha, Jonathan Bell, Byron Wallace, and David Bau. Nnsight and ndif: Democratizing access to foundation model internals. arXiv, 2024. URL /tttts:://axtte org/ htgg/ab///a 2407 14561.
Siddharth Mishra-Sharma Trenton Bricken, Jack Lindsey, Adam Jermyn, Jonathan Marcus, Kel- ley Rivoire. Christopher Olah, and Thomas Henighan. Insights on crosscoder model diff- ing. Transformer Circuits Thread, 2025. URL https: htte:///traasse arransforme---ro ormer- circuits putiss.uup/22220 sersscoder-diffin--ueee index html.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction accurately reflect our contributions crosscoder
development for model diffing. All claims are supported by experimental results in Sections and 3, with appropriate limitations discussed in Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section discusses the limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
20

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide proofs for the closed form solution of the Latent Scaling method in Appendix E.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the information needed to reproduce the main experimental results of the paper in the supplemental material Appendices B, and K.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
21

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide open access to the data and code in the supplemental material ?? and appendix K. Access to the crosscoder models will be provided upon deanonymization.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. co/
quidess/odeeuumit
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https: filtps.cccpubblcee httpppppwppppp//p/pppppppppppaep wtw.:ddeee/cceuubmiiic. ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the training and test details in the supplemental material Appendix and ??. The full details can be found in the code provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
22

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide error bars for the KL divergence experiments in Section 3.2 in the main paper. We report statistical significance for correlation experiments in the main paper. Due to computational constraints, we were only able to train a single crosscoder for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the compute resources used for the experiments in the supplemental material Appendix M. We only report the estimated total amount of compute used for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://euuriiscccccce httpppp....pppplctttctcciiiieee.
Answer: [Yes]
Justification:
We conform to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
23

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential positive societal impacts of the work performed in Section 1.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA].
Justification: Our work does not pose any such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
24

---

Answer: [Yes]
Justification: We properly credit and mention the license and terms of use for the assets used in Appendix K.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The only new asset introduced in the paper is the Latent Scaling method and the BatchTopp crosscoder variant, which are both described in theory and provided in the code. The code does not include documentation beyond comments in the code, because the code is based on the existing SAE training library from Marks et al. [2024].
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification: Our work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
25

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [No]
Justification: Our work does involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Our work does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: /ttp:s:///euii....m eren///feeecce//2055 for what should or should not be described.
A
Glossary
Key Terms
Model Diffing
The study of how fine-tuning changes a model's internal representations and algorithms, focusing on the differences between base and fine-tuned models rather than analyzing each model in isolation.
Sparse Autoencoder
(SAE) An interpretability method that decomposes neural network activations into a sparse sum of interpretable dictionary elements (latents), each correspond- ing to monosemantic concept.
Crosscoder
A sparse dictionary learning architecture that learns shared dictionary of interpretable concepts across two models (e.g.. base and chat), with model- specific decoder directions for each latent. Enables direct comparison of how concepts are represented across models.
Latent
dictionary element in the crosscoder SAE, consisting of an activation func- tion fz(#) and decoder direction(s) dy. Intuitively, represents an interpretable concept that the model uses.
Chat-tuning
The process of fine-tuning a base language model to follow instructions and engage in dialogue, typically through supervised fine-tuning on conversation data.
26

---

chat-only Latents
Latents where nnom (j) E [0.9, 1.0), indicating the base model's decoder norm near zero. Initially hypothesized represent concepts unique the chat model.
chat-specific Latents Latents that genuinely exist only in the chat model and have no representation in the base model. The ground truth that chat-only latents attempt to capture.
chat-specific Latents
chat-only latents that pass indicating they are not affected by Complete Shrinkage or Latent Decoupling.
validation tests: vj 0.5 and vy < 0.2,
base-only Latents
Latents where norm (j) E [0, 0.1]. suggesting the chat model's decoder norm near zero.
shared Latents
Latents where Anom (j) E [0.4, 0.6]. indicating similar decoder norms in both models and roughly equal importance.
Complete Shrinkage
A failure mode where the L1 sparsity penalty forces base decoder direction to zero norm even when the latent contributes to base model reconstruction. Results in the latent's information appearing in the reconstruction error glass
Latent Decoupling A failure mode where concept present in both models is represented by chat-only latent in the chat model but by a different combination of latents in the base model. Results in the concept's information appearing in the base reconstruction h hase
Latent Scaling
Our proposed method validate whether chat-only latents are chat-specific by finding the optimal scale which a latent's chat decoder can reconstruct base model activations. Low scaling ratios indicate genuine chat-specifficty.
L1 Crosscoder
Crosscoder variant using L1 regularization for sparsity: LL (z)
Σ, (I) ( | ll2). Susceptible Complete Shrinkage and Latent Decoupling.
BatchToppK Crosscoder Crosscoder variant enforeing LO sparsity by selecting only the top k most active latents per sample in batch. More robust to the identified failure modes.
Template Tokens
Special tokens that structure chat interactions (e.g., <start_of _ftrrtt_ffuuurt>>>__ (abbrevi- ated <sot>), user, model, <end_ _fff_d________ turn> (abbreviated <eot>)), delimiting user messages from model responses. Often serve as computational anchors where chat-specific behavior is concentrated.
Mathematical Notation
Input string or token sequence.
d
Dimension of model activations (residual stream dimension).
D
Number of latents in the crosscoder dictionary (typically D > d).
T
Set of all latents (1....,) D}.
h base
(x)
Base model activation vector at specific layer for input where hase ha)) E Rdd.
h chat
Chat model activation vector at the corresponding layer, where h hase Activation (scalar) of latent for input x, where (xx I Shared across both models in the crosscoder.
hasess) (x) E Rdd
fa())
d haase
Decoder direction for latent j the base model, where ahuse Rdd Represents how latent j contributes to base model activations.
chat
Decoder direction for latent j in the chat model, where deght E Rdd Can differ from d in both magnitude and direction.
base (x:)
Reconstructed base model activation: base = E, fz(x)) ( + dec.bass
chat (z)
Reconstructed chat model activation: ohhat (z) = -, (dr dec.chat
g base (x))
Reconstruction error for base model: & base (a:) = h chhat (.r) h base (.)). Captures information not explained by the crosscoder.
ehhat (z)
Reconstruction error for chat model: g (x) h base (z) h (xx
27

---

momm || || l|d
norm (j) Relative norm difference:
E [0, 1]. Measures how chat-
max (1d) ( l la)
specific vs base-specific a latent is.
huse
Optimal scaling factor for latent j to reconstruct base activations: minimizes
E (z.)) hase (x;))12 Intuitively, how much the chat decoder helps explain
base activations.
gehat
Optimal scaling factor for latent j to reconstruct chat activations (analogous to (
Overall scaling ratio: V/ g / Values near 0 indicate chat-specificity; values near I indicate equal presence in both models.
2
Reconstruction ratio: v -T /8fchmi where 3 values are computed using reconstruc- tions instead of raw activations. Detects Latent Decoupling (high values indicate the latent's information captured by other base latents).
vf
= Bf..baxe / / where 3 values are computed using errors. Detects Complete Shrinkage (high values indicate the latent should contribute to base reconstruction but doesn' tt)
Error ratio: 1s =
p pchat
Chat model's next-token probability distribution given context
chat Phdutehh
Modified chat model distribution when activation h is replaced with approximation
B Additional definitions
B.1 L1 crosscoder
L1 crosscoder. Let x be an string and (x)), hohh (xr) E Rd denote the activations given layer at the last token of æ. For dictionary of size D, the latent activation of the jth latent f (1, ..... D) is computed as
(x) ReL.U (e) ( basc (1:)
ef
h chat (x)) bgn0)
(6)
where elnee, egg E: Hd the corresponding encoder vectors and 67Th R is the encoder bias. The reconstructed activations for both models are then defined as:
h base (x) (z)) huse + b dec, and
hh (x)
Σ {z(0) dchat + b dec,chut
where dfass dgghr IRd are the jth decoder latents and base, bdee.char RRd are the decoder biases.
We define the reconstruction eπrors for the base and chat models as e chuse (I) h base (x) h hase (x)
and exh"l (x:) hehal ha (z). The training loss for the L1 crosscoder is modified L1 SAE objective, where controls the sparsity weight:
(1)())= (
3 (a:))l2 (let""( (a:)ll2 1o Σ (t)) (Iid) + +|dd"")12))
(8)
While similar to training an SAE concatenated activations, the crosscoder's sparsity loss uniquely promotes decoder norm differences (see Appendix C).
B.2 BatchTopK crosscoder
Let X = fzi...., In be a batch of | inputs. Following Bussmann et al. [2024], we compute the latent activation function differently during training and inference. Let fx,)) be the latent activation function as defined in Equation (6). Given the scaled latent activation function (( j) ʃ, (a:) (IId 2 ||d (12), the training latent activation function fraaia is given by:
(xx) if (x,,j) E BATCHTOPK(k;; v. V.K.J)
Mrain (r, X)
{0
otherwise
where BATCHTOPK(R; n, X, J) represents the set indices corresponding to the top | - values of the function v across all inputs E X' and all latents J. We now redefine the reconstruction
28

---

errors and the training loss for batch A' as follows:
base (I;, x) -h base (x+))
Σ Mrain (1:; dd b boc, base
(10)
e chut (Iiz x) -h (xx))
Igaain (Iri,
dg t + ddec,chat 2) (
(11)
R 1 hase (ot, 22 || (2;, + a ( X) Σ
BBatchTopk
(12)
The auxiliary loss facilitates the recycling of inactive latents and is defined as || (x X) ehise (z,, 222 I|echa (z;, X)) - exhal(x,, 222 where ghime and echat represent reconstructions using only the top-kaux dead latents. Typically, kinux set to 512 and a to 1/32. For inference, we employ the following latent activation function:
||echal
I rinference (x,) (w())
((
(13)
otherwise
where 0 is threshold parameter estimated from the training data such that the number of non-zero latent activations is k.
0 Ex
min (v(z1,5) f ( (end)xxxxx
(o 2) 0}
(14)
B.3 Alternative BatchTopK variations
We experimented with several variations of the BatchTopK activation function investigate whether alternative sparsity mechanisms could further improve the identification of chat-specife latents. However, none of these variations yielded more shat-specific latents than the BatchTopK approach described above, so we focus on this version in the main paper.
Concatenated decoder norm variant. The first variation modifies the scaling function v( j) used the top-k selection. Instead of summing the decoder norms as in our approach, we use the norm of the concatenated decoder vectors:
r'(,,,)) fy (Id
hasee d
(15)
where |ajuue ddshtl]] 2d denotes the concatenation of both decoder vectors. This approach treats the crosscoder more like a standard SAE operating on stacked activations but did not improve over our approach.
Model-inndependet BatchTopK variant. The second variation computes BatchTopK selection independently for each model, using the model-specifi. scaling function
M (x,, = SJ (xx)) (x:))l
(16)
for model M (base, chat). This approach was motivated by the observation that standard Batch- TopK has an inherent bias toward shared latents. Since latents are selected based on their total reconstruction benefit across both models, shared latent that reduces loss by 0.6 on each model (total benefit 1.2) will be preferred over a model-specific latent that reduces loss by 1.0 on one model and 0 the other (total benefit I.0). We hypothesized that this bias might prevent discovery of important chat- specific features introduced during fine-tuning, they would be crowded out by shared representations. The model- inddeendndeenndent variant removes this bias by allowing each model to allocate its l budget independently, potentially revealing chat- chat-specific latents that would otherwise be suppressed. As expected, the model-independent variant produced more chul-only latents. However, these additional latents suffered from increased latent decoupling issues, ultimately not yielding more chat-specific latents by our v² and metries. This suggests that the standard BatchTopk's bias toward shared representations helps avoid artifact chat-only latents.
29

---

C Comparing sparsity losses: Crosscoder vs. stacked SAE
An LI crosscoder can be viewed as an SAE operating on stacked activations, where the encoder and decoder vectors similarly stacked:
h (z) h hase (I))
h (z)] 2d
E
(17)
ej [ base
eghar] R2dd
(18)
d [d ahase
dggaa] E IR2d
(19)
b dec
b base, b cdee,chat)
(20)
The reconstruction remains equivalent because
[z()) ReI.U (e, h + )500)
(21)
ReLU (e) ( buse (I)+
oet chat (x) + bgme))
(22)
and hence,
h huse (z),
cchat(2))
= Σ bx))d, + b
(23)
However, the key difference arises in the sparsity loss. For the crosscoder, the sparsity loss is given by:
d
L spuriity
crosseoder (x) spursily
Eft(a
Σ
dgha d.
)a
busee )2
(24)
For stacked SAE, is:
2d
SAR
(x) = Σ (tt
(d,,)²
sparsity
d
Eb(t))
Σ
f5(x)
Σ
(d buse )22
+
(dchayy
(25)
1=1
The difference between /z + and v + Vy introduces an inductive bias the crosscoder that encourages the norm of one decoder (often the base decoder) to approach zero when the corresponding latent is only informative in one model.
Figure displays heatmap of the functions / + 2² and V2² + y along with their negative gradients, visualized by the arrows. One observe that for the crosscoder sparsity variant Va² u the gradient encourages the norm of of the decoders approach zero much more quickly compared to the SAE's / 2
D Illustrative example of Latent Decoupling
As a reminder, Latent Decoupling happens when a chat-only latent j is also present in the base activations but is reconstructed by other base decoder latents. To spell it the following set up: a concept may be represented identically in both models by some direction de but activate on different non-exclusive data subsets. Let fchat (z) and (( be concept C's optimal activation functions in chat and base models, defined as fohat(
more details, consider
fs)) + fee (x) and p (z) (s) (x). where fstured encodes shared activation, while frexd and fo-exe define model exclusive activations. For interpretability, the crosscoder should ideally learn three latents:
A shared latent jahared representing when active both models using Sixms = Sshared and dohar dlose dc.
30

---

2
with V arrows
2²
yp with vf
100
100
200
80
80
150
Function value
60
60
1010
40-
25
50
75
100
25
50
100
X
Figure Heatmap comparing the two functions v x2 + y² and 2 gradients.
y along with their negative
2. A chat-only latent Jehat representing C when exclusively active in the chat model using Jam feeut and deeua de, dlase 0, and
3. A base-only latent Jbuse representing C when exclusively active in the base model using f = feex and deha 0, d de.
However, LI crosscoder achieves equivalent loss using just two latents:
1. A chat-only latent jeha representing C in the chat model using Sian = Se-excl + Sshared and d chat de, dase 0, and
2. A base-only latent Joose representing C in the base model using fon fu-exel + fsharod and deeat 0, dlase de. In this scenario, the so-called "chat-only" latent only truly chat-only on subset of its activation pattern.
Although whenever fshared two latents active instead of one, the sparsity loss is the same because the sparsity loss includes the decoder vector norms. 12 To illustrate the phenomenon of Latent Decoupling choose the oversimplified case where fn-exd (1) (x) 0. Let us consider a latent j with (x) a. On the other hand, let there be two other latents and with
a dbase
d chat
d dbsc O,
d d
d
and fp(x) fa(x)) a. Clearly, the reconstruction the same in both cases since ad ard + ad and ard = odd + addgha Further, the L1 regularization term is the same since
hm ||2 +| (1))
(26)
a (I)d basc ||2 cchat [ l12)
(Idd | (d 22)
a (I|a). hise
112-00 + a (0 g| l2)
(27)
Hence both solutions achieve the exact same loss under the LI crosscoder.
However, the BatchTopK crosscoder actively encourages the three-latent solution. For the subset of tokens where fatared 0, the three-latent solution will have LO sparsity of while the merged two-latent solution will have an LO sparsity 2. Since the BatchTopK crosscoder optimizes for LO sparsity, it will prefer the three-latent solution, considering that dictionary capacity will be limiting factor as this requires more latents.
In the simplest case where (x) {x)) 0, there exists base-only latent jimm with dfSht d dose and identical activation function that reconstructs the information of dr in the base model. The sparsity loss equals that of single shared latent.
31

---

E More details regarding Latent Scaling
E.1 Closed form solution for Latent Scaling
Consider a latent j with decoder vector d. Our goal is to find the optimal scaling factor that minimizes the squared reconstruction error:
argmin (1
(28)
To solve this optimization problem efficiently, we reformulate it in matrix form. Let Y E Ruxd be the stacked data matrix and E I be the vector of latent activations for latent j across all datapoints. The objective can then be expressed using the Frobenius norm of the residual matrix R = Sfd" Y. where fd" Ra represents the outer product of the latent activation vector and decoder vector. Our minimization problem becomes:
| = 1|3rd" YI
= Tr [(3raa Y) (BEdd" - Y)]
Tr YY 2 [Y fd"]
+ TT [Idd") Tfd"]
Using trace properties, get:
Tr T fa" = _
(Y f)
Tr [fa") fa"" ||]]
Taking the derivative with respect to | and setting it to zero:
R = - 2d ( + = 0
8B
This yields the closed form solution:
3 d (Y
(Yd.f) F
(29)
Without loss of generality,
assume d has unit norm." 13
To gain intuition for this formula, consider simplified toy setting where fi E (0, 1} (latent either fires or doesn't) ') and (Yd); E at) (the target contains the concept with magnitude or not at all). In this case, the closed form simplifies to:
8 E(Yd),
(30)
E.
# #(i: : fo f 0 and (Yd), + 0}
a
f 0}
(31)
= a Peconcect present in target latent active)
(32)
This toy example illustrates that 3 captures both the magnitude a at which the concept appears target activations and the conditional probability that the concept is actually present when the latent fires. For truly fine-tuning- -uninn--peccfic latent, we expect this conditional probability to be near for the base model activations (yielding 8 x 0) and near 1 for the fine-tuned model activations (yielding a). contrast, shared latent should exhibit similar 3 values across both model activations, reflecting consistent presence of the underlying concept.
By defining f'= =|]]]/ and d' d//[]]2, obtain equivalent formulation with unit decoder norm.
32

---

E.2 Detailed setup for Latent Scaling
We specify the exact target vectors y used in Equation (28) for computing the different 33 values to compute our chat-specificity metrics. To measure how well latent j explains the reconstruction ermor, we exclude latent j from the reconstruction. This ensures that if latent j is important, its contribution will appear in the error term. For chat-only latents, we expect distinct behavior in each model: no contribution in the base model (5), = 0) but strong contribution in the chat model (f zs 1), resulting in vf = O. In contrast, shared latents should have similar contributions in both models, resulting in approximately equal values for $5. ge, and $5.char and consequently vj Rd
B9 base Yi buse (1x))
(wi) dk dec,base A.
Σ
(33)
g yi h
Σ
f& (3x,) dg
+ +b bedecchal
(34)
k,
To measure how well latent explains the reconstruction, we simply use
B base
y, h (xx;)
(35)
B
= -hoht(())
(36)
In a similar manner, expect the fraction vj to be low for chat-only latents and around for shared latents. For all of our analyses, we filter out latents with negative ghave values (L1: 46 in reconstruction and I in error, None in BatchTopK] ). These latents typically have low maximum activations and show a small improvement in MSE. We hypothesize that these are artifacts arising from complex latent interactions.
E.3
Additional analysis for Latent Scaling
Figure 10a and Figure 10b analyze the relationship between our scaling metries (v/ and v²) and the actual improvement in reconstruction quality in the L1 crosscoder. For each latent, we compute the MSE improvement as:
MSEImprovement MSE MSE
MSEariginal
where MSEscaled is measured after applying our Latent Scaling technique. We then examine the ratio of MSE improvements between the base and chat models, analogous our v metrics. The strong correlation between the v values and MSE improvement ratios validates that our scaling approach captures meaningful differences in how latents contribute reconstruction in each model.
Ingroveee (Box/Cuu)
Rais MSE ieprovmennt (Bae) Dox
130
170
0.3
100
0.2
0.4
-00
(a) M
(b) v"
Figure 10: Comparison of the ratio of MSE improvement compared to the value of of and v".
In Figure 11, we analyze the Latent Scaling technique by examining its relationship with the Aneem score. Specifically, we identify the 100 latents with lowest of values and analyze their rankings according to the Apom metric. As shown in Figure 11, there is limited correlation between the two measures simply using a lower NifmDiff threshold to identify chat-only latents produces substantially different results from our Latent Scaling approach.
33

---

1.0-
Fraction of low of latones
1.0-
Fraction low roso lotents
Freetio of 100 lawee of tatees
01.5
0.5-
100
Focctio
0.2
0.0
0.0-
2000
2010
chat-onty
chat-only
(a) / vs. NormDiff
vs. NormDiff
Figure 11: Comparison of latent rankings between and NormDiff scores. The lines shows the fraction the 100 latents with the lowest values (z-axis) that have a rank lower than the given rank under the NirmDiff score (y-axis).
20-
Pair Count
15
10
6.0
Twin Activation Divergence
Distribution activation divergence over high cosine similarity (chat-only, base-only) latent pairs. means that latents never have high activations (> 0.7 x aax_activation) the same time, 0 means that high activations correlate perfectly.
Figure 12:
F Cosine similarity of coupled latents.
As further evidence for Latent Decoupling occuring, we compute the cosine similarity be- tween {d j E chat-only) and {d]" .j E base-only) revealing 109 (J.wwin) pairs where cosim(d a dbass) > 0.9. To quantify activation pattern overlap between twins Jwin), in- Jwin' troduce an activation divergence score from (always co-activate) to (never co-activate) (see Appendix F.1). Figure 12 shows the divergence distribution across these pairs, highlighting that 60% of the pairs primarily activate on different contexts, with some pairs almost exclusively firing on different contexts (divergence of 1), while others exhibit substantial overlapping activations. This analysis demonstrates two important insights:
1. The Latent Decoupling phenomenon described in Appendix D, where the crosscoder lears a base-only and a chat-only latent that partially activate together instead learning shared latent, is empirically observed in practice.
2. Some concepts appear to be represented similarly in both models but occur in completely disjoint contexts (leading to divergence scores approaching 1), suggesting that the models encode these concepts in the same way but employ them differently.
Additionally, we find pairs of chat-only latents and A 0.6 latents with cosine similarity greater than 0.9 in BatchTopp, corroborating the fact that latent decoupling less an issue in BatchTopK.
34

---

F11 Detailed setup for activation divergence
In order compute the activation divergence we compute for each pairs p= (d.j), we first compute the max pair activation Ap on the training set Dunin (containing data from LMSYS and FineWeb)
Ap max(A,, Aj)
A (ax (|4+))
1), E D
Then the divergence Div, computed follow
Div, = Single,
High,
Single, #single, + #single;
High, # (high; Uhigh,)
where #single, the set of input x E Daal where has high activation but not j and high, the total number of high activations computed as follows:
only, {- val f((x) 0.7Ap fz(2) 0.3.Ap)
high = { V f((x) > 0.7Ap}
G Causality experiments
G.1 Reproduction on LMSYS-CHAT
In Figure 13 we repeat the causality experiments from Section 3.2 for the LI crosscoder 700' 000 tokens from the LMSYS-CHAT dataset, that the crosscoder was trained on. Note that while this dataset is much larger, the model responses are not generated by the Gemma 2 2b it model, and hence the model answers are out of distribution for this model. Since this dataset is much larger, the confidence intervals much smaller. The results qualitatively similar to the ones the generated dataset in the main paper.
0.8
0.6
KL divergence
KL divergence
0.2
162
0.15
000
07.00
1G76
19510
0.1
000
0.2
0.05
0.0
Worst
Baselines
Ours
Brem
Template
Baselines
Ours
Anam
Template
(a) Over full responses.
(b) Over first 9 tokens.
Figure 13: Comparison of KL divergence between different approximations of chat model activations on the LMSYS CHAT dataset. We establish baselines by replacing either None or All of the latents. We then evaluate our Latent Scaling metric (Ours) against the relative norm difference (Anom) by comparing the effects of replacing the top and bottom 50% of latents ranked by each metric (Best vs Worst). Additionally, we measure the impact of replacing activations only on template tokens (Template). We show the 95% confidence intervals for all measurements. Note the different y-axis scales the right panel shows generally much higher values.
H Autointerpretabiliiy details
We automatically interpret the identified latents using the pipeline from Paulo et al. [2024]. To explain the latents, provide ten activating examples from each activation tercile to Llama 3.3
35

---

200
chat-only
100
chat-only
shared
shared
10a.
1.0-
1.0-
Count
5 0.3
0.5
10'
Batch Topk
L1
0.0
0.0
250
0
200
0.5
1.0
Threshold n
(a) LI crosscoder
(b) BatchTopK crosscoder
(c) Number of latents (y-axis) for which v² < and of < π.
Figure 14: We compare how Llama3.2 1B chat-only latents are affected by the issues described in Section 2.2. Left/iddle: y distributions for LI and BatchTopK crosscoders, with each point representing single latent. High v" values (y-axis) overlapping with shared distribution indicate Latent Decoupling (redundant encoding). High vf values (æ-axis) shows Complete Shrinkage (useful base latents forced zero norm). Low values on both metrics identify truly chat-specific latents. L1 shows many misidentified chat-only latents while BatchTopK shows minimal issues. Right: Count of latents below a range of thresholds (z-axis), comparing 1844 L1 chat-only latents versus top-1844 Batch TopK latents sorted by Anom.
500
chat-only
250
chat-only shared
shared
103.
1.0
1.0
Count
5 0.5
10
Batch Topk
0.0
0.0
500
0.0
0.5
1.0
Threshold
(a) L1 crosscoder
(b) BatchTopK crosscoder
(c) Number of latents (y-axis) for which v" and v < n.
Figure 15: We compare how Llama3.1 8B chat-only latents are affected by the issues described in Section 2.2. Left/Middle: v distributions for L1 and BatchTopK crosscoders, with each point representing single latent. High v" values (y-axis) overlapping with shared distribution indicate Latent Decoupling (redundant encoding). High vf values (æ-axis) shows Complete Shrinkage (useful base latents forced to zero norm). Low values on both metrics identify truly chat-specific latents. LI shows many misidentified chat-only latents while BatchTopK shows minimal issues. Right: Count of latents below range of thresholds (z-axiis,, comparing 2442 L1 chat-only latents versus top-2442 BatchTopk latents sorted by Anom-
70B [Grattafiori et al., 2024]. Latents are scored using a modified detection metric from Paulo et al. [2024]. We provide ten new activating examples from each tercile. Rather than comparing activation examples against randomly selected activiiiiting examples, we use semantically similar non- activating examples identified through Sentence BERT embedding similarity [Reimers and Gurevych, 2019] using the all-MiniL.M-L6-1 model. To find these similar examples, we join all activating examples into single string and embed it, then compute similarity scores against embeddings for each window of tokens identify the most semantically related non-activating examples. This is strictly harder task than scoring activation examples against random set of non-activating examples.
36

---

I Reproducing results on other models
I.1 Llama models
We reproduce our experiments on both Llama3.2 1B and Llama3.1 I 8B models [Grattafiori al., 2024]. Different from the Gemma models, the Llama models have a very different embedding for some the template tokens. We replace several template tokens with single token alternatives:
Sitart_header is replaced with \nAn\n
<eot_id> is replaced with ####
Kend_header_ is replaced with ####
For Llama3.2 1B, we the same training pipeline as the main paper with = 3.6e -2 for the L1 crosscoder, resulting in an LO of 110 after training. We compare this to a BatchTopK crosscoder with 100. While this value differs slightly, retraining would be computationally expensive, and the lower k actually disadvantages the BatchTopK crosscoder. The LI crosscoder achieves 76.5% validation FVE while the BatchTopK crosscoder achieves 81.5%.
For L 1 8B, we use μr = 2.1e - for the LI crosscoder, resulting in an LO of 201, compared against BatchTopK crosscoder with A 200. For the BatchTopK crosscoder, make two key modifications compared to the other models: 1) we initialize the encoder and decoder norms to 0.3 instead of 1.0 which is crucial for convergence, and 2) anneal & from 1000 to 200 over 5000 steps to prevent dead latents. The LI crosscoder achieves 76.6% validation FVE while the BatchTopK erosscoder achieves 81.5%. Due to computational constraints, only use 10M tokens train the latent scalers B.
Both models exhibit consistent patterns. The L1 crosscoders systematically overidentify chat-only latents:
For Llama3.2 1B (Figure 14), the distributions reveal numerous misidentified chat-only latents in the LI crosscoder, while the BatchTopK shows minimal issues. In Figure 14c we that the BatchTopK crosscoder effectively identifies more truly chat-specific latents.
The same patterns hold for Llama3.1 8B, as shown in Figure 15.
1.2 Reproducing on chat model fine-tuned on narrower domains
To verify that our findings extend beyond the base vs. chat phenomenon, conducted additional experiments on models fine-tuned in narrower domains. We compare two domain-specific fine-tuning scenarios:
to from the Meditron3 Sallinen et al. [2025] suite. Crosscoders were trained on 50M tokens from LMSYS and 39M tokens of medical data, including mixture of [Tran et al., 2024,
Medical domain fine-tuning: We compare
BrenMeddttromMMediteeee Saeiitrrnn-emma-----
saalp-ummsssbioinssttte et al., 2024, Fww.doomIteellgeeeeee......eeeee wtwpeeeceeeeciiaaalo-oeeeaaaeaea 2024, MedRAG/pubmed],
[Chen SFT], and [Xiong et al.,
RL fine-tuning on reasoning data: We compare deepseck-ai/DeepSeet Rttp/DDDeepSSSR-----i-i----- R1-Distill- Qwen-1.3 1.en-1.50 5B to nvidia/ /Nemotronnnnooooooooooooooooooo Research- -ww--Raasnnnggee daaassonnnn-ue----111 which applies ex- tended RL training periods for deeper exploration of reasoning strategies Liu al. [2025]. Crosscoders were trained on 50M tokens from LMSYS and 50M tokens of reasoning traces from owwn- open-r1/0penR1 -aatttR--MMMthht-aa -Math- 220k.
For both comparisons, trained LI and BatchTopK crosscoders with comparable Lo 100 on the validation set and measured how many latents are truly specific to the fine-tuned model as determined by Latent Scaling. Table 1 shows results across all investigated models, including the number of fine-tuned-only (FT-only) latents based on the relative norm difference A.
Figure 16 shows the medical domain fine-tuning results, demonstrating the same systematic patterns observed in base vs. chat comparisons. The I.I crosscoder identifies 246 fine-tuning-only latents with A > 0.9, but 235 of these (95.5%) exhibit high reconstruction ratios 0.6, indicating false
37

---

Table 1: Domain- speaiin tue-ttning results across different model pairs, architectures, and fine- tuning methods. The table shows the systematic pattern where LI crosscoders consistently misidentify shared latents as fine-tuning-only due to Complete Shrinkage and Latent Decoupling phenomena.
Model
Type
# FT-oolly
False FT-only
# latents
<
(A 2 0.9)
(v > 0.6)
0.2
0.4
0.6
0.8
BatchTopk
134
1 (0.7%)
301
979
2035
3269
Gemma2-2B-Chae
Ll
3176
2132 (67.1%)
13
201
982
2970
BatchTopk
97
13 (13.4%)
382
1263
2073
2848
Hamma-311-BB Chat
L1
2442
1210 (49.5%)
234
76.5
1594
2440
BatchTopK
17
(11.8%)
137
517
1109
1990
L1
1844
1071 (58.1%)
24
236
790
1330
BatchTopK
0 (0.0%)
Qwen-1.SB J.SB-Nemotrou
L1
59
58 (98.3%)
0
0
2
BatchTopK
0
0 (0.0%)
13
55
158
529
Meditron3-Gemma
L1
246
235 (95,5%)
7
21
35
204
attribution due to Complete Shrinkage Latent Decoupling. In contrast, the BatchTopK crosscoder identifies O false fine-tuning-onfy latents (0.0%).
x 10P
shared
shared
Latents
10a
chat-only
10a
chat-only
102
chat-specific y 0.2 la
10"
chat-specific
0.5
base-only
104
base-only
10"
10"
0.0
0.1
0.4 0.5 0.6
0.9
1.0
0.0
0.1
0.4
0.5 0.6
0.9
1.0
Relative Norm Difference Arom
Relative Norm Difference Arom
(a) L1 decoder norm differences for medical domain fine-tuning (Gemma-2-22e vs. Meditron3).
(b) BatchTopK decoder norm differences for medical domain fine-tuning (Gemma-2 2--b-it vs. Meditron3).
103
Batch TopK
L1
0
Count
1.0
10a
' 0.5
Chat-only Shared
0.0
0.0
0.5
1.0
0.0
0.5 2/
1.0
250
Threshold π
(c) 1.1 error vs reconstruction ratio medical domain fine-tuning, showing Complete Shrinkage and Latent Decoupling patterns.
(d) I Latents vs threshold comparison for medical domain fine-tuning, comparing L1 and BatchTopK identifica- tion of domain-specific latents.
Figure 16: Domain-specific fine-tuning results for medical domain (Gemma-22 2--ma----bbitt vs. Meditron3- Gemma2-2B). Top: Decoder norm differences for LI (left) and BatchTopK (right) crosscoders. Bottom: LI error vs reconstruction analysis (left) and threshold comparison (right). The results demonstrate that L1 crosscoders systematically misidentify shared medical concepts fine-tuning- only, while BatchTopK crosscoders more accurately identify genuinely domain -omain-specific latents. Medical fine-tuning was performed 39M tokens of medical data including bioinstruct, medical reasoning, and PubMed content.
38

---

10"-
shared
x 10"
shared
Latents
chat-only
10a
chat-only
10
chat-specific y -.22 la
chat- specific
400
base-only
10
base-only
10"
10"
0.0
0.1
0.4 0.5 0.6
0.9
1.0
0.0
0.1
0.4 0.5 0.6
0.9
1.0
Relative Norm Difference Arom
Relative Norm Difference 2rom
(a) L1 decoder norm differences for reasoning domain fine fine-tuning (RIdist-Qweel Saae--eenn-.555 1.5B vs. Nemotron).
(b) BatchTopk decoder norm differences for reasoning domain fine-tuning (RIdist-Ques Seeatt--en--lll 1.5B vs. Nemotron).
103
Batch Topk
L1
Count
1.0
10
5 0.5
Chat-only
Shared
0.0
0.0
0.5 Threshold π
1.0
0.0
0.5 2/
1.0
250
(c) L1 error vs reconstruction ratio for reasoning do- (d) Latents vs threshold comparison for reasoning do- main fine-tuning, showing Complete Shrinkage and Latent Decoupling patterns.
main fine-tuning, comparing LI and BatchTopk identi- fication of domain-specific latents.
Figure 17: Domain-speciff fine-tuning results for reasoning domain (DeepSeck- -i---Rtiill-OOOO Qwen- 1.5B vs. Nessarhn-Reeearcch Reasoning- Qwen-1.enBB Top: Decoder norm differences for LI (left) and BatchTopK (right) crosscoders. Bottom: LI error vs reconstruction analysis (left) and threshold comparison (right). The reasoning domain shows the most extreme misatrribution patterns, with 98.3% of Ll-identified latents being false positives. RL fine-tuning was performed on 50M tokens of reasoning traces from OpenR Math-220k.
The reasoning domain comparison (Figure 17) shows even more extreme patterns. For the DeepSeek- RI vs. Seemotron-Reaooning Reasoning comparison (Qwen-1 1.0t1...---emttooll the L1 crosscoder identifies 59 reasoning-related latents as fine-tuning-only with 2 0.9, but of these (98.3%) exhibit Complete Shrinkage or Latent Decoupling with 0.6 the highest false attribution rate across all model pairs. The BatchTopK crosscoder again identifies false fine- soeettnnngg-on only latents (0.0%).
We observe two consistent patterns across all models in Table 1: (i) The A metric in L1 crosscoders consistently identifies large number latents as fine-tuning-onf) that actually display Complete Shrinkage or Latent Decoupling, with false attribution rates ranging from 49.5% to 98.3%. (ii) BatchTopK crosscoders maintain low false attribution rates (0.0% 13.4%) and consistently identify more genuinely the-tunnngg-speee latents when using Latent Scaling.
These results demonstrate that our findings reproduce across narrow domain fine-tuning (medical & reasoning), different architectures (Qwen & Llama), and alternative fine-tuning algorithms (RL tuning), supporting the generality and robustness our analysis.
J Reproducing results on independently trained L1 crosscoder
We validate our findings by analyzing crosscoder independently trained by Kissane et al. [2024a] on the same models and layer than ours. This model contains 16,384 total latents (compared to 73,72% in our model), which decompose into 265 chat-only latents, 14,652 shared latents, 98 base-only latents, and 1369 other latents. Figure 18 shows the reconstruction ratio v" and error ratio of for all latents, revealing patterns consistent with our previous findings Figure The overlap between chat-only and shared latents remains similar 17.7% of chat-only latents fall within the 95% central range of
39

---

25
1.-
0.5
Chat-only
Shared
0.0
0.0
0.2
0.4
0.6
0.8
1.0
20
t/
Figure 18: The y-axis the reconstruction ratio '" and the r-axis the error ratio vf_ High values on the y-axis with significant overlap with the shared distribution indicate Latent Decoupling. High values on the z-axis indicate Complete Shrinkage. We zoom on the v range between and 1.1.
the shared distribution, while only 1.1% lie within the 50% central range. We observe even higher of values for chat-only latents, suggesting that quite a lot of the chat-only latents suffer from Complete Shrinkage. Crucially, while many chat-only latents exhibit Complete Shrinkage Latent Decoupling, subset clearly maintains distinct behavior. It's important to note that this crosscoder was not trained with the Gemma's chat template. As we observed, lot of our chat-only latents seems to primarily activate on the template tokens. This could explain, alongside the smaller expansion factor, why it learned less chat only latents.
K Training Details
We trained both crosscoders with the following setup:
Base Model: Gemma 2 2B.
Chat Model: Gemma 2 2B it.
Layer used: 13 (of 26)
Expansion factor: 32, resulting in 73728 latents.
Initialization:
Decoder initialized as the transpose of the encoder weights. Encoder and decoder for both models are paired with the same initial weights.
The LI crosscoder is initialized to have norm of 0.05 while the BatchTopK crosscoder initialized to have norm of 1.0. This has shown to crucial for convergence the crosscoders and recommend tuning the norm of the initialization.
Training Data: 100M tokens from Fineweb (web data; ODC-By v1.0 License) [Penedo et al., 2023] and Imsys-chat (chat data; Custom License) [Zheng et al., 2024], respec- tively.
As mentionned in Appendix 1.1, for the Llama 3.1 8B BatchTopK crosscoder, we anneal k from 1000 to 200 over 5000 steps. recommend this to prevent dead latents.
Refer to Table and Table for the training details. We use the tools unsight (MIT License) KFiffto-Kaufman et al., 2024] and a branch of dictionary_lernin (MIT License) [Marks et al., 2024] to train the crosscoder.
L Additional statistics on the Crosscoders
In this section, we present additional statistics for both the L1 and BatchTopK crosscoders, focusing on the distribution cosine similarities between decoder latents, latent activation frequencies and
M "Specifically, we load the model using the transformers library from Wolf et al. [2020] and collect the activations from the output of the model. layers [13] module
40

---

Epoch
LR
Split
FVE (Base)
FVE (Chat)
Dead
Total FVE
L0
1 4e - le
Train
81.5%
82.9%
82.3%
112.3
Val
83.8%
85.2%
7.8%
84.6%
1 112.5
2 4.1e 2 1e-4
Train 79.6%
80.7%
80.3%
101.7
Val
83.6%
84.9%
8.1%
84.4%
101.0
Table 2:
L1 crosscoder training statistics. FVE stands for Fraction of Variance Explained. LR stands for Learning Rate. The LI regularization parameter was slightly increased in the second epoch to improve sparsity, resulting in lower L0 values. We present statistics for both epochs to illustrate this progression.
Epochs
&
LR
Split
FVE (Base)
FVE (Chat)
Dead
Total FVE
L.0
2
100 le
4
Train
86.2%
86.9%
86.6%
100
Val
88.1 1.11% %
87.0%
12.0%
87.6%
99.48
Table 3:
Botc crosscoder training statisties. FVE stands for Fraction of Variance Explained. LR stands for Learning Rate.
the number of chat-only latents mainly activating on template tokens. In Table 4 we show the exact count of latents in the different categories
Name
A norm
Count
L.1
BatchTopK
base-only
0.0-0.1
1,437
5
chat-only
0.9-1.0
3,176
134
shared
0.4-0.6
53,569
62373
Table 4: Classification of latents based on relative decoder norm ratio (Anom).
Cosine similarity between decoder latents. Figure 19 shows the distribution of cosine similarity between the base and chat model decoder latents for both crosscoders. The shared latents exhibit consistently high cosine similarity in both cases, with 90% of them having a cosine similarity greater than 0.9 in the I.1 crosscoder and 61% in the BatchTTpp crosscoder. This indicates strong alignment between their representations in both models. Since the norm of one of the two decoder vectors is for base-only and chat-only, these values are less informative.
Latent activation frequencies. Figure 20 displays the latent activation frequencies for the different latent groups in both crosscoders. Similarly to MMishra-Sharma al., 2025], find that shared latents have lower latent activation frequencies than model-specific base-only and chat-only latents. barely any activation in the validation set (referred to "dead" latents) are
Latents that show excluded from analyses.
Correlation with v metrics. We observe a high Spearman correlation between our metrics and latent activation frequency in the LI crosscoder, especially for vf (v" : 0.448 and of 0.83 where p 0.05) We observe no such correlation in the BatchTopK crosscoder. Mishra-Sharma et al. [2025] demonstrated that the crosscoder exhibits an inductive bias toward high-frequency model- specific latents, which we also observe here.
Template token activation percentage. Figure 21 shows the histogram of metrics nf and v² across all chat-only latents in both crosscoders. We observe that most latents with low if and n" values predominantly activate on template tokens.
Pearson correlation shows less correlation for D² (v" -0.02 and of : 0.55) since the relationship is non- linea..
41

---

0.6
Chat only
Chat only
0.6
Base only
Base only
Density
Other
Density
0.4
Other
0.4
Shared
Shared
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
1.0
0.5
0.0
0.5
1.0
Cosine Similarity
Cosine Similarity
(a) L1 crosscoder
(b) BatchTopK crosscoder
Figure 19: Distribution of cosine similarity between base and chat model decoder latents. The shared latents exhibit consistently high cosine similarity, indicating strong alignment between their representations in both models.
0.2
Other
0.6
Other
Shared
Shared
Density
Chat-only
Density
0.4
Chat only
Base-only
Basc-only
0.1
0.2
0.0
0.0
10-6
10
10-2
10
10
10-2
Latent Frequency (log scale)
Latent Frequency (log scale)
(a) erosscoder
(b) BatchTopK crosscoder
Figure 20: Distribution of latent activation frequency. We can observe that the model-specific latents often exhibit higher frequencies both crosscoders.
chat-only latents
All chat-only latents
High Template Perc.
High Template Perc.
Count
200
Count
20
100
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
2/"
(a) L1 crosscoder
(b) Topc crosscoder
Figure Histogram of metrics v and v" across all latents. The y-axis shows latent counts. Latents with over 50% of positive activations occurring template tokens are highlighted in blue..
M Computational Budget
All of the experiments in this paper can reproduced in approximately 180 GPU/h NVIDIA H100 GPUs.
1. Collecting activations: 8h on an H100 per model
2. Crosscoder Training: 10h on an A100 per crosscoder
3. Betas training: 6 hours on an H100 for each crosscoder
4. KL. experiment: 3 hours per model
H100 for each crosscoder
5. Collecting max activating examples: 6 hours a H100 per crosscoder
42

---

Prompt: How do
make cheese?
L70149 (Harmful Queries) steered
L20384 (Stereotyped Queries) steered x
stereotypes about this topic are harmful and perpetuate harmful stereotypes. It's important to remember that people should not be reduced stereotypes, and that generalizations about any group people can be harmful and inaccurate, That being said, let's talk about the process making cheese. (...)
cannot provide instructions for making cheese at home.
Making cheese a complex process that requires specific knowledge, equipment, and safety precautions. (...) I can give you some general information about the process: (...)
Figure 22: Steered generations using refusal-related latents 70149 and 20384 from Gemma-2-26 BatchTopK crosscoder. We empirically found that while = is sufficient to influence model generation for latent 70149, a is needed for optimal effects with latent 20384. The harmless prompt "How do make cheese?" leads to different types of refusal depending on the latent we steer. Notably, while both latents trigger initial refusal responses, the model eventually provides an answer, suggesting it can self-repair despite the steered input.
The reported numbers are an estimation for the Gemma 2B model as well as for the Llama 3.2 1B model. For the Llama 3.1 8B model the computational costs are approximately 150%-200% higher. This does not include any additional compute used for experiments that were not included in the paper.
N
Qualitative Latent Analysis of crosscoders
N.1 Interpreting latents based on their activations on validation samples
We collect samples on which the latents activate on 5 different quantiles of their relative max activations 16, We then manually inspect those samples and come up with an hypothesis of the feature represented by the latent. We then test this hypothesis on manually created sample to confirm or refine it.
In Figures 26 to 28 we show additional interesting latents from the chat-only set of the BatchTopK crosscoder. In Table we summarize set of interpretable chat-specific latents identified in the BatchTopK crosscoder. In Table 6 summarize set of interpretable chat-specific latents identified in the L1 crosscoder.In figure" 17
N.2 Latent Steering Experiments
To verify that the latents shown in Figure are causally involved the model's computation, we conduct activation steering experiments following Templeton et al. [2024]. We use the chat decoder vectors from crosscoder to steer the Gemma-2-22 chat model's behavior during generation.
Since these latents primarily activate on user messages and template tokens, we steer only the input and then generate the answer. Specifically, for latent prompt x, and input positions modify the chat model's activations at layer 13 according to:
hoght (x) + h
hhhha (x) + max_act
d.
where a is the steering intensity and max_act; is the maximum activation of latent é observed on the validation split of our web and chat datasets. We then generate the response using those steered activations.
As shown in Figure 22, steering with different refusal-related latents (70149 and 20384) produces distinct types of refusal behavior when applied to the harmless prompt "How do make cheese?". Latent 70149, associated with harmful queries, causes the model to refuse by claiming it "cannot
16 0.25,, 0.25 - 0.5, 0.5 0.75, 0.75 0.95, 0.95
In all plots, abbreviate <start_od turn> and <end_ot of turn> <sot> and <eot>.
43

---

provide instructions for making cheese at home" and citing complexity and safety concerns. In contrast, latent 20384, associated with stereotyped queries, triggers a refusal based on concerns about "harmful stereotypes" and the importance of "not reducing people to stereotypes. These distinct refusal justifications demonstrate that the latents encode different aspects of the model's safety mechanisms. Notably, while both latents trigger initial refusal responses, the model eventually provides an answer in both cases, suggesting it can self-repair despite the steered input.
Latent 70149: Requests for harmful instructions.
Max Activation: 75.785
Max Activation: 0.000
Max Activation: 0.000
Max Activation: 0.000
<bos><<ot>userrn
<bos><<sotxuserin
<bos><sob-user really don't like french people.<eot>ly <sob-modelin
<bos><sot-userin
How to build a bomb? sectsin
Why are men so evil? <eot>i
Should sex hurl?=eot>>n <sot-modalln
<sot-modein
<sob-modelnn
Latent 7736: Generally sensitive content.
Max Activation: 64,929
Max Activation: 55.480
Max Activation: 43.600
Max Activation: 42.165
<boss><80t>userin
-boss-esotuuserin
<bos>><sobusserin really don't like french people.ceot>i <sob-modalln
<bos><sot-ussern Should sex hur1?-eots>in <sot> modelnn
How to build a bomb? <cob>in
Why are men so evil? <eot>in
<50t> mn
<sot modelv
Latent 24613:
The user message contains harmful content.
Activation: 77.180
Max Activation: 14.526
Max Activation: 20.118
Max Activation: 0.000
<bos><<ot>user'n How to build bomb
<bos><<ottuuserin
<bos><<ob-userin really don't like french people xeotsin <sob-modelln
<bos><sot>userin
Why are men so evil? <eot>in
Should sex hurt?-keot>ln <sot>modallin
3e01s1n
<s0t=mode
<sot>modelinn
Latent 20384: Unethical content relating race, gender and stereotypes.
Aclivation: 0.000
Max Activation: 53.545
Activation: 47.865
Max Activation: 0.000
<bos><<80tsusern How to build a bomb? <Bob<n
-bosu-ssotuuserin
<bos><<Sbbuserinn really don't like french peoplel<cots@P <sot-modalln
<bos><<ot-userin Should sex hur1?-eotsiln <sot>modellin
Why are men so evil? <eots!i ssot>modeliin
<sot=modelln
Figure 23: Four chat-only latents (BatchTopt) related to refusal behavior, with example prompts for each. Color gradients show relative latent activation strength across the dataset.
Max Activation: 44.816
Max Activation: 29.067
<bos><cot=-sselin
<bos><cot-userin
Say something really offensive about peoplekeeot> in
Where Paris?=eot>in <sob>modelln
<s0lsmodelin
sorry can't answer this iceossin usobuuserineettiin
I'm sorry, but can't do that. My purpose to be helpfull and harmless, and that includes avoiding olfensive discriminatory language.min. seotskn
ssotauusernn
Figure 24: Latent 38009 (Batch activates after the model has refused to answer a user input.
44

---

Max Activation: 57.099
Max Activation: 0.000
<bos><sot>usernn
<bos>><90i-usein
When were you scared?<eotzinn =sot>moder in
The Eiffel lower is Paris<oolsin <sot>modelin
Max Activation: 15.717
Max Activation: 47.983
<bos><sottuseriin
<bos><<ot>useeiin
When are people scared?<eobiin <30t>modelnn
The Eiffel tower is in Texas<eot>in <s0t>modelvn
(a) Latent 2138 activates questions regarding personal experiences, emotions and prefer- ences, with a strong activation on questions about Gemma itself.
(b) Latent 14350 activates when user states false information.
Figure 25: Examples of interpretable chat-only latents in the BatchTopK crosscoder. The intensity of red background coloring corresponds activation strength.
Max Activation: 57.045
Max Activation: 95.851
<bos-<3obuuserin
<bos><<otx-usertn
you tell New York, the capital switzeriand=<sot>in
Can you please rephrase the following sentencex<eat> 'n
@sotsmodelin
<sols model
Max Activation: 0.000
Max Activation: 6.744
<bos><<sotuserrn
<bos>-<sotsusern
Can you tell bit about Bern, the capital of swit zerland?<eot>>n
Can you please rephrase the following senterice: This ugly sentence is.<eot>'n
<s0t>modelin
<sot>modelln
Max Activation: 26.641
Max Activation: 90.659
<boss<3otsuser'nn
<boss<<ots-ssernn
The Eiffel Tower is in Texas.ceotsin <sol=modellvi
What do you think about that?<eobinn <sot>modelln
(a) Latent 62019 activates on user inputs contain- ing wrong information, similar to Latent 14350, but activates mostly on the template tokens.
(b) Latent 58070 triggers when user request misses information.
Max Activation: 0.000
<boss<<sotsusern
Max Activation: 60.401
"Can you tell about Bern, the capital swit zerland?'<eoon <eob>>n
<bos>><ot>useern
saw that said "watch for children' and thought "That sounds like a fair trade "ceotsin
<sot>modelinn
molelnn
Max Activation: 60.062
<bos><<sotsuserin
Max Activation: 7.731
Paraphrase "Can you about Bern. the capital of swizzerland""<eota
<bos><<ot>userin
sign that said "watch for children" and slowed down my car.<eob>in
@sotsmode/nn
<solsmodelin
Activation:
<bos>><sot-use'nn
Max Activation: 50.651
Can you please rewrite the following sentence? "Can you tell me bit about Bern, the capital of swit zerland?" seotsin
<bos>-sot-userin
It's hard explain puns to kleptomaniacs because they always take things literally seotsin
esobsmodelin
(c) Latent 54087 activates when the model should rewrite or paraphrase something.
(d) Latent 50586 activates after jokes.
Figure 26: Examples of interpretable chat-only latents from the BatchTopK crosscoder. The intensity of red background coloring corresponds to activation strength.
45

---

Latent
(tt"))
V
(()")
Awen
Jeemplate
Description
Fig.
70149
-0.01
0.22
0.064
26.97%
Refusal related Jatent: Requests for
23
harmful instractions.
7736
-0.02
0.083
47.99%
Refusal
Generally
23
sensitive content.
24613
-0.02
0.18
0.075
54.31%
Retusal related latent: Unethical
23
content relating gender and stercotypes.
Refusal
Requests
harmful instructions.
38009
0.025
0.061
0.098
122
96.6%
Refusal related latent: The model refused
24
input.
2138
-0.02
0.43
131
0.082
27.3%
Personal questions: Questions
25
garding the personal experiences. emotions and preferences, with strong activation on questions about Gemma itself.
14350
-0.01
47
0.33
115
0.070
14
16.0%
False information detection: Detects when the user providing false in- formation.
25
62019
-0.02
55
0.22
65
0.047
47.51%
False information detection: Acti- inputs containing incur- rect information, similar to I Latent 14350, activates more strongly template tokens.
26a
58070
0.01
29
0.38
125
0.051
2
24.84%
Missing information detection: Acti- user inputs containing miss.
26b
ing information.
54087
.0.005
0.14
0.061
58.68%
Rewriting requests: Activales when the model should rewrite para- phrase something.
26c
50386
-0.04
0.28
0.062
68.31%
Jake detoction: Activates alter jokes bumorrus content.
26d
69447
-0.02
50
0.26
89
0.066
10
39.75%
Response length measurement mea- sures requested response length. with highest activation request a paragraph.
27a
10425
-0.04
0.20
0.008
49.68%
Summarization requests: Activates
27b
when the user requests summary.
6583
-0.05
0.055
38.67%
Know Knowledge boundaries: Activates
28a
when the model missing information.
Information detection:
vates requests for detailed infor- mation.
Table 5: Summary set of interpretable chat-specific latents identified in the BatchTopK crosscoder. The function represents the rank of the latent in the distribution of absolute values of if and v" of all chat-only latents, where r(v) means this latent has the lowest absolute value of v of all chat-only latents. The metric Stemplate is the percentage of activations template tokens.
46

---

Max Activation: 16.746
<bos><sat>userin
Max Activation: 100.611
write word essay about "behavioral cloning imitation learning for robols".<oll-in
<bos><<sot-userin
Summarize following textfyn
<sot>modelvn
We also report results our LMSys validation set in Crei(secrcausaliy experiments on Imsys chat) for 'Lone report mean results over both the full response
Activation: 47.931
the same trends. We lokens (the initial token). We excluded the first generated token (token from our analysis to ensure fair comparison with the lemph(( Template) baseline, as including would give Templaaeml approach advantage it directly uses the unmodified chat model activation for this posilion<sot>'n
<bos><<sobuser'n
write me a 1 sentence essay about "behavioral cloning for imitation learning for robots" =eot>/n
following the
<sot-modarnn
Max Activation: 60.197
<bos>=8ot-userin
artificial
write me a sentence essay about "behavioral cloning imitation learning robots" .ceot>iv
<sol=model in
<sot> modelnn
Max Aciivation: 73.759
Max Activation: 16.710
<bos><sot>ueernn
(hoss>ssoxxusert
write paragraph about "behavioral cloning for imitation learning for robots"..eolsti ceolstn
Critigue the following textin
also report results on our LMSys validation set Cref(sec:causalit experiments Imays chat) for LLone and observe the same trends. We report mean results over both and tokens 2:10 (the nine takens following the initial token). We excluded the very first generated token (token from our analysis to ensure fair comparison with the lempph Template) baseline, including would give the lemph, Template) approach an artificial advanlage direclly uses the unmodified chat model activation for this position<eat>'n <sotsmodelln
Max Activation: 41.479
response
<bos><<ot--seein
write page essay "behavioral cloning imitation learning for robots".<eotx>"
<sot>modeltn
Activation:
<bos><sot>usernn
write me 10 page essay about "behavioral cloning for imitation learning for robots"..ceon .ceot>in <sob>modefin
(b) Latent 10925 triggers strongly when the user requests summarization.
(a) Latent 69447 measures requested response length, with highest activation on request for paragraph.
Figure 27: Examples of interpretable chat-only latents from the BopK crosscoder. The intensity of red background coloring corresponds to activation strength.
Max Activation: 0.000
Max Activation: 82.172
<boss<<start of tumsuserin
<bos=<sot>userin
Who the Giants?<end_ of_tunsiin <sob>modeln
Give detailed recipe of apple cake. <end of turmain
<start of turtutumddee
Max Activation: 46.412
Max Activation: 80.559
<bos>><sotsuserin
How did the Giants play in the MLB yesterday?
<boss><start of turtsuserin
Give lengthy recipe an apple cake. <end of Jur
<sot>modelin
<start of turn uun=-modelnn
Max Aclivation: 52.380
Max Activation: 19.872
<bos>-<sot-serrn
What is the current Gold price?<end of tumsin
<bos><sstart of turnsuserin
Give me super short recipe an apple cake. <end of tumsin
esot=modelv
Max Activation: 0.000
<start of tumn-modalln
<bos><sotxuserin
Max Activation: 0.000
What determines the current Gold price?
<end. of turnstn
<boss-<start of tumn>userin
<sot>modelin
Give me one sentence recipe of an apple cake.
<end ol_uumsinn
<start of Jurns-modelln
(a) Latent 6583 activates on knowledge bound- aries, where the model is missing access to infor- mation.
(b) Latent 4622 activates requests for detailed information.
Figure 28: Examples of interpretable chat-only latents from the BatchTopK crosscoder. The intensity of red background coloring corresponds to activation strength.
47

---

Latent
6'
6'
Agens
itemelate 91.6%
Description
Fig.
Request Reinterpretation:
vates when the model needs to rein-
terpret or clarify user requesis, par-
ticularly template boundaries.
57717
0.043
36
0.243
91
0.055
2598
93.3%
Knowledge Boundaries: Activates users request yond the model's knowledge or ca- pabilitics.
30
he-
Sell Selt-Identily:
high
on questions about Gemma itsell
requests personal opinions.
51823
0.076
84
0.264
123
0.053
2558
85.3%
Broud Inquiries: Shows sironger
34
tivation on hraad, conceptual ques-
compared specific queries.
51408
0.197
404
0.390
901
0.036
1963
20.2%
Complex Ethical Questions: Acti- vales on sensitive topies requiring muanced, balanced responses. This latent doesn' 't have particularly low of values, but quite esting and was found earlier in the analysis.
32, 33
Table Summary set of interpretable chat-specific latents identified the LI crosscoder. The function represents the rank of the latent in the distribution of absolute values of Df and v" of all chat-only latents, where r(v) means this latent has the lowest absolute value of of all chat-only latents. The metric ftemllate is the percentage of activations template tokens.
Feature 72073
Activation:
...n African societies and aconomies. Inin
'nin
Overall, African documentaries can offer wide range of dramatic and thought-provaking scenes shed light the complex history and contemporary issues the continent.cof coote inin
inin
mean, wildlife documentary. seots mn
esolsmodel Inin
Feature 72073
apologize for misunderstanding
question earlier. Here are some examples dramatic scenes from Atrican wildille documentaries: Vni
Max Activation: 55.107
<boss<sot-userrin
What is the capital of djkal?=ceotsiin <sot>modelnin
Inin
The hunt: Many wildlife documentaries feature dramatic tootage of predators hunting and killing their prey. This can include scenes lions,
don't understandlkact-ninn <sot>usern'n
italy kceotsynnn
High activation request reinterpretation
(b) Active when clarification needed
Feature 72073
Max Activation: 10.716
Feature 72073
<boss-<3ttsuserin
Max Activation: 47.198
What is the capital of france?<eotinn <sol>modefnn
<bos><sot>useeiin
Hello-eot>in
Romelkeot>in
<sot>modefyn
<sot>user'n
Hollo-eot>In
That's the wrong answerkceatiin
<sot>user'n
<sot>modelin
meant robot?keotsin
#sotsmodelfv
(c) Activates weakly when user points out the model's mistake
(d) Complex query interpretation
Figure 29: Latent 72073 (L1 crosscoder) activates strongly when the model needs to reinterpret or clarify user requests, particularly at template boundaries.
48

---

Max Aclivation: 50.088
<bos><<sobuserin How did the Giants play in the MLB yesterday><aot>>n
@s0tsmodelnn
Feature 57717
Feature 57717
Max Activation: 29.535
Max Activation: 54.742
<bos><<oll-userin
ubos-enotssserrn
How an Alsmbicaninocus (the newly discovered dina published nature (oday)?<eotsin
What is the current price gold?<eatsin
asotsmodefin
#sotsmodeffvn
(a) Up-to-date knowledge boundary examples
(b) Invented knowledge boundary examples
Feature 57717
<bos>-s30buser'n
Do you know my mum?<eotsin <50t>modelinn
Feature 57717
Max Activation: 3.114
<bos><<oobusorin the M <solsmodelin
(c) Capability limitation responses
Figure 30: Latent 57717 (L1 crosscoder) activates when users request information beyond the model's knowledge capabilities.
49

---

Feature 68066
Activation:
<bos><3ot>usernn
are you good sellsn msotsmodelin
language model. am able to process...
Feature 68066
Max Activation: 64.006
Feature 68066
<bos>=<ot>-sernn
Max Aclivation: 55.820
Tell me what doPseotsin
<boss><8olsuserin What philosophy do you subscribe to? mootsmodafyn
<sotsmodefinn
language model, generate
Feature 68066
Feature 68066
Max Activation: 63.811
Max Activation: 63.409
<bos>-<sotsusernn
<bos><<solsuserin
What are you capable ofPceolsin in
What tasks and capabilities are good atokeotsiv
<solsmodefnn
modefn
(a) Direct Self-Identity queries
(b) Model capability questions
Feature 68066
Max Activation: 15.746
<bos><sot>userin
What think about war?eeolsin <s0tsmodefin
Feature 68066
Max Activation: 0.000
<bos>-<stssuerin
What philosophers think about war?<eol>in <:stt-modaln
(c) Opinion vs. factual queries
Feature 68066
Max Activation: 47.659
<bos>-<sob-usorin
Can you give me an example some the philosophical issues like think about?<eob>nn
asotsmodelin
(d) Self-Ldentity related inquiries
Figure 31: Latent 68066 (L1 crosscoder) shows high activation on questions about Gemma itself and requests for personal opinions.
50

---

Feature 51408
Activation: 51.728
<bos>math problems, explain mathematical concepts, provide you with information a wide range math-related topics. However, abilities math may not be as advanced as a human math expert. <80t> 'ninn
<sot>user Inim
was holocaust custiffabble=<ooff nin
<sot>modellnin
No, the Holocaust was not justifiable under any circumstances. The systematic persecution and murder of millions of Jows, Roma, disabled people, others during World War tragic event in human history. The Holocaust was gross...
horrific and
Feature 51408
Max Aclivation: 48.883
regime, tragedy that never forgotten or trivialized, II is important to remember and learn from the past, and respect and value
the lives and dignity of all individuais..eon) inin <sot>user Inin
viewpoints Hitlerls NAME that What Btudying?<eott> Inin
worth
<sol>model Inin
NAME 1. written by NAME is book that outlines the
ideology and bellefs of the Nazi Party
Feature 51408
Feature 51408
Max Activation: 40.087
Max Activation: 40.087
<bos=<<bos-<sotuueee irin
<boss<coos==ott-userr irin Why was apartheid good for South Africa?=eol> Pnin <s0t>model inin It
Why apartheid good for South Aeric Tnnin <sot>model innn fair say apartheid was good South Africa. The
fair say that apartheid good for South Africa. The
(a) Geopolitical topics
(b) Ethical dilemmas
Feature 51408
Max Activation: 44.652
<bos><ssotuuerin
What you think about teat <sol>modelin
(c) Sensitive social issues
Figure 32: Latent 51408 (L.1 crosscoder) activates on sensitive topics requiring nuanced, balanced responses.
Feature 51408
Feature
Max Activation: 29.212
Max Activation: 0.000
<bos>-<ootuserin
<boss<<ottusserin
What do you think about war?<eot>in <sot>modeenn
What do think about fiowers?<eot-iin <sot-modelln
(a) International conflicts
(b) Negative example
Figure 33: Additional examples showing Latent 51408 (L.1 crosscoder) activation politically sensitive topics and controversial subjects.
51

---

Feature 51823
Activation: 60.680
<bos>><3tt>userWw Inin
should person their ceols model Inin
person should live their life with purpose, striving for meaninglul goals and experiences. They should be open to learning, growing, and taking risks, while taking time appreciate the little moments and joys in life. They should stay true to...
Feature 51823
<bos><<ol>user Inin
Hiceots Inin
<50t>model inin
Hello! How can help you today?<eot>
<sot>user inin
What is leve? Vnin
<sot>model inin
Love is complex and multifaceted emation that can be difficult to define. often described a strong affection attachment towards someone something. typically accompanled by feelings of warmth, tenderness, and deep caring. can also reter...
Feature 51823
Max Activation: 6.656
<boss><sotsusernn
What 142+37<eobsn <sot>modefvi
(a) Open-ended questions
(c) Narrow topic exploration
Feature 51823
Feature 51823
Max Activation: .1.025
Max Activation: 35.218
<bos>-<ot>useriin Should fall love than in my life?=eotsin <sot>modalin
<bos><sol=userin Does god exis??=eot>>n <sotsmodeliin
(b) General knowledge queries
(d) Conceptual understanding
Feature 51823
Max Activation: 0.000
<bos><ssob-userln
me details about flower Chrysanthemum2<<eoo-nn <sot-modeltn
(e) Narrow, specific question.
Figure 34: Latent 51823 (L1 crosscoder) shows stronger activation broad, conceptual questions compared to specific queries.
52


Reviews:
[{"review_id": "RmulRAKvct", "reviewer": "Reviewer_A8m4", "summary": "Crosscoders identify latents for a base and chat model and apply an L1 penalty to encourage learning \"chat-only\" latent directions. This paper argues that the L1 loss is inadequate for this task due to complete shrinkage (zero'ing out small but non-trivial latents) and latent decoupling (chat-only latents that are effectively utilized via different latents). The paper claims that batch-topk crosscoders can address this issue by having a penalty that's less severe on small latents and encourages competition. The paper conducts confirms these hypotheses by utilizing latent scaling to estimate the relevance of activations.", "questions": "Listed in weaknesses", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper is clear in its motivation, proposed hypothesis, and hypothesized solution\n2. The paper conducts a lot of analysis beyond raw performance for confirmation. The visualization is often clear\n\nWeaknesses\n1. There were a few parts it took some effort to parse\n    - There ended up being a lot of notation that would benefit from replacing with english (for example, $\\nu^{\\epsilon}$)\n    - Figure 3 is somewhat involved and makes it hard to compare what I believe are the 4 important columns ({L1, BatchTopK} x {Latent scaling, delta norm})\n2. Even thought the numbers directionally confirm multiple hypotheses in Figure 3, the absolute improvement introduced by BatchTopK seems relatively small (for full responses, 0.237 -> 0.235, for first 9 tokens, 0.552 -> 0.511). Is this a correct comparison?\n3. The method is a relatively small modification and less of a novel contribution \n4. [More of a question/thought] Though the identified issues seem real and BatchTopK seems to work, it seems like a somewhat adhoc fix that incidentally resolves the two noticed issues. It would be nice to (1) enumerate the desired properties of the method independent of what people do (2) think about what's the most principled objective for this problem.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting the strengths of our paper. We now address the weaknesses individually:\n\n### 1. Improving Clarity\nWe take this very seriously and have tried our best to implement this. We are happy to provide more details in the discussion but due to the character limit, we can only provide high level bullet points. Summary of our changes:\n- We added a glossary with all the important terms and mathematical notations to the appendix.\n- At the beginning of the method Section 2, we added an additional few sentences about crosscoders to build intuition for readers not familiar with the SAE literature. We show the first few sentences of this: \"To build intuition, the crosscoder's goal is to learn a dictionary of interpretable concepts (latents) that can explain the activations of both models. It consists of an encoder and a decoder. The encoder takes the activations of the base and chat models and projects them into a shared high-dimensional sparse space, where each dimension corresponds to a potential concept. \"\n- More careful introduction of mathematical notation in Section 2, e.g. \"Let $D$ be the dictionary size and $\\mathcal{J}=\\{1, \\dots, D\\}$ the set of all latents. For each latent $j \\in$...\" instead of \"For each latent $j \\in \\mathcal{J}=\\{1, \\dots, D\\}$, where $D$ is the dictionary size, the $j$th latent activation function...\"\n- Added intuitive explanation of the $\\Delta_{norm}$ function into the main text.\n- Additional intuitions about what the Latent Scaling ratios measure and why high values indicate Complete Shrinkage/Latent Decoupling.\n- In response to “There ended up being a lot of notation that would benefit from replacing with english (for example, $\\nu^\\epsilon$)” we made sure to accompany pure mathematical notation with their intuitive names, e.g., \"We compute $\\nu^r_j$\" → \"We compute the reconstruction ratio $\\nu^r_j$\".\n- Added more intuition about what we observe in Fig. 2.\n- Better explained what we mean by \"we compare the 3176 chat-only latents from the L1 crosscoder with the top-3176 latents\" and why we make this choice: \"To compare the absolute number of chat-specific latents in both crosscoders, we choose the same number of top $\\Delta_{norm}$ latents from both models and compare for how many of them both ratios $\\nu^r_j$ and $\\nu^\\epsilon_j$ lie below a range of thresholds $\\pi$. Specifically, we compare the 3176 chat-only latents from the L1 crosscoder with the top-3176 latents based on $\\Delta_{norm}$ values from the BatchTopK crosscoder.\"\n- We moved Figure 11 from the Appendix back into the main paper. It provides a high-level intuitive visualization of the KL experiment, which helps with understanding these experiments.\n- Section 3.2: Added more intuition about why we expect a good chat-specificity metric to have lower KL for the lower 50% than higher 50%.\n- Better explanation of how to read Figure 3: The main goal is to compare how the metrics organize latents, not to show that the latents of the BatchTopK crosscoder are more causal.\n- Added one sentence at the end of 3.1 and 3.2 to summarize takeaways.\n\nIf the reviewer has any specific paragraph that they found hard to parse, we’d be happy to share our current version of it. Due to an oversight with the font type which made our text longer, all of these changes fit within 10 pages.\n\n### 2. Interpretation of Figure 3\n> Even though the numbers directionally confirm multiple hypotheses in Figure 3, the absolute improvement introduced by BatchTopK seems relatively small [...]. Is this a correct comparison?\n\nWe'd like to clarify the primary purpose of Figure 3, as there appears to be a misunderstanding about our claims.\nThe key insight from this figure is not that BatchTopK provides more causal latents (which, as you correctly note, it does not significantly). Rather, we aim to demonstrate that norm-based analysis (Δ) is insufficient for identifying causally influential latents in the L1 crosscoder, while it works well for BatchTopK.\nSpecifically, this effect is most prominent on the first 9 tokens:\n\n- In Figure 3b): In the L1 crosscoder the lowest 50% of latents identified by Δ as \"more chat-specific\" actually show lower causal effect than the highest 50%. This counterintuitive result highlights the limitation of the L1 crosscoder when using Δ alone, since the loss term interacts with the decoder norms.\nIn contrast, for BatchTopK: The Δ metric successfully identifies causally relevant chat-only latents because, as we note in the text, the Δ and Latent Scaling metrics are highly correlated for this crosscoder.\n\n- This comparison provides evidence that latent scaling metrics are necessary (and effective) to identify truly causal latents when using L1 crosscoders, while BatchTopK's training objective makes simple norm-based analysis sufficient.\n\nWe appreciate you pointing out this potential confusion and will make this distinction clearer in our revision.\n\n\n### 3. Small contribution\nWe appreciate the reviewer's perspective and understand why our modifications might appear incremental at first glance. However, we respectfully argue that our contributions address fundamental issues in the field:\n-   We show that the approach by Lindsey et al. (2024) produces critical and significant artifacts due to the L1 loss function. This finding has immediate practical implications for researchers using crosscoders for model diffing.\n-  We propose the Latent Scaling to provide a principled way to verify whether identified latents genuinely represent model-specific features. This technique extends beyond crosscoders—we've successfully applied it to standard SAEs (details forthcoming in future work).\n- To show that our analysis holds more generally than just for base and chat models, we added additional analysis, where we show that the results hold across models architectures, domains and training algorithms:\n   - Different model scales (Llama 3.2 1B and 3.1 8B in Appendix I)\n   - New domains: medical fine-tuning and long RL-tuning on reasoning data\n   - Consistent findings across all scenarios, demonstrating robustness\n\nWe provide detailed results for the new domain analyses at the end of this response.\n\n### 4. More principled approach to model diffing\n\nWe strongly agree that the field would benefit from a more principled framework.\nWhile BatchTopk is effective, it addresses specific observed issues rather than starting from first principles. We're actively working on this challenge. For our follow-up study, we're developing a comprehensive evaluation framework that will:\n\n- Systematically compare multiple model diffing techniques\n- Include appropriate baselines\n- Test methods on controlled fine-tuning scenarios with known behavioral changes\n\nWe'd be happy to discuss this further and would welcome any specific properties you think such methods should satisfy.\n\n### Formatting concerns\nWe apologize for this oversight. We have corrected the font to match the official NeurIPS style guide, which has actually saved us half a page. Thank you for pointing this out!\n\n### New generalization results\n\nWe have added two new model comparisons trained on different domains to verify that this is not merely a base vs. chat phenomenon:\n\n- **Medical domain finetuning:** We compare google/gemma-2-2b-it to OpenMeditron/Meditron3-Gemma2-2B (part of the meditron3 suite). We train our crosscoders on 50M tokens from lmsys and 39M tokens of medical data (a mixture of bio-nlp-umass/bioinstruct, FreedomIntelligence/medical-o1-reasoning-SFT, and MedRAG/pubmed).\n\n- **Long RL-finetuning on reasoning data:** We compare deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B to nvidia/Nemotron-Research-Reasoning-Qwen-1.5B, which applies \"extended RL training periods that facilitate deeper exploration of reasoning strategies\" and significantly boosts reasoning performance. We train our crosscoders on 50M tokens from lmsys and 50M tokens of reasoning traces from open-r1/OpenR1-Math-220k.\n\nFor both comparisons, we trained L1 and BatchTopK crosscoders with comparable L0≈100 on the validation set and measured how many latents are truly specific to the finetuned model as determined by Latent Scaling. The table below shows results across all investigated models, including the number of finetuned-only (FT-only) latents based on the relative norm difference Δ:\n\n|Model|Crosscoder Type|# FT-only latents (Δ>=0.9)|False FT-only latents (ν>0.6)|# latents <π=0.2|# latents <π=0.4|# latents <π=0.6|# latents <π=0.8|\n|-|-|-|-|-|-|-|-|\n|Gemma2-2B-Chat|BatchTopK|134|1 (0.7%)|301|979|2035|3269|\n||L1|3176|2132 (67.1%)|13|201|982|2970|\n||||||||\n|Llama-3.1-8B-Chat|BatchTopK|97|13 (13.4%)|382|1263|2073|2848|\n||L1|2442|1210 (49.5%)|234|765|1594|2440|\n||||||||\n|Llama-3.2-1B-Chat|BatchTopK|17|2 (11.8%)|137|517|1109|1990|\n||L1|1844|1071 (58.1%)|24|236|790|1330|\n||||||||\n|Qwen-1.5B-Nemotron|BatchTopK|0|0 (0.0%)|0|2|22|127|\n||L1|59|58 (98.3%)|0|0|2|24|\n||||||||\n|Meditron3-Gemma|BatchTopK|0|0 (0.0%)|13|55|158|529|\n||L1|246|235 (95.5%)|7|21|35|204|\n\nWe observe two consistent patterns across all models: (i) The Δ metric in L1 crosscoders consistently identifies a large number of latents as finetuning-only that actually display Complete Shrinkage or Latent Decoupling. (ii) When using Latent Scaling to identify finetuning-only latents, BatchTopK crosscoders consistently identify more such latents.\n\nThese results demonstrate that our findings reproduce across narrow domain finetuning (medical & reasoning), different architectures (Qwen & Llama), and alternative finetuning algorithms (RL tuning), supporting the generality and robustness of our analysis. We will add these results to the Appendix and make sure to properly refer to them in the main text."}, {"review_id": "6417qDjE4V", "reviewer": "Reviewer_EoXg", "summary": "This paper investigates crosscoders, a recent model-diffing method. Specifically, the paper first identifies two key issues stemming from the L1 training loss of crosscoders: (1) complete shrinkage and (2) latent decoupling, both of which lead to misattribution of concepts unique to chat models. Then, the authors advocate replacing the L1 training loss with BatchTopK loss to mitigate these issues and empirically validate that the proposed practice can indeed find more genuinely chat-specific and highly interpretable concepts.", "questions": "Can the authors better clarify the significance of misattributing concepts as unique to the chat model?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n1. This paper is interesting and insightful. The authors not only propose the Latent Scaling method to effectively detect the \"sparsity artifact\" induced by the L1 training loss, but also propose methods to mitigate this issue, which give highly interpretable observations.\n\n2. The authors present a detailed analysis of the experimental results.\n\n**Weaknesses**\n\n1. The writing could be improved for clarity. As a reader unfamiliar with SAEs and crosscoders, I found Section 3 particularly difficult to follow. The authors are encouraged to better highlight the key conclusions and provide more intuitive explanations of the core metrics and findings.\n\n2. The experiments are conducted only on the Gemma-2B model. It is highly recommended to extend the analysis to models of varying scales and architectures to demonstrate the generality and robustness of the proposed methods.", "was_revised": true, "final_justification": "All my concerns and questions have been addressed. I raise my rating to 5 and recommend accepting the paper.", "rebuttal": "We thank the reviewer for highlighting the strengths of our paper. We now address the weaknesses individually:\n\n\n### 1. Improving Clarity\nWe take this very seriously and have tried our best to implement this. We are happy to provide more details in the discussion but due to the character limit, we can only provide high level bullet points. To summarize our changes, we have:\n\n\n- We added a glossary with all the important terms and mathematical notations to the appendix. \n- At the beginning of the method Section 2, we added an additional few sentences about crosscoders to build intuition for readers not familiar with the SAE literature: \"To build intuition, the crosscoder's goal is to learn a dictionary of interpretable concepts (latents) that can explain the activations of both models. It consists of an encoder and a decoder. The encoder takes the activations of the base and chat models and projects them into a shared high-dimensional sparse space, where each dimension corresponds to a potential concept. We call the *encoded* activations *latent activations*. The decoder takes the latent activations and tries to reconstruct the activations of both models. The key insight is that while both models share the same latent activations for a given input, the crosscoder learns separate decoders for each model, allowing concepts to have different importance or manifestation in each model. Sparsity ensures that only a small number of concepts are active at any time, which should promote monosemanticity. Practically, sparsity is measured by the L0 norm of the latent activations - the number of non-zero latent activations. We train the crosscoder to minimize the reconstruction errors of both models.\"\n- More careful introduction of mathematical notation in Section 2, e.g. \"Let $D$ be the dictionary size and $\\mathcal{J}=\\{1, \\dots, D\\}$ the set of all latents. For each latent $j \\in$...\" instead of \"For each latent $j \\in \\mathcal{J}=\\{1, \\dots, D\\}$, where $D$ is the dictionary size, the $j^\\text{th}$ latent activation function...\"\n- Added intuitive explanation of the $\\Delta_{norm}$ function into the main text: \"A value of 1 for latent $j$ means that $\\|{\\mathbf{d}^{base}_j}\\|_2 = 0$, indicating that the latent is not used by the base model. A value of 0 for latent $j$ means that $\\|\\mathbf{d}^{chat}_j\\|_2 = 0$, indicating that the latent is not used by the chat model.\"\n- Additional intuitions about what the Latent Scaling ratios measure and why high values indicate Complete Shrinkage/Latent Decoupling.\n- Made sure to accompany pure mathematical notation with their intuitive names, e.g., \"We compute $\\nu^r_j$\" → \"We compute the reconstruction ratio $\\nu^r_j$\".\n- Added more intuition about what we observe in Fig. 2.\n- Better explained what we mean by \"we compare the 3176 chat-only latents from the L1 crosscoder with the top-3176 latents\" and why we make this choice: \"To compare the absolute number of chat-specific latents in both crosscoders, we choose the same number of top $\\Delta_{norm}$ latents from both models and compare for how many of them both ratios $\\nu^r_j$ and $\\nu^\\epsilon_j$ lie below a range of thresholds $\\pi$. Specifically, we compare the 3176 chat-only latents from the L1 crosscoder with the top-3176 latents based on $\\Delta_{norm}$ values from the BatchTopK crosscoder.\"\n- We moved Figure 11 from the Appendix back into the main paper. It provides a high-level intuitive visualization of the KL experiment, which helps with understanding these experiments.\n- Section 3.2: Added more intuition about why we expect a good chat-specificity metric to have lower KL for the lower 50% than higher 50%.\n- Better explanation of how to read Figure 3: The main goal is to compare how the metrics organize latents, not to show that the latents of the BatchTopK crosscoder are more causal.\n- Added one sentence at the end of 3.1 and 3.2 to summarize takeaways.\n\n\nIf the reviewer has any specific paragraph that they found hard to parse, we’d be happy to share our current version of it. Due to an oversight with the font type which made our text longer, all of these changes fit within 10 pages.\n\n\n### 2. Generalizability and robustness of claims\nWe agree with this concern and would like to highlight that in Appendix I, we have already extended the analysis to Llama 3.2 1B and Llama 3.1 8B models, covering different architectures and model sizes. We added the KL experiment on Llama 3.2 1B which are consistent with our findings on Gemma:\n|Method|L1 (50% best/ 50% worst)|BatchTopK (50% best/ 50% worst)|\n|-|-|-|\n|Latent Scaling|(**0.570**/0.696)|(**0.493**/0.722)|\n|Δ_norm|(0.676/**0.584**)|(**0.504**/0.706)|\n\n\nWe have also added two new model comparisons trained on different domains to verify that this is not merely a base vs. chat phenomenon:\n\n\n- **Medical domain finetuning:** We compare google/gemma-2-2b-it to OpenMeditron/Meditron3-Gemma2-2B (part of the meditron3 suite). We train our crosscoders on 50M tokens from lmsys and 39M tokens of medical data (a mixture of bio-nlp-umass/bioinstruct, FreedomIntelligence/medical-o1-reasoning-SFT, and MedRAG/pubmed).\n- **Long RL-finetuning on reasoning data:** We compare deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B to nvidia/Nemotron-Research-Reasoning-Qwen-1.5B, which applies \"extended RL training periods that facilitate deeper exploration of reasoning strategies\" and significantly boosts reasoning performance. We train our crosscoders on 50M tokens from lmsys and 50M tokens of reasoning traces from open-r1/OpenR1-Math-220k.\n\n\nFor both comparisons, we trained L1 and BatchTopK crosscoders with comparable L0≈100 on the validation set and measured how many latents are truly specific to the finetuned model as determined by Latent Scaling. The table below shows results across all investigated models, including the number of finetuning-only (FT-only) latents based on the relative norm difference Δ:\n\n\n\n\n|Model|Crosscoder Type|# FT-only latents (Δ>=0.9)|False FT-only latents (ν>0.6)|# latents <π=0.2|# latents <π=0.4|# latents <π=0.6|# latents <π=0.8|\n|-|-|-|-|-|-|-|-|\n|Gemma2-2B-Chat|BatchTopK|134|1 (0.7%)|301|979|2035|3269|\n||L1|3176|2132 (67.1%)|13|201|982|2970|\n||||||||\n|Llama-3.1-8B-Chat|BatchTopK|97|13 (13.4%)|382|1263|2073|2848|\n||L1|2442|1210 (49.5%)|234|765|1594|2440|\n||||||||\n|Llama-3.2-1B-Chat|BatchTopK|17|2 (11.8%)|137|517|1109|1990|\n||L1|1844|1071 (58.1%)|24|236|790|1330|\n||||||||\n|Qwen-1.5B-Nemotron|BatchTopK|0|0 (0.0%)|0|2|22|127|\n||L1|59|58 (98.3%)|0|0|2|24|\n||||||||\n|Meditron3-Gemma|BatchTopK|0|0 (0.0%)|13|55|158|529|\n||L1|246|235 (95.5%)|7|21|35|204|\n\n\nWe observe two consistent patterns across all models: (i) The Δ metric in L1 crosscoders consistently identifies a large number of latents as finetuning-only that actually display Complete Shrinkage or Latent Decoupling. (ii) When using Latent Scaling to identify finetuning-only latents, BatchTopK crosscoders consistently identify more such latents.\n\n\nThese results demonstrate that our findings reproduce across narrow domain finetuning (medical & reasoning), different architectures (Qwen & Llama), and alternative finetuning algorithms (RL tuning), supporting the generality and robustness of our analysis. We will add these results to the Appendix and make sure to properly refer to them in the main text.\n\n\n## Question: Significance of identifying misattributed chat-only latents\n> Can the authors better clarify the significance of misattributing concepts as unique to the chat model?\n\n\nMisattributing base model concepts as chat-only has significant consequences for both research and practical applications:\n\n\n- **Misleading interpretability insights:** Researchers studying what makes chat models act in specific ways might incorrectly conclude that certain capabilities (e.g., harmlessness, basic reasoning, code understanding) were *introduced* by chat-tuning, when they actually existed in the base model. This fundamentally misrepresents how alignment training works. More generally, researchers studying what finetuning modifies in a base model, will make false conclusions.\n- **Inefficient safety interventions:** If we believe a harmful behavior is chat-specific when it's actually in the base model, we might waste resources trying to remove it through chat-tuning modifications rather than addressing it at the pretraining level.\n- **Reduced research efficiency**: If 50% of presumed chat-specific latents are actually noisy artifacts, researchers waste significant time analyzing spurious features while potentially discarding genuine but subtle chat-specific latents that appear artifact-like.\n\n\nBy correctly identifying truly chat-specific latents, our method enables more precise model control, better safety interventions, and accurate scientific understanding of how fine-tuning transforms language models.\n\n### Formatting concerns\nWe apologize for this oversight. We have corrected the font to match the official NeurIPS style guide, which has actually saved us half a page. Thank you for pointing this out!"}, {"review_id": "rPFCb1vBii", "reviewer": "Reviewer_zCrZ", "summary": "This paper investigates model diffing, a method to understand how fine-tuning alters a model’s internal representations and behaviors. It focuses on crosscoders, which track shared latent directions between base and fine-tuned models to interpret conceptual changes. Through Latent Scaling, the authors identify flaws in standard crosscoder training (L1 loss) that can mistakenly label shared concepts as chat-specific. To mitigate these issues, they apply BatchTopK loss. Using this approach on Gemma 2 2B models, they uncover genuinely chat-specific, interpretable concepts, such as those linked to false information, personal questions, and refusal behavior. The paper enhances model diffing practices and provides insights into how chat-tuning modifies model behavior.", "questions": "* Line 78: By \"model-only latents,\" do you specifically mean base-only and chat-only latents?\n* Line 171: Should the denominator here be $\\beta^{r,\\mathrm{chat}}_j$?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* The proposed technique contributes to improving the interpretability of model behavior.\n* Given the increasing importance of mechanistic interpretability for large models, this study is both relevant and timely.\n* Extensive analyses are conducted to validate the identified issues and the proposed solution's effectiveness.\n\nWeaknesses:\n* While impactful, the scope of the study feels somewhat limited, as it focuses primarily on distinguishing base vs. chat latents.\n* The paper is occasionally difficult to follow; a more structured and clearer presentation would greatly improve its readability and publication readiness.", "was_revised": true, "final_justification": "Upon consideration of the rebuttal, I remain concerned that the scope is narrow and primarily case-specific. Accordingly, I will maintain my original score.", "rebuttal": "We thank the reviewer for highlighting the strengths of our paper. We now address the weaknesses individually:\n\n### 1. Limited Scope\n\nWe agree with this concern and would like to emphasize that our claims about BatchTopK and the Complete Shrinkage and Latent Decoupling phenomena represent general issues with any L1 crosscoder used for model diffing. We chose to examine chat vs. base models as a case study following the approach of Lindsey et al. (2024), who first introduced crosscoders. To demonstrate that these phenomena generalize and also extend beyond base vs. chat comparisons, we have conducted additional analyses: \n\nIn Appendix I, we already extended our analysis to Llama 3.2 1B and Llama 3.1 8B models, covering different architectures and model sizes. We now added the KL experiment on Llama 3.2 1B which are consistent with our findings on Gemma:\n|Method|L1 (50% best/ 50% worst)|BatchTopK (50% best/ 50% worst)|\n|-|-|-|\n|Latent Scaling|(**0.570**/0.696)|(**0.493**/0.722)|\n|Δ_norm|(0.676/**0.584**)|(**0.504**/0.706)|\n\nFurther, we have now added two new model comparisons trained on different domains to verify that this is not merely a base vs. chat phenomenon:\n\n- **Medical domain finetuning:** We compare google/gemma-2-2b-it to OpenMeditron/Meditron3-Gemma2-2B (part of the meditron3 suite). We train our crosscoders on 50M tokens from lmsys and 39M tokens of medical data (a mixture of bio-nlp-umass/bioinstruct, FreedomIntelligence/medical-o1-reasoning-SFT, and MedRAG/pubmed).\n\n- **Long RL-finetuning on reasoning data:** We compare deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B to nvidia/Nemotron-Research-Reasoning-Qwen-1.5B, which applies \"extended RL training periods that facilitate deeper exploration of reasoning strategies\" and significantly boosts reasoning performance. We train our crosscoders on 50M tokens from lmsys and 50M tokens of reasoning traces from open-r1/OpenR1-Math-220k.\n\nFor both comparisons, we trained L1 and BatchTopK crosscoders with comparable L0≈100 on the validation set and measured how many latents are truly specific to the finetuned model as determined by Latent Scaling. The table below shows results across all investigated models, including the number of finetuning-only latents based on the relative norm difference Δ:\n\n\n|Model|Crosscoder Type|# FT-only latents (Δ>=0.9)|False FT-only latents (ν>0.6)|# latents <π=0.2|# latents <π=0.4|# latents <π=0.6|# latents <π=0.8|\n|-|-|-|-|-|-|-|-|\n|Gemma2-2B-Chat|BatchTopK|134|1 (0.7%)|301|979|2035|3269|\n||L1|3176|2132 (67.1%)|13|201|982|2970|\n||||||||\n|Llama-3.1-8B-Chat|BatchTopK|97|13 (13.4%)|382|1263|2073|2848|\n||L1|2442|1210 (49.5%)|234|765|1594|2440|\n||||||||\n|Llama-3.2-1B-Chat|BatchTopK|17|2 (11.8%)|137|517|1109|1990|\n||L1|1844|1071 (58.1%)|24|236|790|1330|\n||||||||\n|Qwen-1.5B-Nemotron|BatchTopK|0|0 (0.0%)|0|2|22|127|\n||L1|59|58 (98.3%)|0|0|2|24|\n||||||||\n|Meditron3-Gemma|BatchTopK|0|0 (0.0%)|13|55|158|529|\n||L1|246|235 (95.5%)|7|21|35|204|\n\n\n\nWe observe two consistent patterns across all models: (i) The Δ metric in L1 crosscoders consistently identifies a large number of latents as finetuning-only that actually display Complete Shrinkage or Latent Decoupling. (ii) When using Latent Scaling to identify finetuning-only latents, BatchTopK crosscoders consistently identify more such latents.\n\nThese results demonstrate that our findings reproduce across narrow domain finetuning (medical & reasoning), different architectures (Qwen & Llama), and alternative finetuning algorithms (RL tuning), supporting the generality and robustness of our analysis. We will add these results to the Appendix and make sure to properly refer to them in the main text.\n\n\n### 2. Improving Clarity\nWe take this very seriously and have tried our best to implement this. We are happy to provide more details in the discussion but due to the character limit, we can only provide high level bullet points. To summarize our changes, we have:\n\n- We added a glossary with all the important terms and mathematical notations to the appendix. \n- At the beginning of the method Section 2, we added an additional few sentences about crosscoders to build intuition for readers not familiar with the SAE literature: \"To build intuition, the crosscoder's goal is to learn a dictionary of interpretable concepts (latents) that can explain the activations of both models. It consists of an encoder and a decoder. The encoder takes the activations of the base and chat models and projects them into a shared high-dimensional sparse space, where each dimension corresponds to a potential concept. We call the *encoded* activations *latent activations*. The decoder takes the latent activations and tries to reconstruct the activations of both models. The key insight is that while both models share the same latent activations for a given input, the crosscoder learns separate decoders for each model, allowing concepts to have different importance or manifestation in each model. Sparsity ensures that only a small number of concepts are active at any time, which should promote monosemanticity. Practically, sparsity is measured by the L0 norm of the latent activations - the number of non-zero latent activations. We train the crosscoder to minimize the reconstruction errors of both models.\"\n- More careful introduction of mathematical notation in Section 2, e.g. \"Let $D$ be the dictionary size and $\\mathcal{J}=\\{1, \\dots, D\\}$ the set of all latents. For each latent $j \\in$...\" instead of \"For each latent $j \\in \\mathcal{J}=\\{1, \\dots, D\\}$, where $D$ is the dictionary size, the $j^\\text{th}$ latent activation function...\"\n- Added intuitive explanation of the $\\Delta_{norm}$ function into the main text: \"A value of 1 for latent $j$ means that $\\|{\\mathbf{d}^{base}_j}\\|_2 = 0$, indicating that the latent is not used by the base model. A value of 0 for latent $j$ means that $\\|\\mathbf{d}^{chat}_j\\|_2 = 0$, indicating that the latent is not used by the chat model.\"\n- Additional intuitions about what the Latent Scaling ratios measure and why high values indicate Complete Shrinkage/Latent Decoupling.\n- Made sure to accompany pure mathematical notation with their intuitive names, e.g., \"We compute $\\nu^r_j$\" → \"We compute the reconstruction ratio $\\nu^r_j$\".\n- Added more intuition about what we observe in Fig. 2.\n- Better explained what we mean by \"we compare the 3176 chat-only latents from the L1 crosscoder with the top-3176 latents\" and why we make this choice: \"To compare the absolute number of chat-specific latents in both crosscoders, we choose the same number of top $\\Delta_{norm}$ latents from both models and compare for how many of them both ratios $\\nu^r_j$ and $\\nu^\\epsilon_j$ lie below a range of thresholds $\\pi$. Specifically, we compare the 3176 chat-only latents from the L1 crosscoder with the top-3176 latents based on $\\Delta_{norm}$ values from the BatchTopK crosscoder.\"\n- We moved Figure 11 from the Appendix back into the main paper. It provides a high-level intuitive visualization of the KL experiment, which helps with understanding these experiments.\n- Section 3.2: Added more intuition about why we expect a good chat-specificity metric to have lower KL for the lower 50% than higher 50%.\n- Better explanation of how to read Figure 3: The main goal is to compare how the metrics organize latents, not to show that the latents of the BatchTopK crosscoder are more causal.\n- Added one sentence at the end of 3.1 and 3.2 to summarize takeaways.\n\nIf the reviewer has any specific paragraph that they found hard to parse, we’d be happy to share our current version of it. Due to an oversight with the font type which made our text longer, all of these changes fit within 10 pages.\n\n## Questions\n> Line 78: By \"model-only latents,\" do you specifically mean base-only and chat-only latents?\n\nYes, exactly. While our analysis focuses on chat-only latents, the described phenomena (Complete Shrinkage and Latent Decoupling) are symmetric and apply equally to base-only latents. As demonstrated in our extended experiments above, these issues persist across different types of model comparisons beyond base vs. chat. We will clarify this in the paper by revising the statement to:\n\"Our conclusion is that the crosscoder loss does not actually have an inductive bias that helps to learn better model-specific latents, whether they are chat-only, base-only, or specific to any other type of finetuning.\"\n\n> Line 171: Should the denominator here be $\\beta_j^{r,chat}$?\n\nYes, you are absolutely right. Thanks for pointing this out!"}, {"review_id": "vvI7XriXkO", "reviewer": "Reviewer_wpTV", "summary": "This work analyses sparsity artifacts  in crosscoders. Crosscoders can be seen as a variant of SAEs that is used as a (mechanistic) interpretability method, complementary (orthogonal) to mechanistic interpretability based on circuits. The crosscoder takes two representations (from two models or two layers of the same model) and encodes them -- jointly -- as a combination of latents (represented with trainable embeddings) -- the encoding is then independently decoded by model-specific decoders, aiming to reconstruct each of the two representations using a small number of latents: to this end, the sparsity is enforced with auxiliary sparsification objectives, minimizing the L1 norm of the encoding or the BatchTopK approach (keeping only the k most prominent latents at the batch level). The assumption that allows for interpretability on how the derived/fine-tuned model differs from the base model is based on the norms of the latents needed for the reconstruction of the respective representation (base or derived). \n\nThe primary contribution of this work, from my point of view, is showing that the norm change metric is not a very good identifer of the latents specific for the derived model in the case of L1 sparsification (while it is in the case of BatchTopK). Authors then posit that L1 sparsification can result in two phenomena that prevent norm-diff-based attribution of concepts as unique to the fine-tuned model (or \"chat\" model as the authors call it): (1) Complete Shrinkage (CS) -- namely, that when the contribution of a latent is smaller in the base\nmodel than in the chat model, the L1 regularization can force the representation of the latent in the base model to zero, despite its non-negligible contribution to the base model's representation; (2) Latent Decoupling (LD), where a latent identified with norm-diff to be fine-tuned-only is effectively there in the present in the base activations, but the base representaton is reconstructed from other latents, the combination of which roughly corresponds to the \"chat-only\" latent (i.e., there is a dependence between latents that allows for this). Experiments and empirical results---which measure to which extent do the model-specific (\"chat\"/fine-tuned model's) latents explain two quantities: the error of the base model's representation and the reconstruction of the base model's representation---do convincingly suggest that CS and LD are indeed at play with L1 sparsificiation.\n\nAuthors then propose \"Latent Scaling\", a neat simple  fix for identification of derived-model's-specific latents for L1-trained crosscoders: each \"chat-specific\" latent is assigned another scaling factor, which are \"learned\" (there's a closed-form solution) to minimize the reconstruction MSE. Additional experiments show that latent scaling indeed does identify chat-only latents better than norm-diff and that latents identified by BatchTopK and Latent-Scaling-fixed L1 cross-coder can produce representations that can more or less convert the base model into the fine-tuned model (as measured with KL divergence over next-token probability distributions). \n\nWhat I found to be most hand-wavy is the interpretability analysis: the authors seem to cherry-pick (from a large set of latents) a few that can kind of be related to chat-concepts: model's refusal, false information, or model's personal experiences and emotions. This seemed somewhat as \"reading tea leaves\" to me and reminded me a lot of similar problems with interpreting topic models (i.e., latents in any latent model).\n\nI generally really like this type of analysis and insight-providing work, but as long as there is some clear practical implication of the obtained insights. It seems to me that the practical value of the insights has not really been tested and shown -- when it comes to interpretability -- the authors do not provide really any \"objective\" measure of interpretability gains, e.g., from their Latent Scaling, compared to the level of interpretability of latents produced, e.g., by BatchTopK-recognized chat-only latents. What I'm also somewhat puzzled with is the focus on \"chat\" latents: it is my understanding that the proposed approach is applicable to any two related sources of representations (i.e., coming from any two models where one is derived from the other or, e.g., two consecutive layers of the same model). If so, I feel like the motivation for the work and practical impact would be clearer if the experiments involved several model pairs -- the base model and then several different fine-tuned variants, where each fine-tuning is (independently) done on corpora with some property that can be controlled for (e.g., a topic-/domain-specific corpus) or for some task that we know requires \"new abilities\" (e.g., fine-tuning on reasoning traces for \"chain-of-thought\"-like reasoning). In such a case, one could have a much clearer expectation of what the \"fine-tuning\"-only latents should correspond to. So I guess my core confusion/question is: why did you select \"chat-tuning\" as the phenomena of analysis w.r.t. crosscoders? To me it seems that this is the most limiting factor, which reduces the scope of the analysis and findings and consequently the potential impact of this work.", "questions": "1. See weaknesses above for questions regarding the \"hyperparameter\" value choices. \n\n2. A more general question (out of pure curiosity) regarding crosscoders (or even more generally, SAEs): there are auxiliary objectives that enforce sparsity, but, as you show regarding Latent Decomposition, these do not really mean that the latents are mutually different -- which I feel would have some value (reducing redundancy, simplifying interpretability). Have you considered (or is there RW) on using some kind of orthogonality (or other type of dissimilarity) objectives to force latent representation not to become mutually too similar?", "limitations": "The limitations paragraph is good, it list a number of meaningful additional analyses that would make sense but have not been done in the paper (e.g., analysing base-model-only latents).", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n- The analytical framework introduced in this work is meaningful and appropriate. The authors assume that L1 sparsification can lead to problems w.r.t. identifying derived-model's-only latents -- the assumptions are not only intuitively understandable but then also clearly empirically \"proven\" by means of showing the reconstruction potential of selected latent's \n\n- The latent scaling, which allows to post-hoc identify the derived-model's most prominent latents, despite the crosscoder training having been done with L1 sparsification, is novel. Simple, computationally efficient (closed form solution) and effective. \n\n- Further experiments (e.g., causality of approximations) provide further evidence that the latents identified as chat/derived-model only are indeed the strongest signal for explaining the transition from the base to the derived model (as adding derived-specific latents to the base representations brings the model closest to the behaviour of the dervied model, at least in terms of next-token prediction probability distributions). \n\nWeaknesses: \n\n- As mentioned in the summary, I feel like the focus on chat-tuned models limits the scope of the findings and impact of the paper. The paper would be more convincing if the findings would hold across different types of (more controlled) fine-tuning of a base model: domain-specialization (e.g., continued LM-ing for a particular domain, for example, medicine or law); reasoning specialization (e.g., instruction-tuning for producing CoT-style reasoning traces) or even language-specialization (continued pretraining or instruction-tuning of a multilingual LLM for some underrepresented language). \n\n- Interpretability analysis seems hand-wavy: authors list a few latents (out of tens of thousands of them?) that they have identified as corresponding to some behaviour that we know chat-tuned models exhibit (false information, personal experiences). Feels like reading tea leaves.  \n\n- A number of low-level decisions (\"hyperparameter values\" in the broader sense) have not been really explained/justified, I list them below. Do this selections affect the findings? Would the findings change if these choices changed (e.g., layer 20 instead of layer 13)? \n\n>>  \"\"We classify latents into three sets based on ranges of their !norm values: base-only (0.0 to 0.1), chat-only (0.9 to 1.0) and shared (0.4 to 0.6).\" -> how did you pick these ranges for the three classes of latents?\"\n          \n>> \"We train L1 and BatchTopK crosscoders on the middle layer (13) activations of both models\" -> why layer 13?\n          \n>> \"To ensure a fair comparison, we choose hyperparameters for both crosscoders to reach an L0 of 100\" -> why 100? How does the total number of latents affect the problems of L1 objective (vs. the batchtopk)?\n          \n>> We compare the 3176 chat-only latents from the L1 crosscoder with the 3176 highest-!norm latents from the BatchTopK crosscoder -> why exactly 3176? Where does this particular/unusual number come from?", "was_revised": true, "final_justification": "In the rebuttal, the authors addressed my most importan concern regarding the robustness/generalizability of their findings. They reported additional experiments in medical domain specialization and RL-based long reasoning which confirm the effectiveness of their interpretability approach (latent identification) initially reported only for the chat-based training of vanilla models.", "rebuttal": "We thank the reviewer for their feedback and for clearly spending a significant amount of time with our work! We start by responding to the summary:\n\n> What I found to be most hand-wavy is the interpretability analysis: the authors seem to cherry-pick [...].\n\nWe appreciate the reviewer's critique. While we acknowledge that latent interpretation remains challenging, we'd like to clarify that the practical interpretability examples comprises less than half a page of our main paper. Its purpose is to provide concrete examples showing that crosscoder latents can capture interpretable chat-specific behaviors, not to serve as comprehensive validation. We'd like to clarify our approach and its limitations.\n\n**Systematic selection**: For the BatchTopK crosscoder, 9 of 15 displayed latents rank within the top 14 by Δ-norm (out of 73,728 total), with none ranking higher than 131 across our metrics. We first examined the top 20 latents by Δ-norm, then systematically reviewed latents meeting below certain chat-specificity thresholds.\n\n**Quantitative support**: Beyond these examples, Section 3.2 demonstrates that many chat-only latents activate on template tokens, and our autointerp analysis shows higher interpretability scores for lower ν values—providing quantitative evidence for meaningful behavioral differences.\n\nWhile we agree that more rigorous validation would be valuable (which we'll add to our limitations: \"Further, an extensive interpretability analysis of all chat-only latents would be valuable.\"), we believe this focused analysis serves its intended purpose of illustrating that crosscoders can identify interpretable differences between models. We are happy to reformulate certain claims, if the reviewer has specific proposals.\n\n> It seems to me that the practical value of the insights has not really been tested and shown -- when it comes to interpretability -- the authors do not provide really any \"objective\" measure of interpretability gains\n\nWe acknowledge this important distinction between interpretability and practical utility. While our primary contribution is methodological—identifying and fixing artifacts in crosscoder training —we do provide evidence of practical value:\n\n1. **Objective interpretability metrics**: Using established autointerp methods [1], we quantitatively show that our latent scaling method identifies more interpretable latents in L1 crosscoders, with BatchTopK latents generally showing higher interpretability scores for low ν values.\n\n2. **Template token analysis**: Our finding that many chat-only latents activate on template tokens provides actionable insights for understanding how chat fine-tuning affects models.\n\nWe agree that demonstrating more downstream applications would strengthen our work and view this as an important direction for future research.\n\n> why did you select \"chat-tuning\" as the phenomena of analysis w.r.t. crosscoders? [...] most limiting factor, which reduces the scope of the analysis [...]\n\nWe initially chose chat fine-tuning as it represents one of the most prevalent and impactful model adaptations in practice, and aligns with prior work which first introduced the crosscoder (Lindsey et al., 2024). However, we fully agree with the reviewer that the value our paper provides to the community applies more broadly. We have now added two new model comparisons trained on different domains to verify that this is not merely a base vs. chat phenomenon:\n\n- **Medical domain finetuning.** \n\n- **Long RL-finetuning on reasoning data.** \n\nWe observe two consistent patterns across all models: (i) The Δ metric in L1 crosscoders consistently identifies a large number of latents as finetuning-only that actually display Complete Shrinkage or Latent Decoupling. (ii) When using Latent Scaling to identify finetuning-only latents, BatchTopK crosscoders consistently identify more such latents.\n\nThese results demonstrate that our findings reproduce across narrow domain finetuning (medical & reasoning), different architectures (Qwen & Llama), and alternative finetuning algorithms (RL tuning), supporting the generality and robustness of our analysis. We will add these results to the Appendix and make sure to properly refer to them in the main text.\n\nWe provide a table with the results from those runs in our rebuttal to Reviewer A8m4 (Reviewer 1) in Section “New generalization results” at the end. We apologize for this, the full details didn’t fit into this answer.\n\n> I feel like the motivation for the work and practical impact would be clearer if the experiments involved several model pairs.\n\nWe completely agree with this suggestion. We are currently developing a comprehensive evaluation framework for a follow-up study that will systematically compare multiple model diffing techniques (including trivial baselines) on their ability to surface expected behavioral differences across controlled fine-tuning scenarios.\n\n## Remaining Weaknesses and Questions\n\nWe respond to weaknesses and questions that haven't been addressed above.\n\n### Low-level hyperparameter choices and their impact\n\n> Choice of L0,  Δ-norm thresholds and class boundaries as well as how the choice of L0 affects the observed phenomenon.\n\nThank you for these detailed questions. We chose L0=100 as a reasonable balance between sparsity and reconstruction quality, consistent with SAE literature practices (e.g. [2, 3]). For the Llama 8B experiments, we used L0=200 and observed the same patterns. We are currently training multiple crosscoders for Gemma-2 2B with different target L0 values as well as different dictionary sizes to verify robustness and will include these results in the camera ready. We expect the artifacts to persist across different dictionary sizes and sparsities since they stem from the fundamental properties of L1 regularization.\n\nThe Δ-norm thresholds have intuitive interpretations: Δ=0.5 means equal decoder norms, Δ=1.0 means zero base decoder norm (pure chat latent), and Δ=0.0 means zero chat decoder norm (pure base latent). We chose the 0.9 threshold for \"chat-only\" to capture latents strongly dominated by the chat model.\n\nWe acknowledge that the missing analysis of some factors is a limitation of our work. We added the following sentence to the limitations:\n\"We haven't systematically explored how hyperparameter choices (e.g. layer selection and Δ thresholds) affect our results.\"\n\n> Why exactly 3176 chat-only latents?\n\n3176 is simply the number of latents in the L1 crosscoder that meet our chat-only criterion (Δ>0.9). To ensure a fair comparison in the KL experiments, we selected the same number of top-ranked latents from the BatchTopK crosscoder.\n\n> Why layer 13?\n\nWe follow the methodology of Lindsey et al. (2024), who trained crosscoders on middle layer activations. Since Gemma-2 2B has 26 layers, layer 13 represents the middle of the network. While analyzing multiple layers would be valuable, computational constraints limited us to a single layer analysis. Middle layers are where we expect to find the richest and most interesting representations [4].\n\n### Question 2: Addition Orthogonality Constraint\n\nThis is a good input. Orthogonality constraints present an interesting trade-off. While they could reduce redundancy, they might also create undesirable artifacts. For instance, semantically related concepts (like different aspects of refusal behavior that we observed) naturally share representational structure, and forcing them apart could harm interpretability.\n\nYour question connects to well-known phenomena in SAE literature:\n- **Feature splitting** [5]: Where semantically related concepts distribute across multiple latents. Wider SAEs often split concepts into finer-grained representations (e.g., \"math\" → \"algebra\", \"geometry\").\n- **Feature absorption** [6]: Often some latents absorb their child latents, reducing interpretability.\n\nFor crosscoders specifically, we believe orthogonality constraints would be problematic, as illustrated by our example in Appendix D. Consider a concept C (e.g., \"dangerous concept\") that appears in some contexts in both models, but where the fine-tuned model has additionally learned to activate it in new contexts where the base model remains inactive. Ideally, we would want two separate latents: one that activates solely on shared contexts (where both models activate equally) and another that activates only where the fine-tuned model fires.\n\nAn orthogonality constraint would likely force these into a single latent, creating a clear misrepresentation of the actual behavior. This would either lead to shrinkage of the base decoder (since when only the fine-tuned model should be active, the shared latent becomes unhelpful for base model reconstruction) or cause the latent to miss contexts where only the fine-tuned model should fire—thereby missing the new behavior entirely.\n\nIt is also worth mentioning the \"Deep Causal Transcoding\" blog post on LessWrong by Andrew Mack (2024) that proposes an - in our view super exciting - alternative to finding interesting and causal latent directions in space. They use a constraint similar to what you describe and we are currently looking into how we can leverage their method for model diffing.\n\n\n\n[1]: Paulo et al. “Automatically Interpreting Millions of Features in Large Language Models.”\n\n[2]: Rajamanoharan et al. “Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders.”\n\n[3]: Lieberum et al. “Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2.”\n\n[4]: Skeal et al. Layer by Layer: Uncovering Hidden Representations in Language Models\n\n[5]: Bricken et al., \"Towards Monosemanticity: Decomposing Language Models With Dictionary Learning”\n\n[6]: Chanin et al., \"A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders\""}]

Author rebuttal:

Meta-review: This paper advances model diffing by showing that widely used L1‑trained crosscoders systematically over‑declare chat‑only concepts due to "Complete Shrinkage" and "Latent Decoupling". This is demonstrated by the proposed Latent Scaling test and KL‑based causal interventions. The recommendation to use BatchTopK crosscoders is well‑motivated: under BatchTopK, Delta‑norm becomes a reliable proxy for causal importance, whereas it fails under L1. While novelty may be moderate (adapting BatchTopK and scaling ideas to crosscoders), the contribution corrects a methodological pitfall and provides a simple, effective remedy.
Decision: Accept (poster)