Title: Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: interpretability

mechanistic interpretability

model diffing

chat-tuning

crosscoder

sparse autoencoder

ai safety

sae
Abstract: Model diffing is the study of how fine-tuning changes a model's representations and internal algorithms. 
Many behaviors of interest are introduced during fine-tuning, and model diffing offers a promising lens to interpret such behaviors. 
Crosscoders are a recent model diffing method that learns a shared dictionary of interpretable concepts represented as latent directions in both the base and fine-tuned models, allowing us to track how concepts shift or emerge during fine-tuning. 
Notably, prior work has observed concepts with no direction in the base model, and it was hypothesized that these model-specific latents were concepts introduced during fine-tuning.
However, we identify two issues which stem from the crosscoders L1 training loss that can misattribute concepts as unique to the fine-tuned model, when they really exist in both models. 
We develop Latent Scaling to flag these issues by more accurately measuring each latent's presence across models.
In experiments comparing Gemma 2 2B base and chat models, we observe that the standard crosscoder suffers heavily from these issues. Building on these insights, we train a crosscoder with BatchTopK loss and show that it substantially mitigates these issues, finding more genuinely chat-specific and highly interpretable concepts. We recommend practitioners adopt similar techniques.
Using the BatchTopK crosscoder, we successfully identify a set of chat-specific latents that are both interpretable and causally effective, representing concepts such as false information and personal question, along with multiple refusal-related latents that show nuanced preferences for different refusal triggers. 
Overall, our work advances best practices for the crosscoder-based methodology for model diffing and demonstrates that it can provide concrete insights into how chat-tuning modifies model behavior.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work analyses sparsity artifacts  in crosscoders. Crosscoders can be seen as a variant of SAEs that is used as a (mechanistic) interpretability method, complementary (orthogonal) to mechanistic interpretability based on circuits. The crosscoder takes two representations (from two models or two layers of the same model) and encodes them -- jointly -- as a combination of latents (represented with trainable embeddings) -- the encoding is then independently decoded by model-specific decoders, aiming to reconstruct each of the two representations using a small number of latents: to this end, the sparsity is enforced with auxiliary sparsification objectives, minimizing the L1 norm of the encoding or the BatchTopK approach (keeping only the k most prominent latents at the batch level). The assumption that allows for interpretability on how the derived/fine-tuned model differs from the base model is based on the norms of the latents needed for the reconstruction of the respective representation (base or derived). \n\nThe primary contribution of this work, from my point of view, is showing that the norm change metric is not a very good identifer of the latents specific for the derived model in the case of L1 sparsification (while it is in the case of BatchTopK). Authors then posit that L1 sparsification can result in two phenomena that prevent norm-diff-based attribution of concepts as unique to the fine-tuned model (or \"chat\" model as the authors call it): (1) Complete Shrinkage (CS) -- namely, that when the contribution of a latent is smaller in the base\nmodel than in the chat model, the L1 regularization can force the representation of the latent in the base model to zero, despite its non-negligible contribution to the base model's representation; (2) Latent Decoupling (LD), where a latent identified with norm-diff to be fine-tuned-only is effectively there in the present in the base activations, but the base representaton is reconstructed from other latents, the combination of which roughly corresponds to the \"chat-only\" latent (i.e., there is a dependence between latents that allows for this). Experiments and empirical results---which measure to which extent do the model-specific (\"chat\"/fine-tuned model's) latents explain two quantities: the error of the base model's representation and the reconstruction of the base model's representation---do convincingly suggest that CS and LD are indeed at play with L1 sparsificiation.\n\nAuthors then propose \"Latent Scaling\", a neat simple  fix for identification of derived-model's-specific latents for L1-trained crosscoders: each \"chat-specific\" latent is assigned another scaling factor, which are \"learned\" (there's a closed-form solution) to minimize the reconstruction MSE. Additional experiments show that latent scaling indeed does identify chat-only latents better than norm-diff and that latents identified by BatchTopK and Latent-Scaling-fixed L1 cross-coder can produce representations that can more or less convert the base model into the fine-tuned model (as measured with KL divergence over next-token probability distributions). \n\nWhat I found to be most hand-wavy is the interpretability analysis: the authors seem to cherry-pick (from a large set of latents) a few that can kind of be related to chat-concepts: model's refusal, false information, or model's personal experiences and emotions. This seemed somewhat as \"reading tea leaves\" to me and reminded me a lot of similar problems with interpreting topic models (i.e., latents in any latent model).\n\nI generally really like this type of analysis and insight-providing work, but as long as there is some clear practical implication of the obtained insights. It seems to me that the practical value of the insights has not really been tested and shown -- when it comes to interpretability -- the authors do not provide really any \"objective\" measure of interpretability gains, e.g., from their Latent Scaling, compared to the level of interpretability of latents produced, e.g., by BatchTopK-recognized chat-only latents. What I'm also somewhat puzzled with is the focus on \"chat\" latents: it is my understanding that the proposed approach is applicable to any two related sources of representations (i.e., coming from any two models where one is derived from the other or, e.g., two consecutive layers of the same model). If so, I feel like the motivation for the work and practical impact would be clearer if the experiments involved several model pairs -- the base model and then several different fine-tuned variants, where each fine-tuning is (independently) done on corpora with some property that can be controlled for (e.g., a topic-/domain-specific corpus) or for some task that we know requires \"new abilities\" (e.g., fine-tuning on reasoning traces for \"chain-of-thought\"-like reasoning). In such a case, one could have a much clearer expectation of what the \"fine-tuning\"-only latents should correspond to. So I guess my core confusion/question is: why did you select \"chat-tuning\" as the phenomena of analysis w.r.t. crosscoders? To me it seems that this is the most limiting factor, which reduces the scope of the analysis and findings and consequently the potential impact of this work.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n- The analytical framework introduced in this work is meaningful and appropriate. The authors assume that L1 sparsification can lead to problems w.r.t. identifying derived-model's-only latents -- the assumptions are not only intuitively understandable but then also clearly empirically \"proven\" by means of showing the reconstruction potential of selected latent's \n\n- The latent scaling, which allows to post-hoc identify the derived-model's most prominent latents, despite the crosscoder training having been done with L1 sparsification, is novel. Simple, computationally efficient (closed form solution) and effective. \n\n- Further experiments (e.g., causality of approximations) provide further evidence that the latents identified as chat/derived-model only are indeed the strongest signal for explaining the transition from the base to the derived model (as adding derived-specific latents to the base representations brings the model closest to the behaviour of the dervied model, at least in terms of next-token prediction probability distributions). \n\nWeaknesses: \n\n- As mentioned in the summary, I feel like the focus on chat-tuned models limits the scope of the findings and impact of the paper. The paper would be more convincing if the findings would hold across different types of (more controlled) fine-tuning of a base model: domain-specialization (e.g., continued LM-ing for a particular domain, for example, medicine or law); reasoning specialization (e.g., instruction-tuning for producing CoT-style reasoning traces) or even language-specialization (continued pretraining or instruction-tuning of a multilingual LLM for some underrepresented language). \n\n- Interpretability analysis seems hand-wavy: authors list a few latents (out of tens of thousands of them?) that they have identified as corresponding to some behaviour that we know chat-tuned models exhibit (false information, personal experiences). Feels like reading tea leaves.  \n\n- A number of low-level decisions (\"hyperparameter values\" in the broader sense) have not been really explained/justified, I list them below. Do this selections affect the findings? Would the findings change if these choices changed (e.g., layer 20 instead of layer 13)? \n\n>>  \"\"We classify latents into three sets based on ranges of their !norm values: base-only (0.0 to 0.1), chat-only (0.9 to 1.0) and shared (0.4 to 0.6).\" -> how did you pick these ranges for the three classes of latents?\"\n          \n>> \"We train L1 and BatchTopK crosscoders on the middle layer (13) activations of both models\" -> why layer 13?\n          \n>> \"To ensure a fair comparison, we choose hyperparameters for both crosscoders to reach an L0 of 100\" -> why 100? How does the total number of latents affect the problems of L1 objective (vs. the batchtopk)?\n          \n>> We compare the 3176 chat-only latents from the L1 crosscoder with the 3176 highest-!norm latents from the BatchTopK crosscoder -> why exactly 3176? Where does this particular/unusual number come from?\n\nQUESTIONS:\n1. See weaknesses above for questions regarding the \"hyperparameter\" value choices. \n\n2. A more general question (out of pure curiosity) regarding crosscoders (or even more generally, SAEs): there are auxiliary objectives that enforce sparsity, but, as you show regarding Latent Decomposition, these do not really mean that the latents are mutually different -- which I feel would have some value (reducing redundancy, simplifying interpretability). Have you considered (or is there RW) on using some kind of orthogonality (or other type of dissimilarity) objectives to force latent representation not to become mutually too similar?\n\nLIMITATIONS:\nThe limitations paragraph is good, it list a number of meaningful additional analyses that would make sense but have not been done in the paper (e.g., analysing base-model-only latents).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper investigates model diffing, a method to understand how fine-tuning alters a model’s internal representations and behaviors. It focuses on crosscoders, which track shared latent directions between base and fine-tuned models to interpret conceptual changes. Through Latent Scaling, the authors identify flaws in standard crosscoder training (L1 loss) that can mistakenly label shared concepts as chat-specific. To mitigate these issues, they apply BatchTopK loss. Using this approach on Gemma 2 2B models, they uncover genuinely chat-specific, interpretable concepts, such as those linked to false information, personal questions, and refusal behavior. The paper enhances model diffing practices and provides insights into how chat-tuning modifies model behavior.

STRENGTHS AND WEAKNESSES:
Strengths:
* The proposed technique contributes to improving the interpretability of model behavior.
* Given the increasing importance of mechanistic interpretability for large models, this study is both relevant and timely.
* Extensive analyses are conducted to validate the identified issues and the proposed solution's effectiveness.

Weaknesses:
* While impactful, the scope of the study feels somewhat limited, as it focuses primarily on distinguishing base vs. chat latents.
* The paper is occasionally difficult to follow; a more structured and clearer presentation would greatly improve its readability and publication readiness.

QUESTIONS:
* Line 78: By "model-only latents," do you specifically mean base-only and chat-only latents?
* Line 171: Should the denominator here be $\beta^{r,\mathrm{chat}}_j$?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03

comments: SUMMARY:
This paper investigates crosscoders, a recent model-diffing method. Specifically, the paper first identifies two key issues stemming from the L1 training loss of crosscoders: (1) complete shrinkage and (2) latent decoupling, both of which lead to misattribution of concepts unique to chat models. Then, the authors advocate replacing the L1 training loss with BatchTopK loss to mitigate these issues and empirically validate that the proposed practice can indeed find more genuinely chat-specific and highly interpretable concepts.

STRENGTHS AND WEAKNESSES:
**Strengths**

1. This paper is interesting and insightful. The authors not only propose the Latent Scaling method to effectively detect the "sparsity artifact" induced by the L1 training loss, but also propose methods to mitigate this issue, which give highly interpretable observations.

2. The authors present a detailed analysis of the experimental results.

**Weaknesses**

1. The writing could be improved for clarity. As a reader unfamiliar with SAEs and crosscoders, I found Section 3 particularly difficult to follow. The authors are encouraged to better highlight the key conclusions and provide more intuitive explanations of the core metrics and findings.

2. The experiments are conducted only on the Gemma-2B model. It is highly recommended to extend the analysis to models of varying scales and architectures to demonstrate the generality and robustness of the proposed methods.

QUESTIONS:
Can the authors better clarify the significance of misattributing concepts as unique to the chat model?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCrosscoders identify latents for a base and chat model and apply an L1 penalty to encourage learning \"chat-only\" latent directions. This paper argues that the L1 loss is inadequate for this task due to complete shrinkage (zero'ing out small but non-trivial latents) and latent decoupling (chat-only latents that are effectively utilized via different latents). The paper claims that batch-topk crosscoders can address this issue by having a penalty that's less severe on small latents and encourages competition. The paper conducts confirms these hypotheses by utilizing latent scaling to estimate the relevance of activations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper is clear in its motivation, proposed hypothesis, and hypothesized solution\n2. The paper conducts a lot of analysis beyond raw performance for confirmation. The visualization is often clear\n\nWeaknesses\n1. There were a few parts it took some effort to parse\n    - There ended up being a lot of notation that would benefit from replacing with english (for example, $\\nu^{\\epsilon}$)\n    - Figure 3 is somewhat involved and makes it hard to compare what I believe are the 4 important columns ({L1, BatchTopK} x {Latent scaling, delta norm})\n2. Even thought the numbers directionally confirm multiple hypotheses in Figure 3, the absolute improvement introduced by BatchTopK seems relatively small (for full responses, 0.237 -> 0.235, for first 9 tokens, 0.552 -> 0.511). Is this a correct comparison?\n3. The method is a relatively small modification and less of a novel contribution \n4. [More of a question/thought] Though the identified issues seem real and BatchTopK seems to work, it seems like a somewhat adhoc fix that incidentally resolves the two noticed issues. It would be nice to (1) enumerate the desired properties of the method independent of what people do (2) think about what's the most principled objective for this problem.\n\nQUESTIONS:\nListed in weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-07"}