SAE as a Crystal Ball: Interpretable Features Predict Cross-domain Transferability of LLMs without Training
Authors: Qi Zhang
Yifei Wang
Xiaohan Wang
Jiajun Chai
Guojun Yin
Wei Lin
Yisen Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Post-training
Transferability
Sparse Autoencoder
Large Language Models
Primary area: foundation or frontier models, including LLMs

Abstract: In recent years, pre-trained large language models have achieved remarkable success across diverse tasks. Besides the pivotal role of self-supervised pre-training, their effectiveness in downstream applications also depends critically on the post-training process, which adapts models to task-specific data and objectives. However, this process inevitably introduces model shifts that can influence performance in different domains, and how such shifts transfer remains poorly understood. To open up the black box, we propose the SAE-based Transferability Score (STS), a new metric that leverages sparse autoencoders (SAEs) to forecast post-training transferability. Taking supervised fine-tuning as an example, STS identifies shifted dimensions in SAE representations and calculates their correlations with downstream domains, enabling reliable estimation of transferability \textit{before} fine-tuning. Extensive experiments across multiple models and domains show that STS accurately predicts the transferability of supervised fine-tuning, achieving Pearson correlation coefficients above 0.7 with actual performance changes. Beyond this, we take an initial step toward extending STS to reinforcement learning. We believe that STS can serve as an {\color{black} interpretable} tool for guiding post-training strategies in LLMs. Code is available at \url{https://github.com/PKU-ML/STS}.

Published as a conference paper at ICLR 2026
SAE AS A CRYSTAL BALL INTERPRETABLE FEATURES PREDICT CROSS-DOMAIN TRANSFERABILITY OF LLMS WITHOUT TRAINING
Qi Zhang 12-
Yifei Wang"
Xiaohan Wang
Jiajun Chai?
Guojun Yin?
Wei Lin²
Yisen Wangldi
State Key Lab of General Artificial Intelligence,
School of Intelligence Science and Technology, Peking University
2 Meituan
3 Amazon AGI SF Lab
4 Institute for Artificial Intelligence, Peking University
ABSTRACT
In recent years, pre-trained large language models have achieved remarkable success across diverse tasks. Besides the pivotal role of self-supervised pre- training, their effectiveness downstream applications also depends critically the post-training process, which adapts models to task-specific data and ob- jectives. However, this process inevitably introduces model shifts that can in- fluence performance in different domains, and how such shifts transfer remains poorly understood. To open up the black box, we propose the SAE-based Trans- ferability Score (STS), new metric that leverages sparse autoencoders (SAEs) to forecast post-training transferability. Taking supervised fine-tuning as an example, STS identifies shifted dimensions in SAE representations and calcu- lates their correlations with downstream domains, enabling reliable estimation of transferability before fine-tuning. Extensive experiments across multiple mod- els and domains show that STS accurately predicts the transferability of super- vised fine-tuning, achieving Pearson correlation coefficients above 0.7 with ac- tual performance changes. Beyond this, we take an initial step toward extend- ing STS to reinforcement learning. We believe that STS can serve as an inter- pretable tool for guiding post-training strategies in LLMs. Code is available at https://gi com/ ML/STS
INTRODUCTION
Recent advances in large-scale neural networks have demonstrated that pre-training massive datasets yields models with strong generalization capabilities (Achiam et al. 2023; Grattafiori et al.]
2024 Yang al., 2025 Liu al. 2024). However, due to discrepancies between the pretraining
objectives and the specific requirements of downstream tasks, pretraining alone is often insufficient achieve optimal performance on these tasks. Post-training, which includes supervised fine-tuning (Zhang et al. 20233 Luo et al., 2023). and reinforcement learning (Schulman et al.] [0017] Shao jet al. 2024; et al. 2025b). play critical role in bridging this gap. By selectively adapting the pretrained model, post-training improves performance in target tasks, and allows models to better capture Nooainn-specife characteristics.
target task However, during the post-training process, it is widely observed that improvements often come at the expense of performance other domains (Dong et al. 20023 Kumar et al. 2022). For instance, Li et al. 2025a) state that improvements in the reasoning ability of large language
Equal Contribution.
Corresponding Author: Yisen Wang (yisen. wang @pku.edu.cn)
Qi Zhang completed this work during an internship at Meituan.
6 This work completed MIT prior Yifei Wangjoining Amazon.

---

Published as a conference paper at ICLR 2026
models come at the cost of reduced model robustness. Despite these observations, the mechanisms underlying how model features are linked and transferred during post-training remain largely un- explored. As result, we currently lack the ability to predict which domain performance is likely to benefit or deteriorate under specific post-training adaptations, limiting both interpretability and principled design of post-training strategies.
In this paper, we analyze the transferability of post-training through the enhanced interpretability provided by sparse autoencoders (Ng al., 2011). The sparse autoencoder (SAE) is an encoder- decoder architecture that reconstructs the internal activations of models while enforcing sparsity constraints on the hidden layer. Previous works have shown that the SAE encoder features achieve monosemanticity (Cunningham et al., 2023; Gao et al., 2024; Zhang et al., (025), where each di- mension only activated by a certain natural concept. Leveraging this property, we observe that post-training only modifies certain SAE dimensions for example, those associated with mathe- matical reasoning. This observation motivates a natural approach to predict the transferability of post-training: we can identify the shifted SAE features and examine their correlations with different domains.
Concretely, our analysis consists of two steps: (1) identifying the dimensions that are shifted during post-training, and (2) assessing their correlations with downstream domains. In the first stage, the primary challenge is to identify the shifted dimensions prior to post-training. Inspired by the observation that in-context learning exhibits behaviors similar to supervised fine-tuning (Wang et al.) 2023 Mosbach et al., 2023), forecast the shifted dimensions by using the supervised answers as demonstrations for in-context learning, and then identify the dimensions that undergo the largest changes. Empirical results show a clear overlap between the predicted and actual shifted dimensions. In the second stage, leveraging the interpretability of SAE activations, we observe that the activation values of these shifted dimensions in domain can capture their correlation. We formalize this the SAE-based transferability score (STS), which quantifies how strongly the shifted dimensions relate to downstream tasks. higher STS suggests a larger expected performance change after supervised fine-tuning. Empirically, we find that our metric consistently correlates well with actual performance shifts. For instance, the Pearson correlation coefficient exceeds 0.7 when evaluating performance variations across domains in the MMLU-Pro dataset (Wang et al. 2024). At last, we provide preliminary exploration of extending our metric to reinforcement learning settings. Together, these results allow to develop an interpretable framework for predicting the cross gross-domain transferability without training. We summarize our contributions as follows:
We propose a method to identify shifted dimensions in supervised fine-tuning without re- quiring access to the fine-tuned models. We observe that when supervised answers are used as context prompts, the shifted dimensions in in-context learning substantially overlap with those in supervised fine-tuning.
We propose the SAE-based Transferability Score (STS), which uses correlations in SAE feature space and estimated shifted dimensions to accurately predict LLM transferabilitt without performing supervised fine-tuning.
We empirically show that higher STS values strongly correlate with larger performance shifts in supervised fine-tuning, achieving Pearson correlations above 0.7 across diverse scenarios. This confirms that STS is reliable, fine-tuning-free metric for predicting L.M cross-domain transferability.
2 RELATED WORK & PRELIMINARY
Post-training. Post-training refers to the stage after large-scale pretraining, where a pretrained model is further adapted to align with specific objectives, user preferences, or downstream applica- tions (Ouyang et al. 2202; Rafailov et al.) 2023; Yu et al. 2025). The methods in post-training can be majorly divided into supervised fine-tuning (SFT) and reinforcement learning (RL). During the SFT process, given set of labeled examples (x,, } the model parameters are updated to min- imize the diserepancy between the model's 's predictions and the ground-truth answers via Negative Log- Lig-Likelihood (NLL) Loss:
(SFr(O)= -Re log p( O).
2

---

Published as a conference paper at ICLR 2026
where ④ denotes the model parameters. After SFT, RL is used to further align pretrained model with human preferences or task-specific objectives. Concretely, the model policy p(y 2 θ) is opti- mized to maximize reward signal from a reward model rule-based function:
(()) Bg-rty =y--pvi=: (= y)]-
In post-training research, transferability has long been a central focus. For example, Huan et al. (2025) empirically examines the transfer of capabilities from mathematical reasoning to other down- stream tasks, while Sun et al. (2025) offers a more fine-grain grained assessment of out-of-distributi generalization performance. Additionally, Chu et al. (2025) compares the generalization behaviors of SFT and RL-based post-training methods. However, most of the current works focus on post-hoc analysis after training. As result, such approaches are less practical because they cannot help pre- dict transfer effects before the fine-tuning process starts. This limitation motivates our work, where aim to build a method that can predict transferability without fine-tuning.
Sparse Autoencoders. Although large language models (LLMs) have demonstrated remarkable performance across wide range of downstream tasks, many of their decisions and internal behaviors remain opaque, which hinders broader deployment in applications. To address this issue, sparse autoencoders (SAEs) have been proposed as a promising framework for improving the mechanistic interpretability of LLMs (Lieberum et al., 2024, Cunningham et al.) 2023). Concretely, given hidden representation
R within the network, an SAE employs an encoder- decoder architecture to project z into a sparse latent representation and reconstruct it back to the original space. For instance, the case of top-K SAE (Gao et al. 2024), the encoding-decoding process can be formulated as:
h = TopK(W.2= b), 8 Wah
The encoder representation h is computed via a linear transformation defined by We E R and a bias 6 E R², while the decoder reconstructs the input features using Wa E Rdxs, The SAE is trained by minimizing the reconstruction loss:
ESAR(We. Wa b) = -
Previous studies (Gao ct al. (e.g., < s), the resulting representations often display monosemanticity. In other words, each feature dimension is only activated by certain natural concept, such mathematical definition, a physical property, or a linguistic pattern.
2024) have shown that when the encoder features are sufficiently sparse
In-context Learning. In-context learning denotes the ability of large pretrained models solve tasks by conditioning on demonstrations provided in the input (Kossen et al. 2023 Wang et al. 2025 Formally, given context consisting of labeled examples
C { 31), (12, U2),.... (Ix, }b)}.
the model receives a new query input øx41 and generates the output Du+1 by leveraging the condi- tional distribution learned during pretraining:
0k+1 po(y) C)+1,C),
where 9 denotes the fixed pretrained parameters. Unlike supervised fine-tur fine-tuning and reinforcement learning, in-context learning (ICL.) adapts to new tasks during inference without requiring additional training. Nevertheless, several studies have shown that models under in-context learning still exhibit many similarities to those trained with supervised fine-tuning and reinforcement learning (Mosbachi ctt al. 2023 Wang et al. 2023).
CAPTURING SHIFTED FEATURES IN SUPERVISED FINE-TUNING
In this paper, we analyze the transferability of supervised fine-tuning across different domains by understanding how feature representations shift within neural networks. Since sparse autoencoders (SAEs) enhance monosemanticity by disentangling overlapping representations, their features pro- vide clearer interpretability for tracking representation shifts in different domains. Consequently, in this section, we start by analyzing how the supervised fine-tuni fine-tuning process modifies SAE features and how the shifted features can be predicted in advance of the fine-tuning process.
3

---

Published as a conference paper at ICLR 2026
16
Activation Shift
14
0.7
0.6
Accuracy
10
0.5
0.3
Random Dimensions
2
0.2
Dimensions with Largest Shifts
0.1
0
10
20
30
40
50
250 500
1000
2000
Dimension Index
Number of Zeroed Dimensions
(a) Activation Shifts SAE Dimensions
(b) Test Accuracy with Zeroed SAE Dimensions
Figure 1: Analysis of feature shifts induced by supervised fine-tuning (SFT). We fine-tune Qwen2.5- 7B-Instruct the LIMO (a mathematical reasoning dataset) and examine shifts of SAE features on the residual stream at layer 25. Figure (a) shows the distribution of shift magnitudes while Figure (b) shows accuracy on Math-LightEval when progressively zeroing the dimensions with the largest shifts. The results indicate that SFT primarily affects small subset of SAE dimensions tied to specific model capabilities.
3.1 SFT-INDUCED CHANGES IN SAE FEATURES
Prior works have shown that SAE encoder features exhibit strong interpretability, with each dimen- sion corresponding to a certain natural concept. Since supervised fine-tuning (SFT) is generally tailored specific downstream tasks and targeted capabilities, fects only small subset of SAE dimensions tied to task-relevant features. Taking the mathematical ability as example, investigate how SAE features change when models fine-tuned on the math dataset LIMO (Ye et al. (022). As noted in (Lieberum et al., 2024), the same dimension in an SAE usually continues to represent similar concepts after fine-tuning. Consequently, we extract monosemantic features before and after the SFT process using the same SAE on the residual streams of 25-th layer in Qwen2.5-77 5- (Team 2024).
wonder whether it primarily af-
As shown in Figure Ta we find that changes in SAE features are largely concentrated in small subset of dimensions. With calculation, we find that the top-100 dimensions account for 25% of the total change, indicating that the SFT process primarily affects only a limited portion of SAE features. Furthermore, to observe the relationship between features shifted during SFT and mathematical ability of the model, rank SAE features according to their magnitude of change during linettuning and then evaluate model performance on the Math-Lighteval dataset (Hendrycks) et al. 202T) by zeroing out different numbers of features. As shown in Figure [bb performance math tasks drops rapidly when the shifted features are removed, whereas the model retains strong mathematical abilities when random SAE features zeroed. These results indicate that the SFT process primarily changes a small subset of SAE features that are closely associated with specific model capabilities.
3.2 IDENTIFYING SHIFTED DIMENSIONS VIA IN-CONTEXT LEARNING
As previously discussed, supervised fine-tuning modifies only a small subset of SAE features that correspond to specific semantics. Intuitively, these features are crucial for studying properties of SFT, such as transferability. However, their identification typically requires examining the model after fine-tuning, which confines the analysis to post-hoc perspective. This limitation motivates central question of our work: can such features be identified prior to the fine-tuning process, thereby enabling predictive understanding of transferability?
To solve this challenge, we draw on the connection between supervised fine-tuning (SFT) and in- context learning (ICL). Previous works demonstrate that ICL can obtain similar performance to SFT in the large language models (Wang et al. 2023 Mosbach et al. (202)). Consequently, this motivates investigate whether the SAE features shifted during ICL and SFT are consistent. To verify this hypothesis, we respectively sort the SAE dimensions according to their changes after SFT and ICL. To be specifie, conduct experiments Qwen2.5-7B. 5-B-5-truuttt For SFT, we employ ground-truth

---

Published as a conference paper at ICLR 2026
Percentage of Overlap (%)
100
SAE Dimensions Raw Dimensions
SAE Dimensions
80
Percentage Total Shifts (%)
35
Raw Dimenslons
30
60
1
25
40
20
20
100
200 Number of Dimensions
300
2 Percentage of Dimensions (%)
3
(a) Degrees of Overlaps between Estimated and Actual Shifted Dimensions
Percentage of Overall Activation Change from Top-Shifted Dimensions
Figure 2: Overlap between estimated dimensions and the training task. Figure (a) demonstrates that the SAE shifted dimensions predicted by ICL substantially overlap with the actual shifted dimen- sions identified after SFT, whereas applying the same method directly raw dimensions is less effective. Figure (b) further shows that raw model dimensions, prior to applying SAE, are influ- enced more uniformly by the SFT process, thereby limiting the ability to identify crucial shifted dimensions.
chain-of-thoughts (CoTs) LIMO supervision, while for ICL, we the same CoTs context prompts. As shown in Figure 2a we observe substantial overlap between the shifted features in SFT and ICL; for example, 57% of the top 100 most shifted SAE dimensions coincide. These findings confirm that the shifted dimensions in ICL and SFT are highly consistent, suggesting that the shifted dimensions can be identified before the SFT process.
We have shown that, by leveraging the connection between ICL and SFT, the shifted SAE dimen- sions can be identified prior the SFT process. A natural question then arises: are SAEs necessary for this method? Intuitively, dimensions in the raw feature space are highly polysemantic and entangled (Elhage et al., 2022). specific abilities of the model (e.g., mathematical reasoning) are distributed across multiple dimensions. As result, the SFT process tends to affect broader set of features, which makes it more difficult to identify crucial shifted dimensions. To verify the analy- sis, further conduct experiments features by their changes during SFT and ICL. As shown in Figure 26] the shifted features before SAE
the raw features before applying SAEs. We again sort the
uniformly distributed. We then investigate the influence the accuracy of identify- ing shifted dimensions. With the same empirical settings, Figure 2a shows that the overlap between shifted features in ICL and SFT is much reduced in the raw feature space. These findings indicate that the enhanced monosemanticity introduced by SAES is crucial for identifying shifted features prior to supervised fine-tuning.
PREDICTING THE TRANSFERABILITY OF SUPERVISED FINE-TUNING ACROSS DOMAINS
In Section we introduced a method for predicting shifted SAE dimensions prior to the SFT pro- Intuitively, when applying fine-tuned models to downstream tasks, if the shifted dimensions are closely related to a given domain, the SFT influence on that domain will be stronger. Thus, understanding the transfcrability of SFT across domains requires analyzing the correlations between shifted dimensions and different domains. In this section, we present metric for evaluating this cor- relation and predict transferability based on that. Specifically, Section 4.1 introduces the proposed metric, while Section 4.2 validates it across diverse scenarios.
4.1 METRICS FOR MEASURING
CORRELATIONS
Due to the enhanced monosemanticity of SAEs (Cunningham et al. 2023). the activations in the SAE feature space become interpretable, meaning that the top-activated sequences within given
'The raw model dimensions refer the representations that used the input of the SAEs.

---

Published as a conference paper at ICLR 2026
dimension usually share similar semantics. This property allows to associate each dimension with specific semantic concepts. Consequently, if the sequences from a particular domain exhibit higher activation values in certain dimension, it implies that this dimension is more strongly correlated with that domain. Building on this intuition, we can use the degrees activations across dimensions to quantify Comain-feature correlations and further analyze the transferability of supervised fine- tuning.
Formally, we define our metric the SAE-based transferability score (STS). In the first step, given an SFT dataset T {Ii, yi we extract SAE features for each sample. Let the SAE features be denoted as h(r;; θ), where è represents the parameters of the pretrained model, and the SAE encoder trained on top of the pretrained model. Similarly, with in-context learning, the features are denoted as h( y/o Lt. ltt, Fy; )), where (No, yo, Vo..... Tt: Yt are the context prompts. We then identify the N dimensions with the largest changes, i.e.,
D = (E,, (1,, Q)
hy (To, yo. It, I1, 2;,;,,,,,;
where hy denotes the j-th dimension of the SAE features.
In the second step, given downstream domain dataset F [ we compute the activation values on the shifted dimensions identified in the first step:
STS (T) = Eg Σ h)( JEDN
It is important to note that we do not use the model after SFT in this estimation, which means that our metric serves as a predictive measure rather than a post-hoc analysis.
Besides directly computing the average activation values, we introduce an alternative method based on in-context learning (ICL) capture the correlation between the downstream domain and the shifted dimensions. We know that ICL leverages multiple demonstrations to guide the model in performing downstream tasks, effectively injecting domain-specifie signals into the representation. Intuitively, by comparing the SAE features extracted with and without ICL demonstrations, can isolate the effect domain knowledge on the shifted dimensions. This difference reflects how strongly the features are modulated by task-relevant information, thereby offering a reliable estimate of the correlation between a domain and the shifted dimensions.
Consequently, estimate the correlation by measuring the difference between features extracted with and without in-context demonstrations. Formally, let {ū,, D: ) denote m ground-truth question-answer pairs in the downstream domain. We define the metric
STSieI (+) = Ez; Σ [ā, To. ... Tim, Hmn Fy; θ - h5( 0)1l JEDx
This metric captures how much the presence of domain-relevant context (the demonstrations) in- fluences the shifted dimensions, providing another reliable estimate of domain-feature correlation besides using maximum activation values.
4.2 EMPIRICAL VERIFICATIONS ON PREDICTING THE TRANSFERABILITY
Based on the proposed metric, we now empirically evaluate the correlation between the STS score and the actual transferability across different downstream domains. Specifically, we first use the LIMO dataset et al., 2025) as the SFT training set, which contains 817 high-quality mathematical examples. We fine-tune three models Sivee2.5-7B-Instrree Llama3-8B -B-Instruct, and Gemma2-9B. Instruct) on LIMO, and compare their performances before and after SFT on different domains of MMLUU-Pr [Wang et al., 2024). To extract SAE features, we apply SAES to the residual streams of the models prior to SFT. For computing the STS metric, employ two ground-truth CoTs from LIMO as in-context demonstrations to identify the top-100 shifted dimensions. When estimating the correlations between the domains and the predicted shifted dimensions, we use five ground- truth CoT's from the domain of MMLU-Pro as prompts to calculate STS1CL. More details of the experiments can be found in the Appendix.
As shown in Figure 3. the correlation between STS and performance changes across different do- mains remains consistently high for all three models, with Pearson correlation coefficients exceeding

---

Published as a conference paper at ICLR 2026
12
gg neett
12
Accurscy Shifts with SFT 69
122
0.71 0.01
Tg neerits
Accurscy Shitz
19.52
Accuracy Shitts
6
goycnelogy
Ptychology
+
5:0
2.55
1:30
1.06
1:30
525 3.75 Co 4.0-1755.00222
755
780
770
STS
(a) Saot Llama3-8B
(b) ST'S Qwen2.5-7B
(c) ST ST'S.s Gemma2-9B
Accuracy Shifts with 160
ssheerin
Accuracy Shifts with SFT (8)
20
0.81 0.01
Acaraacy shirt SF7
10
0.78 = 0.01
p
Lg teetss
tyice
coppuirr
2
yrycolooyy
2acchlogg
135
1.5
2.57
3.05
0.89
0.00
9
12
225
250
2.5
242
245
(d) STCc on Llama3-8B
(c) STCic on Qwen2.5-7B
(f) ST SICE on Gemma2-9B
Figure 3: The Pearson correlation (p) between STS and actual absolute performance shifts on MMLU-Pro induced by SFT on LIMO. Each experiment is repeated three times, and we report the mean and standard deviation of p: the fitted line shown corresponds to one of the runs. We extract SAE features from Haama3-8B-Instruue the evaluation process, select four MMLU-Pro domains with the largest and smallest perfor- mance shifts under SFT. The detailed performance shifts can be found in Appendix A.
Qwen2.5-78 1ren2.5-7B-Instrucct and Gemma2-99 9--nn------ttettt.. During
Table Verification on a code dataset Verifiable-Coding Coding-Problente Problems-Pythonn ical reasoning dataset (CoT Reasoning MieessMenttl Health). The Qwen-2.5-7BB 2-5-rB-nnttuu model is trained on each dataset, and we evaluate: (1) the correlation between ICL feature shifts and SFT, and (2) the correlation between actual performance shifts and our proposed metric.
and a clin-
Training Domain
Correlations between Actual
Overlap between Top 100 Esti- mated and Actual Shifted Dims
Performance Shifts and STS|cL
Code Health
0.77 = 0.01
62
0.71 + 0.02
57
60%. These findings validate that our metric provides a reliable estimation of the transferability of the SFT process. Besides, we note that the performance of STSiCL is more stable than STSact (the coefficients keeps above 75%), which suggests that leveraging in-context learning yields more accurate estimation of the correlation between domains and SAE dimensions than relying solely on activation values.
To better verify the effectiveness of our metric, we further conduct the experiments in different domains to provide more empirical support. Specifically, include code-gencration dataset Sere--Coiing-Probbeec
Peython--kk-Dttaseee (Hugging Face 2025) and a health dialogue dataset (CoT Reasoning Mens Mental Health (Wesney (0255) to examine how well metric STS correlates with actual downstream performance changes. In addition, we further evaluate the overlap between ICL-induced feature drift and SFT-induced shifts. Taking Qwen22.5-BBIInstrucl as an example, the empirical results are summarized in Table I As shown in the table, we observe a strong correlation between our proposed metric and actual performance shifts across different datasets. In addition, our central hypothesis that there is substantial overlap between ICL and SFT shifted dimensions in the SAE representation space continues to hold across different training datasets. These results reinforce the validity of our method and expand the scope of our method by demonstrating its effect across diverse training domains.
4.3 ABLATION STUDY
7

---

Published as a conference paper at ICLR 2026
0.8
0.00
0.80
0.75
Correlation
0.6
0.70
Carrclation Prdatte
0.4
0.65
0.3
0.60
0.1
0.53
16k
131k
0.50
15
20
25
0.50
13
109
Hidden Dimersions in SAE
Layer of Owen2.5-78
LO Norm
(a) Hidden Dims
(b) Model Layers
(c) SAE Sparsity
0.8
0.7
Correlation
Correlation
0.5
0.4
0.3
0.2
0.2
0.1
900-1000
1900
Raw Activations Activationa
Dimensions
(d) Shifted Dims
(c) Prediction Metrics
Figure 4: Ablation studies the implementation of our metric. We evaluate (a) SAEs with varying hidden dimensions the representation space, (b) SAEs trained on different layers of the pre-trained model, (c) different ranges of top-shifted dimensions, (d) different sparsity in SAE representations, e) the comparison between STS and directly using activations.
We conduct several ablation studies to evaluate STS under different conditions. First, we examine the role of monosemanticity by extracting SAE features with different hidden dimensions (16k vs. 131k). As shown in Figure 4a| weaker monosemanticity causes clear drop in prediction accuracy, indicating importance
STS. We also compare SAEs applied to different layers of Qwen2.5-7B- Instruct (layers 15, 20, and 25). Figure 4b shows that STS consistently correlates with performance changes across layers, confirming its robustness. Finally, we vary the sparsity levels of SAE rep- resentations. As illustrated in Figure 44c higher sparsity-typically linked to stronger monoseman- tieity-yields more accurate predictions, further underscoring the essential role of monosemantic features STS.
In addition to evaluating different SAEs, we examine another critical hyperparameter of our metric: the range of estimated shifted dimensions. Selecting dimensions with small shifts risks including those unaffected by the SFT process. As shown in Figure 4d the correlation between STS and downstream performance decreases when smaller-shifted dimensions are selected, supporting this analysis. These findings underscore that the choice of shifted dimensions directly impacts metric reliability, highlighting the need for an appropriate selection strategy that only identify dimensions with largest shifts.
Furthermore, we conducted additional experiments directly predict the performance improve- ments from SFT using model activations. In Figure 4el we compare method with predicting the improvements based on the model activations (using an optimized probe). As shown in the table, neither raw activations nor SAE feature activations exhibit meaningful correlation with the actual performance shifts. These results suggest that simply probing activations is insufficient; identifying the shifted dimensions induced by SFT is essential. Besides, also note that this is task where SAE beats probes, which further shows the potential of the monosemantic representation space.
5 APPLICATIONS: A DATA MIXTURE PRINCIPLE
In Section 4 we demonstrated that our proposed metric, the SAE-based transferability score (STS), exhibits strong correlation with actual performance changes. Building on this result, we now explore a practical application of STS. Specifically, we leverage predicted transferability to opti- mize data mixture strategies during post-training. Using STS, likely to be affected by supervised fine-tuning (SFT). A common approach to mitigate performance degradation in such domains is introduce additional data. Intuitively, with predicted ranking of
identify the domains most
8

---

Published as a conference paper at ICLR 2026
50
Before SFT
Peformane lekee M2
45
Accuracy
0.44
Accuracy
25
0.40
0.315
Fartrrenca belbra
40
20
40
Enginecring
Percentange Engineering
Percentange
(a) Performance with Different Amounts of Engineering Data
(b) Performance with Different
Performance Shifts with Dif- ferent SFT Data
Amounts of Law Data
Figure 5: Comparison of data mixture strategies in the SFT process. We focus the domains with the largest (engineering) and smallest (law) performance shifts induced by SFT Qwen2.5- IB-Instrect LIMO. In total, 220 extra examples from mixture of engineering and law data added. Figure (a) reports enginecring performance with varying amounts of engincering data, while Figure (b) reports law performance with varying amounts of law data. Figure (c) compares the downstream performance without additional data and with additional data mixed according to the ratio of their corresponding STS values.
performance changes across domains, we can allocate more data to those at higher risk of degrada- tion. To validate this idea, examine domains in MMLU-Pro: engineering (with the largest performance drop when fine-tuning Swwen..5-7B-Instren our experiments, we fine-tune Owen2.5-7B-Innttree on LIMO with additional data in MMLU-Pro. We split MMLU-Pro into training and testing sets with a 1:1 ratio. For training, we use Qwen2.5- 7B-Instruct's outputs prior SFT supervised answers. addition to the original LIMO data, we augment the training set with 220 extra examples sampled from the mixture of engineering and law domains.
on LIMO) and law (with the smallest). In
As shown in Figure 5 we observe that domains with larger performance changes require more additional data. For example, allocating more data to the engineering domain leads to substantial improvements (Figure 5ap. whereas allocating extra data to the law domain yields only marginal gains (Figure [b). Moreover, when the data mixture ratio is adjusted to align with the ratio of their corresponding STS values, the resulting model achieves balanced performance across both engineering and law (5ig These findings suggest that STS can serve as an interpretable guide for designing data mixture ratios, enabling more effective post- bost-training while mitigating uneven performance shifts across domains.
EXPLORATIONS ON REINFORCEMENT LEARNING
In this paper, we primarily investigate the impact of supervised fine-tuning across different domains. Nevertheless, reinforcement learning (RL) represents another non negligible post-training paradigm, motivating us to explore whether our method can be extended to RL. begin by applying the STS metric directly, in the supervised fine-tuning setting. Specifically, train Qwen2.5-7BB-Isttmee using the GRPO framework (Shao et al.) 2024) on the Math-LightEEval dataset and evaluate perfor- mance changes across domains in MMLU-PP When computing STS, we follow the same proce- dure as in supervised fine-tuning: ground-truth CoTs from Math-LightEval serve as demonstrations estimate shifted dimensions, and correlations between these dimensions and downstream domains are calculated based on in-context learning. However, as shown in Figure (bb] STS exhibits low cor- relations with performance changes in the RL setting. the following, we try find the reasons behind this discrepancy.
We note that a key distinction between SFT and RI. that RL. lacks access to ground-truth answers, making it challenging to select appropriate demonstrations for in-context learning. As a result, the estimation of shifted features may be inaccurate. To test this hypothesis, we compare the overlaps between the actual and predicted shifted dimensions. As shown in Figure 6al the overlap is substan- tially lower in RL than in SFT. To further validate this observation, we replace the estimated top-100 dimensions with the ground- truth dimensions with the largest changes after RL and recompute the that STS calculated with ground-trus truth shifted dimensions shows
STS metric. strong correlation with performance changes, indicating that the main challenge lies in accurately

---

Published as a conference paper at ICLR 2026
60.0
8 57.5
Correlation Coefficient
0.5
T 55.0
0.4
Percentange Lan447
50.0
0.3
447.5
45.0
0.2
42.5
0.1
40.0
SFT
Post-training Method
0.0
STS,cl
5T5RL
Metric
(a) Overlaps between
Estimated and Actual Shifted Dimensions Induced by SFT and RL
(b) Different Implementations of STS Metric
Figure The explorations on extending STS metric to the RL scenarios. We optimize Qwen2.5-7BB Instruct Math- using GRPO. Figure (a) shows that the identified dimensions in RL show smaller overlap with the actual shifted dimensions compared to SFT, which limits the accuracy of the metric. Figure (b) shows that when the estimated shifted dimensions replaced with the actual ones, the correlation between STS and downstream performance changes becomes much stronger.
estimating shifted dimensions RL prior to training. And this will be the future direction of our explorations. For now, STS in RL can serve as metric to predict transferability without evaluating RL models on downstream tasks.
7 CONCLUSION
In this work, we introduce metric for predicting the transferability of post-training in large-scale neural networks, leveraging the interpretability of sparse autoencoders (SAEs). By identifying the SAE dimensions that are shifted during post-training and measuring their correlations with down- stream domains, we propose the SAE-based transferability score (STS) as a predictive indicator of performance changes. Our experiments show that STS reliably forecasts performance variations across multiple domains, providing new insights into internal mechanisms of post-training. Be- yond supervised fine-tuning, further demonstrate the applicability of approach reinforce- ment learning settings. Overall, we believe our work establishes an interpretable framework for understanding and anticipating post-training effects, paving the way for more targeted and effective post-training strategies.
ACKNOWLEDGEMENT
Yisen Wang is supported by National Natural Science Foundation of China (62376010, 92370129), Beijing Major Science and Technology Project under Contract no. Z251 100008425006, Beijing Nova Program (20230484344, 20240484642), and State Key Laboratory of General Artificial Intel- ligence.
ETHICS STATEMENT
This work makes use of publicly available datasets and models. No private or sensitive data are involved, and harmful content is included. Therefore, we believe this paper does raise any ethical concerns.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, provide comprehensive details of experiments in both the main paper and the appendix. In Section 3] we describe the details of the methodology for identifying shifted dimensions, while Section presents the procedure for computing correlations
10

---

Published as a conference paper at ICLR 2026
between these dimensions and downstream domains. Additionally, Appendix A reports the concrete performance changes across different MMLU-Pro domains, and Appendix B offers further details and complementary discussions.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303, 08774, 2023.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang. Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17166 2025.
Jingyi Cui, Qi Zhang. Yifei Wang, and Yisen Wang. On the theoretical understanding of identifiable sparse autoencoders and beyond. arXiv preprint arXiv:2506. 15963, 2025.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08604 2023.
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint 10.05:2310.05599, 2023.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, al. Toy models of superposi- tion. arXiv preprint arXiv:2209, 10652, 2022.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.040993 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In NeurlPS, 2021.
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du Radha Pooven- dran, Graham Neubig, and Xiang Yuc. Does math reasoning improve general 1lm capabilities? understanding transferability of Ilm reasoning, arXiv preprint 0000:2507700432, 2025.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URI. https: //gi com huggi ng face/ open-r!
Jannik Kossen, Yarin Gal, and Tom Rainforth. In-context learning learns label relationships but not conventional learning. arXiv preprint arXiv:2307.12373. 2023.
Ananya Kumar, Aditi Raghunathan, Robbie Jones. Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform disto--iistriuttion.. arXiv preprint arXiv:2202. 10054, 2022.
Ang Li, Yichuan Mo, Mingjie Li, Yifei Wang, and Yisen Wang. Are smarter llms safer? explor- ing safety-reasoning trade-offs in prompting and fine-tuning. arXiv preprint arXiv:2502.09673. 2025a.
Ang Li, Yifei Wang. Zhihang Yuan, Stefanie Jegelka, and Yisen Wang. Lanpo: Bootstrap- ping language and numerical feedback for reinforcement learning in 1lms. arXiv preprint arXiv:2510. 16.552, 2025b.
11

---

Published as a conference paper at ICLR 2026
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once gemma 2. arXiv preprint arXlv:2408.05147, 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. 2024.
Ziyang Luo, Can Xu, Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306. 200.2066.08888, 2023.
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dictrich Klakow, and Yanai Elazar. Few- shot fine-tuning vs. in-context learning: A fair comparison and evaluation. arXiv preprint UrXiv:2305.16955 2023.
Andrew Ng et al. Sparse autoencoder. CS294A Lecture notes, 72(2011):1-19) 2011.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. In NewrlPS, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. In NeurlPS, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Yiyou Sun, Shawn Hu, Georgia Zhou, Ken Zheng, Hannaneh Hajishirzi, Nouha Dziri, and Dawn Song. Omega: Can Ilms reason outside the box in math? evaluating exploratory, compositional, and transformative generalization. arXiv preprint arXiv: 18880, 2025.
Qwen Team. Qwen2 technical report. arXiv preprint arXiv: 2024.
Qixun Wang, Yifei Wang, Xianghua Ying, and Yisen Wang. Can in-context learning really general- ize to out-of-distribution tasks? In ICLR, 2025.
Xindi Wang, Yufei Wang, Can Xu, Xiubo Geng, Bowen Zhang, Chongyang Tao, Frank Rudzicz, Robert E Mercer, and Daxin Jiang. Investigating the learning behaviour in-context learning: comparison with supervised learning. arXiv preprint arXiv:2307.15411, 2023.
Yubo Wang. Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: more robust and challenging multi-task language understanding benchmark. In NeurlPS, 2024.
Matthew R. Wesney. Cot reasoning mens mental health dataset, 2025. URL https:
//huggingface.
Mens Mental
Health
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 0083:2 2025.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning, arXiv preprint 23030222000.388 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu. Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. /4476, 2025.
12

---

Published as a conference paper at ICLR 2026
Qi Zhang, Yifei Wang, Jingyi Cui, Xiang Pan, Qi Lei, Stefanie Jegelka, and Yisen Wang. Beyond interpretability: The gains of feature monosemanticity on model robustness. In ICLR, 2025.
Yifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi-Chih Yao. Cumulative reasoning with large language models. arXiv preprint arXiv:2308 04371, 2023.
13

---

Published as a conference paper at ICLR 2026
A PERFORMANCE CHANGES AFTER SFT
Table 2: Performance change on different domains of MMLU-Pro (Llama3-8B- LIMO).
-aamtt----Inttructt IB-Instruct fine-tuned
engineering 11.97
physics -5.7
chemistry -5.39
law
Performance Change (%)
-3.72
philosophy -2.61
other -2.49
health -1.1
computer science
Performance Change (%)
-0.97
economics -0.95
math -0,89
psychology -0.63
biology -0.42
Performance Change (%)
history -0.26
business -0.12
Performance Change (%)
Table 3: Performance change on different domains of MMLU-Pro (Qwen2.5-7BB 20-n2.55-BB-nsttruc fine-tuned on LIMO).
engincering -9.8
chemistry -6.36
physics -5.24
computer science -4.88
Performance Change (%)
business -4.43
health -2.93
math -2.89
economics
Performance Change (%)
-2.61
philosophy -2.4
biology +1.53
other -1.3
psychology -0.37
Performance Change (%)
history +0.26
law -0.19
Performance Change (%)
Table 4: Performance change different domains of MMLU-Pro Beemma2-9B-Instruve fine-tuned on LIMO).
engineering -9.29
computer science
other -2.05
law
Performance Change (%)
-3.66
-1.73
health -1.47
math +1.41
business -1.27
history -0.79
Performance Change (%
economics -0.71
chemistrey -0.62
biology -0.56
physics -0.54
Performance Change
psychology -0.05
philosophy
Performance Change
We show the concrete signed performance change of models after SFT on LIMO in Table 23
In our experiments with SFT on LIMO, we observe that performance decreases across nearly all downstream domains, and the primary difference between domains lies in the magnitude of the decrease. This is consistent with the known limitations of SFT in generalization. Consequently, our work focuses on predicting the magnitude of performance change rather than its sign. We consider this meaningful because accurately estimating the degree of decrease provides insights into model behavior under SFT and informs strategics to mitigate these decreases (c.g., the STS STS-guided data mixing strategy in Section 5).
14

---

Published as a conference paper at ICLR 2026
B EXPERIMENTS DETAILS
B.I DETAILS OF APPLIED SAES
For feature extraction with sparse autoencoders (SAEs), we use one SAE for each backbone model on specific layer of the transformer. Each SAE an encoder-decoder architecture. The encoder and decoder are two linear layers while there exists an activation function following the encoder. We introduce the details of SAEs in the following.
Concretely, for Llama3-88 Instruct,, we use ReLU SAE with 16384 hidden dimensions trained on residual-stream activations from layer 25. The SAE trained on the openWebText dataset with context size 1024. The SAE is optimized using the Adam optimizer with 81=0.9, 33=0.999, and weight decay of 0.01. The learning rate is set le-5, An LI sparsity penalty of (with warm-up) is applied on the hidden activations to induce sparse and monosemantic features, following standard SAE training practices.
For Qwen2.5-7B -nntstrut,,,,t stream activations from layer 25. The SAE trained on the openWebText dataset with a context size of 2048. The model is optimized using the AdamW optimizer with $1=0.9, $2=0.999, and cosine-annealing learning-rate schedule starting at 7e-5 with warm-up. An L1 sparsity penalty (with warm-up) applied to encourage sparse and monosemantic features, following standard SAE training setups.
ReLU SAE with 28672 hidden dimensions trained residual-
ReLU SAE with 131072 hidden dimensions trained residual- stream activations from layer 31. The SAE is trained on the open WeenWe dataset with a context size of 1024. The model is optimized using the AdamW optimizer with =0=,.9 32=0.999, and aosine-annealing learning-rate schedule over the first 10,000 steps. We also apply a linear warmup of the sparsity coefficient over the first 10,000 steps stabilize training and encourage sparse feature activations.
For Gemmaa-99-Inntteee
B.2 DETAILS OF ICL PROMPTS
To identify shifted SAE dimensions induced by post-training, we first sample activations from 20,000 tokens before and after in-context learning (ICL), where the ICL demonstrations are con- structed using supervised answers from LIMO. The prompt looks likes 1xI.yI.X2y22x3],, where x1,x2,x3 are the questions in LIMO while y1, y2 are the responses in LIMO. To be specific, we provide a concrete example in the following
In-context Learning Prompts
(f'content': : "A fenced, rectangular field measures 24 meters by 52 meters.
What the largest number square test plots into which the field can be partitioned using all or of the 1994 meters of fence? Let's think step by step and output the final answer within (boxed().". 'role": 'user").
('content': 'Okay, so have this problem where thereś a rectangular field thats meters by 52 meters. The farmer wants partition this entire field into square test plots, with the sides of the squares parallel to the edges of the field.
So. IíI call this over. Thus, the answer is 702. **Final * \boxed(702) "role": 'assistant'),
('content': "A hotel packed breakfast for each of three guests.
Given that the probability each guest got one roll of each type is where and relatively prime integers, find m + n. Let's think step by step and output the final answer within (boxed().".
15

---

Published as a conference paper at ICLR 2026
"role": "user"),
:'content': "Okay, here's 's this problem about hotel packing breakfast for three guests. Each breakfast is supposed to have one nut roll, one cheese roll, and one fruit roll.
The total number of ways to choose three rolls from the remaining 6: C(6,3)=20. So proba- bility Therefore, 9/70 is correct. Thus, m n=79. Therefore, the answer is 79.
**Final #
(
'role": "assistant);
('content': "For how many pairs of consecutive integers in 1000, 1001, 1002, 2000 is no carrying required when the two integers are added? Let's think step by step and output the final answer within {boxed[].". "role": 'user'} }
We then compute the activation differences across SAE dimensions and select those exhibiting the largest shifts. Finally, to quantify the correlations between shifted dimensions and downstream domains, we sample activations from 10,000 tokens for each domain and compute their correlations with the identified dimensions.
B.3 DETAILS OF SUPERVISED FINE-TUNING
When fine-tuning the pre- pre-trained models Qwen2.5-7B 5---ittttBBnnttr Llama3-8B -B-nttrutt-,stt and Gemma2- 9B-Instruct on the LIMO dataset, we adopt a unified experimental setup across models. Specifically, the maximum prompt length is set to 8192 tokens, with sequences truncated from the left to fit within this constraint. All models are fine-tuned for 10 epochs using four H20 GPUs. We train the models with total batch size of 256 and a micro-batch size of 1 per GPU. Models are trained using FSDP with fp32 precision, gradient checkpointing enabled, and no CPU offload. We use the AdamW optimizer with betas (0.9, 0.95), weight decay 0.01, and gradient clipping of 1.0, A cosine learning rate scheduler is applied with a warmup of 10% of the total training steps.
C ADDITIONAL EXPERIMENTS
C.I COMPARISON WITH TRADITIONAL REPRESENTATION SHIFT ANALYSIS
We note that our work not a trivial reframe existing analyses on representation drift or feature correlations. The key distinction is that prior studies examine raw representation shifts after fine- tuning, whereas our paper demonstrates that the sparsity and monosemanticity brought by SAEs enable us to predict feature shifts and find correlations before fine-tuning. As shown in Figure 2, when using raw model features without SAEs, the overlap between predicted and actual shifted dimensions is quite low, indicating that traditional representation analyses cannot accurately identify shifted dimensions.
To further distinguish our method from traditional representation analysis, conduct additional experiments comparing STS with three baselines: (1) raw feature activations in downstream do- mains, (2) representation similarity between downstream and training domains, and (3) representa- tion similarity between models before and after SFT. For Qwen2.5-7B-Inns correlations between these measures and actual performance shifts.
Table I reports the
Table 5: Correlation coefficient between actual performance shifts (Qwen2.5-7BB-nnttuu tuned on LIMO) and different baselines.
Representation Similarity between Downstream Domains and Training Domain
Representation Similarity between Models hefore and after SFT
Feature Activations
STS
Uses Model
No
No
Yes
No
after SFT
Coefficient
0.03
0.11
0.61
0.79
16

---

Published as a conference paper at ICLR 2026
As shown in the table, neither feature activations representation similarity between train- ing and downstream domains strongly correlates with performance shifts. Even when using post- SFT representations, the correlation remains substantially lower than that of our method. These results further demonstrate that our approach is not a minor variation of traditional representation drift analyses.
C.2 REPEATED EXPERIMENTS ON EVALUATING THE CORRELATION COFFFICIENT
We additional experiments with three independent seeds and report the + standard devia- tion in the table below.
Table The Pearson correlation coefficient between STS and actual performance shifts on MMLU- Pro induced by SFT on LIMO.
Metric/ Model
I.L.MA3-8B
Qwen2.5-7B
Gemma2-9B
ST STCiC
0.71 + 0.01
0.90 + 0.02
0.60 + 0.03
0.81 + 0.01
0.78 + 0.01
0.77 + 0.01
As shown in the table, the results further verify that the correlations are statistically significant.
C.3 QUALITATIVE ANALYSIS OF THE IDENTIFIED SHIFTED DIMENSIONS
We annotate SAE dimensions following the auto-interpretabiiity scoring pipeline in (Cunningham et al. 2023). The procedure is as follows:
1. We construct dataset consisting of three domains: math (LIMO), code -Verifiable-Coding- Problems- Paeen---yyhoo--OO- 10k--ttsst)), and dialogue (HH-RLHF).
We encode these samples and extract the corresponding SAE features (using the 25th layer of Qwen2.5-77 Swee2.557B-Iim --B-ttnntt as an example). 3. For each SAE dimension, we collect the top 10 samples with the highest activations.
4. We then prompt an LLM -B--strutt to determine whether these samples belong to math, code, or general dialogue.
5. Finally, each dimension is assigned a label (math/cooe//dallggue)) based on the LLM's judgment.
With the annotated data, we respectively calculate whether the top 50, top 100, and top 200 estimated shifted dimensions belong to the training task, i.e., the math.
Table 7: The percentage of the estimated shifted dimensions that are explained as the math dimen- sion.
Selected Dimensions
50
100
200
89%
93%
92%
As shown in the table, the estimated shifted dimensions show an extremely high correlation with the training task (math), which further verifies the effectiveness our method.
Furthermore, we then evaluate how many math-related features recalled among the top-changed SAE activations. Specifically, we sample 100 dimensions annotated as math and 100 dimensions annotated as code. We then compute the proportion of these dimensions that appear among the top 500 estimated shifted dimensions.
As shown in the table, math-related dimensions recalled at substantially higher rate than code- related dimensions. This demonstrates that our estimation process accurately identifies the SAE dimensions most relevant to the training task.
17

---

Published as a conference paper at ICLR 2026
Table 8: Proportion of 100 annotated dimensions recalled among the top 500 estimated shifted dimensions.
Annotation
Math
Code
Recall
63%
7%
C.4 R2 BETWEEN STS AND ACTUAL PERFORMANCE SHIFTS
We report the corresponding R² results in the following table.
Table 9: R between STS and actual performance shifts on MMLU-Pro induced by SFT on LIMO.
Metric/ Model
LLaMA3-8B
Qwen2.5-7B
Gemma2-9B
STSact STSICL
0.50 + 0.01
0.80 0.04
0.36 + 0.03
0.66 + 0.01
0.61 + 0.02
0.60 + 0.02
As shown in the table, both ST and SICL are effective across models. Notably, the SICL values are highly consistent across the three models (0.60-0.66 with small standard deviations), underscoring the effectiveness of this metric in predicting performance shifts.
C.5 TEST ACCURACY ON THE MATH DATASET
Table 10: Test accuracy (%) of (0wen.55-BBInnstuuu before and after SFT LIMO.
Before SFT
SFT on LIMO
SFT on STS-Guided Data
74.7
81.3 (+6.6)
80.1 (+5.4)
In Figure report downstream accuracy the law and engineering domains. It is important to note that the SFT process was conducted on the math dataset LIMO. In the following table, present the test accuracy on the Math-LightEVAL dataset before and after SFT.
As shown in the table, SFT leads to a substantial improvement on Ligh-LightEVAL Combined with the observations in Figure these results indicate that SFT on STS-guided data effectively enhances math performance while maintaining the model's capabilities in other domains.
C.6 SELECTIVELY FINE-TUNING ON ESTIMATED SHIFTED DIMENSIONS
Table 11: Test accuracy (%) of Jiwen2.5-7B-Instruee after selectively fine-tuning.
Selected Dims
Before SFT
SFT on Random Dimensions
SFT Estimated Shifted Dimensions
1000
74.7
74.2
75.6
3000
74.7
74.0
76.9
To further evaluate whether the selected dimensions are the most relevant to the training task, we conduct selectively fine-tuning based on identified shifted dimensions. To be specific, we first se- lected five layers in Qwen2.5-7B- -ww.srrnec/lllama3555. Instrrct and extracted 3000 dimensions from the SAE representation space at each layer. We then added five linear layers of shape [3000, d], where is the dimension of the raw (pre-SAE) features. During the forward pass, the 3000 SAE dimensions are decoded through these learnalee linear layers and added back to the raw features. We fine-tune only these five linear layers on LIMO, keeping all other model parameters frozen. As shown in Figure 7 selective fine-tuning using the estimated shifted dimensions effectively im- proves math performance with only five linear layers. This result further validates that the selected dimensions are the most relevant to the training domain.
18

---

Published as a conference paper at ICLR 2026
Without SFT
70
SFT on Estimated Shifted Dims
Math Accuracy
60
55
40
Owen
Liama (OpenThoughts) (LIMO) (OpenThoughts)
Qwen
Llama
(LIMO)
Models (Dataset)
Figure 7: Selective Finetuning Different Models on Estimated Shifted Dimensions
Indeed, in selective fine-tuning experiments, tune the top-K dimensions rather than single dimension (K = 3000 in Table IT] We also evaluate K = 1000, with the results shown below.
As shown in the table, selective fine-tuning on the estimated 1000 or 3000 shifted dimensions effec- tively improves math performance. These results further confirm that the identified shifted dimen- sions are highly aligned with the training task.
C.7 ABLATION STUDY THE SPARISTY OF SAES
Following prior work, experiment with different sparsity levels in this representation space. The following Table reports the results using SAEs trained with varying L0 norms.
Pearson correlation coefficients between STS and actual performance shifts on MMLU- Pro induced by fine-tuning Gemma2-993-nntrruct LIMO. STS is computed using SABs with dif- ferent Lo norms.
Table 12:
Lo
13
22
37
63
109
Correlation 0.78
0.77
0.72
0.75
0.65
As shown in the table, increased sparsity in the SAE leads more accurate prediction of per- formance changes. Since stronger sparsity is generally associated with stronger monosemanticity (Cunningham et al. (2023), these results further highlight the critical role of monosemantic repre- sentations in method.
C.8 PREDICTING PERFORMANCE SHIFTS BASED ON MODEL ACTIVATIONS
We conducted additional experiments to directly predict the performance improvements from SFT using model activations. Using (wen2.5--BBInnsrrus LIMO and measure the performance shifts across different MMLU-Pro domains. In the following table, compare our method with predicting the improvements based the model activations (using an optimized probe).
example, fine-tune the model
Table 13: Comparison between STS and predictions based on model activations.
Raw Feature Activations
SAE Feature Activations
STS
Correlation Coefficient
0.03
0.08
0.79
As shown the table, neither raw activations nor SAE feature activations exhibit meaningful cor- relation with the actual performance shifts. These results suggest that simply probing activations is insufficient; identifying the shifted dimensions induced by SET essential for understanding and predicting model behavior during the fine-tuning process.
19

---

Published as a conference paper at ICLR 2026
C.9 ADDITIONAL DISCUSSION ON FIGURE
As shown in Figure 2(b) of the paper, we compare the raw and SAE dimensions in terms of the proportion of total shifts captured by the top shifted dimensions. We observe that the top 1% of raw features account for smaller fraction, indicating that the shifts are distributed relatively uniformly across the raw dimensions, which makes difficult identify the core shifted features. In contrast, shifts in the SAE dimensions are more concentrated, highlighting the effectiveness in capturing key shifted dimensions.
C.10 ADDITIONAL. DISCUSSION ON THE MONOSEMANTICITY ASSUMPTION
We note that recent empirical works and thcoretical studies (Cunningham et al. 2023 Gao al. 2024 Cui et al. 2025) provide consistent evidence that sparse autoencoders trained on LLMSs tend to yield monosemantic representations. These findings support the validity of relying the monose- manticity assumption in our method. We also believe that the monosemantic representation induced by SAEs a key distinction our approach compared with traditional representation analysis tech- niques because it allows us to identify shifted features in semantically interpretable space and thereby predict transferability before conducting SFT. Besides, we clarify that our method does not require any additional curated demonstrations beyond the responses already used for SFT. In prac- tice, we only randomly sample two responses from the SFT training set.
D THE USE OF LARGE LANGUAGE MODELS (LLMs)
In this work, the use of LLMs was limited to minor language editing to improve readability. All conceptual development, theoretical analysis, experimental design, and result interpretation were conducted independently by the authors. Thus, the use of LLMs was purely auxiliary and had no impact on the scientific contributions of this paper.
20


Reviews:
[{"review_id": "e15UKRFhJQ", "reviewer": "Reviewer_czDh", "summary": "This paper proposes a metric for predicting how supervised fine tuning will improve performance without performing the fine-tuning. The metric called STS trains a sparse autoencoder on hidden activations in the LLM to obtain sparse \"monosemantic\" latent features. The central assumption is that SAE dimensions that shift during in-context leraning also shift during fine-tuning. A correlation between these shifted features is the core of STS. Results on presented on a math dataset for several public models with ablations on SAE size, layer, etc.", "questions": "See weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Addresses an interesting and timely problem - forecasting post-training transfer effects for LLMS\n* Builds on recent interpretability work with SAEs\n* Correlations in the experiments are consistent suggesting the metric captures a genuine phenomenon", "weaknesses": "* Limited conceptual novelty. Reframes existing ideas on representation drift and feature correlation under transferability and SAEs. Lacks a theoretical link between SAE features and fine tuning.\n* Limited empirical evidence. While the experiments presented are indictive of some trend, the central hypothesis is only tested on one dataset and adaptation direction. The scope is too narrow to lend credible evidence to a correlation between ICL feature drift and SFT\n* Lack of baselines. No comparison is made against more simple correlations such as cosine distance or linear probe similarity.\n* Overstatement wrt principled prediction", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Qshy7TNaE", "reviewer": "Reviewer_Fh4V", "summary": "This paper proposes to leverage (the change in) SAE activations to predict fine-tuning performance.\n\nThe premises are (well-studied either empirically/theoretically in prior work):\n\n1. The sparse activations in SAE are domain/task-specific.\n2. ICL can approximate taking a few gradient steps during the SFT process.\n\nThe setup is that, given a fine-tuning/source dataset (e.g., Math) and eval datasets (e.g., Engineering/Law), we want to predict the change in performance on the eval datasets from fine-tuning on the source dataset. The authors propose to use SAE to perform this prediction as follows:\n\n- Identify the top-changed SAE activation dimensions before/after adding source-domain ICL examples in zero-shot prompting (ICL simulates fine-tuning).\n- Identify how often those dimensions activate when feeding in the target domain examples, and use this as a measure of transferability (since SAE activation is associated with task relevance)\n\nThey found that the proposed measure\n\n1. Correlates with the absolute change in performance (not the improvement) after fine-tuning.\n2. Has the potential to be used in applications such as dataset mixture setting, e.g., to mitigate catastrophic forgetting due to fine-tuning.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The reviewer finds the topic interesting and timely, and the proposed technique could be useful in LLM training workflows.\n2. The experiment design is generally sound, and the application demonstrated in section 5 is well-motivated.", "weaknesses": "The reviewer feels that this paper has the potential to have greater impact, but is limited by its current presentation and the scope of the experiments.\n\n1. Details on the experiment setup/results are lacking.\n- It appears that all experiments are performed once; ideally, it should be repeated over different random seeds (e.g., for initialization and train/test split) and report the mean + std.\n- Since SAE should be interpretable, a qualitative analysis of the identified SAE activations would be nice as a sanity check to see whether the selected dimension semantically aligns with the task.\n- Following the comment above, another direction is to annotate all SAE features and identify those related to the target task, and then see how much of the relevant features are \"recalled\" in the top-changed SAE activations.\n- The paper would greatly benefit if more source training domains can be evaluated.\n\n2. The fact only absolute change in performance is reported, rather than the signed improvement/decrease in performance, is rather disappointing. Would it be possible to also predict the sign of the change using the proposed approach?\n- Furthermore, in the paper, it was unclear to the reviewer that \"accuracy shift\" meant absolute change rather than the signed improvement/decrease.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lHOR7uxY93", "reviewer": "Reviewer_efyn", "summary": "This paper investigates the problem of the unpredictable transferability of post-training to downstream tasks. Post-training on data can introduce model shifts that improve the performance of some tasks while degrading others. To address this issue, the paper proposes a Sparse Autoencoder (SAE)-based Transferability Score (STS) to forecast post-training transferability. The method leverages supervised answers as demonstrations for in-context learning and identifies the SAE dimensions that exhibit the largest changes, which correlate with downstream task performance. Experiments across multiple models and domains demonstrate that the proposed transferability score accurately predicts the effects of both supervised fine-tuning and reinforcement learning (RL) tuning.", "questions": "Q1: Figure 2: What exactly do “raw model dimensions” refer to?\n\nQ2: What does the ICL prompt template look like?\n\nQ3: Line 243: The statement “the shifted features before SAE are more uniformly distributed” could be clarified. Does this mean less sparsity, or something else?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "S1: The paper focuses on an interesting problem of forecasting changes in downstream task performance without additional training.\n\nS2: The proposed fine-tuning-free approach for estimating transferability is both novel and useful.\n\nS3: The investigation of SAE dimension shifts under both fine-tuning and in-context learning (ICL) is insightful.\n\nS4: The paper evaluates the proposed STS method across multiple major open models (Qwen2.5-7B, Llama3-8B, Gemma2-9B) and demonstrates consistently high correlations.", "weaknesses": "W1: The study is limited to a single training dataset (LIMO) and a single evaluation benchmark (MMLU-Pro). Broader domains (e.g., dialogue, code generation) and larger model scales remain underexplored, even though these are areas where STS would likely be most valuable.\n\nW2: There are potential reproducibility issues, as details on the SAE architectures, training procedures, hyperparameters, and prompt templates are either missing or insufficiently explained.\n\nW3: Methodologically, STS relies heavily on the monosemanticity assumption of SAEs that each latent corresponds to a distinct human-interpretable concept. It also depends on access to high-quality demonstrations to estimate feature shifts (based on the weaker correlations observed in the RL experiments).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFAi62qUMU", "reviewer": "Reviewer_MqTi", "summary": "This paper shows that SAE activations on specific domains (in ICL) can predict gains from SFT on the same domain, with high correlation.\nIt studies what portion of features that are activated during ICL also get bolstered by SFF, and it then does ablation studies to check their impact on model accuracy. It checks the portion of overlapping features between ICL and SFT, and it makes sure that results hold in RL (with a new formulation). The paper finally proposes an application to designing the data mixture used for SFT based on SAE feature activations).", "questions": "Can you unpack the most important features in terms of neurons (that you could later fine tune selectively doing something like a selective LoRA)? Have you tried a sparsity penalty on SAE weights (to prune weights to save compute)?\n\nHave you tried computing sample standard deviations and R2s?\n\nHow do you perform compared to trying to predict improvements form SFT based on model activations directly (using a probe)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The main strengths of this papers lie in the novel insight that SAE activations in ICL can predict improvements from SFT. The link between ICL and SFT was present in the literature, and the authors locate themselves in the literature well.\nThe paper is comprehensive, and it covers different modalities (RL/SFT) and applications (data mix for SFT). It correlates features with SFT gains well (0.7/0.8 correlation), and it shows the (very strong) effects of ablating features.\nThe paper also studies overlaps and uncovers significant portion of feature shifts (increase in activation under ICL) are the same as what SFT causes.", "weaknesses": "The main weaknesses are:\n- no sample standard deviations for statistics you calculate\n- no R2 for Figure 4's linear regressions, it would be interesting to know the fraction of the improvement from SFT you predict\n- the SFT you do is can be ineffective (Figure 5), so predicting it is less interesting\n- SAEs are lossy, and so is your predictive method (compounded lossyness)\n\nI am open to increasing my score if the (first 2 or 3) weaknesses above are addressed\n\nMinor:\n- the contributions in the intro are too verbose", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)