SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMs
Authors: Boyi Deng
Yu Wan
Baosong Yang
Fei Huang
Wenjie Wang
Fuli Feng
Venue: ICLR 2026 Poster (2026)
Keywords: LLMs
interpretability
multilingualism
Primary area: interpretability and explainable AI

Abstract: Large Language Models (LLMs) have impressive multilingual capabilities, but they suffer from unexpected code-switching, also known as language mixing, which involves switching to unexpected languages in the model response. This problem leads to poor readability and degrades the usability of model responses.
However, existing work on this issue lacks a mechanistic analysis and shows limited effectiveness.
In this paper, we first provide an in-depth analysis of unexpected code-switching using sparse autoencoders and find that when LLMs switch to a language, the features of that language exhibit excessive pre-activation values. Based on our findings, we propose $\textbf{S}$parse $\textbf{A}$utoencoder-guided $\textbf{S}$upervised $\textbf{F}$ine$\textbf{t}$uning (SASFT), which teaches LLMs to maintain appropriate pre-activation values of specific language features during training. Experiments on five models across three languages demonstrate that SASFT consistently reduces unexpected code-switching by more than 50\% compared to standard supervised fine-tuning, with complete elimination in one case. Moreover, SASFT maintains or even improves the models' performance on six multilingual benchmarks, showing its effectiveness in addressing code-switching while preserving multilingual capabilities. The code and data are available at https://github.com/Aatrox103/SASFT.

Published as a conference paper at ICLR 2026
SASFT: SPARSE AUTOENCODER-GUIDER SUPER- VISED FINETUNING TO MITIGATE UNEXPECTED
CODE-SWITCHING IN LLMS
Boyi Deng Yu Wan Baosong Yang! Fei Huang' Wenjie Wang" Fuli Feng 'Tongyi Lab, Alibaba Group Inc, "National University of Singapore. 'Anhui Provincial Hospital dengboyi sengboyi@maill Smail..a.........aaa...aaaam..le www..euu.ccc.......ooo.oo on wanyu.wyealibabe inc. com
com
ABSTRACT
Large Language Models (LLMs) have impressive multilingual capabilities, but they suffer from unexpected code-switching, also known language mixing, which involves switching to unexpected languages in the model response. This problem leads poor readability and degrades the usability of model responses. However, existing work on this issue lacks a mechanistic analysis and shows lim- ited effectiveness. In this paper, we first provide in-depth analysis of unex- pected code-switching using sparse autoencoders and find that when LLMs switch language, the features of that language exhibit excessive pre-activation val- ues. Based on our findings, we propose Sparse srreencoder-guideer Supervised Finetuning (SASFT), which teaches L.MSs maintain appropriate pre-activation values of specific language features during training. Experiments on five mod- els across three languages demonstrate that SASFT consistently reduces unex- pected code-switching by more than 50% compared to standard supervised fine- tuning, with complete elimination in one case. Moreover, SASFT maintains or even improves the models' performance ing its effectiveness in addressing code-switching while preserving multilingual capabilities. The code and data are available https: //g thu..ttubb..... com/ Aatrox103/SASSF
six multilingual benchmarks, show-
INTRODUCTION
about recent
understand...
LLaMA
Unexpected code-switching to Chinese
Owen
Can you explain what machine learning is?
The осиовиая concopt of machino leamiing involves...
User
deepseek
Unexpected code-switching Russian
Gemma
What the of artificial intelligence?
@ tl goal creating systems...
LLMss
Unexpected code-switching Korean
Figure 1: Examples of unexpected code-switching to Chinese, Russian, and Korean.
As the demand for multilingual Large Language Models (LLMs) continues to grow (Qin al. 2024
Huang et al.] [024 Zhao et al., (2026). researchers seek to improve the multilingual capabilities of LLMs (Tem et al.
2024: Grattafiori al.) 2024 Yang et al. 2024). For example, Qwen-3 (Yang
et al. 2025) can support 119 languages and performs well multilingual benchmarks (He et al.)
2024a Zhang et al. 20244 Romanou et al. 2024). In addition, Llama-4 is pre-trained on 200 lan- guages, where over 100 languages have more than billion tokens each (Meta) (025).. Moreover, Gemma-3 offers out-of-the-box support for over 35 languages and pretrained support for over 140 languages (Team et al. 2025). While multilingual capabilities important for LLMs, they can lead to unexpected code-switching or language mixing (Guo et al. 2025), where LLMSs switch to
Work done during internships at Alibaba Group.
Corresponding authors.

---

Published as a conference paper at ICLR 2026
This unexpected code-switching makes difficult for users to understand and reduces the model's 's utility (more details please refer to Appendix [A). Therefore, addressing unexpected code-switching in LLMs is essential.
unexpected languages in their response. shown in Figure I
To the best of our knowledge, the only attempt to address unexpected code-switching in LLMs is proposed by Guo et al. (2025), who find that DeepSeek-RI (Guo al. 2025) suffers from un- expected code-switching and attempt to address it by applying GRPO (Shao et al.] (024) with a language consistency reward. However, their method lacks deep understanding of unexpected code-switching mechanisms and shows limited effectiveness. This suggests the need for better anal- ysis and solutions.
Inspired by (Deng et al. 2025), which shows that LLMs have language-specific features through sparse autoencoders (SAEs), we conduct preliminary experiments using SAEs and find that unex- pected code-switching to a specific language occurs with unusually high pre-activation value of that language's 's features. Further experiments show that reducing pre-activation values of these language- specific features during inference can mitigate unexpected code-switching. However, this approach requires external intervention and doesn't change the model, without solving the problem funda- mentally.
Based on our findings, we propose Sparse Autoencoder-guided Supervised Finetuning (SASFT) to address unexpected code-switching. The key idea is to teach LLMs to maintain appropriate pre-activation values of irrelevant language features during training, rather than modifying them during inference. Specifically, we introduce auxiliary loss during supervised fine-tuning (SFT) that encourages the model to keep pre-activation values of specific language features below certain thresholds when generating content in other languages. Since these language features demonstrate strong monolingual characteristics, we aim to reduce code-switching while preserving the model's original capabilities.
Extensive experiments on five widely used models, including the Gemma-2 series Team et al. 2024). Llama-3.1 series (Meta 2024), and Qwen-3 series (Yang al. 2025). demonstrate the ef- fectiveness of our approach. SASFT reduces unexpected code-switching by more than 50% in most cases, with complete elimination (100% reduction) achieved in several scenarios, particularly for the Korean language. Our method significantly outperforms existing methods like GRPO. Notably, SASFT maintains or even improves the models' performance on six multilingual benchmarks, in- cluding MMLU (Hendrycks et al.) (021). HumanEval (Peng et al. [2224] Chen et al.) (021). Flores- 200 (Goyal et al., 20222 Team et al. 2022), among others. Further analysis reveals that applying SASFT across multiple layers achieves better and more stable results compared to single layer.
In summary, our main contributions are:
We provide the first in-depth analysis of unexpected code-switching LLMs using SAEs, revealing that unexpected code-switching is closely related to unusually high pre-activation of irrelevant language features.
We propose Sparse Autoencoder-guided Supervised Finetuning (SASFT), a novel method that addresses unexpected code-switching by teaching LLMs to maintain appropriate pre- activation values of irrelevant language features during training.
We conduct experiments across five models and six datasets, demonstrating that SASFT effectively reduces unexpected code-switching while maintaining multilingual capabilities.
2 PRELIMINARY
Code-switching reduction. Code-switching refers to the linguistic phenomenon of alternating be- tween two or more languages within a single text (Poplack) 1978 Kuwanto et al.; 2024 Winata et al., 2023). Recent studies of code-switching in LLMs (Zhang et al., 2023) Yong et al., 2023
Huzaifah et al. 2024 Winata et al. 2024 Wang et al. 2025b; Yoo et al. 2024 Li et al. (024) over- look important issue: unexpected code-switched content generated by LLMs can confuse users and hinder their comprehension. Therefore, we propose a new task - Code- Switching Reduction in LLMs. which aims to minimize unexpected code-switching while preserving the multilingual ca- pabilities of LLMs. Given a multilingual LLM L, an unexpected code-switching language I, and set of prompts X {x1, 22,... where responses should not contain language the goal of
2

---

We focus on the pre-activation value f(x rather than the feature activation axx). since a(x) only considers positive values and ignores negative pre-activation values that have meaningful negative projections along feature directions (Mayne et al. 2024). Following the notation of (Rajamanoharan Published as a conference paper at ICLR 2026
Code-Switching Reduction can be denoted as:
N
min E'
(CCS((, s.t. Dist(L, <e N
Σ 1=1
(I)
Here, the function CSW( 3) checks text y contains any content language Pr.- Pr.((x;) is the out- put when prompting 2; to LLM I,,, and I[() denotes indicator function. The function Dist L,, T.) measures the difference between the new LLM L and the original LLM L. We want to keep this difference small to make sure L stays similar to L. Since want to minimize unexpected code-switching while preserving the multilingual capabilities, we use the performance difference on multilingual benchmarks "distance".
Code-switching ratio. We define code-switching ratio evaluation metric to measure pected language switching in LLM L. The ratio can be calculated as:
E (I,
Existing tools cannot reliably detect fine-grained code-switching, such as single characters in an- other language (Burchell al., 2024). Thus, we use a script-based approach (see Appendix E.3
SAEs. Sparse Autoencoders (SAEs) are a special type of autoencoder (Hinton & Zemel 1993) They are used break down the hidden states of LLMs into a sparse linear combination of learned feature directions. Given residual stream E R in certain layer, the SAE calculates feature activation a RR where M >> N. It then uses to reconstruct the input as x. The typical reconstruction process is described by the following equations:
f(x) == Wenc e bene:
(3)
a axx) == ReLLUf(x)),
(4)
x(a) = a basc.
(5)
al.) 2024), we define the columns of W dec as for I...., M and refer these columns as "features"; which can be regarded as specific directions within the residual stream x.
3 FEASIBILITY STUDY
3.1
UNEXPECTED CODE- SWITCHWIG IN LLMs
Gemma-2-2b-k Gemma-2-9b-it
Qwen3-1.78 Qwen3-8B
CS Ratio (%)
Llama-3.1-98-Instruct
2
1
Arabic
Thai
English Original Language
French
Vietnamese
Portuguese
Figure 2:
The unexpected code-switching to Chinese for five LLMs in six languages. The results suggest that unexpected code-switching common issue in multilingual LLMs.
We intend to investigate whether there are unexpected code-switches to Chinese. To this end, we select queries whose ideal responses should be in a single language without Chinese from six multi- lingual benchmarks, and generate responses from Gemma-2 (Team et al. 2024), Ilama-3.1 (Meta) 2024]. and Qwen-3 YYagg al.) 2025). We then measure the unexpected code-switching ratio for Chinese according to Eq. (2). The results are shown in Figure 2 and observe that: (I) Unex- pected code-switching occurs various LLMs. (2) The ratio Thai and Arabic content switching to Chinese is higher than others. These findings suggest that unexpected code-switching is com-
mon issue multilingual LLMs across different languages, and needs to be addressed.
More details in Appendix D

---

Published as a conference paper at ICLR 2026
3.2 LANGUAGE-SPECIEL SAE FEATURES
Deng et al. 2025) revealed that LLMs possess language-specific features- dearetes-direetioos in the residual stream that have large projection values only when processing tokens from one particular language. Ablation studies show that removing these features notably impairs the model's performance in the corresponding language while having minimal impact on other languages. Motivated by this, we aim to use these language-speciff features to analyze the mechanism behind unexpected oode-switching.
3.3 UNEXPECTED CODE-SWITCHING IS RELATED TO LANGUAGE SPECIFIC SAE FEATURES
ANNUIII-SPEECIIIC
We aim to explore what causes unexpected code-switching. Inspired by (Deng et al. 2025 we propose that unexpected code-switching the target language might be due unexpectedly high pre-activation values of the target language feature.
3.3.
PRE-ACTIVATION PATTERN BEFORE CODE-SWITCHING
LJama-3. 1-88-Instruutt
Gemma-2-2b-i
Gemma-2-9b-it
Qwen3-1.78
Owen3-8B
Activation Value of Chinere Feuiture
40
40
1.0
20
0-
20
-05
-40
-2 Token Position
0
Token Position
Taken Position
Token Position
Taken Positiin
Figure 3: The average pre-activation values of the Chinese feature at different token positions across various LLMss. Position represents the first token that switches to Chinese. Before code-switehing occurs, the pre-activation values of the Chinese feature gradually increase.
We collect all the unexpected code-switching responses in Figure 2 and caleulate the average pre- activation values the Chinese feature for different positions near the first token that switches to Chinese, shown in Figure We observe that the token immediately preceding the first unexpected code-switching token shows higher pre-activation values of the Chinese feature compared to earlier tokens. This indicates that abnormally high pre-activation of features of another language may indicate an upcoming code-switch to that language.
3.3.2 ABLATING IRRELEVANT LANGUAGE FEATURE MITIGATES CODE SWITCHING
Llama-3.1-4 1-t.--3.--------turr Insttrcctt
Gemma-2-22 -2-b-itt
Gemma-2-9b-it
Qwee3-1..78
Qwen3-88
1.0
0.8
0.8
0.6
0.7
0.6
2h
25
0.4
2h
0.7
2h
6n
0.4
2h
100
200
100
200
10
Valuee of
Value of
Vallue of
Valtue of
Valuue A
Figure 4: The code-switching ratio Chinese after ablating Chinese or English features with dif-
ferent X. (1) Ablating the Chinese feature can reduce the unexpected code-switching ratio. (2) A higher coefficient À leads to better reduction in the unexpected code-switching ratio. (3) Ablating the English feature has little impact on the unexpected code-switching ratio to Chinese.
In Section 3.3.1 we show that unexpected code-switching might be related to high pre-activation values of language features. Here, we investigate how language features impact unexpected code- switching. Specifically, we use directional ablation (Ferrando et al. 2024, Arditi et al. 2024 to subtract the language feature from the residual stream x RN at the final layer of the token imme- diately preceding the first unexpected code-switching token. This process can be expressed as:
x - x Xd,
(6)
where d represents the language feature and À is the coefficient that controls the degree of ablation. After obtaining x', replace with x' and continue the forward pass of the LLMss. We report the
4

---

Published as a conference paper at ICLR 2026
Largest!
Ltraining Jeross-entroppy + Alvretuce
Uo U1 Us
Leross Leross-entrapy
Lyeduce ReLU (fs aj
Ho. Mo.Mi ... Hs
You Vi Ys
Output Probabilities
Mean Activation for Samples
Mean Activation for Samples from Other Languages
Softmax
( Chinese
Feature
Chinese Feature
SAE Encoder
N
Transformer Block
SAE Encoder
LLM
Input Embedding
Multilingual Corpus
Non-Chinese Input
Stanin:Fincing Language Features
Stage2 SASFT
Figure 5: SASFT operates in two steps: First, identifies language-specife features in LLMs (left), then leverages these features training signals reduce code-switching behavior (right).
code-switching ratio with different À in Figure 4 Our observations as follows: (1) Ablating the Chinese feature can reduce the unexpected code-switching ratio. (2) A higher coefficient À leads better reduction in the unexpected code-switching ratio. (3) Ablating English features has little impact on the unexpected code-switching ratio to Chinese. These results suggest that changing language-specific features can mitigate unexpected code-switching.
4 METHOD
SASFT first identifies language-specitn features in LLMs, and then uses these features as training
signals to reduce code-switching L.MMs, as shown in Figure []]
We first briefly review the process
of finding lanuuagesspccife features used in (Deng et al. 2025 in Section 4.1 and then introduce
SASFT for Code-Switching Reduction in Section
4.1 FINDING LANGUAGE- FEATURES
Deng et al. 2025 propose metric to measure the monolinguality feature. Given sets of residual where D, contains the residual streams from language i for a certain layer, they compute how differently feature activates for language L versus other languages. The
streams (Di...., ...... Dx
computation process is as follows:
(x(x),
DDz
xED
1 1
Σ
Σ ax(x), xED,
D (DLH
[Dr]
()ED)(D.)
3 1 7
where a, (x))) is the activation value of feature for residual stream x. We then calculate for all languages and features. For each language, we sort all features based on their values from highest lowest. The top-ranked features are identified sanguage-sppccfff features."
4.2 SASFT
In Section 3.3 we observe that reducing the pre- neeacctivtiion values of language-specit features dur- ing inference help reduce code-switching However, this approach has drawbacks: (1) To effec- tively reduce sode-switching, we must lower the pre-activation values of specific language features significantly. We believe this because specifie language features aren't just in the final layer; they
5

---

Published as a conference paper at ICLR 2026
appear in earlier layers too. Changing just the final layer does not affect features from previous lay- ers, so a big reduction is needed. But making large changes or modifying multiple layers can harm the model's other abilities (Deng et al. (025), making this method impractical. (2) This method re- quires external intervention and doesn't fundamentally change the model, leading to extra overhead and complexity during inference.
Considering the effectiveness of reducing the pre-activation values of specific language features and its drawbacks during inference, we propose method teach LLMs when lower the pre-activation values of these features during the training process. Specifically, we introduce an auxiliary loss during supervised fine-tuning (SFT) to ensure that LLMs keep the pre-activation values of specific language features below a certain threshold across several layers. Formally, consider a language I that we aim to avoid code-switching to. We have sets of residual streams DD = {Di..... Dx ). where each D; contains the residual streams from training data in language for specific layer. The auxiliary loss can be defined as follows:
Leeduce = EDDwD((DL)) EXND,
Σ LoESE
ReLU (f, - ag)
(8)
where (x) is the pre-activation values of feature 8 for the residual stream x. The set St. denotes the language-specific features for language L. For each feature in language j. we use aj represent its pre-estimated average pre-activation value. We don't set aj to zero because the pre-estimated average pre-activation value can be negative. In such cases, zero would be too large as a baseline value. Additionally, Dz is the set of residual streams for language I,, which we exclude because generating language from language L does not count as sode-switching.
For SASFT, we combine two losses to get the final training loss:
Ltraining Lecoss- entropy + XLreduue
where is hyperparameter we can adjust control how much Leeduce contributes to the total loss.
Another straightforward idea is to enhance the pre-activation values of original language features, which might reduce the ratio of code-switching from this language to others. However, our experi- ments in Appendix |F] show that this method is less effective than reducing the pre-activation values of unexpected language features. Therefore, we mainly focus on the "reducing" approach.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Training data. We study unexpected code-switching to Chinese, Korean, and Russian. Specifi- cally, we construct six SFT datasets using open-source data (see Appendix C for details). For each language (Chinese, Korean, and Russian), we create two datasets: a larger dataset with 210k sam- ples (100k English, 100k target language, 10k others) and smaller dataset with 110k samples (50k English, 50k target language, 10k others).
Models. We use base models for our experiment as they are suitable for further fine-tuning. Our study includes five models of different sizes and series: Gemma-2-2B, Gemma-2-9B (Team al. 2024), Llama-3.1-80 1--BB (Meta) 2024), Qwen3-1.7B- 1---1.7B-Base, and Qwen3-8B -BB-BBae (Yang et al.) 2025). For Gemma-2 models, we use SAEs from Gemma Scope (Lieberum al. 2024)). while for Llama- 3.1, we use SAEs from Llama Scope (He et al. 2024b). For Qwen3 models, we train our own set of SAEs on the residual stream of each layer.
Baselines. We compare our method with three baseline methods. (1) SFT: standard SFT ap-
proach trained with the cross-entropy loss. (2) SFT+GRPO: following the work Gf et al.) (2055) who use GRPO to handle unexpected code-switching in DeepSeck-RII (Guo et al., 2025), ), we apply GRPO (Shao et al. 2024) with language consistency reward on SFT-trained model. The lan- guage consistency reward is computed as the percentage of target language words in the model's output. We refer this baseline SFT+GRPO. (3) SFT+Penalty:
SFT variant that augments the cross-entropy objective with an additional penalty term that discourages tokens from a specified non-target language by reducing their predicted probabilities. Details can be found in Appendix B

---

Published as a conference paper at ICLR 2026
Implementation. We use identical hyperparameters for SFT and SASFT. For GRPO, we total of 10k samples, consisting of 1k samples for each of the 10 languages. Detailed hyperparameter settings can be found in Appendix E
Evaluation. Our evaluation focuses on two key aspects: (1) the code-switching ratio defined in Eq. 2 and (2) the model's performance on multilingual benchmarks. The code-switching ratio calculated using the same query set as described in Section [3.]] while the benchmarks include
the multilingual versions of MMLU (Hendrycks et al., (2,12.02). HumanEval (Peng et al. [024 Chen Team
al. (02T), Flores-200 (Goyal et 20.2, Team et al. [2022), HellaSwag (Zellers et al. 2019).
LogiQA (Liu et al. 2020). IFEval (Zhou et al. 2023), and MGSM (Shi et al. 2022 from pm-
meval (Zhang et al. 2024p.
5.2 MAIN RESULTS
Comparison of code-switching ratios (%) across different methods and models. For each target language (Chinese, Russian, and Korean), we train models on two dataset settings: a 210k dataset and a 10k dataset, then evaluate their code-switching ratio to Chinese, Russian, and Korean, Bold numbers indicate the best results. Results show SASFT consistently outperforms the baselines, achieving over 50% reduction in most cases.
Table 1:
Training Data 210k
Training Data 110k
Model
Method
CS: any ch
CS: any
CS: any
CS: any
4h
CS: any
CS: any ko
SFT (Baseline)
0.74
0.57
3.45
0.68
0.42
1.06
Remma-2-2B
SFT+GRPO
0.74 (0%)
0.49 (-14%)
3.44 (0%)
0.72 (45%)
0.36 (-16%)
1.08 (+2%)
SFT+Py
0.67 (-10%)
0.41 (-27%)
1.18 (-66%)
0.66 (-4%)
0.32 (-25%)
0.70 (-34%)
SASFT
0.42 (-43%)
0.22 (-61%)
0.73 (-79%)
0.32 (-53%)
0.16 (-62%)
0.34 (-68%)
SFT (Baseline)
0.78
0.12
0.81
0.80
0.05
0.50
Gemma-2-99
SFT+Ponalty
SF++GRPO
0.96 (+23%)
(0%)
0.16 (+32%)
0.11 (-9%)
0.63 (-22%)
(-14%)
0.78 (-3%)
(-2%)
0.04 (-14%)
0.07 (+43%%
0.48 (-5%)
0.29 (-42%)
SASFT
(-47%)
0.01 (-94%)
0.13 (-84%)
(-58%)
0.01 (-79%%
0.24 (-53%)
SET (Baseline)
1.16
0.67
0.57
0.48
0.79
0.37
SFT+GRPO
(-55%)
(-55%)
0.37 (-35%)
(-16%)
0.34 (-57%)
0.95 (+157%)
Llama 3.1 8B
SFT+Penalty
0.42
(-64%)
0.33 (-51%)
0.54 (-4%)
0.39 (-20%))
0.27 (-66%%
0.31 (-17%)
SASFT
(-76%)
0.28 (-59%)
0.46 (-19%)
%
0.22 (-72%)
0.20 (-46%)
SFT (Bascline)
0.81
0.19
0,36
0.68
0.19
0.23
Qwen3-1.7B. BBB-B-
SFT+GRPO
(.19%)
0.11 (-42%)
0.34 (-6%)
(0%)
0.19 (+11%)
0.20 (-16%)
SFT+Penalty
0.53 (-35%)
0.09 (-53%)
0.06 (-84%)
0.49 (-28%)
0.07 (-62%%
0.06 (-73%)
SASPT
(-72%) )
0.03 (.85%) %)
0.00 (-100%)
(-55 %)
0.03 (-87%) %)
0.02 (-93%)
SFT (Baseline)
0.96
0.16
0.43
0.83
0.17
0,25
SFT GRPO
(-14%)
0.09 (-40%)
0.22 (-27%)
(-26%)
0.06 (-65%)
0.12 (-20%)
Qwen3-8B-Bas
SFT Ponaltyy
0.70 (-27%)
0.12 (-24%)
0.23 (-47%)
0.76 (-9%)
0.08 (-50%)
0,18 (-27%)
SASPT
0.66 (-31%)
0.07 (-56%)
0.07 (-83%)
0.62 (-26 %c)
0.07 (-59%)
0.05 (-80%)
Code-switching ratio comparison: SASFT consistently reduces code-switching. We present the results for code-switching ratio to Chinese (zh), Russian (ru), and Korean (ko) in Table I and we observe that: (1) SASFT demonstrates superior performance reducing code-switching across all scenarios, with more than 50% reduction in 23 out of 30 cases compared to the SFT baseline. (2) SASFT consistently outperforms GRPO across different models and languages. While GRPO shows unstable results with both improvements and deteriorations (e.g., +157% for Llama-3.1-88 with Korean), SASFT maintains consistent reductions across all settings. (3) The effectiveness of SASFT is particularly evident in Qwen-3, while also showing significant improvements in other models like Gemma-2, demonstrating its general applicability across model scales. These results demonstrate that SASFT is robust and effective method for reducing unexpected code-switching in LLMs, consistently outperforming existing approaches while maintaining stability across different languages and model architectures.
Performance on multilingual benchmarks: SASFT preserves multilingual capabilities. We evaluate our method on six multilingual benchmarks to assess its impact on the multilingual capa- bilities of LLMs, as shown in Table The results demonstrate that: (1) SASFT generally maintains or slightly improves model performance across different benchmarks. For instance, Llama-3. 1-BB with SASFT shows notable improvements on several tasks, including MMMLU (+3.13), humaneval (+4.14), and hellaswag (+1.07) compared to the SFT baseline. (2) Even for models where slight performance decreases are observed, the degradation is minimal (usually within 1-2%), suggesting that SASFT effectively reduces code-switching while preserving the model's multilingual capabil-
7

---

Published as a conference paper at ICLR 2026
Performance comparison on six benchmarks across different methods. We evaluate models trained on the Chinese 110k dataset setting. Results demonstrate that SASFT successfully maintains model capabilities while reducing code-switching, even showing improvements in several cases, The red numbers indicate performance improvements compared to the SFT. More results are provided in Appendix
Table 2:
Model
Method
MMLU
HumanEval
Flores
HellaSwag
LogiQA
IFEval
MGSM
Acc (%)
Acc (%)
Bleu (%)
Acc (S)
Acc
Acc (%)
Acc (56
SFT
29.88
76.63
22.56
24.97
28.00
14.86
12.05
Gemma-2-22
SFT-GRPO
SPTt-Pealty
29.66 (-0.22)
30.81 (+0.93)
76.35 (-0.28)
80.62 (+3.99)
22 80 (+0.24)
22 (+0.31)
26.41 (+1.4)
(+1.94)
26.62 (-1,38)
(-0.62)
14.71 (-0.15)
(+0.42)
11.97 (-0.88)
10.99 (-1.66
SASFT
30.24 (+0.36)
79.09 (+2.46)
22.28 (-0.28)
24.75 (-0.22)
25.75 (-2.25)
15.18 (+0.52)
12.24 (-0.19)
44.31
95.62
30.59
32.95
21.61
44.61
Gemma-2-9B
SIT+GRPO
44.21 (-O.
95.72 (+0.10)
30.71 (+0.12)
33,86 (+0.91)
31.63 (:2.49)
21.80 (+0. 19)
45.88 (+1.23)
SFT-Penalyy
46.39 (+2.08)
97.02 (+1.40)
30.09 (-0.50)
32.37 (-0.58)
34.63 (-0.51)
21.26 (-0.35)
46.35 (+1.74)
SASPT
45.91 (+1.60)
95.67 (+0.05)
(-1.8)
32.18 (-0.77)
(+0.26)
22.44 (+0.83)
44.96 (+0.35)
SET
29.99
87.74
22.81
32.39
32,88
20.08
19:92
Llama-3.1-88
SFT-GRPO
SET Penalty
29.67 (-0.52)
29.70 (-0.29)
85.43 (-2.31)
85.58 (-2.16)
24.36 (+ (55.5
(-0.47)
28.63 (-3.76)
(-4.22)
30.37 (-2.51)
(40.66)
20.00 (-0.08)
18.91 (-1.17)
15.81 (-4.11)
(+2.91)
SASFT
33.12
91.88 (+4.14)
(+0.92)
33.46 (+1.07)
30.63 (:2.25)
19.85 (40.23)
18.35 (-1.5)
SET
37.47
90.29
23,70
33.53
32.38
20.27
32.91
Qwea3-1.78 (aan3-1.7BBBaat
SPT-GRPO
(+0.33)
90.48 (+0.19)
23.45 (.0.25)
35.74 (+2.211
31.37 (-1.01)
20.19 (-0.08)
32.67 (-0.24)
PenaPty
37.78 (+0.31)
89.13 (-L.1)
23.55 (-0.15)
(+2.71)
(-0.62)
20.44 (+0.
(+0.69)
SASPT
38.38 (+0.91)
(-1.25)
23.67 (-0.03)
33.71 (80.18)
32.38 (0.00)
20.22 (-0.05)
30.85 (-2.06)
SFT
52.15
95.87
29.99
42,48
42.25
33,64
58.03
Qwead-8B-Base
SFT-GRPO
50.85 (-1.30)
96.44 (70.57)
30.14 (+0.15)
44.48 (+2.00)
41.50 (40.75)
33.42 (-0.22)
58.28 (-2.75)
SPT Penalty
(0.74((1144)
94.71 (-1.16)
30. (+0.1
34.51 (-7.97)
39,88 (-2.77)
34.04 (+0.40)
56.29 (-1.74)
SASET
50.09
98.27 (+2.44)
29.97 (.0.22)
39.60 (-2.88)
(+0.50)
33.91 (+0.27)
58.45 (+0.42)
ities. These results indicate that our SASFT method effectively addresses the code-switching issue without substantially affecting the model's overall performance on multilingual tasks; in some cases, SASFT even improves performance.
5.3
IN-DEPTH ANALYSIS
Llama-3,1-88
Gemma-2-2b
Gamma-2-9b
Qwan3-1.78
Qwan3-88
2.5
0.8
Single
Single
Single
Single
0.8
1.5.
1.0
0.4
0.4
0.6
0.5
0.2
Layer (roverse order)
Layer (reverse order) Layer (reverse order)
Layer (reverse order)
Layer (reverse order)
Figure 6: Impact of layer selection on code-switching ratio across different models. Single-layer (solid lines) represents applying SASFT to individual layers, while Multi-layer (dashed lines) rep- resents applying SASFT to consecutive layers starting from the final layer. Layers are counted in reverse order (0 represents the final layer). Results show that multi-layer consistently achieves bet- ter and more stable performance than the single-layer approach, while the single-layer effectiveness decreases when moving towards earlier layers.
Effect of layers used in SASFT: multi-layer outperforms single-layer in redueing code- switching, We investigate how different layer selections (in reverse order from the final layer) affect SASFT's performance code-switching reduction, shown in Figure 6 The results demonstrate that: (1) Multi-layer SASFT consistently shows better performance than the single-layer approach across all models. This is particularly evident in Gemma-2 and Qwen3, where the multi-layer ap- proach (dashed lines) maintains lower code-switching ratios throughout different layer selections. (2) For single-layer SASFT, the performance generally deteriorates as we move towards earlier lay- ers, with the code-switching ratio showing an increasing trend across most models. (3) The impact of layer selection is more pronounced in single-layer interventions, showing higher variability in performance, while multi-layer approaches demonstrate more stable performance across different layer combinations, suggesting better robustness.
multi-Featurr outperforms single-feature in reducing code- switching, We examine how different feature selection strategies affect SASFT's performance in code-switching reduction, comparing fingle-feature versus multi-feature approaches across models,
Effect of features used in SASET:
8

---

Published as a conference paper at ICLR 2026
Cs Ratio to chinese
Gemma-2-2b
Qwen3-1.7b
CS Ratio to Russian
Gemma-2-2b
Qwen3-1.7b
0.75
0.8
0.3
0.50
0.6
0.2
0.10
0.25
0.4
0.05
0.00
0.2
Chinese Feature Index
Chinese Feature Index
Russian Feature Index
Russian Feature Index Single
Single
Multi
Single
Multi
Single
Multi
Multi
Figure 7: Impact of feature selection code-switching ratio across different models. Single-feature (solid lines) represents applying SASFT to individual features, while Multi-feature (dashed lines) represents applying SASFT to consecutive features starting from the rank-1 language feature. represents the rank-1 language feature. Results show that multi-feature intervention consistently achieves better and more stable performance than single-feature approach.
shown in Figure 7 We observe that: (1) Multi-feature SASFT consistently shows better per- formance than the single-feature approach for Chinese features, maintaining lower code-switching ratios with reduced variance. (2) The performance difference between Chinese and Russian features suggests language-dependeng effectivvnees, possibly due to models' stronger Chinese language ca- pabilities compared Russian. (3) Notably, the optimal code-switching reduction achieved when applying the multi-feature approach.
5.4 ABLATION STUDY
To validate the rationality of setting a; to pre-estimated average values rather than zero in Eq. (8) compare SASFT zero (a; 0) with SASFT in Table We observe that: (I) zen effectively reduces code-switching and shows comparable performance to SFT+Penalty on Gemma-2-2B, while achieving notably better results on Dwen3-1.7B-Baal (2) SASFT outperforms SASFT zero across most configurations, demonstrating that using pre-estimated average pre-activation values is more effective than simply setting them to zero.
Table 3: Comparison of code-switching ratios between different a, settings. Bold numbers indicate the best results while underlined numbers represent the second best. Both SASFT (a, = 0) and SASFT show effectiveness in reducing code-switching, with SASFT achieving better performance across different settings.
Model
Method
Training Data 210k
Training Data 110k
CS: any + zh
CS: any - mu
CS: any
ko
CS:any any -> sh
CS: any - n
CS: any
ko
SFT (Baseline)
0.74
0.57
3.45
0.68
0.42
1.06
SFT-GRPO
0.74 (0%)
0.49 (-14%)
3.44 (0%)
0.72 (+6%)
0.32 (:24%)
1.08 (+2%)
Gemma-2-28
SFT-Penalty
0.67 (-9%)
0.41 (-28%)
1.88 (-66%)
0.66 (-3%)
0.28 (-33%)
0.70 (-34%)
SASFT.o
0.58 (-22%)
0.90 (+58%)
1,99 (-42%)
0.49 (-28%)
0.36 (-14%)
0.58 (-45%)
SASPT
(-43%)
0.22 (-61
0.73 (-79%)
(-53%)
(-62%)
(-68%)
SFI (Baseline)
0.81
0.19
0.36
0.68
0.19
0.23
SFT+GRPO
0.66 (-19%)
0.11 (-42%)
0.34 (-6%)
0.68 (0%)
0.19 (0%)
0.20 (-13%)
Qwen3-1.7B 1B7B-Base
SFT-Penalty
0.53 (-35%)
0.09 (-53%)
0.06 (-83%)
0.49 (-28%)
0.07 (-63%)
0.06 (-74%)
SASFT
0.49 (-40%)
0.07 (-63%)
0.05 (-86%)
0.40 (-41%)
0.04 (-79%)
0.02 (-91%)
SASFT
0.22 (-73%)
0.03 (884%)
0.00 (-100%)
0.31 (-54%)
0.03 (844%)
0.02 (-91%)
RELATED WORKS
Code-switching refers to the linguistic phenomenon of alternating between two or more languages within single text (Poplack) 1978 Kuwanto al.) 2024] Winata et al. 2023)
Code-switching,
While recent studies make significant progress in processing code-switching content (Zhang et al. 2023 Yong et al. 2023) and leveraging code-switched data to enhance LLMs (Wang et al. 2025b
Yoo et al. 2024). they overlook critical issue: unexpected code-switched content generated by LLMs can significantly impair user comprehension. Guo et al.] (2025) first attempts to tackle this challenge by applying GRPO (Shao et al. 2024) with language consistency reward
SFT- trained model. Recently, WWang et al. (2025a); Nie et al. (2025) show that code-switching closely aligns with that of the model's 's internal representations.

---

Published as a conference paper at ICLR 2026
SAEs. SAEs serve powerful interpretability tool by decomposing a model's internal represen- tations into meaningful feature directions, enabling researchers to mechanistically explain various phenomena within LLMs (Bricken et al. 2023] Cunningham al., [023] Shi et al. 20255. Fer- rando et al. )024) employs SAEs to discover features indicating L.Ms' entity recognition, while Cunningham al. (2023) identifies features associated with apostrophes. [Galichin al. (2025);
Fang et al. (202)) use SAEs to identify and validate reasoning features in reasoning models like DeepSeck-RI (Guo et al., 2025). Particularly noteworthy is the work by Deng et al. (2025), which reveals that certain features are strongly correlated with specific languages, and ablating these fea- tures only impacts the model's performance in one language. Inspired by their findings on language- specific features, we employ SAEs to analyze unexpected code-switching behavior and solve it.
7 CONCLUSION
We focus on the issue of unexpected code-switching in multilingual LLMs. Through analysis with SAEs, we discover that unexpected code-switching is linked to unusually high pre-activation val- ues of irrelevant language features. Based on this finding, propose SASFT, novel approach that guides LLMs to maintain appropriate pre-activation values of language-specificc features during training. Extensive experiments on five different models demonstrate that SASFT effectively re- duces unexpected code-switching by more than 50% while maintaining or improving performance on various multilingual benchmarks. Our work provides practical solution for developing more reliable multilingual LLMs, contributing to the advancement of multilingual L.Mss.
ACKNOWLEDGEMENTS
This work was supported by Alibaba Group through Alibaba Research Intern Program.
REFERENCES
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver. BC, Canada, December 10 15, 2024, 2024. URL http: httpt:tpppppppaaaapppppppp nips www.nnipss.ppppeee cc/paper wille/paper//0// hhash// htmml
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-oodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https//transformer- httpp///0///mmoommatttffeftiiiie
Laurie Burchell, Alexandra Birch, Robert Thompson, and Kenneth Heafield. Code-switched lan- guage identification is harder than you think. In Proceedings of the 18th Conference of the Eu- ropean Chapter of the Association for Computational Linguistics (Volume I: Long Papers), pp-
646-658. Association for Computational Linguistics, 2024. doi: 10.18653w1120044ccalllooo
https: http:://wc..aamooolooooo.//////0
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemyy Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1416. 14168, 2021.
10

---

Published as a conference paper at ICLR 2026
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.0860/ 2023.
Boyi Deng, Yu Wan, Baosong Yang, Yidan Zhang, and Fuli Feng. Unveiling language-specific features in large language models via sparse autoencoders. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 4563- 4608, Vienna, Austria, July 2025, Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 10.18653/w 10t0l////222eeeee hwt.222222..-ooo.2222222
Yi Fang, Wenjie Wang, Mingfeng Xue, Boyi Deng, Fengli Xu, Dayiheng Liu, and Fuli Feng. Con- trollable Ilm reasoning via sparse autoencoder-based steering, arXiv preprint urXiv:2601.33595 2026.
Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, and Neel Nanda. Do i know this entity? knowledge awareness and hallucinations in language models. arXiv preprint a414422411.14257, 2024.
Andrey Galichin, Alexey Dontsov, Polina Druzhinina, Anton Razzhigaev, Oleg Y Rogov, Elena Tutubalina, and Ivan Oseledets. I have covered all the bases here: Interpreting reasoning features in large language models via sparse autoencoders. arXiv preprint arXiv:2503. 18878, 2025.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The flores- 101 evaluation benchmark for low-resource and multilingual machine translation. Trans. As- soc. Comput. Linguistics, 10:522 538, 2022. doi: 10.1162TACLI 00474. URL httpss
//doi. 1ttpg///..166/aaao.. __00444
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXii:240..21788 2024.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501.12948, 2025.
Yun He, Di Jin, Chaoqi Wang. Chloe Bi, Karishma Mandyam. Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, et al. Multi-if: Benchmarking Iims on multi-turn and multilingual instructions following. arXiv preprint arXiv:2410. 15553, 2024a.
Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, et al. Llama scope: Extracting millions of features from llama-3.1 1-bb with sparse autoencoders. arXiv preprint arXiv:2410.20522. 2024b.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- 3mm@
view.net, 2021. htt. //tp:::::////e/eeveeeeee eww net. feet:/formmidddd77e
Geoffrey E. Hinton and Richard S. Zemel. Autoencoders, minimum description length and helmholtz free energy. In Jack D. Cowan, Gerald Tesauro, and Joshua Alspector (eds.), Advances in Neural Information Processing Systems 6, [7th NIPS Conference. Denver, Colorado, USA, 1993), pp. 3-10. Morgan Kaufmann, 1993. URL http: httppesspppppppeeeeeeee wtw.:scpppcpccccpppp.ppppppppp 798 wwwaaauttcooddrrrreeeem www.miimmmmmmmmemeeeeemm wtwpeerrrriiioonneeneeeeeeeee-ee http::/ewii......oooooooooooaaaa
Kaiyu Huang, Fengran Mo, Xinyu Zhang, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, et al. A survey on large language models with multilingualism: Recent advances and new frontiers. arXiv preprint arXiv:2405. 10936, 2024.
Muhammad Huzaifah, Weihua Zheng, Nattapol Chanpaisit, and Kui Wu. Evaluating switessit translation with large language models. In Nicoletta Calzolari, Min- Min-Yen Kan, Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint In- fernational Conference Computational Linguistics, Language Resources and Evaluation,
11

---

Published as a conference paper at ICLR 2026
LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pp. 138816694 ELRA and ICCL, 2024. www.::22....mmmm....... 565
URL htttttts///caaaaae
Amir Hossein Kargaran, François Yvon, and Hinrich Schütze. GlotSeript: A resource and tool for low resource writing system identification. In Nicoletta Calzolari, Min- Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evalu- ation
2024), pp. 7774-7784, Torino, Italia, May 2024. ELRA and ICCL. URL org/
ht
main. 687.
Garry Kuwanto, Chaitanya Agarwal, Genta Indra Winata, and Derry Tanti Wijaya. Linguistics the- ory meets Ilm: Code-switched text generation via equivalence constrained large language models. arXiv preprint 2rXiv:2410.220660 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. 2024.
Jiahuan Li, Shujian Huang, Aarron Ching, Xinyu Dai, and Jiajun Chen. Prealign: Boosting cross-lingual transfer by early establishment of multilingual alignment. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-
16, 2024, pp. 10246-102577 Association for Computational Linguistics, 2024. URL https: //aclanccoology.orrr/22222 emn lammlppmet halpmmman.555 572
Tom Lieberum, Senthooran Rujamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05 147, 2024.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninh International Joint Conference on Artificial In telligence, IJCAI 2020, pp. 3622 3628. ijcai.org, 2020. doi: 10.2966/JCCAL22200550. URL https://doi .o../r///.. 2496344963222 jcai 2020/501
Harry Mayne, Yushi Yang, and Adam Mahdi. Can sparse autoencoders be used decompose and interpret steering vectors? In NewrIPS 2024 Workshop on Foundation Model Interventions, 2024.
net httprriwwwweeeefoooooeooooaee-eo
Meta. Introducing Llama 3:1: Our most capable models to date, 2024. URL URLhttps::/ //as meta.
com/ hoom/blog/metan 11attt---1aaaa31-11111
Meta. The llaaa herd: The beginning of new era of natively multimodal innovation, 2025.
URL https: //ap:::://a..e att.a//aaaeaaaaaaaaaaaaamaaaaaaa m com foom/blog/// 11 aallammaaa 4 mult Limodal-innt 11 igence/
Ercong Nie, Helmut Schmid, and Hinrich Schuetze. Mechanistic understanding and mitigation of language confusion in English-centric large language models. In Christos Christodoulopou- los, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for
Computational Linguistics: EMNLP 2025, pp. 690-706, Suzhou, China, November 2025. As- doi: 10.18653/w1/2025. Endings-emnlp..3 URL https: https: httt::://acaaatholoocoo org. 2025. www.:sfiiiiiiiigg..
sociation for Computational Linguistics. ISBN 779-8-89176-335-7
377
Qiwei Peng, Yekun Chai, and Xuhong Li. Humaneval-xl: A multilingual code generation bench- mark for cross-lingun natural language generalization. arXiv preprint arXiv:2402. 16694, 2024.
12

---

Published as a conference paper at ICLR 2026
Shana Poplack. Syntactic structure and social function of code-switching, volume 2. Centro de Estudios Puertoriquetios,(Cty University of New York], 1978.
Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip Yu. Multilingual large language model: A survey of resources, taxonomy and fron- tiers. arXiv preprint 0rXiv:2404.04923 2024.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407. 14435. 2024.
Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shiv- alika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A Haggag, Alfonso Amayuelas, et al. Include: Evaluating multilingual language understanding with regional knowledge. arXiv preprint arXiv:241 197999, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multi- lingual chain-of- thain-of-thought reasoners. arXiv preprint arXhv:2210.03057 2022.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das. and Jason Wei. Lan- guage models are multilingual chain-of-thougle reasoners. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, fR3wGCk
2023. URL https:tp//.c http:ttt////peeeeeeee./. net cotoommmmiii fannnndd-fffffce IXp
Wei Shi, Sihang Li, Tao Liang. Mingyang Wan, Guojun Ma, Xiang Wang, and Xiangnan He. Route sparse autoencoder to interpret large language models. In Christos Christodoulopou- los, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 6801-6815 Suzhou, China, November 2025, Association for Computational Linguistics. ISBN 979-8-89176-332. 6. doi: 18.18653/v 1/201/ 2e0.0055mmmlmmmain36666666 URL httph: /tle://acllaan //aaaaaaaaanlooyoo..o.....oo org/2025. emn p ma in.iii.... 3467]
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, al. Gemma 2: Improving open language models a practical size. arXiv preprint 2rXiv:2408.00/ 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
NLLB Team, Marta R. Costa-jussă, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shan- non Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left be- hind: Scaling cumaan-eenteed machine translation, 2022. URL httth:://aaaiit //tr/avvvvv/////////a//aa///.a.v org. abs/
22077 04672
13

---

Published as a conference paper at ICLR 2026
Mingyang Wang, Lukas Lange, Heike Adel, Yunpu Ma, Jannik Strötgen, and Hinrich Schuetze. Language mixing in reasoning language models: Patterns, impact, and internal causes. In Chris- tos Christodoulopoulls, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceed- ings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 2637- 2665, Suzhou, China, November 2025a. Association for Computational Linguistics. ISBN 979- 8-89176-332-6. doi: 10. 10.18653/w//2225.mmmll/ org/2025, emn main.132/
URL bettps://aclantholl.
Zhijun Wang, Jiahuan Li, Hao Zhou, Rongxiang Weng. Jingang Wang. Xin Huang, Xue Han, Junlan Feng, Chao Deng, and Shujian Huang. Investigating and scaling up code-switching for multilin- gual language model pre-training. arXiv preprint 2504:2500000801. 2025b.
Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. Polylm: An open source polyglot large language model. arXiv preprint arXiv:2307.06011 2023.
Genta Winata, Alham Fikri Aji, Zheng Xin Yong, and Thamar Solorio. The decades progress on code-switching research in NLP: A systematic survey on trends and challenges. In Anna Rogers, Jordan Boyd-Graber. and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics:: ACI. 2023, pp. 2936-2978 2978, Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 10. 1/20////////1122.2 org/2023. www.ffiininngiime htt.://../00.....nnnnnneneeee 18. httpt:ttt://caaatttttt
1857
Genta Indra Winata, Ruochen Zhang, and David Ifcoluwa Adelani. MINERS: multilingual language models as semantic retrievers. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12, 2024 pp. 2742 2766. Association for Computational Linguisties, 2024. URL hnttps://aclann
org/202442202 f findings-emnlp... emnlp. 155
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint arXiv:2412.15115 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint NXXv:25005.0938, 2025.
Zheng Xin Yong, Ruochen Zhang, Jessica Forde, Skyler Wang, Arjun Subramonian, Holy Love- nia, Samuel Cahyawijaya, Genta Winata, Lintang Sutawika, Jan Christian Blaise Cruz, Yin Lin Tan, Long Phan, Long Phan, Rowena Garcia, Thamar Solorio, and Alham Fikri Aji. Prompt- ing multilingual large language models to generate code-mixed texts: The case of south East Asian languages. In Genta Winata, Sudipta Kar, Marina Zhukova, Thamar Solorio, Mona Diab, Sunayana Sitaram, Monojit Choudhury, and Kalika Bali (eds.), Proceedings of the 6th Work- shop on Computational Approaches to Linguistic Pode-Switching, pp. 43 63, Singapore, Decem- ber 2023. Association for Computational Linguistics. URL ht..t://///// acc.acllllll hatpa/accaacoooog.....oooo org/ 2023. ca www.calllll.... .57
Haneul Yoo, Cheonbok Park, Sangdoo Yun, Alice Oh, and Hwaran Lee. Code-switching curriculum learning for multilingual transfer in llms. arXiv preprint arXiv:2411.0246.0 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Marquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2. 2019, Volume I: Long Papers, pp. 4791- 4800. Association for Computational Linguistics, 2019. doi: 10.18653/V1/P19-110 1472. URL http:///ooooog///.////o 11016/5///11199111
Ruochen Zhang, Samuel Cahyawijaya, Jan Christian Blaise Cruz, Genta Indra Winata, and Al- ham Fikri Aji. Multilingual large language models are not (yet) code-switchers. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference Empiri- cal Methods in Natural Language Processing, EMNLP 2023, Singapore. December 6-10, 2023, pp. 12567-12582. Association for Computational Linguisties, 2023. doi: 10.18653/V1/2023.
14

---

Published as a conference paper at ICLR 2026
MAI..74 774
URL httt:://b...rr/00006337/11....12
Yidan Zhang, Boyi Deng, Yu Wan, Baosong Yang. Haoran Wei, Fei Huang, Bowen Yu, Junyang Lin, Fei Huang, and Jingren Zhou. P-mmeval: A parallel multilingual multitask benchmark for consistent evaluation of llms. arXiv preprint wrXiv:2411.09111 2024.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL htRlhttt:: http:///ppeevvvveeeeeeee net. forum?ic=R) 8u222R11
Ziyi Zhao, Chongming Gao, Yang Zhang, Haoyan Liu, Weinan Gan, Huifeng Guo, Yong Liu, and Fuli Feng. Don't start over: A cost-effective framework for migrating personalized prompts between llms. arXiv preprint arXiv:2601. 2026.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followingg evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
A UNEXPECTED CODE-SWITCHING IN LLMS: GROWING CONCERN
The phenomenon unexpected code-switching, where language models abruptly switch between different languages during generation, has become increasingly prevalent in various open-source LLMs. This issue significantly impacts user experience and model reliability. For instance, multiple users have reported unexpected code-switching in models like DeepSeek and Qwen, particularly between English and Chinese
This phenomenon has been widely documented across different community platforms. For DeepSeck, users have reported the code-switching issuc both on GitHub, where the model occa- sionally switches to Chinese mid-conversationf| and Reddit, where multiple random switches to Chinese characters, particularly when generating longer r Similar issues have been observed with the Qwen model, where Reddit users reported unexpected Chinese outputs during other language interaction
experienced
B BASELINE DETAILS
We propose a more intuitive SFT baseline named SFT+Penalty. The main idea is to add penalty term the SFT loss that minimizes the probabilities of tokens from certain language.
Specifically, to reduce code-switching to a target language L (c.g., Chinese), we first identify all tokens belonging to language from the tokenizer, denoted as Vz. During training on samples in languages other than I., in addition to the standard cross-entropy loss LCE, we add a regularization term at each token position in the response to penalize the model's prediction probability for tokens in Vz. The training objective can be formulated as:
LTLP= = + 3
1
ΣΣ LEV
pe(ulz,Vec).
(10)
where T' is the response length, Po(u|æ, Vet) is the model's probability of predicting token e at response position and 3 the penalty coefficient.
experienced ox
httt:ps://ww...eddiit output_chinese
15

---

Published as a conference paper at ICLR 2026
C DETAILS OF SFT TRAINING DATA
We construct six SFT datasets using variety of open-source data, with the statistics summarized Table 4 and Table 5 Each dataset represents distinct setting in which we carefully control the total sample size and language composition. Specifically, in each configuration, the datasets include either approximately 210k or 110k samples, focusing on three target languages: Korean (ko), Russian (ru), and Chinese (zh).
Among the data sources, KULLM Tulu3 (Lambert et al., 2024). WildChat (Zhao et al. 2024). and BelleGroug each provide single-languaee samples: specifically, KULLM for Korean, Tulu3 for English, WildChat for Russian, and BelleGroup for Chinese. The remaining data, Multialpaca (Wei et al. 2023), Flores (Goyal et al. 2022), and GSM8KInstruet (Cobbe al. 2021 offer multilin- gual data, contributing samples across various languages.
Table 4: Number of samples of each language in different dataset settings. Each row shows the dis- tribution of samples across languages for different dataset sizes (either 210k or 110k). For Russian (ru), the sample size is approximate due to limited available data.
Samples per Language
Dataset
ko
zh
th
Fr
01
de
ja
id
TU
other
ko-210k
100000
100000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
2276
ko-110k
50000
50000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
2276
ru-210k
1000
86354
ru- 11kk
50000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
50000
2276
zh-210k
100000
1000
1000
100000
1000
1000
1000
1000
1000
1000
1000
1000
1000
2276
zh-110k
50000
1000
1000
50000
1000
1000
1000
1000
1000
1000
1000
1000
2276
Table 5: Number of samples from each source in different dataset settings. Each row shows the sample counts contributed by different data sources under various dataset sizes.
Samples per Source
Dataset
KULLM
Tulu3
Multialpaca
Flores
GSM8K
BelleGroup
WildChat
ko-210k
97834
97697
9774
4775
1250
985
961
ko- k00k
48892
48831
8829
3691
1087
985
961
ru-210k
984
97697
10823
4878
1628
985
82635
ru- 10k
984
48831
9549
3768
1296
985
47863
zh-210k
984
97697
7854
6088
1581
98111
961
zh-1 10k
984
48831
7854
4339
1266
49041
961
D DETAILS OF THE CODE-SWITCHING EVALUATION DATA
For the preliminary experiments presented in Figure use prompts in Arabic, Thai, English, French, Vietnamese, and Portuguese, totaling 34,996 examples. However, we observe that some of these languages exhibit relatively low code-switching ratios. Consequently, in our subsequent main experiments, we replace these low-ratio languages with alternatives that demonstrate more pronounced code-switching behavior.
For our main experiments, we use prompts from the multilingual versions of MMLU (Hendrycks) al. 2021). MGSM (Shi al. 2023), HellaSwag (Zellers
2019). LogiQA (Liu al. 2020). IFEal (Zhou et al., 2023 ) and Flores-200 (Goyal et al.) 2022: Team et al. 2022). all provided by
pmmeval (Zhang et al. 2024). In total, our evaluation set comprises 1,756 examples in Chinese (zh), 1,146 in Arabic (ar), and 1,150 examples each in Thai (th), Vietnamese (vi), Korean (ko), and
Japanese (ja).
16

---

Published as a conference paper at ICLR 2026
Table 6: Code-switching evaluation dataset: source-to-target language pairs and sample counts.
CS Target
Prompt Source
# Examples
# Runs
Total Samples
1,146
8
zh vi 1,150
1,150
27,568
8
ar
1,146
8
ru
th 1,150
8
27,568
ko
1,150
zh 1,756
ko
th 1,150
8
32,448
ja 1,150
We investigate code-switching behavior three target languages: Chinese (zh), Russian (ru), and Korean (ko). For each target language, we evaluate prompts from three different source languages. Table 6 presents the composition of our code-switching evaluation dataset, where each example is tested times to ensure robust detection of code-switching patterns.
E IMPLEMENTATION DETAILS
E. TRAINING
We the Hugging Face TRL library conjunction with DeepSpeed PersSpeedffr for SFT, and the combina- tion of TRL. and vLLM (Kwon et al. 2023 for GRPO.
For SFT and SASFT, both learning rate and in Eq. intervals, with the learning rate ranging from 1 x 10 to x 10-1 and X from x 10-5 to 10-2. For SASFT, all main experiment results obtained using the first language features from the last two layers. The following table summarizes the optimal hyperparameters and corresponding training times for SFT 110k samples for each model:
selected via grid search over respective
Table Optimal hyperparameters and SFT training time for 1 110k samples across different models.
Model
Learning Rate
À
SFT Training Time
Deepspeed Optimization Level
Gemma-2-25
5.0 10-5
5.0 x 10-1
lh
None
Gemma-2-9E
5.0 10-6
5.0 x 10-1
ZeRO2
Llama-3.1-88
5.0 10-5
1.0 x 10-3
3h
ZeRO1
Qwen3-1.7B
1.0 10
1.0 10
40min
None
Qwen3-8B
5.0 10 5
5.0 x 10
3.1h
ZeRO1
For all experiments, the batch size is set to 256, weight decay to 0.1, warmup steps to 100, and cosine learning rate scheduler is employed. AdamW (fused) serves as the optimizer, and training is performed using bf16 precision. Further, for SASFT, we select the last two layers and the first two features. All reported training times correspond nodes equipped with 8 NVIDIA A100 H20 GPUs; times may vary based on model size and hardware.
For GRPO, employ the TRL library in combination with vLLM, conducting grid search for the learning rate within the range 10 to 1 x 60-6, We use a batch size of 256 and set the number of rollouts to 8. The following table presents the optimal GRPO learning rates and corresponding training times:
17

---

Published as a conference paper at ICLR 2026
Table 8: Optimal GRPO learning rates and training times.
Model
GRPO Learning Rate GRPO Training Time
Gemma-2-2B
7.0 10-7
40 min
Gemma-2-99
7.0 x 10-s
3.5 h
Llama-3.1-8 1-8B
5.0 x 10
2
Qwen3-1.7h
1.0 10-7
35 min
Qwen3-8B
5.0 x 10-7
2 h
All GRPO experiments are performed under similar hardware configurations as SFT, utilizing NVIDIA A100 or H20 GPUs, with training duration depending on model size and hardware speci- fications.
E.2 INFERENCE
During inference, we the following decoding parameters:
top-p sampling: 0.8
repetition penalty: 1.0
temperature: 1.0
To reduce the inference time, we utilize the no-thinking mode for Qwen-3.
E.3 CODE-SWITCHING DETECTION
We GlotScript (Kargaran al. 2024) for code-switching detection. GlotScript identifies dif- ferent writing systems based on Unicode character ranges. We focus on Chinese, Russian, and Korean because their writing systems (Han, Cyrillic, and Hangul, respectively) are distinct from other scripts. This makes them easily distinguishable, unlike languages such as English and French that share the Latin alphabet and cannot be reliably separated based on script alone.
In our detection process, if Han characters appear in response that should not contain Chinese, we mark as unexpected code-switching to Chinese. The same rule applies to Cyrillic and Hangul characters for detecting unexpected code-switching to Russian and Korean, respectively.
F SASFT VARIANT
F.1 METHOD
Another idea is that enhancing the pre-activation values of original language features should be able to reduce the ratio of code-switching from this language to other languages. Therefore, we extend Eq. { to enhance the pre-activation values original language features, which can be defined as follows:
Lenhanse
ExewDsr
Σ L
ReLU (Bwr - f((x))
(11)
where M is the language intended for enhancement, and By is the pre-estimated average pre- activation values of feature language M. We call this variant as SASFTT Enhance. Ewhhnce.
F.2 EXPERIMENTS
In this section, we focus SASFT
SASFTEnhanee Enhance which enhance original language features using Eq.
II
Cude-Switching Ratio Comparison: Our Methods Effectively Reduce Code-Switching.
Ta-
ble 9 presents code-switching ratios from Arabic and Thai to Chinese, Russian, and Korean. We
18

---

Published as a conference paper at ICLR 2026
observe that SASFT Enhance generally reduces code- rode-switching compared to the SFT baseline, outper- forming GRPO in most cases (7 out of 12). Importantly, SASFTReduS achieves the lowest ratios in all settings, consistently providing the best results. Overall, both enhancement and reduction approaches are effective, with the reduction method showing superior performance.
Table 9: Evaluation of code-switching reduction for Arabic and Thai as enhanced source languages. Models are tested their tendency to switch from these source languages to Chinese, Russian, and Korean. Bold numbers indicate the best results while underlined numbers represent the second best in each column.
Enhunced Langugg:: ar
Enhanced Language:
Model
Method
CS: zh
CS:
CS: - ko
CS: th zh
CS: th
CS: ko
SFT (Baseline)
1.14
1.22
0.17
0.43
0.43
0.00 (0%)
Gemma-2-2B
SFT+GRPO
(-31%)
(-505)
(.47%)
(+121%)
(-60%)
(0%)
SASF Vndaance
1.31 (+15%)
0.70 (-43%)
0.00 (-100%
0.43 (018)
0.26 (-40%)
0.00 (0%)
SASFT Reducx
0.61 (-46%)
0.26 (-79
0.00 (-100%)
0.17 (-60%%
0.09 (-79 %
0.00 (0%)
(Baseline)
1.04
Qwen3-1.78 BBBB
SFI -CRP
0.61 (-41%)
0.26 (0%)
0.26 (0%)
0.53 (+51%)
0.09 (-30%8)
0.09 (0%)
SASFT Enkaaoc
(-75%)
0.17 (-35%)
0.09 (-65%)
0.44 (+26%)
0.17 (-6%)
(0%)
STS Redusx
0.17 (-84%)
0,00 (-100% %
0.00 (-100%)
0.26 (-26%)
0.00 (-10(%)10%
0.00 (-100%)
G LIMITATIONS AND FUTURE WORK
Our study has several limitations that we plan to address future work: First, we only explore un- expected code-switching Chinese, Russian, and Korean. Adding more languages would make the study more complete. Second, while we experiment with 5 LLMs from 3 model families of different sizes, all models are under 9B. Testing on larger models would provide more comprehensive un- derstanding of our method's effectiveness. Third, theoretically. our method only requires constraints on the model's hidden states, so should be possible extend to other fine-tuning approaches like DPO and GRPO. We believe this is promising direction for future research. Finally, although using pre-estimated average pre-activation values as thresholds works well in our experiments, finding a fine-gratie fine-grained token-level threshold could potentially improve performance further.
H LLM USAGE STATEMENT
In this work, LLMs are utilized as general-pure purpose assist tools for programming and writing. Specif- ically, L.MSs assist in code generation and debugging, checking for grammatical errors, and refining the language of the manuscript. No novel research ideas, analyses, or conclusions are contributed by LLMs.
I EXTENDED PERFORMANCE COMPARISONS
This section provides additional results comparing model performance across six benchmarks under alternative settings, as shown in Tables 10 to 14 We include detailed comparisons among different methods support our findings the main text. The results further demonstrate that SASFT ef- fectively maintains model capabilities while reducing code-switching, and in several cases, achieves improved performance relative to SFT. These additional experiments validate the robustness and consistency of our conclusions.
19

---

Published as a conference paper at ICLR 2026
Table 10: Performance comparison on six benchmarks across different methods. We evaluate mod- els trained on the Korean 110k dataset setting. The red numbers indicate performance improvements compared to the SFT.
Model
Method
MMLU
HumanEval
Flores
HellaSwag
LogiQA
IFEval
MGSM
Acc (%)
Acc (16
Bleu (58)
Acc (%)
Ace (%)
Ace (%)
Acc (%)
SFT
27.56
77.60
18.39
26.43
26.00
15:85
11.9
Gemma-2222
SPT+Ptnalty
27.82 (+0.26)
26.77 (-0.79)
77.07 (-0.53)
(-1.35)
18.49
18.48 (40.09)
22.12 (1)
26.25 (40.25)
(+0.87)
16.26 (4041)
(+0
12:77 (+0.88)
SASFT
26.68 (-0.88)
75.29 (-2.31)
17.96
22.01 (-4.42)
26.25 (10.25)
15.81 (-0.44)
11.31 (-0.58)
SET
47.56
96.63
29.23
34.52
30.87
24.54
48.67
Gemma-2-9B
SPT+Penalty
SPT+GRPO
46.06 (-0.900
47.47 (-0.09
9%.96 (-0.67)
96.35 (-0
29 (44):45)
29.18 (-0.05)
34.38 14)
33.33
29.25 (-1.62)
33.75 (+2
25 (41).60)
24 (-042)
46, (-2.30)
50.88 (+2.21)
SASFT
46.85 (-0.71)
94,62 (-2.01)
(-1.04)
33,60 (-0.92)
29.12 (-1.75)
25.24 (+0.70)
47, (-1.55)
SFT
32.07
24.73
Ll-ma-3.1-8B
SPT+GRPO
27.73 (-4.34)
77.16 (2.98)
20.75 (-3.98)
25.68 (-1.92)
30.63 (-1.62)
17.75 (-4.24)
9.44 (-6.48)
SPT+Penalty
32.30 (+0.23)
89.18 (-0.96)
24.68 (-0.05)
29.20 (+) (+1.50)
30.63 (-1.62)
22.03 (+0.01)
18.27 (+2.45)
SASET
32.40
87,55 (-2.59)
24.05 (-0.68)
30,20 (x2.60)
32,00 (-0.25)
20.95 (-1.03)
13.49 (-2.43)
SPT
22.49
31.00
Qwen3-1.78- Basc
SPT+GRPO
37.47 (+0.60)
88.32 (+2.41)
23.04 (+0.55)
34.14 (+1.64)
31.52 (+0.62)
19.3 (+0.33)
32.03 (0.00)
SPT Penaiiy
37.94 (-0.13)
87.02 (- (-1,11)
22.58 (60.09)
33,53 (+-1.03)
34.38 38)
19:17 (40. 19)
33.31 (+1.28)
SASET
37.49 (-0.58)
(-4.448)
22,97
34.15 (+1.65)
(40.14)
(+0.85)
SFT
49:67
97.74
22.86
34.52
39,00
35.92
59:47
Qwen3-SBB Bann3-SB-Base
SPE+GRPO
45.27 (-4.40)
96.3 (-1.39)
24.81 (+1 995)
22.53 (-11, (99)
39.12 (40.
34.47 (-1 45)
55.23 (-4.24)
SFT+Penalty
47.68 (-1.99)
95,10 (-2.64)
26.12 (+3.26) 26)
30.33 (-4,19)
38.12 (-0.88)
35.52 (-0.10)
60.72 (+1.25)
94,90 (+3.21)
18.96 (-3.90)
(++.66))
(+2.50)
(-1.03)
(+2.45)
Table 11: Performance comparison six benchmarks across different methods. We evaluate mod- els trained on the Korean 210k dataset setting. The red numbers indicate performance improvements compared to SFT.
Model
Method
MMLU
HumanEvvl
Floces
HellaSwag
LogiOA
IFEval
MGSM
Acc (%)
Ace (%)
Bleu (%)
Acc (%)
Acc (%)
Ace (%)
Acc (9)
SFT
25.96
75.87
19.31
19.97
24.62
16.24
14,00
Gemma-2.20
SFT+GRPO
25.98 (+0.02)
78.22 (+2.35)
19.35 (+0.04)
19.55 (-0.42)
25.25 (+0.63)
16.27 (+0.03)
13.63 (-0.37)
SFT+Fenalty
26.58 (40.62)
79.76 (+3.89)
15.45 (-3.86)
22.09 (+2. 12)
29.12 (+1.50)
16.66 (+0-12)
(-0.24)
SASET
(+1.21)
(-0.44)
(-0.97)
(+2.11)
(+0.63)
(+0.19)
(+0.08)
SET
50.14
92.02
29.15
42.09
33.88
23.89
49.60
Gemma- 2-9B
SET&GRPO
49.21 (-0.93)
91 11.54 (-0)48)
28.68 (-0.47)
42.31 (+0.22)
32. 12 (-1.76)
23 (-4.20)
$3.44 (+3 (+3.84)
SFT+Penalty
(+0.24)
(+0.83)
SASET
49.33 (-0.81)
92.69 (+0.67)
28.87 (-0.28)
40.13 (-1.9))
34.75 (+0.87)
23,60 (-0.29)
50.88 (+1.25)
SET
34.98
$9.57
23.68
33.72
28.50
22.29
22,67
Llama- 3.1.a-3.1.60
GPPOO
Penalty
35.15 (10.17)
(10.28)
89.2
(-0.72)
23.79 (60.11)
(-0.24)
$1.38 (-2.34)
(1 (.34)
31.00 (v2.55) 20)
(v2.25)
22 (10. 32)
(+0.22)
22 29.69 (+0.02)
(+4.77)
SASFT
35.27 (40.29)
86 83 (-2.74)
23.25 (-0:45)
$3.01 (-0.71)
33.50 (45 000)
23 (-1.26)
25.09 (+2.42)
SFT
37.02
85,
22.40
31.73
31.25
20.19
36.69
Qwen3-1.778 1..8---1.7--Bag
Penalyy
SFT+GRPO
36.57 (-0.45)
13 (-0.97)
(+0.09)
22.50 (+0.10)
35.41 (+3.68)
(+2.74)
33.00 (+1.75)
21 (+0.82)
(-0.12)
36.72 (+0.03)
36,93 (+0.24)
SASFT
37.36 (40.34)
X5. 19 (+0.09)
22,55 (+0.11)
31,17 (-D.56)
31.001 (-0.25))))
20.14 (4U05)
37.12 (+0.43)
SFT
26.37
34.78
65.71
Qwen
SFT+GRPO
SET S
50.80 (+1.16)
48.19 (-1.45)
96. 15 (-0.73)
97.7 (+0.86)
24.83 (.1.54)
27.07 (+0.70)
40.32 (+2.92)
34.87 (-2.53)
40,50 (2)
41.38 (+2.00)
34.18 (40.60)
(+1.16)
64. (-1.58)
61.87 (-3.84)
SASET
51.36 (+1.72)
95.77 (-1.11)
21,80 (-4.57)
45.68 (+8.25)
42.88 (+3.50)
35.27 (+0.19)
63.92 (-1.79)
20

---

Published as a conference paper at ICLR 2026
Table 12: Performance comparison six benchmarks across different methods. We evaluate mod- els trained on the Russian 11k dataset setting. The red numbers indicate performance improvements compared to SFT.
Model
Method
MMLU
HumanEval
Flores
HellaSwag
LogiOA
IFEval
MGSM
Acc (%)
Acc (%)
Bleu (%)
Acc (%)
Acc
Acc (%)
(9)
SET
24.74
82.45
23.25
17:35
24.87
11.71
Gemma: 1aamma-2222 2:2B
SPT&GRPO
25.14 (10.40)
83.85 (+1.40)
23.54 (+0.29)
14.58 (-2.77)
27.25 (+2.38)
16.81 (60.16)
10.80 (-0.91)
SFT+Poually
26.77 (+2.03)
(+2.65)
22.18 (-1.07)
19.65 (+2.30
29,87 (+5.00)
16.81 (40.16)
12.11 (+0.40)
SASFT
26.01 (+1.27)
80.96
23.31 (+9.06)
19.24 (+1.89)
(+0.63)
(-0.39)
(-0.75)
SET
42.97
94.23
31,82
33.84
33.38
23.62
44.48
Gemma: 2-98
GRPO
42,92 (-0.05)
93 89 (-0.34)
31,85 (-0.27)
36.08 (12.24)
31,75 (-1 (.)
23.57 (40.25)
43.63 (-0.85)
SFT+Ponalty
(+2.21)
30.32 (-1.50)
SASFT
40.76 (-2.21)
96.68 (+2/45)
31.31 (-0.51)
29.86 (-3.98)
31.87 (-1.51)
22.23 (-1.39)
44,40 (-0.08)
SET
29.96
92.40
21.45
23.71
29.38
19.59
15.76
1.1na-3.1-83
Penalty
SPT&GRPO
29.84 (-0.12)
(+3.92)
91 (-0.91)
89.23 (-3.17)
21,82 (+0.33)
(+4.04)
21,80 (-1.91)
(+6.73)
29 29.62 (10 24)
29.75 (+0.37)
19.19 (-0.40)
(+0.65)
15.15 (-0.61)
SASPT
32.06 (+2.10)
92.98 (+0.50 58)
23.52 (+2.007)
29.53 (+5.83) 82)
32.88 (+3.550)
20.37 (40.78)
17 (+1 (+1.68)
SFT
37.22
90,00
23.46
35.53
32:25
19 9.88
33.25
Qwen3-1 (wen3-11.B--Raee
PPT+Peeallty
SFT+GRPO
37.77
37.47 (+0.25)
90.72 (+0.72)
90.05 (+0.05)
23.68 (+0.22)
32.79 (:2.74)
31.63 (-0.62)
20.54 (+0.76)
20 20
33.47 (+0.22)
32.69
SASFT
38.20 (10.98)
91.11 (+1.11)
(+1.00
34,92 (-D.61)
33,62 (x1.37)
19 (10.06)
32.43 (-0.82)
SFT
47.21
94.13
25.77
35,42
41.38
50.03
Qweni-88--BB
SFT+GRPO
45.04 (-2.17)
94.33 20)
26.86 (+1 (9)
03 (-7.39)
40.62 (-0.76)
29 1.30)
48. (-1.28)
SET+Penally
45.73 (-1.48)
95.00 (+0.87)
26.89 (+1.12)
28.35 (-7.07)
41.00 (-0.38)
30,80 (-0.12)
50.03 (0.00)
SASFT
50.28 (+-3007
88. (-5.24)
26.95 (=1.18)
38.47 (+3.05)
41,50 (+3.12)
32.35 (+1.43)
$3.89 (+3.86)
Table 13: Performance comparison six benchmarks across different methods. We evaluate mod- els trained on the Russian 210k dataset setting. The red numbers indicate performance improvements compared to SFT.
Model
Method
MMLU
HumanEval
Flores
HellaSwag
LogiOA
IFEval
MGSM
Acc (%)
Acc (%)
Blcu (%)
Acc (%)
Acc
Ace (%)
(%)
SET
23.19
17.22
Gemma-2-2B
SFT+GRPO
28.04 (-0.32)
88 65 (6)
23.35 (+0.16)
25.63 (40 79)
29.38 (-2.12)
17.17 (-0.05)
14.08 (-0.75)
SFT Pennlyy
28.32 (-0.04)
86 88 (-0.81)
25.06.--0.1)) (-0.13)
25.34 (40.50)
27.00 (8)
17.08 (-1.14)
13.84 (-D-99)
(-0.27)
85.46 (+0.77)
23.25 (+0.06)
(+1.83)
(-1.75)
13.44 (-1.99)
SFT
45.55
96.78
30.51
35.44
33.75
20.82
50.05
Gemma-2-99
GGP+
44.99 (-0.56)
96-92 (+0.14)
30.65 (+0.14)
36.64 (+1.20)
33.25 (-0.50)
22.76 (-0.06)
50.75 (+0.70)
Pennlly
44.51
96
30.67 (40.166
(40.67)
(+1 .25
(40) 36)
SASFT
43.55 (-2.00)
94.81 (-1.97)
22.93 (-7.58)
32,71 (-2.73)
31.87 (-1.88)
21.83 (-0.99)
49.79 (-0.26)
SFT
33.97
93.94
23.24
29.72
30.25
21.24
14.51
3.1-8
SFT+Ptnalty
SET+GRRPO
33.29 (-0.68)
33.71 (-0.26)
94.37 (+0.43)
(+2.45)
23:47 (+0.23)
(+0.76)
31,76 (+2.04)
(+1.59)
28.38 (-1
(+1.75)
20
13.25 (-1.26)
14.35 (-0.16)
SASET
34.53 (10.56)
96.59 (+2.65)
23.24 (0.00)
29.87 (10.15)
30.75 (10.50)
21.70 (10.46)
18.88 (+4.37)
SFT
38.06
93.75
23.76
33.65
31.75
20.72
35.04
Qwen3.11.B.BBan
SFT+Penalty
SFT+GOROO
37.88 (-0.18)
38.38 (+0.32)
94.47 (+0.72)
92.
23,29 (-0.17)
33.55 (+0.10)
(+1
36.00 (+1.25)
(-0.75)
20.37 (-0.35)
20
34,99 (-0.05)
34,80 (-0.43)
SASET
38.23 (+0.17)
93. (-0.63)
23.14 (-0.62)
33.96 (+0.31)
32.38 (10.63)
21.14 (+0.42)
34,53 (-0.51)
SPT
50.73
96
28.31
38.99
43.12
35.08
60.27
Owen3-8B-Ban
SFTAGRM)
48.63 (-2.10)
95. (-1.30)
28,40 (+0.
34.01 (-4.98)
43.12 (0,00)
33.98
57.87 (-2.40)
SFT+Penalty
51.56 (+0.83)
95.72 (-0.72)
28.66 (+0.35)
40.60 (+1 61)
42.62. (-0.50)
34.82 (-0.26)
55 (-1.99)
SASFT
52.11 (+1.55)
95.22 (-1.20)
26.69 (-1
44.83 (+5 84)
42.62. (-0.50)
35.74 (40.66)
58. (-2.08)
21

---

Published as a conference paper at ICLR 2026
Table 14: Performance comparison six benchmarks across different methods. We evaluate mod- els trained on the Chinese 210k dataset setting. The red numbers indicate performance improve- ments compared to SFT.
Model
Method
MMLU
HumanEval
Flores
HellaSwag
LogiQA
IFEval
MGSM
Acc (%K
(56
Bleu
Acc (%)
Acc
(%)
1emma-2.2B
SET+GRPO
28.99 (40.41)
90.87 (-0.38)
23.25 (-0.45)
28.50 (4)
25.75 (-3.75)
(40.4
14,80 19)
SFT+Penalty
28.50 (40.22)
90.77 (-0-48)
(30.26-))2)
27.85 (+0.38)
26.00 (-3.50)
15.94 (+0.29)
15.44 (=0.83)
SASFT
(-0.72)
(+1.50)
(-1.38))
SET
45.77
93:70
29.37
33.92
31.63
24.58
49.63
Gemma-2-98
SETRGRPO
Penaliy
45.39 (-0.38)
46,22 (40.45)
94-09 (40.39)
91.73 (-1.97)
29.33 (-0.04)
79.22 (-D.15)
36.22 (42. 50)
34.78 (10.86)
32.38 (10.75)
29:12 (-2.51)
24.24 (-1.34)
23 (-0.74)
48.99 (-0.64)
48 (-0-91)
SASFT
47.04 (+1.27)
92.50 (-1,20)
28.79 (-0.58)
34.11 (40. 19)
33.13 (+1.50)
25 (40.92)
$0.29 (+0.66)
SET
31.53
91.35
22.70
88
30.00
21.28
16.13
Llama-3.11 1-18B
SFT+GRPO
Penalty
30.35
(-1.18)
84)
89.33 (-2.02)
22,42 (-0.28)
2.39)
29.93 (+1.05)
(10.91)
30.62 (+0.62)
21.22 (-0.06)
13.65 (-2.48)
(+3.10)
SASFT
33.37 (4)
95. 38 (+4. (3)
24.68 (+1.98)
33.80 (+4.92)
31.62 (+1.62)
23.01 (+1.73)
20.56 (+4.43)
SET
37.27
93. 22
23.59
32.30
34.001
20.53
32,48
Qwen3-1.77 1.en3-1,,B-Baee
SET Penalty
36.99
37.76 (10.49)
92 69 (-0.53)
23.21 (-0.38)
(+0.09)
35.66 33.66)
31.38 (-2.65 (2))
21 (+0.54)
34.51 (+2.03)
SASET
38.10 (+0.83)
92.12 (-1,10)
23.56 (+0.03)
34,20 (+1.90)
33.50 (-0.50)
20.93 (+0.40)
33.01 (+0.53)
SFT
$5.09
Qwen3 -B-ease
GRROO
(-4.68)
(-0.53)
(+0.61)
(-6.6)
42. (-0.38)
(-0.94)
50.75 (-4.34)
SFT+Pomllty
48.64 (-0.89)
96 83 (0.00)
30.75 (+0.55)
33.42 (+1 84)
40.58 (-1 (2)
35 36.66 (+0.99)
57.25 (+2.16)
SASET
49.60 (+0.07)
96.92 (+0.09)
30.81 (+0.61)
37.18 (+5.60)
43.38 (+0.88)
33,90 (-0.77)
51.95 (:3.14)
22

---

Published as a conference paper at ICLR 2026
J ROBUSTNESS OF LANGUAGE FEATURES ACROSS SAE CONFIGURATIONS
To investigate the robustness of language-specifie features different SAE hyperparameters, conduct experiments using SAEs with varying sparsity (I0) and dimensionality (width) from Gemma Scope (Lieberum al. 2024). Specifically, we examine six different SAE settings: 10_38 width_16k, 10.34 width 65k, 10_73 width 16k, 10_63 width 65k, 10 158 width 16k, and 10_ 10_122 width 65k.
For each SAE configuration, we identify the rank #0 language-specific feature for Chinese and Korean using the method described in Section 4.1. We then compute the pairwise cosine similarity between these features across different SAE configurations for layers through 25. The results are visualized as heatmaps in Figures 21
Our findings demonstrate that language- -peecfii features exhibit remarkable consistency across dif- ferent SAE hyperparameters. For both Chinese and Korean, the cosine similarities between rank #0 features from different SAE configurations typically exceed 0.85, with many similarities above 0.90. This high degree of similarity persists across all examined layers (19-25), indicating that:
Language-spest specific features robust to variations in SAE sparsity targets (10 values rang- ing from 34 to 158)
Feature identification stable across different SAE dimensionalities (16k vs. 65k width)
The consistent patterns across multiple layers suggest that language features are fundamen- tal properties captured by SAEs regardless of specific training configurations
These results provide strong evidence that our language feature identification method is reliable and that SASFT's effectiveness is not critically dependent on specific SAE hyperparameter choices.
Similarity (Layer
Feature Similarity (Layer 19)
L,004
3
88300
6.646
6/644
E.655
1.3006
33222
E:6611
0.8822
6,531
wid
0.798
1.09
0.1044
0.8775
03379
Defferent SACs
0. 1.9
1.119
0.091
0:10:00
0,111
666
Differun
47/0
89244
11104
0.811
0.3111
5.5597
2.10.4
0.502
CA
0,0007
5902
8177
63333
0.832
3,009
0,004
83337
0.0000
0.5345
3,0003
8:46
8,73
80:00
8,30
10.155
$:40
8,73
43
8,127
11.15
Differant SAC
Different SACs
Figure& 8: Similarity of rank #0 Chinese features across SAE configurations at layer 19.
Figure 9: Similarity of rank #0 Korean features across SAE configurations at layer 19.
23

---

Published as a conference paper at ICLR 2026
Rank #o Chinese Feature Similarity Across SAEs (Layer
Rank 40 Korean Feature Similarity Across SAEs (Layer 20)
with
with CEC
6:644
Different SACa
1.804
00301
1.413
18031
1:49.22
14.99
1:14:1
Offirennt
web 10.00
0,3002
3.704
8.522
64064
1,0300
6.602
WIPLESO WRIT
39325
0.5533
6:0042
1.0000
6.4335
AMEE 3Rc
50
Figure 10: Similarity of rank #0 Chinese fea-
Figure 11: Similarity of rank #0 Korean features across SAE configurations at layer 20.
tures across SAE configurations layer 20.
Rank #0 Chinese Feature Similarity Across SABs (Layer 21]
Rank do Korean Feature Similarity Across SAES (Layer 21)
3.0
30
mot.
keh.00
DU0T3
1.300
83771
F3441
8.007
1.33
5.1432
F18.00
Different SAAE
Different
0.0033
3,341
83322
2.000
6.32%
6.779
0,204
33322
8.5445
2,0006
0.5452
6.621
WAR
62
WAR OIEE
0J7772
83300
6S775
0.3244
2.000
3350
E5533
0.0144
0,5527
Different
Differant SAEs
Figure 12: Similarity rank Chinese fea- tures across SAE configurations at layer 21.
Figure 13: Similarity of rank #0 Korean features across SAE configurations at layer 21.
Similarity (Layer
Similarity (Layer
1,000
25335
E3727
0U9931
E3533
1,3000
5432
E5725
C/6005
0,553
E3537
Different SACx
WER.CO
0.399
7.99
8.99
6.44
0.000
0.371
1.990
6.9425
0,599
E.541
Diffoentt
201
with 00L
88400
3,0000
0.8230
33300
3,0000
0.524
0.0633
88510
6.2006
E.033
31111
6.5244
1.6000
THE
183U
11522
11602
10 30k
8.75 Scc
1.116
WER
30
widh Difffront SAES
8.31
U/
12 116
Different SAEs
Figure 14: Similarity of rank #0 Chinese fea- tures across SAE configurations at layer 22.
Figure 15: Similarity of rank #0 Korean features across SAE configurations at layer 22.
24

---

Published as a conference paper at ICLR 2026
Rank #o Chinese Feature Similarity Across SAEs (Layer
Rank 40 Korean Feature Similarity Across SAEs (Layer
with
KERL with ECC
6.8223
833M
Different SACs
18801
5.111
1.0381
03201
1999
11031
1:4410
1:0033
Offirent
web 10.00
0.333
0.7504
2,000
6.700
1,0300
E.227
WRI
0.9944
2.0000
6.0005
1.0000
6:071
13
wirk 3Rc
AME 3ec
NET 1/
50
Figure 16: Similarity of rank #0 Chinese fea- tures across SAE configurations layer 23.
Figure 17: Similarity of rank #0 Korean features across SAE configurations at layer 23.
Rank #0 Chinese Feature Similarity Across SABs (Layer 24)
Rank do Korean Feature Similarity Across SAES (Layer 24)
3.0
30
meh.
Keb.00
0.904
1.300
19811
E300
0,492
1.330
53411
0.199
Different SAAE
Different
0.923
3.347
8.333
2.000
6,6530
65513
0,003
33331
8.500
2,0006
0.674
6G45L
WAR
89011
62
WAR CIEE
0.9033
3304
83231
65516
0.3300
2.000
E5231
0.4523
0.5957
Different
Differant SAEs
Figure 18: Similarity rank Chinese fea- tures across SAE configurations at layer 24.
Figure 19: Similarity of rank #0 Korean features across SAE configurations at layer 24.
Similarity (Layer
Similarity (Layer
1,000
3.933
FP227
E2000
0,683
E:735
1,3000
1214
E3331
0.132
C/243
Different SAts
WERLO0
0.017
1.90
1.89%
8:602
0.9990
8:102
0.310
57994
1.993
0.841
EX517
Difiment
with 02C
0,3009
1332
3,0000
0.8355
0.003
33770
3,0000
0.7931
0.9655
8890
6.2446
E.5075
84833
6.7603
1.6000
0.745
4.95
6922
1.311
0.122:
11532
6-1.11
11602
KISE I6c
338
13.33
28
20
widh
40
136
13
Different SAES
Dilferont SAES
Figure 20: Similarity of rank #0 Chinese fea- tures across SAE configurations at layer 25.
Figure 21: Similarity of rank #0 Korean features across SAE configurations at layer 25.
25

---

Published as a conference paper at ICLR 2026
K CAUSAL EVIDENCE: ENHANCING LANGUAGE FEATURES INDUCES
CODE-SWITCHING
Gemma-2-2b-it
Gemma-2-9b-it
Qwen3-1.78
Qwen3-88
0.8
zh
0.20
4h
th
0.8
zh
an
0.15
0.30
0.4
0.4
0.2
0.09
0.0
0.00
0.0
0.0
Value
Value of
Value of
Value
Value of
Figure 22: Code-switching ratio to Chinese after enhancing Chinese English features with dif- ferent À values. (1) Enhancing the Chinese feature can induce unexpected code-switching. (2) A higher coefficient leads higher code-switching ratio. (3) Enhancing the English feature has little impact on the code-switching ratio to Chinese
To establish causal relationship between -speuacie-seeeccceeeee feature activation and code-switching, we conduct the inverse experiment of Section 3.3.2. While ablation demonstrates that reducing language feature activation decreases code-switching, we now test whether artificially increasing the activation target language feature can induce code-switching. Specifically, we use directional enhancement to add the language feature to the residual stream E RN at the final layer of randomly selected token. This process can expressed as:
x' x Xd,
(12)
where d represents the language feature and À is the coefficient that controls the degree of enhance- ment. After obtaining x', we replace x with x' and continue the forward pass of the LLMs. We test this on 100 samples that originally contained code-switching Chinese and report the code- switching ratio to Chinsese with different À in Figure 22 Our observations are as follows: (1) En- hancing the Chinese feature induces unexpected code-switching across all models. (2) higher co- efficient leads to higher code- seilessitthiing ratios. (3) Enhancing English features has minimal impact on code-switching behavior. These results, combined with our ablation experiments, provide bidi- rectional causal evidence: artificially manipulating language-specific feature activations can both induce and suppress code-switching behavior, strongly supporting our hypothesis that language- specific feature activation causally determines language selection in LLM generation.
26


Reviews:
[{"review_id": "tf6cXBY8qM", "reviewer": "Reviewer_dtNW", "summary": "The paper studies unexpected code-switching in multilingual LLMs and links the phenomenon to unusually high pre-activation on language-specific SAE features. Building on this observation, it proposes SASFT: add an auxiliary loss during SFT that penalizes pre-activation of irrelevant language features across several layers.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear story from SAE analysis to a concrete training modification.\n2.  The results are good to show the proposed method's effectiveness.\n3. The paper is well-writen and easy to follow.", "weaknesses": "1. GRPO is run with only 10k samples (1k per language). That seems light for a control-behavior objective. Consider stronger RL baselines , or simple supervised baselines that directly penalize language-ID tokens . Without stronger baselines, it’s hard to attribute gains purely to SASFT. Does the author could ensure the GRPO have true convergence?\n2. The study focuses on zh/ru/ko. It’s unclear if results hold for low-resource scripts (e.g., Amharic, Khmer), closely-related Latin languages where CS is subtler (es/pt/fr/it)? \n3. SASFT relies on high-quality SAEs and on accurate language-feature identification; for Qwen the authors train their own SAEs, for others they reuse published ones. How sensitive are results to (1)SAE training corpora, dimensionality, sparsity target? (2)Which layer(s) the features are extracted from? (3) Feature drift after fine-tuning (do features stay monosemantic)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "82WiIoR9Cn", "reviewer": "Reviewer_fhn1", "summary": "This paper addresses the problem of unexpected code-switching in multilingual large language models (LLMs). The authors first employ Sparse Autoencoders (SAEs) to analyze the internal representations of LLMs and discover that unexpected language switches are correlated with over-activation of target-language-specific features. Based on this finding, they propose a novel fine-tuning method, SASFT (Sparse Autoencoder-guided Supervised Finetuning), which introduces an auxiliary loss term during supervised fine-tuning to constrain the pre-activation values of specific language features below a threshold, thereby reducing activations of irrelevant languages. Experiments on five models and three target languages (Chinese, Russian, and Korean) show an average >50% reduction in unexpected switching, including four cases of complete elimination, while maintaining or improving performance on six multilingual benchmarks.\n\nMain Contributions:\n- Using sparse autoencoders, the paper reveals that when a model is about to unexpectedly switch to language L, the language-specific features of L show significantly elevated pre-activation values in the residual stream.\n- Proposes SASFT, a training-stage approach that suppresses the activation of irrelevant language features without requiring inference-time intervention.\n- Provides comprehensive experiments demonstrating that SASFT is effective, robust, and preserves multilingual capabilities.", "questions": "1. Could the authors provide a more comprehensive comparison to strengthen the empirical validity of SASFT?\n2. The paper suggests that increased pre-activation of language-specific features precedes unexpected code-switching. Have the authors examined whether artificially increasing the activation of a non-target language feature can induce code-switching?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a Sparse Autoencoder-guided Supervised Finetuning (SASFT) approach that combines sparse autoencoders with supervised fine-tuning. \n2. The effectiveness of SASFT is validated through extensive experiments across multiple language pairs and model families. The results demonstrate consistent mitigation of code-switching phenomena in diverse multilingual settings.\n3. Experimental evidence indicates that the proposed method effectively reduces unintended language switches, thereby improving the accuracy, consistency, and usability of multilingual model outputs.", "weaknesses": "1. Limited Baseline Comparison：The paper only compares SASFT with GRPO, which, although relevant, is insufficient to establish the method’s relative advantage.\n2. Lack of Mechanistic or Causal Analysis：While the paper empirically observes that the pre-activation values of target-language features increase prior to code-switching and validates this via directional ablation, this evidence remains correlational. The work does not provide a mechanistic explanation of why such activation leads to language switching, nor tests the causal hypothesis.\n3. Potential Model-Specific Bias. The reported improvements may partially stem from inherent differences in model multilingual balance, rather than from SASFT’s general efficacy. The paper does not control for or analyze how such model-specific language priors influence the observed reductions in code-switching.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5uvkS4klTl", "reviewer": "Reviewer_iweR", "summary": "This paper proposes a novel method, SASFT, to mitigate the issue of unexpected code-switching in large language models (LLMs). The method is grounded in Sparse Autoencoders (SAEs), which are used to identify language-specific feature. An auxiliary loss is introduced during supervised fine-tuning to suppress pre-activation values of irrelevant language features, thereby reducing unexpected code-switching. The method is evaluated on five multilingual LLMs across six benchmarks and three code-switching target languages, and shows strong reduction in code-switching rates, while mostly maintaining or improving performance on standard benchmarks.", "questions": "1. How does the proposed method perform on multilingual mathematical reasoning tasks, such as MGSM？", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses an underexplored yet practically important problem, unexpected code-switching, which impacts user experience and model usability.\n\n2. Demonstrates consistent reductions in code-switching across various models and languages, outperforming previous methods (e.g., GRPO) in most settings.\n\n3. The paper offers detailed analysis on factors such as layer depth and feature selection.\n\n4. The paper is clearly written and easy to follow.", "weaknesses": "1. The method for identifying language-specific features relies on rankings without justification. This introduces sensitivity to hyperparameter selection and limits the interpretability of the results, especially in multilingual settings where features may vary across tasks.\n\n2. The paper reports a substantial +327% increase in Korean code-switching under the GRPO method, but does not provide sufficient explanation for this anomaly. A deeper analysis is needed to clarify the cause of such a drastic change.\n\n3. It remains unclear why SASFT underperforms on certain benchmarks, such as MMLU and HellaSwag, for the Qwen3-8B model. This raises questions about robustness and generalizability across tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y0iFGJbzJ3", "reviewer": "Reviewer_aWvq", "summary": "This paper addresses the problem of unexpected code-switching in multilingual Large Language Models (LLMs). The authors first conduct a mechanistic analysis using sparse autoencoders, identifying that the issue is caused by the excessive pre-activation of language features. Based on this finding, they propose Sparse Autoencoder-guided Supervised Finetuning (SASFT), a novel method that trains LLMs to control these pre-activation values. Experimental results on five models and three languages show that SASFT reduces unexpected code-switching by over 50% (eliminating it entirely in four cases) while maintaining or even improving performance on standard multilingual benchmarks.", "questions": "1.how were these code-swithing issues for testing are constructed or prompted? Overall, it seems that this issue occurs relatively infrequently and is difficult to reproduce, especially across different models. Such cases seldom happen, making it challenging to trace and resolve the problem on a large scale.\n\n2.Page 4 Line 184， how many unexpected code-switching responses are collected？\n\n3.Have you analyzed for different languages which are prone to code-switch with each other?\n\n4.On which layers were the main supervised SFT experiments conducted? Regarding the time efficiency of autoencoder computation, does using more layers lead to higher computational efficiency, and how was this trade-off balanced in the ablation studies?\n\n5.Why compare with RL (GRPO), it is more likely to make alignment during the RL phase, not for multi-lingual extension.\n\n6.According to Table2, the proposed method does not show a significant improvement in effectiveness.\n\n7.Figure 6, pls give detailed description for multi-layer, like the specific number of layers and which layer.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This work's key strength lies in its mechanistic, root-cause analysis of the code-switching problem, moving beyond superficial fixes. By using sparse autoencoders, it identifies the core issue: excessive pre-activation of language features.\nThe proposed solution, SASFT, is highly effective, consistently reducing unexpected code-switching by over 50% and even eliminating it entirely in several cases. Crucially, it achieves this without compromising performance, as it maintains or even improves the models' capabilities on standard multilingual benchmarks.\nIn short, its main strengths are its diagnostic depth, highly effective solution, and ability to fix the problem without sacrificing general multilingual proficiency.\nThe paper is well-written and easy to follow.", "weaknesses": "1.Limited scope of evaluation: The solution is only tested on three languages and five models. Its effectiveness across a wider range of languages, especially low-resource ones, remains unverified.\n\n2.Uncertain generalizability: The method's performance is demonstrated on \"six multilingual benchmarks,\" but it is unclear if it generalizes well to other critical tasks like reasoning, complex translation, or creative writing.\n\n3.Computational overhead: The approach relies on sparse autoencoders, which likely introduce significant computational cost and complexity compared to standard fine-tuning, a trade-off not mentioned.\n\n4.Lack of comparative baselines: While it outperforms standard supervised fine-tuning, it is not compared against other specialized techniques aimed at reducing code-switching, making its relative advancement unclear,like the works listed in related works.\n\n5.Superficial treatment of intentional Code-Switching: The method focuses on \"unexpected\" code-switching but may risk suppressing intentional and culturally appropriate code-switching (e.g., in bilingual communities), potentially reducing linguistic flexibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)