When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners
Authors: Weixiang Zhao
Jiahe Guo
Yang Deng
Tongtong Wu
Wenxuan Zhang
Yulin Hu
Xingyu Sui
Yanyan Zhao
Wanxiang Che
Bing Qin
Tat-Seng Chua
Ting Liu
Venue: NeurIPS 2025 spotlight ()
Keywords: Multilingual Reasoning
Large Language Models
Cross-Lingual Generalization
Primary area: applications

Abstract: Multilingual reasoning remains a significant challenge for large language models (LLMs), with performance disproportionately favoring high-resource languages. Drawing inspiration from cognitive neuroscience, which suggests that human reasoning functions largely independently of language processing, we hypothesize that LLMs similarly encode reasoning and language as separable components that can be disentangled to enhance multilingual reasoning. To evaluate this, we perform a causal intervention by ablating language-specific representations at inference time. Experiments on 10 open-weight LLMs spanning 11 typologically diverse languages show that this language-specific ablation consistently boosts multilingual reasoning performance. Layer-wise analyses further confirm that language and reasoning representations can be effectively disentangled throughout the model, yielding improved multilingual reasoning capabilities, while preserving top-layer language features remains essential for maintaining linguistic fidelity. Compared to post-training methods such as supervised fine-tuning or reinforcement learning, our training-free language-reasoning disentanglement achieves comparable or superior results with minimal computational overhead. These findings shed light on the internal mechanisms underlying multilingual reasoning in LLMs and suggest a lightweight and interpretable strategy for improving cross-lingual generalization.

When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners
Weixiang "hao" Guo Yang Deng", Tongtong Wu', Wenxuan Zhang', Yulin Hu! Xingyu Sui', Yanyan Zhao't, Wanxiang Che', Bing Qin', Tat-Seng Chua Ting Liu' 'Harbin Institute of Technology. "Singapore Management University, 'Monash University *Singapore University of Technology and Design, NNationa University of Singapore (wxzhao, jhguo, yyzhao) ) 8 i r. hit. edu. cn
Abstract
Multilingual reasoning remains a significant challenge for large language models (LLMs), with performance disproportionately favoring high-resource languages. Drawing inspiration from cognitive neuroscience, which suggests that human reasoning functions largely independently of language processing, we hypothe- size that LLMs similarly encode reasoning and language as separable compo- nents that can be disentangled to enhance multilingual reasoning. To evaluate this, we perform causal intervention by ablating language-specificc represen- tations at inference time. Experiments on 10 open-weight LLMs spanning 11 typologically diverse languages show that this language-specifec ablation consis- tently boosts multilingual reasoning performance. Layer-wise analyses further confirm that language and reasoning representations can be effectively disentan- gled throughout the model, yielding improved multilingual reasoning capabili- ties, while preserving top-layer language features remains essential for maintain- ing linguistic fidelity. Compared to post-training methods such supervised fine- tuning or reinforcement learning, our training-free language-reasoning dis- entanglement achieves comparable
superior results with minimal computa- tional overhead. These findings shed light on the internal mechanisms underly- ing multilingual reasoning in LLMs and suggest lightweight and interpretable strategy for improving cross-lingual generalization. Our code is available at: https//gtthbbbomm/mmteeeeePPllge www.aggguaaeeeasnnnnnnnniiiieeee
Introduction
Recent advances in reasoning large language models as exemplified by OpenAl's o1/03/04 mod- els [Jacch et al., 2024 OpenAl (225) and the DeepSeck-RI series [Guo et al. 2025] have signifi- cantly ad advanced the capacity of language models to handle complex reasoning tasks. These improve- ments stem from multi-stage post-training pipelines, notably supervised fine-tuning tonittunng and large-scale
reinforcement learning with bussoning-enbancer objectives [Guo al., 2025 Kimi al., 2025
Qwen 2025 Muennighoff et al.) 2025 Yeo et al. 2025). As result, these models demonstrate
increasingly human-like like deliberative abilities and can generate extended chains of thought (CoT) to
support long-horizon reasoning [Li et al. 2025a Xu et al. 2025 Chen et al. 2025a
Despite the remarkable advancements of LLMS' reasoning capabilities, progress has been heavily concentrated high-resource languages such as English and Chinese [Glazer et al.] 2024 Phan
Equal contribution
Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

100
LLM
90
MMSIU
8
Reusoning
NORR
5-80%%
17.%
70
SS
INT R4458
60
(Dientanglemeut
-21133 Hi-> 85
First,
Secondry,
30
Quea S.s. 18
Qwen 7B Thinking
Owne LB- Thinking
Qwen sap Sa--------iiiitit Thinking
Binnill Quen-78
M--Disill-LLAn Diiiinl-LLLMMaaa
H--DDutll-ga (weuull--wa--l
GLM-ZI-99
QW12-128
Hello -Hlllo
Thus....
Figure I: Overview of our hypothesis and findings. Left: Motivated by cognitive neuroscience, we hypothesize that reasoning and language processing in LLMs can be disentangled. Right: To val- idate this hypothesis, we perform a causal intervention by removing language-specific components from hidden representations. We evaluate this intervention across 10 open-weight LLMs and mul- tilingual reasoning benchmarks. Results show consistent performance improvements, supporting our claim that disentangling language and reasoning enhances multilingual generalization.
While these models exhibit strong performance these dominant languages, they continue to face challenges in reasoning tasks across low- and mid-resource languages (Karim et al., 2025| Gao et al. 2025]. This growing disparity multilingual reasoning capabilities raises critical concerns: it hinders the global applicability of LLMs, exacerbates existing linguistic inequities in Al access, and perpetuates the marginalization of underrepresented languages due to limited data and investment (Okolo and Tano 2024, Ghosh et al. 2025 Qin et al. 2025] Despite the significance of this issue, the multilingual reasoning gap remains largely under-examined current research.
al., 2025
In this work, we investigate the interplay between language and reasoning in LLMs, aiming to un- cover the key factors that influence cross-lingual reasoning generalization. Our key hypothesis is that reasoning and language processing can be explicitly disentangled, allowing reasoning abili- ties-once acquired in high-resource language-to transfer more broadly across languages. This idea is supported by cognitive neuroscience findings that the human brain's 's language network responsible for comprehension and production remains largely inactive during reasoning tasks [Monti et al.] [2009] [012]Amalric Ammlric and Dehaene 2019]. and that human language itself is evolution- arily optimized for communication rather than for reasoning Feddrrenoo al., 2024].
To further validate this hypothesis, we perform causal interventions within LLMs' latent spaces during multilingual reasoming by subtracting language-speccfic components from each hidden state ($3). This intervention aims to disentangle linguistic features from the underlying reasoning pro- cess. Notably, previewed in Figure [ we observe consistent performance gains in multilingual reasoning across diverse set of tasks, including mathematical problem solving (Shi et al.) 2023), commonsense inference MMuenighhoff et al., 2023). and knowledge-intensing question answering [Heddrycks et al.) 2021). These improvements are robust across all 10 open-weight LLMs, en- compassing both -aronnnn-orreented across 11 languages of varying resource availability. This consistent pattern provides compelling empirical evidence that reasoning and linguistic processing can be disentangled, enabling the effec- tive transfer of reasoning capabilities from high-resource to low-resource languages.
oriented and ceneral-purpose LLMs. Moreover, the benefits generalize
To further assess the impact of our intervention, we examine the correlation between the intensity of language-specifie components in hidden states and reasoning performance ($4.J). Our analysis reveals that stronger language-specifie signals tend correlate with lower reasoning accuracy, im- plying that excessive linguistic information may disrupt reasoning processes. We further perform layer-wise ablation study pinpoint where language and reasoning are most intertwined ($4.2). We find that language reasoning decoupling improves performance across almost all layers, but inter- ventions the upper layers significantly degrade output fidelity, indicating that language-specific signals in later layers are crucial for maintaining language-specific generation. In contrast, low and middle layers provide the best trade-off between reasoning gains and linguistic coherence. To further evaluate the practical value of these findings, we compare our training-free intervention with stan-

---

dard multilingual post-training methods, including both supervised fine-tuning and reinforcement learning ($4.3). Our training-free intervention- based approach achieves comparable or even supe- rior performance. This suggests that structural disentanglement of language and reasoning can serve as a lightweight and effective alternative, or complement, to post-training, opening up promising directions for future cross-lingual model enhancement.
2 Disentangle Language and Reasoning in the Activation Space
In 2.1 we identify language-specifie subspaces by isolating components that consistently encode linguistic variation across inputs. In $2.2 we introduce projection- intervention that removes these components during inference to disentangle language-specific information from the reasoning process. Finally, in $2.3 we provide empirical evidence that the removed components indeed corre- spond to language-specific signals, validating the effectiveness of our approach and the plausibility of language-reasoning disentanglement in practice.
2.1 Language-Specific Subspace Identification
Assuming the backbone model processes reasoning inputs from different languages, we compute a mean representation for each language at every layer as follows:
e
(1)
Here, ej E Rd denotes the embedding of the final token from the i-th sample in language 1, and is the total number of samples for that language. By concatenating the mean vectors my for all L languages along the column axis, we construct mean embedding matrix M Rdxt that characterizes the multilingual latent space.
Building on prior works Pires et al. 2019 Libovický et al. 2020 Yang et al. 2021 the multilin-
gual latent space M can be decomposed into orthogonal components: (1) language-agnostie
subspace M. , that encodes language-dependent variations linguistic expression. Following the formulation of Piratla et al. 2220), Xie et al. [2022] Liu et al. [2025], the decomposition objective is:
subspace M, a that captures cross-lingual shared semantics, and (2) a language-specine
min M.
M M. T M.T
s.t. Span (M.) 1 Span (M,),
where M. Rdx11 M, Rdxr, and T Lxr represents the language-specific coefficients along the r basis directions of M, The lower dimensionality of M, is justified by the observation that shared semantic content across languages often structurally simpler. In contrast, M, typically requires higher dimensionality to capture the rich diversity of language-specifie features.
The optimal solution to Equation 2 can be efficiently obtained via Singular Value Decomposition Finally, the identified
(SVD), with the detailed procedure provided in Algorithm [ in Appendix C language-speciff subspace M, & serves as the foundation for our subsequent intervention aimed at signals from the model's 's reasoning representations.
disentangling
2.2 Activation Ablation for -anguage-Reasoning Disentanglement
Given that M, characterizes the language-specifie subspace within the model's activation space, leverage a ablation mechanism to disentangle anguage-specitns signals from multilingual reasoning. Specifically, for any hidden representation derived from multilingual input over a specific rea- soning task, we project out its components along the subspace spanned by M.:
h A M,h
(3)
where À is the coefficient to control the ablation strength. This effectively removes language- specific variations, allowing the remaining representation h to better reflect language-agnostic rea- soning processes. By default, this projection applied throughout all the model layers, focusing on the final input token representation.
3

---

2.3 Verifying the Linguistic Nature of Removed Components
To assess the effectiveness of the projection-based intervention (Equation 3) in disentangling language-specific information from the model's 's reasoning process, we conduct empirical validation from two complementary perspectives: (I) representation space visualization, and (2) language fidelity between input and output on a specific reasoning query. We perform this validation on Qwen 5 7B Instruct, covering languages different resource levels: French and Japanese (high-resource), Thai (medium-resourcee) and Swahili (low-resource).
First, shown in Figure 2 visualize the hidden representations of the final token across different languages, before and after applying the language-specifie ablation defined in Equa- tion After removing components along the language-specific subspace M., the language clustering effect is reduced. More notably, rep- resentations of non-English languages exhibit clear tendency shift toward the English cluster, while English representations remain largely unaffected. This convergence results in tighter cross-lingual clustering and reveals transition toward more language-invarin resentations centered around English. Interest- ingly, although Qwen is pretrained primarily on both Chinese and English, we observe that even Chinese representations (the green cluster) tend to converge toward English rather than forming definitive expla-
Zb
Jp
En (Ablation)
Zh (AHutte))
Fr (Abbativ))
Jp (tatatio))
(Abante
-to
-5
rep-
-30
Figure 2: PCA visualization of final-token hidden states before and after projection in Qwen-2.5- 7B- In-Instt (middle layer). Post-ablation, non- English languages converge toward English, indi- cating increased language invariance.
their own central anchor. nation likely relates to the distributional domi- nance of English in the pretraining data, though the exact ratios remain unknown. This phenomenon warrants further investigation in future work. Additional visualizations for other model layers and model families are provided in Appendix D
To further examine how this alignment in representation space affects the model's behavior, we evaluate the fudeity [Holtermann et al. 2024], , a metric that quantifies consistency between the input and output languages. We use GlotLID Kargaran et al.] [023], multilingual language identifier supporting over 1,600 languages, to detect the language of model-generated responses. As shown in Figure 3 increasing the degree of ablation (i.e., projecting out more components of M,) leads to a substantial drop in language fidelity across all languages: models increasingly default to English in their responses, even when prompted in another language. This behavioral shift mirrors the representational alignment observed in Figure [ confirming two key findings: (i) the removed components do encode language-specife signals, and (ii) once removed, the model tends to revert to its dominant pretraining language, i.e. English, as the default output anchor, consistent with prior work multilingual representation aligment [Chen al.) 2023 Wu al. 2024 Wang al. 2024
Dumas et al.) 2024 Wendler et al. 2024 Chen et al. 2024 Zhao et al. 2024a
3 Causal Intervention within the Activation Space
In this section, we perform causal interventions on the hidden states of LLMs during multilingual reasoning tasks, with the goal of empirically validating our central hypothesis that language and reasoning are functionally separable within LLMs.
Models We conduct intervention experiments across a diverse set of LLMss, covering both reason- ing and non-reasonigg models. Specifically, for non-reasoning (instruction-tunr -tnnedd models, we in- clude the 1wwn-22.5Inntrree (1.7B, 4B, and 8B) [Tem 2025 For reasoning-oriente
family (3B and 7B) (Yang et al., (0244) and the Qwen-3-Instruct series models, we study DeepSeek-R1-Dinn (7B and 14B) Guo et al. 2025]. GLM--Z19B [GL.M et al. 2024), and QwQ-32B [Qwen] 2025 Notably, for the Qwen-3 series, we employ the Th nk ng mode to enable explicit reasoning.

---

Table 1: Multilingual reasoning performance on MGSM datasets across different languages, before and after language-reasoning disentanglement within the activation spaces of the backbone models (+ L-R Disentangle). The best results are highlighted in bold. The values in parentheses indicate language fidelity to indicate input-output consistency.
High-Resource
Mid- Resourcs Te
Low-Resource Bn
AVG.
En
Es
Fr
Jp
0wen-2.5-Insteet-BB + L-R Disentangle
86.0
75.2
70.8
70.0
68.8
59.6
60.8
61.6
10.0
37.2
12.4
56.36 (90.33%)
85.6
76.8
72.0
72.0
72.4
61.2
73.6
64.8
10.8
39.6
14.8
58.51 (91.20%)
Qwen-2.5 2.5-Instruet-77
92.4
82.8
78.8
77.2
82.8
72.4
81.2
79.2
36.8
67.6
16.8
69.82 (79.75%)
L-R Disentangle
92.8
84.0
79.6
80.4
84.4
73.6
82.8
79.2
37.2
64.4
20.0
70.76 (84.44%)
91.6
84.4
78.8
76.8
79.6
73.6
76.4
76.4
38.8
62.4
10.4
68. 11 (27.78%)
+ L-R Disentangle
91.6
85.2
79.2
79.2
83.2
75.2
79.2
78.8
41.2
64.8
8.8
69.67 (27.64%)
Qwen-3-4B-1h 1oon---4--Thinkimg
95.6
88.8
79.2
83.2 2.
88.0
81.2
85.6
85.2
72.4
68.88
32.0
79.56 (27.30))
+ L-R Disentangle
96.0
89.2
82.0
83.2
87.6
83.6
85.2
84.8
73.6
82.8
32.4
80.04 (27.31 %)
Owen-3-8B- Thinkkng
96.4
88.0
82.4
2.
87.22
85.2
89.2
90.0
80.8
88.0
84.15 (27. (27.24%%
+ L.R Disentangle
96.0
90.4
84.4
83.6
89.2
87.6
89.2
90.4
81.2
87.2
60.4
85.42 (27.27%) %
RI-Distill-Qwern
70.0
65.2
67 6
74.0
78.0
54.4
70.4
55.2
27.2
48.8
6.4
56.11 11 (90, 98%)
+ L-R Disentangle
71.2
69.2
68.8
73.2
81.2
57.2
73.6
56.8
29.6
54.8
7.6
58.51 (92.18%)
RI-Distill-LLaMA-s
79.2
51.6
$1.6
49.2
70.4
52.0
56.4
38.8
12.8
26.4
3.2
44.69 (84.44%)
I.R Disentangle
84.8
56.0
55.6
52.4
73.6
55.2
56.0
46.4
14.0
29.6
6.4
49.19 (84.44%)
RI-Distil-QQeen-LAD
70.8
71.2
74.4
75.6 65.6
85.6 6
68.8
82.8
76.8
20.8
67.2
23.6
65. 24 (93993%)
+ L-R Disentangle
71.6
73.6
75.6
76.8
86.4
73.2
82.4
81.2
26.4
66.8
25.6
67.24 (95.20%)
GLM-Z1-9B
94.0
80.8
74.0
75.6
84.8
73.2
83.2
70.8
41.6
41.6
44,4
69. 64 (55, 96%)
+ L-R Disentangle
94.8
85.2
72.8
76.0
87.2
79.2
80.8
71.6
42.4
43.6
47.2
71.16 (56.04%)
QwQ-32B
95.6
89.2
82.8
80.4
90.0
86.4
85.6
89.2
58.4
82.4
76.4
83.31 (56. (56.44%)
+ I..R Disentangle
97.6
90.8
84.4
83.6
91.2
88.0
86.4
88.8
62.0
83.6
74.8
84.62 (60.47%)
Languages We select 11 target languages to evaluate the multilingual reasoning under interven- tion. These languages span diverse linguistic families and resource levels, ensuring broad typologi- cal coverage. Specifically, we include high-resource languages such as English (En), Spanish (Es), French (Fr). German (De). Chinese (Zh). Japanese (Jp), and Russian (Ru); redium-resoure lan- guages such as Thai (Th) and Telugu (Te); and low-resource languages such Bengali (Bn) and Swahili (Sw). This selection is designed to balance linguistic diversity with practical considerations of data availability and benchmark coverage.
Benchmarks We evaluate the impact of language-reasoning disentanglement in multilingual rea- soning across three major benchmarks, including MGSM (Shi et al. 2023]: mathematical reason- ing, XWinograd [Muennighoff et al.) 2023]: commonsense reasoning, and M-MMLU [Hendrycks) et al. [021]: knowledge- intensive question answering. We use Accuracy as the evaluation metric for all benchmarks. detailed description of each benchmark is provided in Appendix E
inteeiige-ntensive
Implementation Details Our language-reasoning disentanglement are implemented based on the VLLM framework (Kwon et al. 2023 for efficient inference. For each model, we disentangle the kunuuage-specifs components from mid-layer hidden states and then re-inject them at higher layers. This strategy preserves overall input-output consistency while allowing to isolate the causal effect of aanguage-specifii information on reasoning. A detailed layer-wise analysis of this mechanism is provided in 4.2 and hyperparameter configurations can be found in Appendix F
Results and Analysis The results for MGSM are presented in Table I while the outcomes for XWinograd and M-MMLU are shown in Table We draw the following conclusions:
disentanglement consistently improves multilingual reasoning perfor- mance. This improvement holds across all model types and architectures. For non-reasoning mod- els, both the Qwen-2.5 and Qwen-3 Instruct series exhibit clear gains in multilingual reasoning after lenguagg-reasoning disentanglement. Similarly, for reasoning models, the intervention proves ef- fective across models trained with different paradigms: both the DeepSeek-R1-Dee are distilled from stronger models via supervised fine-tuning, and models optimized with large-seale reinforcement learning objectives like QwQ-32B, show substantial multilingual improvements.
series, which
5

---

Table 2: Multilingual reasoning performance on XWinograd and M-MMLU datasets across different languages, before and after rnasoing disentanglement within the activation spaces of the backbone models (+ L-R Disentangle). The best results are highlighted in bold. The values in parentheses indicate language fidelity to indicate input-output consistency.
XWinograd Jp
M-MMLU Jp
En
Fr
Zh
Ru
AVG.
En
Es
Fr
De
Zh
Bn
Sw
AVG.
Qwen 2.5-Insttuct-3BB
71.5
63.9
68.5
06.0
55.5
65.07
(98. 3%)
71.5
57.0
55.0
52.5
54.0
55.0
42.0
34.0
52.62
(20.566))
L-R Divemangle
75,5
69.9
73.5
71.0
61.0
70.18
(99.52%)
71.5
60.5
57.5
53.5
57.5
56.5
49,5
38.5
56.63
(31.06%)
Duven-25-Insttmettt
73.0
62.7
73.5
53.0
68.13 (98, 7056)
14.5
69.0
70.5
65.5
66.5
62.5
52.5
29.5
61.25 125))
Diventangle
78.0
(99.90%)
74.0
70.0
56.0
35.5
(34.56%))
Quen-3-1.78-11 T.iiiiinn
72.5
63.9
82.0
71.5
60.0
69 09.97 (60..00%)
74.5
62.5
63.5
65.0
65.5
59.5
47.0
27.5
S8. 12 (12.88%)
LRR Disentangle
76.0
71.1
80.5
63.5
73.42 (59. 10%)
79.0
63.5
66.0
67.0
64.5
52.5
33.0
61.75 (13.31%)
Quen- 3. T-in-ihinning
83.0
84.5
(60))0))))
76.5
75.5
67.5
42.0
71. (25 00%
+LLRR Disesrangle
86.0
83.1
N8.5
78.5
84,53 (60.10%)
81.5
77.5
78.5
76.5
74.5
73.0
70.0
40.5
71.69 (25.06
Qwen:3-8B- T-Bikininn + +LR Disentangle
87.0
30.7
81.5
90.0
83.5
85.
83.5
78.5
77.5
76.0
79.5
77.5
73.5
45.5
73 73.94 (20.69%)
89.0
$8.0
87.0
90.0
86.0
87.99
(59.70%%
84.0
82.5
75.0
81.0
190.5
81.5
73.0
49.0
76.19 13%)
RI-Dinill- Qwen-7B
54.5
45.8
49.0
50.06 (40.0001))
61.0
51.5
52.0
45.5
16.5
41 (24.75%)
+ LRR Disentangle
54.5
49.4
53.0
54.5
47.5
53.90 (50.50%)
61.0
56.5
55.0
53.0
37.5
$1.5
33.0
16.5
45.50 (24.63%)
RI RI-Diatll-LLMMAA
78.0
68.7
56.5
53.5
63.0
63.93 (75 505))
67.5
62.5
57.5
52.5
41 D
48.5
30.0
26.5
48 50 (19 19%)
+ L.R Disentangle
79.5
67.5
62.5
54.5
63.5
65.49 (76.20%
72.0
61.0
61.0
61.0
45,5
52.5
36,0
31.0
$2.50 (19.37%)
RI RI-Diailll-O Seiill--ee--44B
90.5
78.
74.5
85.5
77.0
81. 16 (65, 50%))
81.5
78.5
75.5
77.5
64.0
76.5
600.5
31.0
68. 13 (23 AP%)
L-R Disentangle
79.5
78.0
83.50
(65.40
85.5
50.0
80.0
62,0
36.0
71 (24.50%)
GLM-ZI-90 21-98
87.5
$1.9
80.5
78.0
82.0
81 11.99 (4))00%)
82.5
77.5
75.0
79.0
71.5
71.0
59.0
44.0
69 99 87%)
--R Diventangle
89.0
80.7
82.5
83.0
81.0
83.00
(40.20 20%)
84.5
82.0
78.5
79.0
76.0
75.5
65.5
47.5
73.56 (25.00%)
QuQ-32B LR Disensangle
92.5
88.0
85.5
89.5
78.5
86.29 (65, 30%)
84.0
81.5
81.5
81.0
81.5
79.5
75.5
45.5
76.25 19%
94.5
91.6
85.5
77.5
(69.90%)
87.0
85,5
83.0
75.5
49.5
79.06
(13.25%%
Moreover, the benefits of language-reasoning disentanglement are even more pronounced on XWinograd and M-MMLU benchmarks that emphasize commonsense inference and knowledge- intensive reasoning. Nearly all models show larger absolute gains in average accuracy after in- tervention. This suggests that linguistic interference is particularly detrimental in reasoning tasks requiring subtle context integration or factual grounding, and that removing language-specific noise in the representation space can yield stronger improvements under these conditions.
The performance gains are consistent across languages with different resource levels. High- resource languages such as English, French, and Chinese show steady improvements, indicating that even well-represented languages benefit from suppressing language-specifie interference. More notably, the enhancement is also substantial in medium- and low-resource languages. For example, Swahili despite being minimally present in pretraining data- achieves accuracy gains exceeding 10% in several models, with some cases more than doubling the original performance. These results demonstrate that our intervention provides balanced benefits across the linguistic spectrum and is also impactful in underrepresented settings, contributing to more equitable multilingual reasoning.
4
Deeper Analysis
4.1 Language-Speciffic Activation Negatively Correlates with Reasoning Accuracy
To further understand the impact of gegguage-speciffee signals on multilingual reasoning, we quanti- tatively analyze how the intensity of these signals affects model performance. Specifically, we vary the ablation strength, defined as the proportion of language-specific components M, removed (positive values) or injected back (negative values) during inference. This allows us to observe both the benefits of suppressing and the effects of amplifying language-specific information.
As shown in Figure 3 we report MGSM accuracy alongside two fidelity metrics: reasoning fi- delity measures whether the model's intermediate reasoning aligns with the input language, while response fidelity captures whether the final answer is delivered in the same language as the input. There are three representative models involved: QwQ-32B. We observe a consistent negative correlation between sunguage-specing activation and reasoning performance. As the ablation strength increases (i.e., more language-specific components are removed), MGSM accuracy improves. Conversely, when language-specific components are am- plified (negative ablation strength), reasoning performance degrades most notably in the QwQ model, where accuracy drops steeply as language-specific activation increases.
7-- DeepSeek-R1 -I-Dsstill-7BB, and

---

Reasoning Fidelity
Response Pidelity
Performance
Qwen2.5-78-lattrre
Deepöedk-RI Meaeec------iiil--eeeeeee
QWQ
60.0
85.0
0.84
70.5
0.82
58.0
84.0
0.66
Performance
70.0
0.80
0..60
56.0
0.8
83.0
Pidelity
69.5
0.78
0.55
0.76
82.0
0.50
0.74
68.5
0.60
81.0
0.72
68.0
0.50
80.0
-04
-0.2
0.0
0.4
-0.4
-0.2
0.0
0.2
0.4
-0.4
-0.2
0.0
0.4
Figure Effects of ablation strength on multilingual reasoning performance and output fidelity. Positive values indicate the proportion of language-specto components in M, removed, while negative values indicate their injection (reinforcing language-specific signals). We report MGSM accuracy (reasoning performance, red), reasoning fidelity (green), and response fidelity (yellow). Dashed lines indicate the original model's performance without intervention.
Reasoning Fidelity
Response Fidelity
Performance
Owen2.557lB-Instterr
DeepSeck-Rt Baai--RR--Diill-eeeeeeee Disiillll--Oee--11
QNQ
86.0
0.65
0.84
70.5
0.82
58.0
85.0
0.60
Performance
70.0
0.80
56.0
0.80
84.0
Fidelity
69.5
0,78
83.0
0.55
0.76
0.53
69.0
0.74
68.5
52.0
0.60
81.0
0.48
68.0
50.0
0.50
80.0
0.45
4
16
20
24
12
16
20
24
24
32
40
48
56
Figure 4: Layer-wise impact of language-reasoning disentanglement on MGSM accuracy and out- fidelity. The x-axis denotes the starting layer index of the intervention. Most layers support effective disentanglement that improves reasoning performance. Dashed lines indicate the original model's performance without intervention.
These trends reaffirm our central hypothesis: excessive entanglement of linguistic signals can in- terfere with multilingual reasoning. By suppressing such components, we guide the model toward more language-invariant internal representations and thereby enhance reasoning performance. At same time, we observe trade-offs response fidelity, suggesting that fine-grained control over ablation strength may be required to balance accuracy and output fluency.
We further analyze the effect of ablation strength on each individual language. The reasoning perfor- mance trends align with the overall average, improving as language-specific signals are suppressed. However, response fidelity degrades more noticeably in low-resource languages, indicating their higher dependence on language-specing components for fluent generation. Full results and analysis are provided Appendix G.1
4.2 Layer-wise Effeets of Language-Reasoninn Disentanglement
To investigate where language and reasoning most effectively separable, conduct layer- layer-wise analysis. Specifically, we apply the disentanglement intervention to different layers of the model, from lower to middle upper, and evaluate its impact reasoning performance and output fidelity.
As shown in Figure we observe that ablation applied at most layers, especially from lower to middle depths, consistently improves reasoning accuracy while maintaining stable output fidelity. This suggests that language-reasoning disentanglement is broadly effective across the network and does not require precise targeting specific layers to yield benefits.
7

---

Table 3: Multilingual reasoning accuracy MGSM across different languages using Qwen-2.5- Instruct-3B, comparing three approaches: baseline, inguage-reasonine Disentanglement (+ L-R Disentangle), and multilingual post-training (SFT and RL). The best result for each language is highlighted in bold, and the second-best is marked with underline.
High- Rees-Reesurree
Mid-Ran Resouree
Now-Resource Sw
AVG.
En
Es
Fr
De
Zh
Jp
Ru
Th
Te
Bn
Qwen-2.5-Innse 2.5-Instruct-3B
86.0
74.0
70.0
70.0
69.2
56.0
71.2
61.6
8.0
35.2
12.4
55.78
w/ L-R Disentangle
87.2
76.8
72.0
71.2
73.6
61.6
73.6
65.2
12.0
40.8
15.2
59.02
w/ SFT
80.0
60.0
60.0
60.0
70.0
30.0
70.0
70.0
20.0
10.0
10.0
49.09
w/ RL (PPO)
82.0
72.8
75.2
70.0
72.8
63.2
74.8
70.4
14.8
53.2
14.4
60.33
However, interventions at the upper layers lead to sharp drop in both reasoning and response fidelity. This indicates that while the model's core reasoning logic can operate more language- invariantly, the top layers still encode important information for language fluency. Removing language-specific signals at this stage disrupts generation consistency and harms output quality.
These findings reaffirm the central claim that reasoning and language can be disentangled across wide range of model depths, but highlight that middle layers offer the best trade-off: substantial gains in reasoning performance without sacrificing linguistic coherence. This also aligns with prior findings showing that multilingual models tend to form language-agnosie semantic spaces their
intermediate layers Wuu et al. 2024
Wang al. 2024
4.3 Comparison with Multilingual Post-fraining
We compare the performance of models after language-specific ablation with those enhanced via multilingual post-training, including both supervised fine-tuning (SFT) and reinforcement learning (RL) on mathematical reasoning datasets. This comparison serves to evaluate the practical utility of proposed intervention and further validate the central hypothesis: that disentangling lan- guage from reasoning offers an efficient and effective pathway for improving multilingual reason- ing- tog-potentially rivaling expensive post-traing post-training strategies.
Experimental Setup We use Brren2.5-3BB-Intteeee as the base model due to computational con- straints; this the largest model we can post-train on 8xA100 (80GB) GPUs. Both SFT and RL experiments are conducted using the 7,500-sample MATH benchmark [Hendrycks et al. [021]] which provides verifiable ground-truth answers. For SFT, golden reasoning traces are distilled from QwQ-32B, a model optimized for high-quality chain-of-thought reasoning. Since MATH is English- only, we translate both prompts and reasoning traces into the 10 target languages from 33 the Google Translate API. SFT is implemented via the LLaMA-Fact repository (Zheng et al.)
2024 while RL training uses PPO [Schulman et al.] (0177) implemented in OpenRLHF [Hu et al.
2024
Further training details are provided in Appendix
Results and Analysis The comparison between different methods are demonstrated in Table We derive two key insights from this comparison:
Training-free intervention achieves performance comparable to post-training. By disentan- gling language and reasoning during inference, our training-free intervention not only surpasses supervised fine-tuning, but also achieves results on par with reinforcement learning. This high- lights the practical potential of our core finding: disentangling language-specific information from reasoning dynamics can yield substantial multilingual gains- even without additional training-
Moreover, these results point to promising direction for future work: Combining reason- ing-lang disentanglement with SFT or RL may offer a principled way to improve multilingual reasoning while mitigating the inefficiencies of current training pipelines.
RL remains effective in multilingual settings, while SFT yields limited Despite its simplicity, PPO-based RL demonstrates strong potential for enhancing multilingual rea- soning, achieving consistent gains across both high- and low-resource languages.
negative gains.
In contrast, SFT fails to provide meaningful benefits, and in many cases, significantly degrades performance. We attribute this to two key factors. First, the multilingual mathematical data used for

---

SFT is obtained via automatic translation, which may introduce inaccuracies and inconsistencies. This aligns with recent findings that mathematical reasoning data is particularly difficult translate due to its reliance on precise terminology and logical structure (L.iu et al. 2024, She et al., 2024 Wang et al. 2025]. highlighting a critical data quality bottleneck multilingual supervision.
Second, recent works have shown that small models often struggle to mimic the step-by-step reason- ing behaviors of larger teacher models, even when high-quality traces are available [Li al. 2025b Yu et al. 2025a Zhao et al. 2025a Our results corroborate this conclusion to some extent: direct distillation from QwQ-32B into a 3B model fails to reproduce effective reasoning across languages. This underscores the challenges of multilingual distillation and calls for further research into more effective supervision strategies in pross-lingual contexts.
Related Works
Multilingual Reasoning of LLMs Due to the imbalance in language distribution, most L.Mss
exhibit strong English bias and limited generalization low- and mid-resource languages (Zhang
ct al.
2024a Dou et al. 2025 Zhao et al. 2025b Zhou et al. 2025].
To address this limitation, prior research has primarily focused on rraining-time strategies that fall into two broad categories. The first line of work aims to construct high-quality multilingual reason-
ing datasets [Zhu al.) 2024 She al., 2024 Wang et al., 2025 Shimabucoro al. 2025 Ko al.
2025]. while the second leverages supervision signals from nigh-resource languages (typically En-
glish) to enhance reasoning in low-resource settings (She et al. [024 Zhao et al., 2024b Huo al.
2025 Ruan et al. 2025 Fan et al. 2025]. Complementary to these are test-time scaling approaches,
which adapt models at inference time without modifying model weights [Qnn et al. 2023 Zhang
et al. 2024b Yong et al. 2025 Gao et al. 2025 Tran et al. 2025 Yu et al. 2025b Son et al. 2025
In contrast, our work shifts the focus the internal of multilingual LLMs. By analyzing how LLMs represent language and reasoning in their latent spaces, we uncover structural entanglement that hin- ders generalization, and propose lightweight, training-free intervention that improves multilingual reasoning by explicitly disentangling language- specific signals from the reasoning process.
Mechanical Interpretation of Multilingualism Recent studies have explored how LLMs process multilingual inputs from two main perspectives: language-specii specific encoding and language-agnosst abstraction. The former reveals that certain neurons are selectively activated by specific languages, subnetworks [Tang et al. [2024] Kojima et al.) [2224] Saito jet al. 2024, Zhhng et al.) 2024c). In contrast, other works show that LLMs form shared semantic spaces across languages, enabling cross-lingual generalization through language-invaring represen- tations [Wu et al. 2024 Wang et al. 2024 Brinkmann et al. 2025 Chen et al. 20256).
suggesting internal language-specialize
We examine how language-specific signals interact with reasoning processes and find that suppress- ing such signals improves cross-lingual reasoning. This provides new evidence of their entanglement and highlights the potential of explicit disentanglement path to more robust generalization.
Representation Engineering Representation-leve interventions in LLMSs are gaining traction for
their transparency and efficiency (Zou et al. 2023 Grounded in the I Linear Representation Hy-
pothesis [Mikolov et al., 2013 Nanda et al. 2023
Park et al.) [024]. prior work explores manip- ulating hidden states at inference time to enhance truthfulness [Li et al. 2023 Campbell et al.]
2023 Zhang et al. 2024d] or reduce harmful behavior (Lee et al. 2024 Uppaal et al. 2224 Zhao 2025c]. We adapt this paradigm multilingual reasoning by targeting language-specific
components in the representation space and demonstrate that their removal improves cross-lingus wwthhot
performance without any additional training or model modification.
Discussion, Limitations, and Future Work
While our work provides strong empirical evidence for the effectiveness of langung--reasoning disentanglement in large language models (LLMss, also opens up new questions and highlights several important limitations and future directions:

---

Why English as the Language-Invarian Anchor? In our projection- based intervention, we ob- serve that representations across languages tend to converge toward English- even even bilingual mod- els like Qwen, which are trained with significant Chinese data. This suggests that English serves as the dominant anchor the model's ' latent space. While this may reflect the disproportionately high presence of English in pretraining corpora, the exact causes remain opaque due to the lack of transparency in model training data. Future work could benefit from more controlled experiments or synthetic training settings to better understand how anchor languages emerge in multilingual LLMs.
Connection to Latent Reasoning Our findings on aanuuage-reasonig disentanglement resonate with the emerging paradigm of latent reasoning in LLMs (Deng et al. 2024 Goyal et al. [204]] Shen et al.) 2025a Recent work by Hao et al. (2024) introduces the Coconut framework, which enables LLMs to perform reasoning within continuous latent space, rather than relying solely on explicit language tokens. This approach allows models to internally process reasoning steps as high-dimensional representations termed "continuous thoughts" before generating any output.
Both our projection- based intervention and the Coconut framework aim to disentangle reasoning processes from language-specift features. While our method removes iagguage-specffe compo- nents from hidden states to enhance cross-lingual reasoning. Coconut bypasses the need for linguis- tic expression during intermediate reasoning steps altogether. These complementary approaches suggest that reasoning L.LSs can more effectively modeled and enhanced by focusing on latent representations, rather than being constrained by surface-level language.
This convergence opens up promising avenues for future research, such integrating lan- guage-reasoning disentanglement techniques with latent reasoning frameworks to further improve the generalization and interpretability of LLMs across diverse tasks and languages.
Conclusion
In this work, we explore the internal mechanisms underlying multilingual reasoning in large lan- guage models. Motivated by cognitive insights, we hypothesize that reasoning and language process- ing can be explicitly disentangled. Through targeted interventions in the LLMs' activation space, we demonstrate that removing language-specing information significantly improves reasoning per- formance across languages. Our empirical analysis reveals a strong inverse correlation between language-specific signal strength and reasoning accuracy, and identifies mid-level layers as the most effective location for intervention. Furthermore, show that training- free approach achieves results comparable to, and in some cases exceeding, those of multilingual post-training methods. These findings suggest that structural disentanglement offers lightweight and scalable alternative for enhancing cross- cross-lingual reasoning, and open up promising opportunities to integrate such inter- ventions with future multilingual training and adaptation strategies.
Acknowledgments
We thank the anonymous reviewers for their comments and suggestions. This work was supported by the New Generation Artificial Intelligenc-Nattinn National Science and Technology Major Project 20232D0121100 1000 the National Natural Science Foundation of China (NSFC) via grant 62441614 and 62176078, the Fundamental Research Funds for the Central Universities, and the National Re- search Foundation, Singapore under its National Large Language Models Funding Initiative (AISG Award No: AISG-NMI MANNNMIP-20044 2-24---002)) Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
References
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint wxXiv:2412.166200 2024.
OpenAl. Openai 03-mini system card. OpenAl's Blog, 2025. URL htt.:https//peee// .cmm/ index/o3 mamxx/o3-ming ni system-------- card
10

---

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms 12942 2025.
via reinforcement learning. arXiv preprint
Team Kimi, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.12599 2025.
Team Qwen. Qwq-32b: Embracing the power of reinforcement learning. Qwen's 's Blog, 2025. URL https://qwen.mm
-a0po///www3333000 32b
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, arXiv preprint arXiv:2501. 2025.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain- of-thought reasoning in Ilms. arXiv preprint wrxiv22502.03373, 2025.
Zhong-Zhi Li, Duzhen Zhang, Ming-Liam Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, al. From system system 2: A survey of reasoning large language models. arXiv preprint 2.NNv:2250../41999 2025a.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang. Yunke Zhang. Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09688 2025.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng. Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Gao, and Wangxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models. arXiv preprint arXiv:2503.0955 2025a.
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caro- line Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, et al. Fron- tiermath: A benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint arXiv:241 1.04872, 2024.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249, 2025.
Aabid Karim, Abdul Karim, Bhoomika Lohana, Matt Keon, Jaswinder Singh, and Abdul Sattar. Lost cultural translation: Do llms struggle with math across cultural contexts? arXiv preprint arXiv:2503. 18018, 2025.
Changjiang Gao, Xu Huang, Wenhao Zhu, Shujian Huang, Lei Li, and Fei Yuan. Could thinking multilingually empower Ilm reasoning? arXiv preprint arXiv:2504.1 1833, 2025.
Chinasa T Okolo and Marie Tano. Closing the gap: A call for more inclusive language technologies. 2024.
Akash Ghosh, Debayan Datta, Sriparna Saha, and Chirag Agarwal. The multilingual mind: A survey of multilingual reasoning in language models. arXiv preprint arXiv:2.00. 099557, 2025.
Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip Yu. A survey of multilingual large language models. Patterns, 6(1), 2025.
Martin M Monti, Lawrence M Parsons, and Daniel N Osherson. The boundaries of language and thought in deductive inference. Proceedings of the National Academy of Sciences, 106(30): 12554-12559, 2009.
Martin M Monti, Lawrence M Parsons, and Daniel N Osherson. Thought beyond language: neural dissociation of algebra and natural language. Psychological science, 23(8):914 9222,99222222222 2012.
Marie Amalric and Stanislas Dehaene. distinct cortical network for mathematical knowledge in the human brain. Neurolmage, 189:19-31, 2019.

---

Evelina Fedorenko, Steven T Piantadosi, and Edward AF Gibson. Language is primarily a tool for communication rather than thought. Nature, 630(8017):555-586, 2024.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models multi- lingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Rep- resentations, 2023.
Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. In The 61sl Annual Meeting The Association For Computational Linguistics, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual multilingual BERT? In Pro- ceedings of the 57th Annual Meeting of the Association for Compulational Linguistics, pages 4996-5001. Association for Computational Linguistics, 2019.
Jindrich Libovický, Rudolf Rosa, and Alexander Fraser. On the language neutrality of pre-trained multilingual representations. In Findings of the Association for Computational Linguisties: EMNLP 2020, pages 1663-1674, 2020.
Ziyi Yang, Yinfei Yang, Daniel Cer, and Eric Darve. A simple and effective method to eliminate the self language bias in multilingual representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5825-5832, 2021.
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common-specific low-rank decomposition. In International conference on machine learning, pages 7728-7738. PMLR, 2020.
Zhihui Xie, Handong Zhao, Tong Yu, and Shuai Li. Discovering low-rank subspaces for language- agnostic multilingual representations. In Proceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 5617-5633, 2022.
Xiaohao Liu, Xiaobo Xia, See-Kiong Ng, and Tat-Seng Chua. Principled multimodal representation learning. arXiv preprint arXiv:2507.173443 2025.
Carolin Holtermann, Paul Röttger, Timm Dill, and Anne Lauscher. Evaluating the elementary mul- tilingual capabilities of large language models with MultiQ. In Findings of the Association for Computational Linguistics ACL 2024, pages 4476,4494, 2024.
Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schütze. GlotLID: Lan- guage identification for low-resource languages. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: /ttp::///pereeeeeee net/ forum? dd
d14e3EB25
Nuo Chen, Ning Wu, Shining Liang, Ming Gong, Linjun Shou, Dongmei Zhang, and Jia Li. Is bigger and deeper always better? probing llama across scales and layers. arXiv preprint arXiv:23/2.04333, 2023.
Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, Jiasen Lu, and Yoon Kim. The semantic hub hypothesis: Language models share semantic representations across languages and modalities. arXiv preprint arXiv:2411.04986, 2024.
Weixuan Wang, Barry Haddow, Minghao Wu, Wei Peng, and Alexandra Birch. Sharing matters: Analysing neurons across languages and tasks in llms. arXiv preprint arXiv:2406.09265. 2024.
Clément Dumas Veniamin Veselovsky, Giovanni Monea, Robert West, and Chris Wendler. How do llamas process multilingual text? a latent exploration through activation patching. In ICML. 2024 Workshop on Mechanistic Interpretability, 2024.
12

---

Chris Wendler, Veniamin Veselovsky. Giovanni Monea, and Robert West. Do llamas work in en- glish? on the latent language of multilingual transformers. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 15366- 15394, 2024.
Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Journey to the center of the knowledge neurons: Discoveries of language-independen knowledge neurons and degenerate knowledge neurons. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17817-17825, 2024.
Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. How do large language models handle multilingualism? In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024a.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiw:2412.15115 2024.
Qwen Team. Qwen3, April 2025. URL
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang. Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang. Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang. Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv. Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang. Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang. Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang. Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm-130b
130b to glm-4 all tools, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhanghhi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguis- tics.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405. /1143, 2024.
Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, and Lidong Bing. Is translation all you need? a study on solving multilingual tasks with large language models. arXiv preprint arXiv:2403.10258, 2024.
Shuaijie She, Wei Zou, Shujian Huang, Wenhao Zhu, Xiang Liu, Xiang Geng, and Jiajun Chen. Mapo: Advancing multilingual reasoning through sattlinngullllignnee-------ee prreeeeeeeeeeeeeeeeeee opti- mization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10015-10027, 2024.
Yiming Wang, Pei Zhang. Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, et al. Polymath: Evaluating mathematical reasoning in multilingual contexts. arXiv preprint 2504 18428, 2025.
Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ra- masubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. arXiv preprint 2502 2025b.
13

---

Qianjin Yu, Keyu Wu, Zihan Chen, Chushu Zhang, Manlin Mei. Lingjun Huang, Fang Tan, Yong- sheng Du, Kumlin Liu, and Yurui Zhu. Rethinking the generation of high-quality cot data from the perspective of Ilm-adaptive question difficulty grading. arXiv preprini arXiv:2504. 1919, 2025a.
Weixiang Zhao, Xingyu Sui, Jiahe Guo, Yulin Hu, Yang Deng, Yanyan Zhao, Bing Qin, Wanx- iang Che, Tat-Seng Chua, and Ting Liu. Trade-offs large reasoning models: An empirical analysis of deliberative and adaptive reasoning over foundational capabilities. arXiv preprint arXiv:2503. / 17779, 2025a.
Wenxuan Zhang, Hou Pong Chan, Yiran Zhao, Mahani Aljunied, Jianyu Wang, Chaoqun Liu, Yue Deng, Zhiqiang Hu, Weiwen Xu, Yew Ken Chia, et al. Seallms 3: Open foundation and chat mul- tilingual large language models for southeast asian languages. arXiv preprint arXiv:2407. 19672, 2024a.
Longxu Dou, Qian Liu, Fan Zhou, Changyu Chen, Zili Wang, Ziqi Jin, Zichen Liu, Tongyao Zhu, Cunxiao Du, Penghui Yang, et al. Sailor2: Sailing south-east asia with inclusive multilingual Ilms. arXiv preprint arXiv:2502. 12982, 2025.
Yiran Zhao, Chaoqun Liu, Yue Deng. Jiahao Ying, Mahani Aljumied, Zhaodonghui Li, Lidong Bing, Hou Pong Chan, Yu Rong, Deli Zhao, et al. Babel: Open multilingual large language models serving over 90% of global speakers. arXiv preprint arXiv:2503 00000005, 2025b.
Runtao Zhou, Guangya Wan, Saadia Gabriel, Sheng Li, Alexander J Gates, Maarten Sap, and Thomas Hartvigsen. Disparities in IIm reasoning accuracy and explanations: A case study on african american english. arXiv preprint (rxxi:2503304099 2025.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Multilingual machine translation with large language models: Empirical results and analysis. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2765-2781, 2024.
Lufsa Shimabucoro, Ahmet Ustun, Marzieh Fadaee, and Sebastian Ruder. A post-trainer's guide to multilingual training data: Uncovering cross-lingual transfer dynamics. arXiv preprint arXiv:2504. 16677, 2025.
Hyunwoo Ko, Guijin Son, and Dasol Choi. Understand, solve and translate: Bridging the multilin- gual mathematical reasoning gap. arXiv preprint arXiv:2501.0244 2025.
Weixiang Zhao, Yulin Hu, Jiahe Guo, Xingyu Sui, Tongtong Wu, Yang Deng, Yanyan Zhao, Bing Qin. Wanxiang Che, and Ting Liu. Lens: Rethinking multilingual enhancement for large language models. arXiv preprint arXiv:2410.04400 2024b.
Wenshuai Huo, Xiaocheng Feng, Yichong Huang, Chengpeng Fu, Baohang Li, Yangfan Ye, Zhirui Zhang, Dandan Tu, Duyu Tang, Yunfei Lu, et al. Enhancing non-english capabilities of english- centric large language models through deep supervision finettunig. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24185 241933299,,-- 2025.
Zhiwen Ruan, Yixia Li, He Zhu, Longyue Wang, Weihua Luo, Kaifu Zhang, Yun Chen, and Guan- hua Chen. Layalign: Enhancing multilingual reasoning in large language models via layer-wise adaptive fusion and alignment strategy. In Findings of the Association for Computational Lin- guistics: NAACL 2025, pages 1481-1495, 2025.
Yuchun Fan, Yongyu Mu, Yilin Wang, Lei Huang, Junhao Ruan, Bei Li, Tong Xiao, Shujian Huang, Xiaocheng Feng, and Jingbo Zhu. Slam: Towards efficient multilingual reasoning via selective language alignment. In Proceedings of the 31st International Conference on Computational Lin- 2025.
guistics, pages
Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. Cross-lingual prompting: Improving zero-shot chain-ot of-thoughh reasoning across languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2695-2709, 2023.
Yongheng Zhang, Qiguang Chen, Min Li, Wanxiang Che, and Libo Qin. Autocap: Towards auto- matic cross-lingual alignment planning for zero-shot thain-of-though In Findings of the Associ- ation for Computational Linguistics ACL 2024, pages 9191-9200, 2024b.
14

---

Zheng-Xin Yong, M Farid Adilazuarda, Jomibek Mansurov, Ruochen Zhang, Niklas Muennighoff, Carsten Eickhoff, Genta Indra Winata, Julia Kreutzer, Stephen H Bach, and Alham Fikri Aji. Crosslingual reasoning through test-time scaling. arXiv preprint arXiv:2505.0544 2025.
Khanh-Tung Tran, Barry O' D'ulliian,, and Hoang D Nguyen. Scaling test-time compute for low-
resource languages: Multilingual reasoning llms. arXiv preprint arXiv:2504.02888 02850, 2025.
Zhiwei Yu, Tuo Li, Changhong Wang, Hui Chen, and Lang Zhou. Cross-lingual consistency: A novel inference framework for advancing reasoning in large language models. arXiv preprint arXiv:2504.01887, 2025b.
Guijin Son, Jiwoo Hong, Hyunwoo Ko, and James Thorn. Linguistic generalizability of test-time scaling in mathematical reasoning. arXiv preprint arXiv:2502. 2025.
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Wayne Xin Zhao, Furu Wei, and Ji-Rong Wen. Language-speciffe neurons: The key to multilingual capabilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5701-5715, 2024.
Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. On the mul- tilingual ability of decoder-basse pre- rre-trained language models: Finding and controlling language- specific neurons. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6919-6971. Association for Computational Linguisties, 2024.
Koshiro Saito, Sakae Mizuki, Masanari Ohi, Taishi Nakamura, Taihei Shiotani, Koki Maeda, Youmi Ma, Kakeru Hattori, Kazuki Fujäi, Takumi Okamoto, et al. Why build local large lan- guage models: An observational analysis from 35 japanese and multilingual lms.. arXiv preprint arXiv:2412. 14471, 2024.
Zhihao Zhang, Jun Zhao, Qi Zhang, Tao Gui, and Xuan-Jing Huang. Unveiling linguistic regions in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 6228-6247, 2024c.
Jannik Brinkmann, Chris Wendler, Christian Bartelt, and Aaron Mueller. Large language mod- els share representations of latent grammatical concepts across typologically diverse languages. arXiv preprini arXiv:2501 00346666..66.... 2025.
Yuxin Chen, Yiran Zhao, Yang Zhang. An Zhang, Kenji Kawaguchi, Shatiq Joty, Junnan Li, Tat- Seng Chua, Michael Qizhe Shich, and Wenxuan Zhang. The emergence of abstract thought in large language models beyond any language. CoRR, abs/2506.09890, 2025b.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip
Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engincering: A top-down approach ai transparency. arXiv preprint arXiv:2310001405, 2023.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746-751, 2013.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self self-supervised sequence models. In Proceedings of the 6th BlackboxNLP Workshop: Analyz- ing and Interpreting Neural Networks for NLP, pages 16-30, 2023.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning, 2024.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2023.
James Campbell, Phillip Guo, and Richard Ren. Localizing lying in llama: Understanding instructed dishonesty on fule- questions through prompting, probing, and patching. In Socially Respon- sible Language Modelling Research, 2023.
15

---

Shaolei Zhang, Tian Yu, and Yang Feng. TruthX: Alleviating hallucinations by editing large lan- guage models in truthful space. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Pupers), pages 8908-8949 2024d.
Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg. Jonathan K Kummerfeld, and Rada Mi- halcea. A mechanistic understanding of aligument algorithms: case study on dpo and toxicity. In Forty-first International Conference on Machine Learning. 2024.
Rheeya Uppaal, Apratim De, Yiting He, Yiquao Zhong, and Junjie Hu. Detox: Toxic subspace projection for model editing, arXiv preprint (tti::2400.339661 2024.
Weixiang Zhao, Jiahe Guo, Yulin Hu, Yang Deng, An Zhang, Xingyu Sui, Xinyang Han, Yanyan Zhao, Bing Qin, Tat-Seng Chua, et al. Adasteer: Your aligned 11m is inherently an adaptive jailbreak defender. arXiv preprint arXiv:2504.094400 2025c.
Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to inter- nalize cot step by step. arXiv preprint arXiv:2405. 14838, 2024.
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference on Learning Representations, 2024.
Xuan Shen, Yizhou Wang, Xiangxi Shi, Yanzhi Wang, Pu Zhao, and Jiuxiang Gu. Efficient reasoning with hidden thinking. arXiv preprint arXiv:2501. 19201, 2025a.
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing Ohain-of-thought into continuous space via self-distillation. arXiv preprint arXiv:2502.21074, 2025b.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in continuous latent space. arXiv preprint arXiv:2412.06669, 2024.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. CLUE: A Chinese language understanding evaluation bench- mark. In Proceedings of the 28th International Conference Computational Linguistics, pages 4762-4772, 2020.
Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyê"n, n, Franck Dernoncourt Ryan Rossi, and Thien Nguyen. Okapi: Instruction-tuie large language models in multiple languages with reinforce- ment learning from human feedback. In Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing: System Demonstrations, pages 318-327, 2023.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pages 3505-3506 2020.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper's contributions and scope can be found in the section of Abstract and Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work described in Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificatton, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper is empirical study and does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation Details can be found in Section 3, Appendix and Appendix
H.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data also be provided via
often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models. it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Our data and codes could be found in supplementary files.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (tteiiinssttttt//////aaa /eettps:/niis.co/
ssoonPossssooooooooooo ooloo olicy) for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g,, for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips: ps. cc/publi c/qu des /CodeSoodeeemmeeeeee as onPol i lyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Implementation Details can be found in Section 3, Appendix and Appendix
H.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
Statistical significance the experiments can be found Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g,, Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Experiments compute resources can be found in Section 3, Appendix F and Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://sss//euuuss.....tt. http:////weerrre...cccooo//ll/ts
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts can be found in Appendix B.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
Safeguards can be found in Appendix B.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The creators original owners of assets (e.g., code, data, models), used in the paper, are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, haperswithcor com/ datasetss has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
This can be found in Appendix F and our supplementary files.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM is used only for writing, editing, formatting purposes.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http:///eeeiigsceeofeeeeeeeeenee
2025/LLX for what should or should not be deseribed.
23

---

model improvement. A Further Discussion, Limitations, and Future Work
While our work provides strong empirical evidence for the effectiveness of language reasonnin disentanglement in large language models (LLMs), also opens up new questions and highlights several important limitations and future directions:
Trade-off Between Reasoning and Language Fidelity. Although language-specffee ablation im- proves reasoning accuracy. we also observe degradation in language fidelity, especially when in- tervention is applied higher layers. This trade-off highlights the challenge of balancing language- invariance with surface-level coherence. Future efforts could explore more fine- fine-grained or dynamic projection mechanisms that adaptively control which components are suppressed and which are pre- served based on task goals or input context.
Applicability Beyond Reasoning. While our focus is on multilingual reasoning tasks, the prin- ciple of disentangling task-relevant and sanguage-specifte signals may extend to other high-level cognitive abilities in LLMs, such as planning, reflection, or explanation generation. Investigating how this approach generalizes to other domains, and whether other types of entangled information can be removed, offers a promising avenue for broader interpretability-gun
From Post-hoc Intervention to Training-time Integration. Our method operates at inference time and requires no model modification, which makes it lightweight and broadly applicable. How- ever, the benefits of projection-base disentanglement could potentially be amplified if integrated into training objectives. For example, encouraging intermediate representations to be language- invariant during supervised fine-tuning or reinforcement learning could improve generalization more systematically. Future work could explore hybrid objectives that explicitly regularize the disentan- glement of reasoning and language components during model optimization.
Toward Principled Multilingual Model Diagnosis. Finally, our findings suggest that analyzing language-specific activation patterns offers a diagnostic signal for cross-lingual model behavior. This could form the basis for more principled evaluation frameworks that go beyond surface-level accuracy, helping to reveal when and where models fail to generalize across languages.
B Broader Impact
As LLMs increasingly deployed in global applications, ranging from education to healthcare and decision support, ensuring their equitable performance across languages has become both a techni- cal and ethical imperative. However, most recent progress in reasoning-enhanced LLMs remains concentrated in English and few high-resource languages. This reinforces existing disparities in Al access, limiting the utility of advanced models for speakers of low- and mid-resource languages.
Our work addresses this issue by investigating the internal mechanisms behind multilingual reason- ing in LLMs and introducing a lightweight, training- wraning-frree free approach to enhance reasoning capabilities across languages. By explicitly performing language- reasoning disentanglement, we show that rea- soning can be made more This has the potential to democratize access to strong reasoning models for underrepresented lan- guages, particularly in settings where resources for large-scale finetuning are limited.
requiring costly additional data or retraining.
In addition, our analysis contributes to the broader goal of making LLMs more interpretable and controllable. Rather than relying solely post-hoc evaluation, explore how internal representa- tions can be causally intervened upon to improve specific capabilities. This aligns with the growing movement toward transparent, mechanism-driven Al development and offers tools that can be inte- grated into responsible deployment pipelines.
Nonetheless, our intervention raises new questions about faimess and control: the choice to remove language-specific features must be carefully balanced against preserving cultural and linguistic iden- tity. Future work should examine the societal implications of in real-world multilingual applications, and how such techniques may interact with biases training data or user experience.
disentanglement
24

---

Algorithm 1: Language Subspace Probing
In: languages' mean embeddings M, rank of subspace
Out: language-agnostic subspace M,, language-specific subspace M,, coordinates
/* 1) Approximate M in low rank
M', M
2 M ,,,-I + Top-r -VD(M-M )
M' t- M + M
/* 2) Force orthogonality
Ma < | M 1;
s M,, + Top-r SVD (M' 1
Overall, we hope that this work inspires more inclusive and principled approaches multilingual reasoning, helping to bridge the capability gap across languages and contributing to the development of more equitable, transparent, and linguistically fair Al systems.
C Probing for Language Subspace
The optimal solution of Equation 2 can be computed efficiently via Singular Value Decomposition (SVD). Algorithm presents the detailed procedure. Readers interested in more details can consult the proof provided in Xie et al.) (2022). The only hyperparameter of anguage-speecife information captured by the identified subspace. The larger r is, the more laneuage-specific signals we can identify.
L controls the amount
D Representation Space Visualization
To complement the main analysis presented Section 2.3
we provide additional visualizations of final-token hidden representations across multiple models and layers, before and after applying the projection-basee ablation defined Equation 3 These results offer further evidence of the role played by language-specific subspaces in shaping multilingual representations.
Layer-wise Comparison Owwen-255-7B-Instte
Figure 5 shows the visualizations Qwen- 2.5-7B-Instruc three different layer depths: lower, middle, and upper. Across all layers, observe a consistent trend after projection, multilingual representations become less language- specific and tend to converge toward the English cluster. However, the degree of convergence varies: the middle layers exhibit the most prominent language alignment, while the lower layers show weaker separation begin with, and the upper layers retain stronger language-specific traits even after ablation. These results suggest that while the disentanglement effect holds across the model. the middle layers offer the most effective separation between language and reasoning repre- sentations.
Syven---B-Thinninng Ri-Distill-Qwenn-B and QwQ 32B. Figures 6 Figures Figures [ analogous visualizations for Ei-Distill-Qwer 7B QwQ-32B, respectively. The same convergence trend can be observed: representations of various languages tend to collapse toward the English representation after projection, especially in the middle layers. This confirms that the observed phenomenon generalizes beyond a single model family or training paradigm.
Visualizations in -wen-2.5-3B-nnstte
E Benchmark
We evaluate the impact of removing language-specific information on multilingual reasoning across three major benchmarks, including:
25

---

(AMaadia)
(AMardii) (Ailarii))
(ANlsios)
BY (ARtetioo)
Br (Aldatiin))
Jp (ANarton
Jp (Aliatt))
TE (AMlatio))
Th (Atrution)
Se (AMalioi)
Sn (Abazion)
(a) PCA visualization of final-token hidden states (b) PCA visualization of final-token hidden states before and after ablation in -wwen2.55-7B-nnstrrccl before and after ablation in Qwen-2.5-7B 2---strruttttt (14, middle layer).
(5, low layer).
Br (Altaiie:
3 (attaee))
Ti (Atteie))
So (Alutio))
(c) PCA visualization of final-token hidden states before and after ablation in Qwen-2.5-77 I-Inrutucc (27, top layer).
Figure 5: Layer-wise PCA visualizations in Qwen-2.5-7BB 2.5-7BBInstruct. Each subfigure shows hidden states at lower, middle, and upper layers, before and after projection.
MGSM [hh et al. 2023 designed to evaluate the reasoning abilities of large language models in multilingual settings. It comprises 250 grade-school math problems, originally from the GSM8K dataset, each translated by human annotators into 10 diverse languages: English, Spanish, French, German, Russian, Chi- nese, Japanese, Thai, Swahili, Bengali, and Telugu. These problems require multi-step reasoning, making MGSM valuable resource for assessing models' capabilities in mathematical problem- solving across different languages. The dataset includes both the questions and their step-by-sted solutions, facilitating comprehensive evaluation of multilingual reasoning performance.
The Multilingual Grade School Math Benchmark (MGSM) a dataset
XWinograd [Muennighoff al. 2023 A well-established tool for evaluating coreference res- olution (CoR) and commonsense reasoning (CSR) capabilities of computational models. The dataset the translation of the English Winograd Schema datasets and adds 488 Chinese schemas from CLUEWSC2020 [Xu et al. 2220]]. totaling 6 languages. Formulated as fill-in- a-blank task with binary options, the goal requires commonsense reasoning. In our experimental setup, this benchmark covers English (En), French (Fr), Chinese (Zh) and Japanese (Jp) and Russian (Ru).
choose the right option for given sentence which
M-MMLU [Hendrycks et al. 2021 Lai et al. 2023 A benchmark designed to measure knowl- edge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot set- tings. The datasets is a machine translated version of the MMLU dataset by GPT-3.5-turbo and covers 34 languages. This is massive multitask test consisting of multiple-choice questions
26

---

(ABlstio))
(AMardii)
(Allatic))
2h (MMlarii)
Br (Aldatiin)))
Jp (AAorio))
Jp (Aliatto)
(AMlatio))
Th. (Atration)
(Ablatioa)
Sm (Abbiinn))
(a) PCA visualization of final-token hidden states (b) PCA visualization of final-token hidden states before and after ablation in Qwen-225-3B-IIntrruc before and after ablation in Qwen-2.5-3P B--nSrrurtttt (2, low layer).
(16, middle layer).
Ian
In
Zh (Xllriiee)
Ir (Albbiinn
Ap (Sbiddas
Th (Ablrtiie)
Sw (AMatio))
(c) PCA visualization of final-token hidden states before and after ablation in Qwen-2.5 2.5--B 3B--Instruct (35, top layer).
Figure 6: Layer-wise PCA visualizations in Qwen-2.5-23 2.5-3B-Insttruc Each subfigure shows hidden states at lower, middle, and upper layers, before and after projection.
from various branches of knowledge. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. In our experimental setup, this bench- mark covers English (En), Spanish (Es), French (Fr), German (De), Chinese (Zh), Japanese (Jp), Bengali (Bn) and Swahili (Sw).
F Implementation Details of Causal Intervention
We use 7,500 math problems from the Google Translate version of the MATH dataset (Hendrycks) et al.] 202T]. translated into 10 annuages--Bengali (bn), German (de), Spanish (es), French (fr), Japanese (jp). Russian (ru). Swahili (sw), Telugu (te), Thai (th), and Chinese (zh)-as the data for extracting the language-specific subspace. For each input, we ablate activations only on the tokens in the prompt. the main experiments, we perform a grid search find the best results, applying À in the range of O to 0.4 at the middle layers and -0.4 to 0 at the higher layers. For various LLMs, the specific range of middle and high layers we have selected are shown in Table Furthermore, in the analysis of ablation strength presented in S.ction[4.1] our focus is on evaluating model performance the middle layers and examining language fidelity in the higher layers. Also, We conduct significance tests on the multilingual results, showing that our method yields statistically significant gains (p < 0.05) on most benchmarks. We believe this further supports the effectiveness of our approach, especially given its low cost and wide language coverage.
27

---

(AMardii))
(AHotoo)
Br (Aldatiio))))
Jp (ARorio))
Jp (Atisrto))
(ANasioo)
Th (Atrattio)
3w (Allatioa)
Sm (Abbiion))
(a) PCA visualization of final-token hidden states (b) PCA visualization of final-token hidden states before and after ablation in Owen-3-8B-Thinking before and after ablation in (wen-3-8B-Thinkiee (5, low layer).
(17, middle layer).
Ian
In
Zh (MMlriee))
Ir (Alibtii
Ap (Abladae;
Th (Allrtiee)
Sw (AMaaiio)
(c) PCA visualization of final-token hidden states before and after ablation in Qwen-3-8B 3-en----B-hhinking (35, top layer).
Layer-wise PCA visualizations in Owen-3-8B-- 3-8----inBBTnng Thinking. Each subfigure shows hidden states at lower, middle, and upper layers, before and after projection.
Figure 7:
The exact layer ranges for each model, detailing the total number of layers along with specified middle and higher layer ranges. Number of Layers
Table 4:
Model Name
Middle Layers
Higher Layers
0wen-2.5-Innstruct
36
12-26
27-35
2ren-2.5-Instruct-777
28
10-19
20-27
28
10-19
20-27
Dwen-3-4B-Think-e
36
12-26
27-35
Bren-3-8B-Thinkine
36
12-26
27-35
RI-Distill-Qwen.
28
10-19
20-27
R1-Distill--.Lamaa
32
12-22
23-31
RI-Distill-Qwen-Il
48
16-33
34-47
GLM-Z119B
40
12-30
31-39
QwQ-32B
64
20-46
47-63
G Additional Experimental Results
G.1
Correlation Analysis on Each Language
Figure 10 presents the effects of varying ablation strength on both reasoning performance and out- put fidelity for each of the 10 evaluation languages, using QwQ-32B as the backbone model. Over- all, we observe that increasing ablation strength-iee. removing more language-speciff compo- nents-leads improved reasoning performance across most languages, consistent with the ag-
28

---

(ABlatioo)
7h: (AMariis)
Br (Aldatina))
Jp (AARbR))))
Jp (Allarto))
(ANasiio)
T1 (Atdation)
(Ablation)
5m (A3b2lnn)
(a) PCA visualization of final-token hidden states (b) PCA visualization of final-token hidden states before and after ablation in RI-Distill-Qwen- before and after ablation in R1-Distil-Qwen-77B (5, low layer).
(14, middle layer).
Inn
(Ahbilee)
7h (Ablrdee)
Fr (Aliitin
2p (SSiidon)
Th (Allericn:
Sw (AM4a194)
(c) PCA visualization of final-token hidden states before and after ablation in RI S21-istill-Qwar Qwen-7B (27, top layer).
Figure 8: Layer-wise PCA visualizations in RI-Distill-Qwen- Each subfigure shows hidden states at lower, middle, and upper layers, before and after projection.
gregate trend discussed in Section 4.1 However, the fidelity curves show more nuanced patterns, particularly across languages of different resource levels.
High-resource languages (En, Es, Fr, Zh, Jp, Ru). In high-resource languages, reasoning per- formance is relatively stable improves steadily with increased ablation. Output fidelity remains high and shows only minor degradation (e.g.. English and Chinese maintain near- near-perfect fidelity throughout). This indicates that high-resource languages are less dependent on language-specift activations for surface realization, likely due to stronger coverage pretraining.
Mid- and low-resource languages (Th, Bn, Sw). In contrast, mid- and low-resource languages exhibit sharper trade-off'. For Thai, Bengali, and Swahili, fidelity drops steeply as ablation strength increases- despite clear gains in reasoning accuracy. For instance, in Thai, output fidelity declines from 0.9 to below 0.4 at high ablation levels, even as performance improves by over 10 points. This suggests that these languages rely heavily language-specifie signals to maintain fluent generation, possibly due to weaker anchoring in the shared representation space.
Implication. These findings highlight the need for balanced adaptive ablation strategy in multilingual settings: while removing inguage-specinfe components can enhance reasoning perfor- mance, overly aggressive suppression may compromise output fluency, especially for underrepre- sented languages. Future work could explore language-aware projection schedules or hybrid control mechanisms dynamically balance reasoning abstraction and language preservation.
We further replicate this per-language analysis on two additional
models:
R1 R1-Distill 11-0ee11- and Qwen-2.5 - as shown in Figure I and
29

---

(AStstio))
(AHotio))
Br (Aldatiia)) 2o (Alistoo)
3p (ARbrtio)
(ANlatio))
Tb (Aboatio))
(Ablativa)
Sm (Abbria))
(a) PCA visualization of final-token hidden states (b) PCA visualization of final-token hidden states before and after ablation in QwQ-32B (12, low before and after ablation in QwQ-32B (35, middle layer).
layer).
Inn
In
Zh (MMlriee))
Ir (Albiiii
Ap (Sbidde
Th (Ablrtiee)
Sw (AHutin)
(c) PCA visualization of final-token hidden states
before and after ablation in QwQ-32B (63, top layer).
Figure 9: Layer-wise PCA visualizations in QwQ-32B. Each subfigure shows hidden states at lower, middle, and upper layers, before and after projection.
Figure [2] respectively. Across both models, we observe consistent trends with those reported for QwQ-32B: increasing ablation strength generally improves multilingual reasoning performance, while response fidelity declines more significantly in lower-resource languages.
These results reinforce the generality of our findings across different models and training paradigms. They further suggest that the entanglement between language-specific activation and reason- ing is robust phenomenon, and that language-reasome disentanglement can benefit diverse LLMs-though may require language-aware tuning preserve fluency in less represented lan- guages.
G.2 Layer-wise Analysis on More Models
To further verify the robustness of our findings, we extend the layer-wise analysis introduced in Section 4.2 to additional models, including Owen-2.
Qwen-3-1. 7B, Owen-3-8B, and R1 R1-Distill dowen-14B. For each model, apply the same intervention procedure by ablating language-specifiic components at different layer middle, and upper layers- and evaluate the effects on multilingual reasoning performance and out- put fidelity. Results are shown Figure 13
lower,
The results consistently align with earlier observations: (1) Performance improves across all mod- els when language-specific signals are removed middle layers, reinforcing the hypothesis that reasoning representations are most disentangled from language at this depth; (2) Upper-layer ablation often degrades output fidelity, particularly high-resource languages, indicating that language-specific features in later layers are critical for surface realization.
30

---

1.00
Janacring Fideliy
Response Fidilly
0.25
99-0
MOSM Perfonsance
0.90
1.20
Pertoraussee
58.0
Wesformmans
Performaatee
0.30
Videity
Fidelit
0.15
92.0
0.90
97.0
0,70
0.10
90.0
96.0
0.60
88.0
0.05
79.0
86.0
0.00
-044
-0.2
0.0
0.4
0.4
-0.4
-0.2
0.0
(a) Effects of ablation strength (b) Effects of ablation strength (c) Effects of ablation strength
English reasoning performance and Spanish reasoning performance and French reasoning performance output fidelity.
output fidelity.
output fidelity.
1.00
0.99
90.0
90.0
90.5
0.98
88.0
Pertooaaios
90.0
0:97
Pidelity
1080
Pidelity
Perfoemaats
0.80
89.5
0.96
86.0
89.0
84.0
84.0
88.5
0.94
8..0
0.60
82.0
88.0
0.93
$0.0
86.0
-044
-0.2
0.0
0.2
0.4
-0.4
-0.2
0.0
0.2
0.4
-0.4
-0.2
0.0
0.2
0.4
(d) Effects of ablation strength Effects of ablation strength (f) Effects of ablation strength
on Chinese reasoning performance on Japanese reasoning performance Russian reasoning performance and output fidelity.
and output fidelity.
output fidelity.
82.0
94.0
1.00
90.0
86.0
99.0
0.80
78.0
Perfemaanos
perfoemancs
86.0
88.0
0.00
Fideiiy
0.80
Viddky
0.00
84.0
74.0
36.0
0.40
80.0
0.40
86.0
70.0
0.20
78.0
80.0
76.0
66.0
0.4
0.22
0.0
0.2
0.4
00.1
-0.2
0.0
0.2
0.4
-0.4
-0.2
0.0
0.2
0.4
(g) Effects of ablation strength (h) Effects of ablation strength (i) Effects of ablation strength Thailand reasoning performance Bengali reasoning performance and Swahili reasoning performance and output fidelity.
output fidelity.
output fidelity.
Figure 10: Effects of ablation strength on each language. The backbone is QwQ-32B.
31

---

1.00
1.0
1.00
Janacring Pideliy
74.0
Response Fiddlly
MOSM Perfonzeace
1.90
70.0
0.90
Pertormazze
72.0
0.80
Phoellay
Pesformaaass
68.0
0.60
Fidelit
Performases
0,70
70.0
66.0
66.0
0.40
64.0
66.0
0.49
62.0
0.30
0.40
-0.4
0.2
0.4
-0.4
-0.2
0.4
-0.4
-0.2
(a) Effects of ablation strength (b) Effects of ablation strength (c) Effects of ablation strength English reasoning performance and Spanish reasoning performance and French reasoning performance output fidelity.
output fidelity.
output fidelity.
1.00
60.0
0.98
0.55
74.0
0.95
58.0
0.95
Pertocaanee
$0.0
0.93
Videlit
Pesformans
56.0
1ug0
Videlity
72.0
0.888
68.0
0.85
0.80
66.0
0.8g
50.0
-0.4
-0.2
0.0
0.2
0.4
-0.4
-0.2
0.0
0.2
0.4
-04
-0.2
0.0
0.2
0.4
(d) Effects of ablation strength Effects of ablation strength (f) Effects of ablation strength
on Chinese reasoning performance on Japanese reasoning performance Russian reasoning performance output fidelity.
and output fidelity.
and output fidelity.
62.0
56.0
0.90
0.45
54.0
Perfetmanos
0.70
Fldcky
perfsemanne
Vileeky
Perfocmance
0:35
vidckyy
56.0
48.0
0.60
6.0
54.0
0.60
45.0
0.50
4.0
59.0
0.40
2.0
0,50
42.0
50.0
0.4
0.2
0.0
0.2
0.4
-0.4
0.2
0.0
0.2
0.4
-0.4
-0.2
0.0
0.2
0.4
(g) Effects of ablation strength (h) Effects of ablation strength (i) Effects of ablation strength Thailand reasoning performance Bengali reasoning performance and Swahili reasoning performance output fidelity.
and output fidelity.
output fidelity.
Figure 11: Effects of ablation strength on each language. The backbone is RI-Distill Swill--een--BB
32

---

85.0
1.00
1.0
100
95-0
Sopusac Fidblly
82.0
MCS3 Perfommanc
86.0
0.90
81.0
85.0
Pertoransss
93.0
Merformaeaee
84.0
Performassee
$6.0
0.80
Vidalit
0.80
Pidelity
0.80
99.0
R3.0
91.0
0,70
80.0
77.0
90.0
0.60
0.60
76.0
75.0
79.0
-0.4
0.2
0.4
-0.4
0.4
-0.4
-0.2
(a) Effects of ablation strength (b) Effects of ablation strength (c) Effects of ablation strength
English reasoning performance and Spanish reasoning performance and French reasoning performance output fidelity.
output fidelity.
output fidelity.
1.00
76.0
1.00
1.00
86.0
75.0
84.0
0.90
85.0
74.0
Perbcaanee
82.0
84-
0.80
Fidelity
0.80
Pidelity
0.80
$3.0
72.0
80.0
71.0
81.0
0.60
70.0
0.60
78.0
69.0
-0.4
-0.2
0.0
0.2
0.4
-0.4
-0.2
0.0
0.2
0.4
-0.4
-0.2
0.0
0.2
0.4
(d) Effects of ablation strength Effects of ablation strength (f) Effects of ablation strength
on Chinese reasoning performance on Japanese reasoning performance Russian reasoning performance output fidelity.
and output fidelity.
and output fidelity.
83.0
71.0
0.30
80.0
70.0
0.90
0.90
0.25
61.0
69.0
19.0
086
Pertemanno
80.0
68.0
0.20
18.0
Flatiiy
67.0
Viádky
0.80
0.15
16.0
77.0
66.0
0.60
15.0
76.0
64.0
7500
0,50
63.0
13.0
0.4
0.22
0.0
0.2
0.4
-0.4
0.22
0.0
0.2
0.4
-0.4
-0.2
0.0
0.2
0.4
(g) Effects of ablation strength (h) Effects of ablation strength (i) Effects of ablation strength on Thailand reasoning performance Bengali reasoning performance and Swahili reasoning performance output fidelity.
and output fidelity.
output fidelity.
Figure 12: Effects of ablation strength on each language. The backbone is Qwen-2.5-7B. IBIIntuuct..
33

---

QweaZ,5-3B-inn B-Instruct
Qwen3-1.77
0.90
58.0
0.85
70.0
0.70
Performanie
56.0
Fidelity
Performanee
68.0
0.60
Fidelity
0.75
0.90
54.0
0.70
0.40
0.65
62.0
0.60
50.0
60.0
8
16
24
(a) Results Qwen-2.5-3B-Instiee
(b) Results on Qwen-3-1.7B.
Qwen3-8R
DeppSec-RI Baaeid-RR--isii--aaaaa
85.0
0.95
84.5
0.90
84.0
0.50
Performanee
Performancs
Hidelty
66.0
0.85
Pidelity
83.5
0.50
0.80
64.0
0.75
83.0
0.40
62.0
82.5
0.65
82.0
0
8
16
20
24
28
32
8
12 16
20
24
28
36
(c) Results on Qwen-3-8B.
(d) Results on RI-Distill DI-Dstilll-Qeen- 14B.
Figure 13: Layer- impact of language-reasoning disentanglement on MGSM accuracy and out- put fidelity. The x-axis denotes the starting layer index of the intervention. Most layers support effective disentanglement that improves reasoning performance.
These consistent trends across model families and sizes further support the generality of clusions and suggest that middle-layer intervention broadly effective strategy for improving multilingual reasoning LLMs.
G.3 Impact of Dataset Difficulty and Translation Quality
While prior experiments primarily focus established multilingual reasoning datasets such as MGSM, these benchmarks may not fully capture the difficulty spectrum of modern reasoning tasks. To more comprehensively assess our approach under varying task complexity and translation fi- delity, we further evaluate on PolyMath [Wang et al. 2025]. a recently introduced human-verified multilingual dataset for mathematical reasoning.
The Math dataset contains problems annotated with three difficulty levels-low medium, and high- high-allowing us to examine reasoning robustness across linguistic and cognitive complex- ity. For each target language, we randomly sample 50 questions per difficulty level and compute difficulty-weighted accuracy score. Experiments are conducted on two representative models, DeepSeek- R1 Sel--iitill--weett -wen-7B and Qwen-2.C 5-an-2.5-Insteeee S-nsttruct-t.. 7B, both before and after lengungg-reasoning disentanglement.
Results in Table 5 show that higher-quality and lifficulty-bllancee datasets substantially reduce the reasoning performance gap between English and other high-resource languages (e.g., Span- ish, French). However, mid- and low-resource languages such as Swahili and Telugu continue to underperform, suggesting that structural capability disparities persist even with improved data. Im- portantly, our intervention helps close this gap: after language-reasonin disentanglement, weaker languages (c.g., Russian in DeepSeck, Thai in Qwen) exhibit notable gains, achieving performance
34

---

Table 5: Multilingual reasoning performance PolyMath datasets across different languages, be- fore and after language-reaoning disentanglement within the activation spaces of the backbone models (+ L-R Disentangle). The best results highlighted in bold. The values in parentheses indicate language fidelity to indicate input-output consistency.
High-Resouree
Hld-Reeourcee Th
Low-Resource Bn
AVG.
Fr
De
Zh
Ru
Te
Sw
Qwen-2.5 S0wen-2,5-Instreee I--struuut--7
23.14
23.14
22.29
21.43
18.29
20.00
28.86
16.57
14.29
17.14
9.43
19.51 (42 001)
L-R Disentangle
28.86
25.14
20.57
22.00
23.43
28.00
21.43
17.14
16.29
22.47 (42.00%)
RI-Distill-ueen:
46.29
41.14
49.14
45.71
39.71
42.29
28.
42.29
21.14
40.91 94%)
+ -I.R Disentangle
50.29
50.86
48,00
51.14
46.57
43.43
46.86
41.71
34,00
45.14
21.71
43.31 (44.18%)
levels comparable to stronger languages. These findings demonstrate both the practical value and the robustness of our approach under more controlled and linguistically diverse evaluation conditions.
H Configuration for Post-training
Supervised Fine-tuning (SFT) All training experiments are conducted on eight A100 GPUs us- ing the LLaMA- Factory repository [Zheng al. 2024 For distributed training. we leverage the DeepSpeed [Rasley et al.) [020] framework with ZeRo-2 optimization. The optimizer is Adam W. We train the model for epochs on the multilingual version of the MATH dataset (7,500 samples), with total batch size of 32, a learning rate of le-5 and the max context length of 8, 1922 The learning rate follows a cosine annealing schedule with 10% warm-up steps.
Reinforcement Learning (RL) For RL, we apply the PPO algorithm (Schulman et al. 2017 using the OpenRLHF framework (Hu et al. 2024]. Each training run consists of 3 episodes, with rollouts per sample. We train for epoch on the same multilingual MATH dataset, using batch size of 96. The maximum input length is set to 1,024 tokens, and the maximum output length is 8,192
tokens to accommodate long reasoning traces. The actor model is optimized with a learning rate of 5e-7, while the critic uses a higher learning rate of 9e-6. We apply a KL penalty coefficient 0.01 to stabilize training and prevent the actor from drifting too far from the initial policy.
35


Reviews:
[{"review_id": "EjDA0JAryB", "reviewer": "Reviewer_trBL", "summary": "This paper proposes a method to improve multilingual reasoning in large language models (LLMs) by disentangling language and reasoning representations. The authors hypothesize that, similar to the human brain, language and reasoning are separable components in LLMs. They introduce a technique to ablate language-specific information from the model's hidden states during inference. Their experiments, conducted on 10 open-source LLMs across 11 languages and 3 benchmarks, show that this intervention consistently improves multilingual reasoning performance. The paper also provides an analysis of the effects of this ablation, showing that it encourages the model to rely on a more language-agnostic representation for reasoning, with English often serving as an anchor.", "questions": "1.  The paper's results seem to depend heavily on the choice of layers for ablation. Could you clarify which specific layers were selected for the results presented in Tables 1, 2, and 3, and what was the rationale for this choice?\n\n2.  Was language re-injected (decoupling) for the results in the main tables to achieve the high language fidelity, or was it pure ablation? If it was re-injected, could you provide the implementation details (e.g., which layers, what rule was used, similar to Equation 3)?\n\n3. For a given prompt language, maybe it would be more relevant to only ablate/suppress (idiosyncrasies of) *other* languages and not the prompted one. Maybe this would allow to strongly ablate to get better reasoning accuracy while preserving language fidelity (without re-injecting/decoupling).", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The paper addresses the important and challenging problem of improving multilingual reasoning in LLMs.\n- The proposed method is intuitive, training-free, and shows promising results on a wide range of models and languages.\n- The paper includes a good amount of analysis to understand the effect of the proposed intervention.\n\n## Weaknesses\n\n### Unclear methodology\n\n- The difference between *ablation* and *decoupling* seems very important and is unclear in the paper as is.\n- As far as I understand, ablation = suppressing language in early/mid layers and decoupling = ablation + re-injecting language in later layers.\n- Apart from Figure 4 which is explicitly about decoupling, it seems that all results are about ablation, is it really the case?\n- There are no implementation details on the re-injection. Which layer did you choose and why? Do you use a similar rule as (3)?\n\n### A claim is not supported\n\nHere I will assume that the dashed lines on Figure 3 and 4 represent the performance of base models.\n\nFrom lines 230-233: \"As shown in Figure 4, we observe that ablation applied at most layers, especially from lower to middle depths, consistently improves reasoning accuracy while maintaining stable output fidelity. This suggests that language–reasoning decoupling is broadly effective across the network and does not require precise targeting to specific layers to yield benefits.\"\n\n- While it is true that reasoning and response language fidelity remain high, only QwQ actually benefits from the decoupling in terms of reasoning accuracy, while Qwen2.5's accuracy is even impaired.\n- Figure 3 and 4 show that there is a trade-off and that it seems not possible to improve reasoning without impairing language with this decoupling method. Therefore not successfully decoupling language and reasoning.\n- This makes me wonder:\n   * Which ablation layer has been chosen for the results in the different tables?\n   * Has language been re-injected (decoupling instead of ablation) to ensure the high language fidelity displayed?\n\n### Poor visualization of results\n\nThe main tables (1, 2, and 3) are very cumbersome to read. They would be easier to parse and the global effect of the ablation would be clearer if the results were presented as barplots or lineplots (e.g., Table 1 could be a series of barplots, one for each language, with models on the x-axis and performance on the y-axis, using shading to distinguish base vs. ablated models). Additionally, the captions for Figures 3 and 4 are incomplete, as they do not define the dashed lines, which presumably represent the baseline performance.\n\n\n### Misc\nThe paper would be stronger if the limitation section from the appendix was moved to the main body. Furthermore, the paper's checklist incorrectly claims that statistical significance is reported. The checklist states, \"Statistical significance of the experiments can be found in Appendix F,\" which is not true; Appendix F contains no such information.", "was_revised": true, "final_justification": "The authors' clarifications have successfully addressed all of my concerns.\n\nThe explanation of the distinction between ablation and decoupling, along with the provided table detailing the specific layers used for each model, resolves the main ambiguities in the methodology. Including this information in the revised manuscript will significantly improve the paper's clarity and reproducibility.\n\nGiven these comprehensive revisions, I am confident that the paper will be much stronger. I revised the scores accordingly.", "rebuttal": "Thank you for your thoughtful and constructive review. We sincerely appreciate your feedback and address each of your concerns point by point below.\n\n---\n\n> **W1: Unclear methodology**\n\nThank you for the valuable feedback, and we apologize for the confusion.\n\nYour understanding is correct: ablation suppresses language signals in mid layers, while decoupling further re-injects them in higher layers. All reported main experimental results are based on decoupling, as stated in Implementation Details (Lines 172–173).\n\nRegarding re-injection, we use a similar layer-wise rule as in Equation (3), which is used for ablation; the only difference is that we add back the language component instead of subtracting it. The exact layers for ablation and re-injection per model are listed below:\n\n| Model Name                | Number of Layers | Ablation Layers | Re-Injection Layers |\n|---------------------------|------------------|---------------|---------------|\n| Qwen-2.5-Instruct-3B      | 36               | 12-26         | 27-35         |\n| Qwen-2.5-Instruct-7B      | 28               | 10-19         | 20-27         |\n| Qwen-3-1.7B-Thinking       | 28               | 10-19         | 20-27         |\n| Qwen-3-4B-Thinking         | 36               | 12-26         | 27-35         |\n| Qwen-3-8B-Thinking         | 36               | 12-26         | 27-35         |\n| R1-Distill-Qwen-7B        | 28               | 10-19         | 20-27         |\n| R1-Distill-LLama-8B       | 32               | 12-22         | 23-31         |\n| R1-Distill-Qwen-14B       | 48               | 16-33         | 34-47         |\n| GLM-Z1-9B                 | 40               | 12-30         | 31-39         |\n| QwQ-32B                   | 64               | 20-46         | 47-63         |\n\n---\n\n> **W2: Unsupported claim**\n\nThank you for your detailed comments. We would like to clarify the experimental settings and improve our descriptions accordingly.\n\nIn our experiments, we roughly divide the model layers into lower, middle, and higher layers.\n- In **Figure 3**, we apply **ablation only in the middle layers**.\n- In **Figure 4**, we apply **ablation only on small groups of consecutive layers**.\n- In **Table 1 and Table 2**, we apply **ablation in the middle layers followed by re-injection in the higher layers**—i.e., full **decoupling**.\nThe exact ablation and re-injection layers for each model are listed in our response to W1.\n\nAs you correctly observed in **Figure 4**, ablating different layers does not always improve reasoning accuracy. Our wording here was not sufficiently rigorous. Compared to language fidelity, ablation at certain isolated layers can have unstable effects on reasoning. While **Figure 3** shows that middle-layer ablation consistently boosts reasoning, **Figure 4** focuses on the impact of ablation on **language fidelity**—especially the sharp drop when ablation is applied in **higher layers**.\n\nThis is precisely why our main setup performs **middle-layer ablation and top-layer re-injection**: to improve reasoning while preserving output fluency and consistency.\n\nWe fully agree with your point about the **reasoning–fidelity trade-off**. This trade-off is indeed present under ablation alone, and motivates our decoupling strategy.\n\nThank you again for the constructive feedback. We will revise the manuscript to make these distinctions and justifications clearer.\n\n---\n\n> **W3: Poor visualization of results**\n\nThank you for the helpful suggestion.\n\nWe fully agree that the main tables could be made more readable and the overall effect clearer through visualizations. In the revision, we will include bar/line plots for Tables 1–3 and improve the figure captions by explicitly defining the dashed lines (which indicate baseline performance).\n\nWe appreciate your feedback and will revise accordingly.\n\n---\n\n> **W4: Misc**\n\nWe agree that moving the Limitation section into the main body will improve clarity and completeness, and we will make this change in the revision.\n\nRegarding the checklist: you are right—Appendix F does not currently include statistical significance results. This was an oversight on our part. We have conducted significance tests on the multilingual results, showing that our method yields statistically significant gains (p < 0.05) on most benchmarks. We believe this further supports the effectiveness of our approach, especially given its low cost and wide language coverage.\n\nWe will include the statistical analysis and correct the checklist in the revised version. Thank you again for your careful review.\n\n---\n\n> **Q1: choice of layers for ablation**\n\nFor Tables 1, 2, and 3, we apply **ablation in the middle layers** and **re-injection in the higher layers**. The specific layer ranges used for each model are provided in our response to W1. This choice is inspired by recent interpretability studies on multilingual representations in LLMs (see Lines 241–242).\n\nWe will include this information directly in the revised paper for clarity.\n\n---\n\n> **Q2: Regarding Re-Injection**\n\nYes, language was re-injected in the higher layers for all main results—i.e., we used **decoupling**, not pure ablation. The specific re-injection layers for each model are listed in our response to W1.\n\nThe implementation is nearly identical to ablation, except that we **add** the language component back instead of subtracting it.\n\n---\n\n> **Q3: only ablate/suppress (idiosyncrasies of) other languages**\n\nThank you for the insightful suggestion.\n\nWe have tried using a language subspace that excludes the prompt language. However, in this case, the prompt has very little projection onto the subspace, so the ablation has minimal effect—leading to negligible changes in reasoning performance.\n\nWe agree this is an interesting direction and worth further exploration.\n\n---\n\nOnce again, thank you for your constructive feedback. We hope our response has addressed your concern."}, {"review_id": "wcUHuNjk3n", "reviewer": "Reviewer_RfPJ", "summary": "The work uses a method to disentangle language specific and reasoning processing in language models. This is done by first constructing mean embedding matrix M for multilingual latent space (using embeddings of the final token triggered by samples of all available languages) and computing via singular value decomposition (SVD) the language specific subspace M_s (based on formulation of M introduced in previous works), which is used to project out components of hidden activations h from any network layer. This is supposed to remove language specific variations and to ensure remaining representation \\hat{h} reflects only language agnostic reasoning relevant content. Authors perform then interventions on activation space on a diverse set of language and reasoning models. For the interventions, they remove language specific components from activations in middle layers, while plugging those back in upper layers. Doing so, authors observe improvements on broad range of language understanding and reasoning benchmarks (MGSM, XWinograd, M-MMLU), across multiple languages. Further, they compare for Qwen 2.5 Instruct 3B the training free intervention with SFT and RL posttraining, observing in majority of the cases activation intervention being as effective or even outperforming posttraining. Authors conclude that their training free method to dampen language specific activations in mid layers while keeping those intact in top layers is significantly improving problem solving performance and argue for benefits of disentanglement of core reasoning and language specific processing.", "questions": "Following the weaknesses section,\n\n1. How is language generation affected by the intervention, is there any way to check whether it is still intact or breaking down?\n2. Can we be sure that intervention based on M_s is special and eg even random projection to subspace of same rank would not give similar positive effects?\n3. Is there any conclusion from observations done in the study on how to improve pre-training in general using language-specific and language-agnostic reasoning disentanglement?\n4. Is the benefit of training free vs SFT/RL posttraining limited to smaller scale 3B only? Will it hold eg for 7B as well?", "limitations": "Following the weaknesses section -\n1. No conclusion can be made for effect of intervention on language generation capability. UPDATE: experiment done\n2. Claimed benefit of training free intervention vs SFT/RL posttraining is limited to one small scale only (3B)", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The strengths of the study:\n\n1. Training free method that according to evaluations results in significant improvement on number of language understanding and reasoning benchmarks across various high, mid and low resource languages\n2. Thorough experiments looking at the effect of activation interventions across various layers, allowing for detailed conclusion about differential effect of the intervention on the middle and top layers.\n3. Evaluations done on variety of relevant models.\n4. Important for dealing with multi-lingual setting, offers interesting angle on language specific and language agnostic function.\n\n\nThe weaknesses:\n1. It is not clear how ablations affect the language generation quality in multi lingual setting. Authors show improvements on problem solving reasoning benchmarks. Those can be entirely decorellated from language generation capability, eg model can solve reasoning tasks posed in language X but entirely fail to generate fluent text in the same language. As authors argue for benefits in multi lingual setting, I think it is important to understand how language generation is affected by the intervention (eg using evals like ROUGE, BLEU, METEOR, etc)\n2. Control for the causal effect of the intervention is missing. It can be that other various projections into subspace of same rank would affect the reasoning quality. As authors explicitly argue for causal effect, control eg, comparison of authors methods to projection on random subspace of same rank with corresponding activation interventions based on that, is required to check this claim.\n3. Authors do not offer any hints or experiments for improving training. If it is true that disentanglement of language specific and language agnostic reasoning core is beneficial for reasoning function, while retaining language comprehension and generation capabilities, it would be important to provide at least some thoughts on how pretraining can be shaped to improve the model function accordingly (even better would be of course experiment showing that it works)\n4. The comparison to SFT and RL is done only for rather small scale 3B model. It is well known that larger scale models benefit more from posttraining. It would be insightful to see how training free intervention compares to post training on larger scales, eg at least 7B. Claims that training free intervention matches or outperforms posttraining are in current form limited to 3B scale only and claim should be correspondingly toned down.\n5. Authors repeatedly refer to open-weights models as open-source models. This is not correct. Open source models allow full study and modification of their origins including pre-training procedure, for which fully open and available data is required. Many of the models referred by authors, eg Llama or Qwen families, do not have open data, and are thus not open-source, but only open weights.\n6. The source code offered by authors in supplementary for reproduction is hardly documented, it is not straightforward to repeat experiments in the current form as described in the paper.", "was_revised": true, "final_justification": "Authors provided extensive experiments per request that substantially strengthened the work. I increase therefore the score accordingly from initial 3 to 5.", "rebuttal": "Thank you for your thoughtful and detailed review. We appreciate your feedback and address each point below.\n\n---\n\n> **W1: Impact on multilingual language generation quality**\n\nThank you for the suggestion. To address your concern, we conducted additional **machine translation** experiments by translating English sentences into target languages, in order to assess the impact of our method on language generation quality.\n\nSpecifically, we used the **Flores-101** dataset and sampled **200 sentences per language**. We evaluated **ROUGE** and **BLEU** scores on **Qwen2.5-7B-Instruct** and **DeepSeek-R1-Distill-Qwen-7B**.\n\n- DeepSeek-R1-Distill-Qwen-7B\n\n|Model|ROUGE-1|ROUGE-2|ROUGE-L|BLEU|\n|---|---|---|---|---|\n|Origin Model|26.80|10.15|24.17|8.63|\n|Ours|**27.74**|**10.67**|**25.30**|**9.07**|\n\n- Qwen2.5-7b-Instruct\n\n|Model|ROUGE-1|ROUGE-2|ROUGE-L|BLEU|\n|---|---|---|---|---|\n|Origin Model|38.21|19.82|35.36|17.00|\n|Ours|**38.82**|**20.18**|**36.03**|**17.43**|\n\nResults show that our method does **not degrade language generation quality** in the prompted language. We will include these results in the revision.\n\n---\n\n> **W2: Causal effect of projection intervention**\n\nThank you for this important point. We fully agree that without a **random subspace control**, it is difficult to establish the **specificity** of the language subspace used in our method.\n\nTo address this, we generated **random subspaces of the same rank** as our extracted multilingual subspace and replaced the original subspace in the intervention. We then evaluated the performance on **MGSM** using **Qwen2.5-7B-Instruct** and **DeepSeek-R1-Distill-Qwen-7B**.\n\n- DeepSeek-R1-Distill-Qwen-7B\n\n|Model|Performance|Fidelity|\n|---|---|---|\n|Origin Model|56.11|90.98|\n|Ours|**58.51**|**92.18**|\n| w/ random subspace|55.27|89.67|\n\n- Qwen2.5-7b-Instruct\n\n|Model|Performance|Fidelity|\n|---|---|---|\n|Origin Model|69.82|79.75|\n|Ours|**70.76**|**84.44**|\n|w/ random subspace|68.65|79.20|\n\nThe results show that with random subspaces, both **performance and fidelity degrade**, whereas our method yields **consistent improvements**. This supports a **causal link** between the extracted language subspace and the observed reasoning gains. We will include these control results in the revision.\n\n---\n\n> **W3: Implications for training**\n\nThank you for the suggestion.\n\nWe have discussed the potential integration of our **language–reasoning decoupling** approach into training in **Appendix A (Lines 640–646)**. For example, using the ablated representation as a **supervision signal** could encourage **language-invariant intermediate representations** during supervised fine-tuning or reinforcement learning, potentially improving generalization.\n\nHowever, implementing and validating this at training time is beyond the scope of this paper and is left as future work.\n\n---\n\n> **W4: Limited scale of post-training comparisons**\n\nThank you for the suggestion. We agree that comparing against post-training methods at larger scales would strengthen the paper. Due to **limited computational resources**, we were unable to run such large-scale post-training baselines.\n\nHowever, as shown in **Tables 1 and 2**, our **training-free intervention consistently improves multilingual reasoning** across a wide range of model sizes, from **1.7B to 32B**. These results demonstrate the scalability and effectiveness of our method, even without additional training.\n\n---\n\n> **W5: Clarification on open-source terminology**\n\nThank you for pointing this out. We acknowledge the misuse of the term “open-source models.” In the revision, we have corrected all instances to “open-weight models” to accurately reflect that these models only release weights, not training data or pretraining details.\n\n---\n\n> **W6: Reproducibility of code**\n\nThank you for the feedback. We apologize for the incomplete documentation.\n\nThe released code covers all core components, including **vLLM-based intervention**, **language subspace extraction**, and **main evaluation scripts**, and supports reproducing key results.\n\nWe are committed to improving reproducibility and will release a fully documented version shortly. Thank you again—we appreciate the suggestion and will address it promptly.\n\n---\n\nFor each of the question you raised, we have provided point-by-point responses above. Please kindly refer to our replies to Weaknesses 1–6 for detailed clarifications and corresponding experimental or revision plans.\n\nWe truly appreciate your constructive feedback and hope our responses address your concerns."}, {"review_id": "haMZoGdz99", "reviewer": "Reviewer_iAev", "summary": "The paper proposes a method to disentangle the language-agnostic reasoning capabilities of a multilingual LLM from its language-specific knowledge. They showcase how attenuating the language-specific signals enhance the reasoning performance of multiple LLMs across all the languages supported by the corresponding LLM. Their method, while being pretty lightweight, yields some impressive results comparable to post-training techniques like Reinforcement Learning (PPO).", "questions": "(Q1) In the ‘Implementation Details’ section the authors say \"For each model, we ablate the language-specific components from mid-layer hidden states and then re-inject them at higher layers.” What specific layers are these higher layers?  \n(Q2) In figure 3, it looks like the inverse of the expectation happens in the vicinity of 0. For example, the increase in performance for DeepSeek-R1-Distill-Qwen-7Band QwQ when the ablation strength is -0.1, and the decrease in performance for Qwen2.5-7B-Instruct when the ablation strength is 0.1. What are the authors' intuitions for these?  \n(Q3) In Figure 4, ablation in the later layers leads to much sharper drops in reasoning fidelity, even if not the worst performance in the case of  DeepSeek-R1-Distill-Qwen-7Band QwQ.This appears to influence the author’s decision to reintroduce language-specific components at higher layers (as mentioned in the implementation section). A discussion on why preserving reasoning fidelity is important would be useful.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths:\n(S1) The paper is well written, and the method is simple and easy to integrate in a pre-existing deployed LLM with no training required.  \n(S2) Various types of LLMs like non-thinking Qwen, distilled reasoning models like R1-Distill-Qwen and reasoning models training using RL like QwQ are studied which covers the broader categories of language models.  \n(S3) Solid analysis across models and layers and good choice of metrics like reasoning fidelity, generation fidelity alongside performance. This helps paint a good picture of how and where the proposed method helps.  \n\n## Weaknesses:\n(W1) The language choices are diverse but there are only two languages in the mid and low-resource categories so it’s tough to see the performance impact especially on these categories.  \n(W2) On page 7, the authors say “we observe that ablation applied at most layers, especially from lower to middle depths”. It is not clear how this method affects the performance when language is entangled with the knowledge we are measuring the performance on. For example, how does this affect the performance on INCLUDE (https://arxiv.org/abs/2411.19799) which measures the efficacy of LLMs on regional knowledge corresponding to each language’s home countries?", "was_revised": true, "final_justification": "As stated in my review, this is a good paper and the authors have adequately addressed my concerns.", "rebuttal": "Thank you for your thoughtful feedback. We address each of your concerns below.\n\n---\n\n> **W1. Limited number of mid/low-resource languages**\n\nThank you for your comment and suggestion. While we include **11 typologically diverse languages** covering high-, mid-, and low-resource cases, we acknowledge that the number of mid/low-resource languages is still limited. This is mainly due to the **lack of widely adopted multilingual benchmarks** with sufficient coverage of truly low-resource languages. In future work, we aim to incorporate more underrepresented languages, **once suitable benchmarks become available**.\n\n---\n\n> **W2: Entanglement of language and knowledge (e.g., INCLUDE benchmark)**\n\nThank you for raising this important point. To address this, we conduct additional experiments on the **Regionality** subset of the INCLUDE benchmark, using **Qwen2.5-7B-Instruct** and **DeepSeek-R1-Distill-Qwen-7B**. The results are summarized below:\n\n- Qwen2.5-7B-Instruct\n\n|Model|Bn|Te|Zh|Jp|Es|Fr|De|Ru|Avg.|Average Fidelity|\n|---|---|---|---|---|---|---|---|---|---|---|\n|Origin Model|49.40|35.34|67.81|71.06|65.00|63.07|**53.93**|55.98|57.70|11.57|\n|Ours|**51.00**|**36.14**|**70.02**|**75.65**|**67.40**|**67.12**|51.69|**60.56**|**59.95**|**11.66**|\n\n- DeepSeek-R1-Distill-Qwen-7B\n\n|Model|Bn|Te|Zh|Jp|Es|Fr|De|Ru|Avg.|Average Fidelity|\n|---|---|---|---|---|---|---|---|---|---|---|\n|Origin Model|34.54|29.72|50.30|56.09|54.60|49.87|43.82|**49.80**|46.09|12.37|\n|Ours|**41.37**|**32.13**|**51.11**|**57.09**|**56.00**|**53.10**|**51.69**|48.61|**48.89**|**12.41**|\n\nWe find that even under strong language–knowledge entanglement, our method still yields **consistent improvements**, suggesting that it maintains **robust reasoning** while preserving **localized knowledge fidelity**. We will include this analysis and discussion in the revised version to highlight the **generality and practical value** of our approach.\n\n---\n\n> **Q1: What are the higher layers used for re-injection?**\n\nThank you for the question. The specific middle and higher layers used for ablation and re-injection vary by model. The selection is roughly based on dividing the model into three segments (lower, middle, higher), as inspired by recent interpretability studies. The exact layer ranges for each model are as follows:\n\n|Model Name|Number of Layers|Middle Layers|Higher Layers|\n|---|---|---|---|\n|Qwen-2.5-Instruct-3B|36|12-26|27-35|\n|Qwen-2.5-Instruct-7B|28|10-19|20-27|\n|Qwen-3-1.7B-Thinking|28|10-19|20-27|\n|Qwen-3-4B-Thinking|36|12-26|27-35|\n|Qwen-3-8B-Thinking|36|12-26|27-35|\n|R1-Distill-Qwen-7B|28|10-19|20-27|\n|R1-Distill-LLama-8B|32|12-22|23-31|\n|R1-Distill-Qwen-14B|48|16-33|34-47|\n|GLM-Z1-9B|40|12-30|31-39|\n|QwQ-32B|64|20-46|47-63|\n\n---\n\n> **Q2: Unexpected effects near ablation strength 0**\n\nIndeed, some models exhibit **minor deviations from the overall trend** when the ablation strength is near 0. Upon further analysis, we found that this is primarily due to **Japanese and Thai**, which behave anomalously in both performance and language fidelity in this narrow region.\n\nWe believe this is related to the **surface-form similarity** between Japanese/Thai and Chinese. \nAs illustrated in **Figures 4 (a)–9 (a)**, the lower-layer representations of Japanese and Thai are quite similar, and as shown in **Figures 4 (c)–9 (c)**, at higher layers, both languages become **closer to Chinese** than to English. In practice, we observed that when slight ablation is applied, the model may even **respond in Chinese to Japanese or Thai prompts**, in contrast to its usual tendency to **reply in English to prompts in other languages**.\n\nOur intuition is that these **“neighboring” languages** do not form fully independent anchors in the multilingual semantic space. As a result, **small changes in ablation strength may trigger abrupt shifts in representation**, leading to local irregularities in both reasoning and language output.\n\nThis highlights the **complex internal dynamics of multilingual LLMs**, particularly for **lower-resource languages that are structurally close to high-resource ones**. We plan to investigate this further in future work by analyzing language clustering in representation space and exploring **language-specific or adaptive interventions** to improve stability and fairness.\n\n---\n\n> **Q3: Why is preserving reasoning fidelity important?**\n\nThank you for the question. We believe that preserving language fidelity is crucial for multilingual LLMs.\n\nOn one hand, it ensures that users can interact with the model **fluently in their native language**, which is essential for usability and accessibility. On the other hand, language consistency reflects the model’s **ability to understand and maintain context**, supporting accurate and coherent information exchange.\n\nTherefore, we view maintaining **alignment between input and output languages** as a necessary goal for practical multilingual applications.\n\n---\n\nThank you again for your valuable feedback and we will incorporate your suggestions to improve the paper."}, {"review_id": "VThhpHXS27", "reviewer": "Reviewer_EnSg", "summary": "This paper studies multilingual reasoning in LLMs. The issue in multilingual reasoning is that the reasoning ability is usually better in high-resource languages such as English and Chinese. \n\nThe paper argues that (i) language and (ii) reasoning are separate components in LLMs (similar to in humans as studied in neuroscience), and thus they can be disentangled to enhance multilingual reasoning. \n\nSpecifically, the paper takes the embedding of the final token as a representation of a document (in a certain language), and takes the average over all documents from the same language to obtain the mean embedding vector $\\textbf{m}_l$. With the mean vectors of all languages, SVD is applied to decompose them into two orthogonal spaces: (1) $M_a$ shared semantic content (of lower rank = 1) and (2) $M_s$ language-specific. To remove language-specific features, any hidden representation $h$ can be projected to the $M_s$ subspace, and then subtracted from $h$ to obtain the language-agnostic part. The paper empirically validates this process by providing PCA visualisation showing clusters split by languages.\n\nBased on the decomposition framework, the intervention (i.e., making the representation language-agnositic), is applied at inference time on MGSM, XWinograd, M-MMLU on multiple LLM families. Results show consistent performance improvements across languages, LLM families, and datasets.\n\nFurthermore, ablation4.1 shows that language-specific features (controlled by the amount to be subtracted in the intervention step) negatively correlate with reasoning accuracy. Ablation4.2 shows that the findings generally hold across layers (except the very last one). Ablation4.3 shows that the proposed intervention can provide performance improvements as much as post-training.", "questions": "Please refer to the weakness, especially weakness1. I believe the work (and its findings) can have more impact to the broader ML/NLP community if the invention can be integrated into post-training, allowing trained models to perform better at inference without any additional computation cost.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper provides useful insights related to multilingual reasoning, that language and reasoning could be decoupled.\n2. The intervention method, originally serving as a probing mechanism, allows improved performance (as much as post-training)\n3. The paper provides several ablation studies to strengthen the general finding about language and reasoning decomposition.\n\nWeaknesses\n1. The findings in this paper could have a limited impact. First, from the performance perspective, although the intervention can provide performance improvement, its additional computation could make it less practical. Second, it’d be better if the paper went deeper into suggesting how post-training could adopt this intervention into improving multilingual reasoning at the post-training stage (instead of having to apply it at inference time).\n2. [minor point] I’m not sure if the datasets (e.g., MGSM) are too easy for current LLMs, or if the drop in performance in low-resource languages might actually come from poor machine translation (when the datasets were constructed)?", "was_revised": true, "final_justification": "Given the strengths of the paper with the weaknesses -- computational overhead and potentially limited benefit in maths. The main contribution will lie in the insights & findings of the paper (e.g., time-time decomposition & intervention) -- the rebuttal provides further details about these two points; however, they can still be seen as limitations for the work considered as a methodology contribution. My original assessment is already leaning accept, so I'll keep this original assessment, but I raised its clarity", "rebuttal": "Thank you for your thoughtful feedback. Below we address your concerns point by point.\n\n---\n\n> **W1.1: its additional computation could make it less practical**\n\nThank you for your feedback. Regarding the concern on computational overhead, we conducted a speed test on the **MGSM dataset** using **DeepSeek-R1-Distill-Qwen-7B** on a **single NVIDIA A100 GPU**. \n\n|Model|Inference speed (token/s)|\n|---|---|\n|Origin Model|2618.63|\n|Ours|2342.98|\n\nThe results show that our method causes **minimal slowdown** in inference speed.\n\nImportantly, the intervention itself involves only a **lightweight projection and addition operation**, which is negligible compared to the total cost of running the LLM. The slight speed drop mainly comes from our **vLLM-based implementation**, not from the method itself.\n\nWe are actively working on optimizing the code to further reduce overhead and bring the inference speed closer to the original model, enhancing the method’s practicality.\n\n---\n\n> **W1.2: post-training integration**\n\nWe have discussed the potential integration of our **language–reasoning decoupling** approach into training in **Appendix A (Lines 640–646)**. For example, using the ablated representation as a **supervision signal** could encourage **language-invariant intermediate representations** during supervised fine-tuning or reinforcement learning, potentially improving generalization.\n\nHowever, implementing and validating this at training time is beyond the scope of this paper and is left as future work.\n\n---\n\n> **W2: Dataset difficulty and translation quality**\n\nThank you for the thoughtful observation. We agree that some of the datasets used, such as MGSM, are relatively easy for modern LLMs.\n\nTranslation quality is also a valid concern. To address this, we introduced **PolyMath** [1], a **human-verified multilingual dataset**. For each language, we sampled **50 questions from each of three difficulty levels (low, medium, high)** and computed a **difficulty-weighted score**.\n\n- DeepSeek-R1-Distill-Qwen-7B\n\n|Model|Bn|Sw|Th|Te|En|Es|Fr|De|Zh|Jp|Ru|Avg.|Average Fidelity|\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|Origin Model|42.29|21.14|**42.29**|28.29|48.86|46.29|41.14|49.14|45.14|**45.71**|39.71|40.91|41.94|\n|Ours|**45.14**|**21.71**|41.71|**34.00**|**50.29**|**50.86**|**48.00**|**51.14**|**46.57**|43.43|**46.86**|**43.31**|**44.18**|\n\n- Qwen2.5-7b-Instruct\n\n|Model|Bn|Sw|Th|Te|En|Es|Fr|De|Zh|Jp|Ru|Avg.|Average Fidelity|\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|Origin Model|17.14|9.43|16.57|14.29|23.14|23.14|22.29|**21.43**|18.29|20.00|28.86|19.51|42.00|\n|Ours|**18.57**|**16.29**|**21.43**|**17.14**|**25.71**|**28.86**|**25.14**|20.57|**22.00**|**23.43**|**28.00**|**22.47**|**42.00**|\n\nExperiments on DeepSeek-R1-Distill-Qwen-7B and Qwen-2.5-Instruct-7B show that, while higher-quality and balanced datasets reduce the performance gap between English and other high-resource languages (e.g., Spanish, French), **mid- and low-resource languages** like Swahili and Telugu still underperform—confirming that capability gaps persist.\n\nOur method helps **close this gap**, notably **elevating performance** in weaker languages (e.g., Russian in DeepSeek, Thai in Qwen) to levels comparable with stronger languages. This demonstrates the **practical value** and **effectiveness** of our intervention.\n\nReference:\n\n[1] PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts.\n\n---\n\nWe sincerely appreciate your comments and will revise the manuscript to better reflect these clarifications."}]

Author rebuttal:

Meta-review: This paper describes interventions that can be applied to improve the performance of LLMs by decoupling the aspects that are responsible for the details of language encoding from those concerned with logical reasoning.

Strengths:

The reviewers appreciated the fact that the method did not involve extra training and could be applied to a wide range of existing models as shown in the provided results.

In the rebuttal the authors were able to address concerns regarding computational effort with measurements. They also clarified how they chose layers in the model for ablation and re-injection.

The authors also successfully addressed the concerns raised by reviewer  RfPJ regarding evaluating the disfluencies that could be introduced by the intervention in the language generation phase.


All of the reviewers were favorably inclined towards this submission and felt that it represented a useful contribution to improving LLM performance by decoupling language interpretation from reasoning in a way that would allow lower resourced languages to take more advantage of the reasoning power gleaned from higher resourced languages.
Decision: Accept (spotlight)