Title: When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multilingual Reasoning

Large Language Models

Cross-Lingual Generalization
Abstract: Multilingual reasoning remains a significant challenge for large language models (LLMs), with performance disproportionately favoring high-resource languages. Drawing inspiration from cognitive neuroscience, which suggests that human reasoning functions largely independently of language processing, we hypothesize that LLMs similarly encode reasoning and language as separable components that can be disentangled to enhance multilingual reasoning. To evaluate this, we perform a causal intervention by ablating language-specific representations at inference time. Experiments on 10 open-weight LLMs spanning 11 typologically diverse languages show that this language-specific ablation consistently boosts multilingual reasoning performance. Layer-wise analyses further confirm that language and reasoning representations can be effectively disentangled throughout the model, yielding improved multilingual reasoning capabilities, while preserving top-layer language features remains essential for maintaining linguistic fidelity. Compared to post-training methods such as supervised fine-tuning or reinforcement learning, our training-free language-reasoning disentanglement achieves comparable or superior results with minimal computational overhead. These findings shed light on the internal mechanisms underlying multilingual reasoning in LLMs and suggest a lightweight and interpretable strategy for improving cross-lingual generalization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies multilingual reasoning in LLMs. The issue in multilingual reasoning is that the reasoning ability is usually better in high-resource languages such as English and Chinese. \n\nThe paper argues that (i) language and (ii) reasoning are separate components in LLMs (similar to in humans as studied in neuroscience), and thus they can be disentangled to enhance multilingual reasoning. \n\nSpecifically, the paper takes the embedding of the final token as a representation of a document (in a certain language), and takes the average over all documents from the same language to obtain the mean embedding vector $\\textbf{m}_l$. With the mean vectors of all languages, SVD is applied to decompose them into two orthogonal spaces: (1) $M_a$ shared semantic content (of lower rank = 1) and (2) $M_s$ language-specific. To remove language-specific features, any hidden representation $h$ can be projected to the $M_s$ subspace, and then subtracted from $h$ to obtain the language-agnostic part. The paper empirically validates this process by providing PCA visualisation showing clusters split by languages.\n\nBased on the decomposition framework, the intervention (i.e., making the representation language-agnositic), is applied at inference time on MGSM, XWinograd, M-MMLU on multiple LLM families. Results show consistent performance improvements across languages, LLM families, and datasets.\n\nFurthermore, ablation4.1 shows that language-specific features (controlled by the amount to be subtracted in the intervention step) negatively correlate with reasoning accuracy. Ablation4.2 shows that the findings generally hold across layers (except the very last one). Ablation4.3 shows that the proposed intervention can provide performance improvements as much as post-training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper provides useful insights related to multilingual reasoning, that language and reasoning could be decoupled.\n2. The intervention method, originally serving as a probing mechanism, allows improved performance (as much as post-training)\n3. The paper provides several ablation studies to strengthen the general finding about language and reasoning decomposition.\n\nWeaknesses\n1. The findings in this paper could have a limited impact. First, from the performance perspective, although the intervention can provide performance improvement, its additional computation could make it less practical. Second, it’d be better if the paper went deeper into suggesting how post-training could adopt this intervention into improving multilingual reasoning at the post-training stage (instead of having to apply it at inference time).\n2. [minor point] I’m not sure if the datasets (e.g., MGSM) are too easy for current LLMs, or if the drop in performance in low-resource languages might actually come from poor machine translation (when the datasets were constructed)?\n\nQUESTIONS:\nPlease refer to the weakness, especially weakness1. I believe the work (and its findings) can have more impact to the broader ML/NLP community if the invention can be integrated into post-training, allowing trained models to perform better at inference without any additional computation cost.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to disentangle the language-agnostic reasoning capabilities of a multilingual LLM from its language-specific knowledge. They showcase how attenuating the language-specific signals enhance the reasoning performance of multiple LLMs across all the languages supported by the corresponding LLM. Their method, while being pretty lightweight, yields some impressive results comparable to post-training techniques like Reinforcement Learning (PPO).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n(S1) The paper is well written, and the method is simple and easy to integrate in a pre-existing deployed LLM with no training required.  \n(S2) Various types of LLMs like non-thinking Qwen, distilled reasoning models like R1-Distill-Qwen and reasoning models training using RL like QwQ are studied which covers the broader categories of language models.  \n(S3) Solid analysis across models and layers and good choice of metrics like reasoning fidelity, generation fidelity alongside performance. This helps paint a good picture of how and where the proposed method helps.  \n\n## Weaknesses:\n(W1) The language choices are diverse but there are only two languages in the mid and low-resource categories so it’s tough to see the performance impact especially on these categories.  \n(W2) On page 7, the authors say “we observe that ablation applied at most layers, especially from lower to middle depths”. It is not clear how this method affects the performance when language is entangled with the knowledge we are measuring the performance on. For example, how does this affect the performance on INCLUDE (https://arxiv.org/abs/2411.19799) which measures the efficacy of LLMs on regional knowledge corresponding to each language’s home countries?\n\nQUESTIONS:\n(Q1) In the ‘Implementation Details’ section the authors say \"For each model, we ablate the language-specific components from mid-layer hidden states and then re-inject them at higher layers.” What specific layers are these higher layers?  \n(Q2) In figure 3, it looks like the inverse of the expectation happens in the vicinity of 0. For example, the increase in performance for DeepSeek-R1-Distill-Qwen-7Band QwQ when the ablation strength is -0.1, and the decrease in performance for Qwen2.5-7B-Instruct when the ablation strength is 0.1. What are the authors' intuitions for these?  \n(Q3) In Figure 4, ablation in the later layers leads to much sharper drops in reasoning fidelity, even if not the worst performance in the case of  DeepSeek-R1-Distill-Qwen-7Band QwQ.This appears to influence the author’s decision to reintroduce language-specific components at higher layers (as mentioned in the implementation section). A discussion on why preserving reasoning fidelity is important would be useful.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work uses a method to disentangle language specific and reasoning processing in language models. This is done by first constructing mean embedding matrix M for multilingual latent space (using embeddings of the final token triggered by samples of all available languages) and computing via singular value decomposition (SVD) the language specific subspace M_s (based on formulation of M introduced in previous works), which is used to project out components of hidden activations h from any network layer. This is supposed to remove language specific variations and to ensure remaining representation \\hat{h} reflects only language agnostic reasoning relevant content. Authors perform then interventions on activation space on a diverse set of language and reasoning models. For the interventions, they remove language specific components from activations in middle layers, while plugging those back in upper layers. Doing so, authors observe improvements on broad range of language understanding and reasoning benchmarks (MGSM, XWinograd, M-MMLU), across multiple languages. Further, they compare for Qwen 2.5 Instruct 3B the training free intervention with SFT and RL posttraining, observing in majority of the cases activation intervention being as effective or even outperforming posttraining. Authors conclude that their training free method to dampen language specific activations in mid layers while keeping those intact in top layers is significantly improving problem solving performance and argue for benefits of disentanglement of core reasoning and language specific processing.\n\nSTRENGTHS AND WEAKNESSES:\nThe strengths of the study:\n\n1. Training free method that according to evaluations results in significant improvement on number of language understanding and reasoning benchmarks across various high, mid and low resource languages\n2. Thorough experiments looking at the effect of activation interventions across various layers, allowing for detailed conclusion about differential effect of the intervention on the middle and top layers.\n3. Evaluations done on variety of relevant models.\n4. Important for dealing with multi-lingual setting, offers interesting angle on language specific and language agnostic function.\n\n\nThe weaknesses:\n1. It is not clear how ablations affect the language generation quality in multi lingual setting. Authors show improvements on problem solving reasoning benchmarks. Those can be entirely decorellated from language generation capability, eg model can solve reasoning tasks posed in language X but entirely fail to generate fluent text in the same language. As authors argue for benefits in multi lingual setting, I think it is important to understand how language generation is affected by the intervention (eg using evals like ROUGE, BLEU, METEOR, etc)\n2. Control for the causal effect of the intervention is missing. It can be that other various projections into subspace of same rank would affect the reasoning quality. As authors explicitly argue for causal effect, control eg, comparison of authors methods to projection on random subspace of same rank with corresponding activation interventions based on that, is required to check this claim.\n3. Authors do not offer any hints or experiments for improving training. If it is true that disentanglement of language specific and language agnostic reasoning core is beneficial for reasoning function, while retaining language comprehension and generation capabilities, it would be important to provide at least some thoughts on how pretraining can be shaped to improve the model function accordingly (even better would be of course experiment showing that it works)\n4. The comparison to SFT and RL is done only for rather small scale 3B model. It is well known that larger scale models benefit more from posttraining. It would be insightful to see how training free intervention compares to post training on larger scales, eg at least 7B. Claims that training free intervention matches or outperforms posttraining are in current form limited to 3B scale only and claim should be correspondingly toned down.\n5. Authors repeatedly refer to open-weights models as open-source models. This is not correct. Open source models allow full study and modification of their origins including pre-training procedure, for which fully open and available data is required. Many of the models referred by authors, eg Llama or Qwen families, do not have open data, and are thus not open-source, but only open weights.\n6. The source code offered by authors in supplementary for reproduction is hardly documented, it is not straightforward to repeat experiments in the current form as described in the paper.\n\nQUESTIONS:\nFollowing the weaknesses section,\n\n1. How is language generation affected by the intervention, is there any way to check whether it is still intact or breaking down?\n2. Can we be sure that intervention based on M_s is special and eg even random projection to subspace of same rank would not give similar positive effects?\n3. Is there any conclusion from observations done in the study on how to improve pre-training in general using language-specific and language-agnostic reasoning disentanglement?\n4. Is the benefit of training free vs SFT/RL posttraining limited to smaller scale 3B only? Will it hold eg for 7B as well?\n\nLIMITATIONS:\nFollowing the weaknesses section -\n1. No conclusion can be made for effect of intervention on language generation capability. UPDATE: experiment done\n2. Claimed benefit of training free intervention vs SFT/RL posttraining is limited to one small scale only (3B)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to improve multilingual reasoning in large language models (LLMs) by disentangling language and reasoning representations. The authors hypothesize that, similar to the human brain, language and reasoning are separable components in LLMs. They introduce a technique to ablate language-specific information from the model's hidden states during inference. Their experiments, conducted on 10 open-source LLMs across 11 languages and 3 benchmarks, show that this intervention consistently improves multilingual reasoning performance. The paper also provides an analysis of the effects of this ablation, showing that it encourages the model to rely on a more language-agnostic representation for reasoning, with English often serving as an anchor.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper addresses the important and challenging problem of improving multilingual reasoning in LLMs.\n- The proposed method is intuitive, training-free, and shows promising results on a wide range of models and languages.\n- The paper includes a good amount of analysis to understand the effect of the proposed intervention.\n\n## Weaknesses\n\n### Unclear methodology\n\n- The difference between *ablation* and *decoupling* seems very important and is unclear in the paper as is.\n- As far as I understand, ablation = suppressing language in early/mid layers and decoupling = ablation + re-injecting language in later layers.\n- Apart from Figure 4 which is explicitly about decoupling, it seems that all results are about ablation, is it really the case?\n- There are no implementation details on the re-injection. Which layer did you choose and why? Do you use a similar rule as (3)?\n\n### A claim is not supported\n\nHere I will assume that the dashed lines on Figure 3 and 4 represent the performance of base models.\n\nFrom lines 230-233: \"As shown in Figure 4, we observe that ablation applied at most layers, especially from lower to middle depths, consistently improves reasoning accuracy while maintaining stable output fidelity. This suggests that language–reasoning decoupling is broadly effective across the network and does not require precise targeting to specific layers to yield benefits.\"\n\n- While it is true that reasoning and response language fidelity remain high, only QwQ actually benefits from the decoupling in terms of reasoning accuracy, while Qwen2.5's accuracy is even impaired.\n- Figure 3 and 4 show that there is a trade-off and that it seems not possible to improve reasoning without impairing language with this decoupling method. Therefore not successfully decoupling language and reasoning.\n- This makes me wonder:\n   * Which ablation layer has been chosen for the results in the different tables?\n   * Has language been re-injected (decoupling instead of ablation) to ensure the high language fidelity displayed?\n\n### Poor visualization of results\n\nThe main tables (1, 2, and 3) are very cumbersome to read. They would be easier to parse and the global effect of the ablation would be clearer if the results were presented as barplots or lineplots (e.g., Table 1 could be a series of barplots, one for each language, with models on the x-axis and performance on the y-axis, using shading to distinguish base vs. ablated models). Additionally, the captions for Figures 3 and 4 are incomplete, as they do not define the dashed lines, which presumably represent the baseline performance.\n\n\n### Misc\nThe paper would be stronger if the limitation section from the appendix was moved to the main body. Furthermore, the paper's checklist incorrectly claims that statistical significance is reported. The checklist states, \"Statistical significance of the experiments can be found in Appendix F,\" which is not true; Appendix F contains no such information.\n\nQUESTIONS:\n1.  The paper's results seem to depend heavily on the choice of layers for ablation. Could you clarify which specific layers were selected for the results presented in Tables 1, 2, and 3, and what was the rationale for this choice?\n\n2.  Was language re-injected (decoupling) for the results in the main tables to achieve the high language fidelity, or was it pure ablation? If it was re-injected, could you provide the implementation details (e.g., which layers, what rule was used, similar to Equation 3)?\n\n3. For a given prompt language, maybe it would be more relevant to only ablate/suppress (idiosyncrasies of) *other* languages and not the prompted one. Maybe this would allow to strongly ablate to get better reasoning accuracy while preserving language fidelity (without re-injecting/decoupling).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}