Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs

## Abstract

Abstract Enhancing the complex reasoning capabilities of Large Language Models (LLMs) attracts widespread attention.
While reinforcement learning (RL) has shown superior performance for improving complex reasoning, its impact on cross-lingual generalization compared to Supervised Fine-Tuning (SFT) remains unexplored. We present the first systematic investigation into cross-lingual reasoning generalization of RL and SFT. Using Qwen2.5-3B-Base as our foundation model, we conduct experiments on diverse multilingual reasoning benchmarks, including math reasoning, commonsense reasoning, and scientific reasoning. Our investigation yields two significant findings: (1) Tuning with RL not only achieves higher accuracy but also demonstrates substantially stronger cross-lingual generalization capabilities compared to SFT. (2) RL training on non-English data yields better overall performance and generalization than training on English data, which is not observed with SFT.
Furthermore, through comprehensive mechanistic analyses, we explore the underlying factors of RL’s superiority and generalization across languages.
Our results provide compelling evidence that RL enables the model with more robust reasoning strategies, offering crucial guidance for more equitable and effective multilingual reasoning.

## 1 Introduction

Figure: Figure 1: SFT and RL performance improvement using English, Chinese and German training data on the same base model, Qwen2.5-3B-Base. Performance improvements are measured relative to the base model. We report the performance improvement in six language settings.
Refer to caption: x1.png

Multilingualism plays a significant role in human society and occupies a critical position in the development of large language models (LLMs). With over 7,000 languages worldwide, each encapsulates unique cultural contexts and expressive modalities (Campbell & Grondona, 2008). LLMs not only break down language barriers and facilitate cross-cultural communication, but also enable equitable global Artificial Intelligence benefits (Howard & Ruder, 2018; Sharma et al., 2025).

With advances in LLMs reasoning (Hao et al., 2023; Yue et al., 2025), cross-lingual reasoning has received increasing attention (Wang et al., 2024a; Chai et al., 2025; Payoungkhamdee et al., 2025).
Multilingual reasoning requires models to not only comprehend the semantic content of different languages, but also to possess the ability to perform logical inference and problem-solving across diverse linguistic environments (Alam et al., 2024). Current research demonstrates that while large-scale pre-trained language models have achieved remarkable progress in English comprehension and generation, significant performance gaps persist for other languages (Qiu et al., 2024).
Consequently, enhancing multilingual reasoning capabilities and achieving effective cross-lingual generalization has emerged as a significant challenge (Pham et al., 2024; Hu et al., 2025).

Reinforcement Learning (RL) is considered a pivotal tool for enhancing reasoning capabilities (Wang et al., 2024b; Guo et al., 2025). Recent studies show that RL training exhibits superior performance on complex tasks such as mathematical and logical reasoning (Xie et al., 2025). Compared to Supervised Fine-tuning (SFT), Reinforcement Learning, through reward-guided mechanisms, enables models with more robust and generalizable reasoning strategies (Huan et al., 2025). Notably, researches reveal that RL not only significantly improves model performance, but also enables stronger cross-task generalization capabilities (Shao et al., 2024).
In this work, we conduct the first investigation into whether RL exhibits strong generalization capabilities across languages in reasoning. Experimentally, we compare the performance of RL and SFT on diverse languages, exploring their performance across various languages to examine their cross-lingual generalization abilities.
To fully reflect reasoning capability, we evaluate performance on diverse multilingual reasoning
benchmarks, including math reasoning, commonsense reasoning, and scientific
reasoning (Shi et al.,; She et al., 2024; Son et al., 2025; Xuan et al., 2025; Qi et al., 2025).

The empirical investigation yields two notable findings: (1) As illustrated in Figure [1](https://arxiv.org/html/2509.23657v1#S1.F1), RL demonstrates superior performance improvements compared to SFT, with enhanced cross-lingual generalization capabilities. Our results indicate that models trained with RL can more effectively transfer reasoning abilities learned in one language to another. This finding of cross-lingual reasoning is consistent with existing findings for cross-task transfer (Korkmaz, 2024; Huan et al., 2025; Cheng et al., 2025; Chu et al.,). (2)
Given that the pre-training corpora of most existing LLMs are predominantly English-centric (Morishita et al., 2024; Rytting & Wingate, 2021; Singh et al., 2024), the conventional expectation is that RL training with English data would maximally leverage the model’s potential (Yoon et al., 2024; She et al., 2024). However, as shown in Figure 1, our findings surprisingly reveal a counter-intuitive phenomenon: RL training using non-English data (such as Chinese and German) yields better cross-lingual reasoning performance and superior generalization than using English data.
This contrasts with SFT, where no such phenomenon is observed, as performance remains comparable and, in some datasets, even shows an opposite trend.

To investigate underlying reasons for the findings, we conduct preliminary analyses: (1) First, we analyze whether the language used in reasoning is consistent with the language of input question. Our investigation reveals that language inconsistency serves as a potential factor of RL’s cross-lingual generalization, and the superiority of the non-English data in RL. (2) Second, we examine the role of sampling mechanisms in RL’s superior performance. We find that the sampling mechanism in RL explores sufficient and diverse solution paths, allowing models to learn more robust and generalizable strategies. (3) Third, we explore the semantic shift of the model after training. We find that the stability of the semantic space contributes to RL’s superior cross-lingual generalization. Our preliminary explorations provide insights for future research in multilingual reasoning.

The main contributions of this work are as follows:

(1) We present the first systematic analysis of the differences between RL and SFT in cross-lingual reasoning generalization, filling an important gap in this research area.

(2) We reveal two significant findings: 1) RL excels over SFT in cross-lingual generalization, and 2)
Counterintuitively, non-English data is superior to English data for RL training.
To our knowledge, we are the first to demonstrate that using non-English data for RL more effectively enhances performance and cross-lingual generalization, although most models are pre-trained mainly on English.

(3) Through comprehensive analyses, we explore three potential factors underlying RL’s superiority: linguistic inconsistency in reasoning, sampling-driven policy optimization, and the semantic shift after training, which provides a crucial foundation for multilingual reasoning.

## 2 Related Work

Multilingual Reasoning.
Multilingual reasoning is a challenging and representative task for evaluating the intelligence of large language models (Ahn et al., 2024; She et al., 2024; Yoon et al., 2024).  Shi et al. establish the foundation for this field by translating English mathematical problems from GSM8K (Cobbe et al., 2021) into multiple languages, creating the multilingual benchmark MGSM (Shi et al.,). To enhance multilingual reasoning capabilities, existing work primarily employs prompting strategies. Qin et al. (2023) and  Huang et al. (2023) propose a translate-then-solve approach that first translates non-English questions into English before problem-solving, achieving promising results on closed-source models like ChatGPT (Ouyang et al., 2022).
However, less attention has been paid to how different training paradigms affect the model’s intrinsic cross-lingual generalization capabilities. Our work addresses this gap by comparing SFT and RL at the model’s foundational level, demonstrating RL’s unique advantage in learning reasoning strategies without relying on specific languages.

Supervised Fine-tuning For Reasoning.
Supervised fine-tuning (SFT) effectively enhances LLM reasoning abilities by distilling expert-level chain-of-thought (CoT) examples (Huang et al., 2024). Synthetic data generation is a key strategy: Large teacher models are used to generate solutions for mathematical problems (Yue et al., 2023; Tang et al., 2024), enhancing the reasoning process. Additionally, recent research examines the impact of data quality factors on the model performance  (Toshniwal et al., 2025; Yu et al., 2023; Ye et al., 2025). Beyond mathematics, other works (Kim et al., 2023; Xu et al., 2024) expand reasoning tasks to larger domains, broadening the scope and complexity of problem-solving in various fields.
Although SFT is successful in enhancing reasoning, its learning approach is fundamentally based on imitating and memorizing given “expert” trajectories (Ge et al., 2023), leading to overfitting to the language and pattern in the training data. Our research differs from these works by focusing on the limitations of SFT in cross-lingual scenarios. By contrasting it with RL, we demonstrate that merely imitating high-quality CoT data is insufficient for achieving the robust cross-lingual generalization that RL provides.

Reinforcement Learning For Reasoning.
Reinforcement learning (RL) has become a widely adopted technique for post-training large language models (LLMs) to better align their outputs with human preferences (Ouyang et al., 2022; Achiam et al., 2023). Recent studies extend its application to enhancing reasoning abilities, encouraging longer, structured CoT traces and occasional breakthrough moments (Jaech et al., 2024; Guo et al., 2025). These approaches treat chain-of-thought (CoT) reasoning as an RL problem, utilizing various reward mechanisms such as final-answer correctness (Xie et al., 2025; Wen et al., 2025), verifier-based scoring (Gehring et al., 2025), and step-level rewards (Zhang et al., 2025). While online RL approaches (Schulman et al., 2017; Shao et al., 2024) are commonly used, high computational costs motivate the development of offline RL methods (Zhang et al., 2024; Yuan et al., 2025).
Existing work primarily relies on English-centric data in RL.
In this work, we not only validate the effectiveness of RL in a multilingual setting but also innovatively uncover the unique advantages of non-English training data within the RL framework.

## 3 RL improves the generalization across languages

### 3.1 Experimental Setup

Base Model and Datasets.
We adopt Qwen2.5-3B-Base (Yang et al., 2024) as the base model to clearly explore the impact of RL and SFT. The training datasets are translations of GSM8K (Cobbe et al., 2021) and LUFFY (Yan et al., 2025). We use Qwen3-30B-A3B (Yang et al., 2025) to translate the training data into other languages and utilize the model’s verification to further guarantee the quality of the translation.
To further examine the generality of the observed phenomena, we also include SmolLM3-3B-Base (Bakouch et al., 2025) as an additional base model for verification.
To fully assess the reasoning ability, we evaluate the model on multilingual reasoning benchmarks from three kinds of reasoning: MGSM (Shi et al.,), MMath500, and MAIME2024 (Son et al., 2025) for mathematical reasoning, MMLU-ProX-Lite (Xuan et al., 2025) for commonsense reasoning, and MGPQA-D (Qi et al., 2025) for scientific reasoning.

Learning Algorithms and Evaluation Metrics.
We compare the performance of various tuning algorithms, including SFT and RL. Specifically, we use GRPO to explore the performance of RL. The final answer is explicitly distinguished and encapsulated with in a \boxed{}. To evaluate model performance, we calculate the accuracy of each reasoning dataset. We test 6 times for MMath500 and MGSM, and 16 times for MAIME2024. This paper evaluates the reasoning capabilities of LLMs across ten languages: Bengali (Bn), Thai (Th), Swahili (Sw), Japanese (Ja), Chinese (Zh), German (De), French (Fr), Russian (Ru), Spanish (Es), and English (En).
To measure the relative improvement over the base model’s potential, we introduce a generalization score (Gen). This score is calculated by averaging the normalized gains across all test languages, which represents the model’s ability to capitalize on the potential for improvement in each language. For a given tuned model $M_{\text{tuned}}$, the generalization score is defined as:

$$ $Gen(M_{\text{tuned}})=\frac{1}{|L|}\sum_{l\in L}\frac{\text{Acc}(M_{\text{tuned}},l)-\text{Acc}(M_{\text{base}},l)}{1-\text{Acc}(M_{\text{base}},l)}$ $$

where $L$ is the set of evaluation languages, $\text{Acc}(M,l)$ is the accuracy of model $M$ on language $l$, and $M_{\text{base}}$ is the base model before tuning.

Implementation Details.
All experiments utilize full-parameter tuning during both the SFT and RL phases to enable a thorough evaluation of model capabilities. The SFT experiments are carried out within the LlamaFactory (Zheng et al., 2024) framework, employing a learning rate of $2\times 10^{-5}$, a cosine learning rate scheduler, and a batch size of 32. For RL, the verl (Sheng et al., 2024) platform is used for implementation. To guarantee a fair comparison among different RL approaches, a uniform set of parameters is adopted: the learning rate is set to $1\times 10^{-6}$, the rollout batch size to 512, and the sampling temperature to 1.0, along with a KL-divergence coefficient of 0.001.

### 3.2 Finding 1: RL exhibits superior cross-lingual generalization than SFT

Table: Table 1: Performance of base, SFT, and RL models on MGSM. “Base” denotes Qwen2.5-3B-Base. “SFT (zh)” and “RL (zh)” indicate tuning on Chinese data. We report accuracy on 10 linguistic settings; $\Delta$ (RL–SFT) denotes the performance gap. Each value is averaged over six runs. “Avg” and “Gen” refer to the mean accuracy and generalization score, respectively.

Significant performance improvement.
As shown in Table [1](https://arxiv.org/html/2509.23657v1#S3.T1), RL consistently outperforms SFT across ten languages. The improvements range from +9.4 points (evaluating Russian when trained in English) to +34.5 points (evaluating Bengali when trained in German), with an overall average improvement of +17.5 to +25.2 points depending on the training language. This establishes a strong baseline for RL’s superiority. The complete results are provided in Appendix [A.3](https://arxiv.org/html/2509.23657v1#A1.SS3).

Robustness in cross-lingual transfer.
Notably, RL’s advantage is most prominent in cross-lingual transfer scenarios, suggesting it learns more robust reasoning strategies rather than optimizing for the training language. For instance, when trained on Chinese, RL not only excels on Chinese evaluation (+17.6 points over SFT) but also generalizes significantly better to typologically distant languages like German (+25.8 points) and Spanish (+25.4 points). The consistency of these improvements across diverse language pairs (e.g., English-Bengali: +21.3 points) indicates that RL fosters the development of multilingual reasoning capabilities.

Table: Table 2: Performance of base, SFT, and RL models on MMath500. We report the accuracy score on 6 linguistic settings.

Coherent validation.
The validity of this conclusion is further strengthened by consistent results on the MMath500 dataset in Table [2](https://arxiv.org/html/2509.23657v1#S3.T2). For example, when trained on Chinese data, the RL model achieves an average accuracy of 61.3%, substantially surpassing SFT’s 39.8% (a +21.5 point improvement). This cross-dataset corroboration confirms that the enhanced generalization ability of RL is not an artifact of a single benchmark.

Effective generalization across languages in other reasoning tasks.
The superiority of RL extends beyond multilingual mathematical reasoning to challenging out-of-distribution tasks. On benchmarks like MMLU-ProX-Lite and MGPQA-D, RL consistently maintains positive generalization scores, while SFT models often exhibit negative transfer. For instance, on MMLU-ProX-Lite, an RL model trained on German data achieves a generalization score (Gen) of 30.8, starkly contrasting with SFT’s 8.0. This demonstrates that the robust reasoning representations acquired via RL are highly transferable across both linguistic and task boundaries. Notably, this advantage holds even under a double-cross setting: when trained on mathematical data in German (De) and evaluated on a commonsense reasoning task in Chinese (Zh), RL achieves a +20.0 point improvement over SFT.

In summary, results from cross-lingual, cross-dataset, and cross-task evaluations robustly support that RL enables models with superior generalization in multilingual reasoning compared to SFT.

### 3.3 Finding2: RL using non-English training data yields superior performance to English training data, while SFT does not

Superiority performance gains in non-English RL. Analyzing the average performance, RL training on non-English data systematically surpasses the English baseline. Specifically, RL trained on German achieves the highest average performance at 71.5%, followed by French (70.7%) and Japanese (70.9%), shown in Table [7](https://arxiv.org/html/2509.23657v1#A1.T7) in Appendix [A.3](https://arxiv.org/html/2509.23657v1#A1.SS3), all substantially exceeding English-based RL training (62.7%), with the German advantage being a significant +8.8 points.

**Table 3: Performance comparison on MMLU-ProX-Lite and MGPQA-D. “Avg” denotes the average score across languages (En/Zh/De), and “Gen” represents the generalization score.**
| Model | MMLU-ProX-Lite | MGPQA-D |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| En | Zh | De | Avg | Gen | En | Zh | De | Avg | Gen |  |
| Base | 9.2 | 2.4 | 3.6 | 5.0 | 0.0 | 21.1 | 20.0 | 20.7 | 20.6 | 0.0 |
| SFT(En) | 28.9 | 13.0 | 24.1 | 22.0 | 17.9 | 12.5 | 5.1 | 11.7 | 9.8 | -13.7 |
| RL(En) | 40.6 | 31.6 | 25.6 | 32.6 | 29.1 | 30.0 | 23.5 | 25.2 | 26.2 | 7.0 |
| $\Delta$ (RL-SFT) | +11.6 | +18.6 | +1.6 | +10.6 | +11.2 | +17.4 | +18.4 | +13.5 | +16.4 | +20.7 |
| SFT(Zh) | 24.4 | 11.6 | 21.3 | 20.3 | 7.4 | 20.7 | 18.1 | 14.2 | 17.7 | -3.7 |
| RL(Zh) | 40.8 | 35.0 | 34.4 | 36.7 | 33.4 | 25.0 | 27.3 | 28.3 | 26.9 | 7.8 |
| $\Delta$ (RL-SFT) | +16.3 | +23.4 | +13.1 | +16.4 | +19.8 | +4.3 | +9.2 | +14.1 | +9.2 | +11.5 |
| SFT(De) | 15.8 | 7.3 | 15.0 | 12.7 | 8.0 | 7.2 | 12.8 | 8.8 | 9.6 | -13.9 |
| RL(De) | 39.9 | 27.2 | 35.5 | 34.2 | 30.8 | 26.2 | 27.2 | 25.3 | 26.2 | 7.1 |
| $\Delta$ (RL-SFT) | +24.1 | +20.0 | +20.5 | +21.5 | +22.8 | +19.0 | +14.4 | +16.6 | +16.7 | +21.0 |

Further more, the superiority is also pronounced in cross-lingual scenarios. For example, RL trained on German not only excels on German evaluation (80.5%) but also shows remarkable transfer to distant languages like Bengali (63.3% vs 47.5% for English, +15.8 pts) and Thai (75.3% vs 61.0% for English, +14.3 pts), indicating learning of transferable representations.

Similar phenomenon across diverse reasoning tasks. The pattern is consistently validated on diverse benchmarks. On MMLU-Pro-X Lite, from Table [3](https://arxiv.org/html/2509.23657v1#S3.T3), RL trained on Chinese achieves 36.7%, outperforming RL trained on English (32.6%). On MGSM, RL trained on German attains a generalization score of 41.2%, significantly higher than RL trained on English (30.9%), confirming the robust and generalizable benefits of non-English RL training.

The Different Phenomenon observed in SFT.
This phenomenon appears exclusively with RL. In contrast, SFT results exhibit minimal variation across training languages, with Avg scores ranging from 46.3% on German to 47.6% on Japanese (see Table [7](https://arxiv.org/html/2509.23657v1#A1.T7) in Appendix [A.3](https://arxiv.org/html/2509.23657v1#A1.SS3)). The performance differences remain within statistical noise, ruling out data quality as the sole explanation and highlighting the critical role of the RL objective.

### 3.4 Same phenomenon on another base model

In Table [4](https://arxiv.org/html/2509.23657v1#S3.T4), we report the performance of SmolLM3-3B-Base under the same configuration of Qwen2.5-3B-Base. We find that our observations are consistently.

Finding 1: RL exhibits superior cross-lingual generalization than SFT.
Across all training languages, RL consistently and significantly outperforms SFT. The improvements are substantial, ranging from +11.7 points (evaluating Swedish when trained on German) to +34.7 points (evaluating Chinese when trained on German).

Finding 2: RL using non-English training data yields superior performance to English training data, while SFT does not.
Similar to the trend observed on Qwen2.5-3B-Base, RL trained on non-English data surpasses RL trained on English. RL (De) reaches the highest average accuracy at 69.9 and the strongest generalization score at 64.9.
In contrast, SFT models remain far behind.

Table: Table 4: Performance of base model, SFT, and RL tuning models on MGSM. Base denotes the orginal SmolLM3-3B-Base model. We report the accuracy score on 10 linguistic settings.

## 4 Mechanics Analysis of RL’s Cross-lingual Generalization

To investigate why RL exhibits stronger generalization than SFT, and why RL training on non-English data outperforms that on English, we present an abbreviated set of responses generated by RL-trained models on the test set in Table [5](https://arxiv.org/html/2509.23657v1#S4.T5). The complete responses are provided in Appendix [A.4](https://arxiv.org/html/2509.23657v1#A1.SS4).
Our analysis reveals that when models are trained using German instructions during RL training, the resulting models do not strictly adhere to German when generating thinking and responses. Instead, they employ non-German or mixed languages for reasoning processes. This observation attracts our attention and leads us to propose a hypothesis: could this inconsistent language usage in reasoning contribute to the enhanced generalization observed in RL training?

### 4.1 Exploration of language consistency in RL

To empirically validate this hypothesis, we conduct comparative experiments using two distinct approaches: (1) employing prompts that strictly constrain language usage, and (2) incorporating the language consistency reward that encourages the model to adhere to the language of the question into the RL training process. The details are as follows:

$$ $r_{\text{overall}}=0.5r_{\text{acc}}+0.5r_{\text{consistency}}$ (1) $$

Table: Table 5: Case study of on MMath500. We present a case of German question.

Figure: Figure 2: Scores on MMath500. The chart compares the average accuracy of different models. “RL (Zh)” indicates training on Chinese data.
Refer to caption: x2.png

We investigate the impact of language consistency by forcing the model to use a specific language during inference and observing performance changes. The results are presented in Figure [2](https://arxiv.org/html/2509.23657v1#S4.F2) and Table [4.1](https://arxiv.org/html/2509.23657v1#S4.SS1), with a specific case study provided in Table [5](https://arxiv.org/html/2509.23657v1#S4.T5).

We observe two key aspects:
(a) Language inconsistency serves as a potential source of cross-lingual generalization capability, and (b) Building upon this mechanism, RL (De) exhibits greater language inconsistency than RL (En), resulting in superior cross-lingual performance. These observations suggest that the degree of flexibility in deviating from training language constraints may determine the extent of cross-lingual generalization achieved by RL-trained models.

Figure: Figure 3: Model performance comparisons among the Base, SFT, RFT, and RL models on MGSM. We use German data in LUFFY in SFT, RL, RFT for training.
Refer to caption: x3.png

Language inconsistency enhances cross-lingual generalization.
Enforcing language consistency significantly degrades performance. As demonstrated in Figure [2](https://arxiv.org/html/2509.23657v1#S4.F2), both RL (Zh) and RL (De) models achieve strong baseline performance (61.3% and 61.4%, respectively). However, when constrained to use their training languages through prompting, performance drops substantially—RL (Zh) falls to 53.7% and RL (De) to 60.5%. The degradation becomes even more pronounced when language consistency rewards are applied, with RL (De) plummeting to 52.0%. This pattern strongly indicates that enforced language consistency impairs cross-lingual reasoning capabilities.

Table [4.1](https://arxiv.org/html/2509.23657v1#S4.SS1) reveals that unconstrained RL models show low consistency in their training languages—both RL (Zh) and RL (De) achieve 0.0% consistency, indicating they spontaneously adopt other languages during reasoning. Conversely, constrained models exhibit high consistency rates (up to 99.9% for RL (De) with consistency rewards), but at the cost of reduced performance. This negative correlation between language consistency and performance suggests that linguistic flexibility enables models to leverage more powerful, multilingual reasoning modules.

Case analysis of the language inconsistency. As shown in Table [5](https://arxiv.org/html/2509.23657v1#S4.T5), When solving German question, the unconstrained model (RL (De)) employs mixed English and German reasoning and reaches correct solutions, while the consistency-constrained model, despite only reasoning in German, produces flawed logical steps and wrong answers. This demonstrates that constraining models to specific languages may inhibit access to more robust reasoning patterns established during pre-training.

Language inconsistency in non-English RL.
The performance comparison in Figure 2 shows that RL (De) achieves 61.4% average accuracy compared to RL (En)’s 55.7%. More importantly, RL (De) maintains strong performance across different target languages, while RL (En) shows more pronounced degradation in non-English tasks. This suggests that German-based training may provide advantages for cross-lingual generalization.

Different source languages yield distinct generalization patterns. The superior performance of RL (De) may stem from German’s grammatical complexity and its linguistic distance from other languages, potentially encouraging the development of more multilingual reasoning strategies. In contrast, English-based training might lead to more language-specific reasoning patterns that transfer less effectively across languages.

Language consistency in SFT. Unlike RL models, SFT models exhibit high language consistency (e.g., SFT (Zh) maintains 99.7% consistency) due to their imitation-based training paradigm. While this consistency appears desirable, it actually constrains generalization by trapping models within language-specific thought patterns established during training, leading to impaired cross-lingual performance when facing problems in other languages.

### 4.2 Exploration of sampling in RL

To further investigate the source of RL’s advantage over SFT, we analyze the role of sampling in performance enhancement. We introduce Rejection Sampling Fine-Tuning (RFT) (Touvron et al., 2023) as an intermediate baseline between SFT and full RL.
The RFT we use involves sampling multiple times from the model after RL training.
It then fine-tunes the model using only the samples that yield the correct answer. This represents a more on-policy exploration mechanism than SFT.

As shown in Figure [3](https://arxiv.org/html/2509.23657v1#S4.F3), across all languages, accuracy increases progressively from the Base model to SFT, RFT, and finally to the RL-tuned model. Specifically, SFT achieves 46.3% average accuracy, RFT improves to 66.8%, and RL reaches 71.5%. This trend underscores the importance of the model’s exploration of solution paths in enhancing its reasoning abilities.

Better Performance with Data Aligned to the Model’s Distribution.
Although SFT follows a completely correct off-policy solution path, RFT data, more aligned with the model’s distribution, enables the model to explore reasoning chains better suited to its own configuration through sampling. This alignment helps the model capture reasoning patterns and optimization trajectories more effectively, allowing it to generalize beyond memorized solutions.

The Importance of Online Optimization in RL.
Compared to RFT, RL (GRPO in our experiments) continuously performs more on-policy sampling with both positive and negative examples during training. This not only further aligns the data with the model but also goes beyond mere imitation learning. As shown in Figure [5](https://arxiv.org/html/2509.23657v1#A1.F5), RL consistently outperforms the other methods across all languages, demonstrating that the online policy optimization process in RL is more effective at enhancing the model’s generalizable reasoning than RFT.

Figure: Figure 4: Feature of LLM’s hidden state of last layer, in training (dataset LUFFY) configuration of (a)RL-De, (b)RL-De+Prompt, and (c))RL-De+Prompt+Reward. “+Prompt” adds language control prompts, and “+Reward” adds a language consistency reward.
Refer to caption: x4.png

### 4.3 Exploration of model Semantic Feature shift

To investigate why RL training with different languages yields varying generalization capabilities, we analyze the semantic feature shifts in learned representations.

Methodology. We extract final layer hidden states from base and RL-trained models when processing MMath500 test data across six languages. These representations are projected to 2D space using PCA, and we compute difference vectors: $\mathbf{h}_{diff}=\mathbf{h}_{RL}-\mathbf{h}_{Base}$.

Results. Figure [4](https://arxiv.org/html/2509.23657v1#S4.F4) reveals distinct shift patterns across training configurations. RL-De exhibits the most concentrated distribution around the origin, indicating minimal deviation from base representations, while RL-En displays more scattered distributions. This ordering directly correlates with cross-lingual performance in Table [2](https://arxiv.org/html/2509.23657v1#S3.T2). Similarly, language consistency interventions in RL-De progressively increase representational scatter: baseline RL-De maintains compact distributions, while RL-De+Prompt+Reward shows greater dispersion, mirroring the performance degradation pattern.

Interpretation. These findings suggest that pre-training establishes multilingual reasoning structures crucial for cross-lingual transfer (Hua et al., 2024; Merchant et al., 2020). Models preserving these structures through minimal representational drift maintain stronger generalization capabilities. Conversely, larger shifts disrupt universal reasoning mechanisms (Luo et al., 2025), explaining why RL’s linguistic inconsistency paradoxically enhances cross-lingual performance by preserving pre-trained reasoning structures (Lai et al., 2025).

## 5 Conclusion

We systematically investigated the differences between Reinforcement Learning and Supervised Fine-Tuning for enhancing cross-lingual reasoning and the generalization across languages.
Multiple experiments demonstrate that RL not only achieves substantially higher accuracy than SFT but also exhibits superior cross-lingual generalization. Contrary to conventional cognition, we find that RL training on non-English data yields superior performance, challenging English-centric training.
Our preliminary mechanistic analysis investigates the potential reasons for the superior cross-lingual generalization of RL from three perspectives: the linguistic inconsistency during the reasoning process, the unique explore-and-optimize sampling strategy, and the semantic shift after training. The understanding of these potential factors not only provides crucial insights into understanding RL’s advantages in multilingual reasoning but also establishes a foundation for effectively enhancing cross-lingual reasoning in the future.

## 6 Ethics statement

This study acknowledges several ethical implications of its investigation into cross-lingual reasoning in LLMs. Data and fairness concerns arise from potential biases in multilingual benchmarks, which may introduce performance disparities across languages. Our evaluations incorporate diverse linguistic settings and different multilingual reasoning tasks to mitigate such biases, though future work must further scrutinize cultural and linguistic influences on model behavior.

Beyond technical limitations, societal impact requires careful consideration. While improved multilingual reasoning could enhance accessibility for non-English speakers, reducing barriers in education and professional settings, it also risks misuse—such as automated disinformation generation or harmful content propagation across languages. We advocate for responsible deployment, emphasizing robust safeguards, human oversight, and ongoing risk assessments to balance innovation with ethical constraints.

## References

- Achiam et al. (2023)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Ahn et al. (2024)
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin.
Large language models for mathematical reasoning: Progresses and challenges.
In *Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop*, pp. 225–237, 2024.
- Alam et al. (2024)
Firoj Alam, Shammur Absar Chowdhury, Sabri Boughorbel, and Maram Hasanain.
Llms for low resource languages in multilingual, multimodal and dialectal settings.
In *Proceedings of the 18th conference of the European chapter of the association for computational linguistics: tutorial abstracts*, pp. 27–33, 2024.
- Bakouch et al. (2025)
Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Carlos Miguel Patiño, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallouédec, Kashif Rasul, Nathan Habib, Clémentine Fourrier, Hynek Kydlicek, Guilherme Penedo, Hugo Larcher, Mathieu Morlon, Vaibhav Srivastav, Joshua Lochner, Xuan-Son Nguyen, Colin Raffel, Leandro von Werra, and Thomas Wolf.
SmolLM3: smol, multilingual, long-context reasoner.
[https://huggingface.co/blog/smollm3](https://huggingface.co/blog/smollm3), 2025.
- Campbell & Grondona (2008)
Lyle Campbell and Verónica Grondona.
Ethnologue: Languages of the world.
*Language*, 84(3):636–641, 2008.
- Chai et al. (2025)
Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xinnian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, et al.
xcot: Cross-lingual instruction tuning for cross-lingual chain-of-thought reasoning.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, pp. 23550–23558, 2025.
- Cheng et al. (2025)
Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, et al.
Revisiting reinforcement learning for llm reasoning from a cross-domain perspective.
*arXiv preprint arXiv:2506.14965*, 2025.
- (8)
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma.
Sft memorizes, rl generalizes: A comparative study of foundation model post-training.
In *Forty-second International Conference on Machine Learning*.
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.
Training verifiers to solve math word problems.
*arXiv preprint arXiv:2110.14168*, 2021.
- Ge et al. (2023)
Yubin Ge, Devamanyu Hazarika, Yang Liu, and Mahdi Namazifar.
Supervised fine-tuning of large language models on human demonstrations through the lens of memorization.
In *NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following*, 2023.
- Gehring et al. (2025)
Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Taco Cohen, and Gabriel Synnaeve.
Rlef: Grounding code llms in execution feedback with reinforcement learning.
In *Forty-second International Conference on Machine Learning*, 2025.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Hao et al. (2023)
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu.
Reasoning with language model is planning with world model.
*arXiv preprint arXiv:2305.14992*, 2023.
- Howard & Ruder (2018)
Jeremy Howard and Sebastian Ruder.
Universal language model fine-tuning for text classification.
In *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 328–339, 2018.
- Hu et al. (2025)
Peng Hu, Sizhe Liu, Changjiang Gao, Xin Huang, Xue Han, Junlan Feng, Chao Deng, and Shujian Huang.
Large language models are cross-lingual knowledge-free reasoners.
In *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pp. 1525–1542, 2025.
- Hua et al. (2024)
Tianze Hua, Tian Yun, and Ellie Pavlick.
mothello: When do cross-lingual representation alignment and cross-lingual transfer emerge in multilingual models?
*arXiv preprint arXiv:2404.12444*, 2024.
- Huan et al. (2025)
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Poovendran, Graham Neubig, and Xiang Yue.
Does math reasoning improve general llm capabilities? understanding transferability of llm reasoning.
*arXiv preprint arXiv:2507.00432*, 2025.
- Huang et al. (2023)
Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei.
Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting.
In *Findings of the Association for Computational Linguistics: EMNLP 2023*, pp. 12365–12394, 2023.
- Huang et al. (2024)
Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu.
O1 replication journey–part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson?
*arXiv preprint arXiv:2411.16489*, 2024.
- Jaech et al. (2024)
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
Openai o1 system card.
*arXiv preprint arXiv:2412.16720*, 2024.
- Kim et al. (2023)
Seungone Kim, Se Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo.
The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 12685–12708, 2023.
- Korkmaz (2024)
Ezgi Korkmaz.
A survey analyzing generalization in deep reinforcement learning.
*arXiv preprint arXiv:2401.02349*, 2024.
- Lai et al. (2025)
Song Lai, Haohan Zhao, Rong Feng, Changyi Ma, Wenzhuo Liu, Hongbo Zhao, Xi Lin, Dong Yi, Min Xie, Qingfu Zhang, et al.
Reinforcement fine-tuning naturally mitigates forgetting in continual post-training.
*arXiv preprint arXiv:2507.05386*, 2025.
- Luo et al. (2025)
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang.
An empirical study of catastrophic forgetting in large language models during continual fine-tuning.
IEEE, 2025.
- Merchant et al. (2020)
Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney.
What happens to bert embeddings during fine-tuning?
*arXiv preprint arXiv:2004.14448*, 2020.
- Morishita et al. (2024)
Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa.
Enhancing reasoning capabilities of llms via principled synthetic logic corpus.
*Advances in Neural Information Processing Systems*, 37:73572–73604, 2024.
- Ouyang et al. (2022)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Training language models to follow instructions with human feedback.
*Advances in neural information processing systems*, 35:27730–27744, 2022.
- Payoungkhamdee et al. (2025)
Patomporn Payoungkhamdee, Pume Tuchinda, Jinheon Baek, Samuel Cahyawijaya, Can Udomcharoenchaikit, Potsawee Manakul, Peerat Limkonchotiwat, Ekapol Chuangsuwanich, and Sarana Nutanong.
Towards better understanding of program-of-thought reasoning in cross-lingual and multilingual environments.
*arXiv preprint arXiv:2502.17956*, 2025.
- Pham et al. (2024)
Trinh Pham, Khoi Le, and Luu Anh Tuan.
Unibridge: A unified approach to cross-lingual transfer learning for low-resource languages.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 3168–3184, 2024.
- Qi et al. (2025)
Jirui Qi, Shan Chen, Zidi Xiong, Raquel Fernández, Danielle S Bitterman, and Arianna Bisazza.
When models reason in your language: Controlling thinking trace language comes at the cost of accuracy.
*arXiv preprint arXiv:2505.22888*, 2025.
- Qin et al. (2023)
Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che.
Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 2695–2709, 2023.
- Qiu et al. (2024)
Xiaoyu Qiu, Yuechen Wang, Jiaxin Shi, Wengang Zhou, and Houqiang Li.
Cross-lingual transfer for natural language inference via multilingual prompt translator.
In *2024 IEEE International Conference on Multimedia and Expo (ICME)*, pp. 1–6. IEEE, 2024.
- Rytting & Wingate (2021)
Christopher Michael Rytting and David Wingate.
Leveraging the inductive bias of large language models for abstract textual reasoning.
In *Proceedings of the 35th International Conference on Neural Information Processing Systems*, pp. 17111–17122, 2021.
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*arXiv preprint arXiv:1707.06347*, 2017.
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Sharma et al. (2025)
Nikhil Sharma, Kenton Murray, and Ziang Xiao.
Faux polyglot: A study on information disparity in multilingual large language models.
In *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pp. 8090–8107, 2025.
- She et al. (2024)
Shuaijie She, Wei Zou, Shujian Huang, Wenhao Zhu, Xiang Liu, Xiang Geng, and Jiajun Chen.
Mapo: Advancing multilingual reasoning through multilingual-alignment-as-preference optimization.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 10015–10027, 2024.
- Sheng et al. (2024)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
Hybridflow: A flexible and efficient rlhf framework.
*arXiv preprint arXiv: 2409.19256*, 2024.
- (39)
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al.
Language models are multilingual chain-of-thought reasoners.
In *The Eleventh International Conference on Learning Representations*.
- Singh et al. (2024)
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, et al.
Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation.
*arXiv preprint arXiv:2412.03304*, 2024.
- Son et al. (2025)
Guijin Son, Jiwoo Hong, Hyunwoo Ko, and James Thorne.
Linguistic generalizability of test-time scaling in mathematical reasoning.
*arXiv preprint arXiv:2502.17407*, 2025.
- Tang et al. (2024)
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei.
Mathscale: Scaling instruction tuning for mathematical reasoning.
In *International Conference on Machine Learning*, pp. 47885–47900. PMLR, 2024.
- Toshniwal et al. (2025)
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman.
Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Touvron et al. (2023)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
*arXiv preprint arXiv:2307.09288*, 2023.
- Wang et al. (2024a)
Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, Aiti Aw, and Nancy Chen.
Seaeval for multilingual foundation models: From cross-lingual alignment to cultural reasoning.
In *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pp. 370–390, 2024a.
- Wang et al. (2024b)
Shuhe Wang, Shengyu Zhang, Jie Zhang, Runyi Hu, Xiaoya Li, Tianwei Zhang, Jiwei Li, Fei Wu, Guoyin Wang, and Eduard Hovy.
Reinforcement learning enhanced llms: A survey.
*arXiv preprint arXiv:2412.10400*, 2024b.
- Wen et al. (2025)
Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al.
Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms.
*arXiv preprint arXiv:2506.14245*, 2025.
- Xie et al. (2025)
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo.
Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning.
*arXiv preprint arXiv:2502.14768*, 2025.
- Xu et al. (2024)
Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, and Li Yuan.
Llava-cot: Let vision language models reason step-by-step.
*arXiv preprint arXiv:2411.10440*, 2024.
- Xuan et al. (2025)
Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, et al.
Mmlu-prox: A multilingual benchmark for advanced large language model evaluation.
*arXiv preprint arXiv:2503.10497*, 2025.
- Yan et al. (2025)
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang.
Learning to reason under off-policy guidance.
*arXiv preprint arXiv:2504.14945*, 2025.
- Yang et al. (2024)
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al.
Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement.
*arXiv preprint arXiv:2409.12122*, 2024.
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Ye et al. (2025)
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu.
Limo: Less is more for reasoning.
*arXiv preprint arXiv:2502.03387*, 2025.
- Yoon et al. (2024)
Dongkeun Yoon, Joel Jang, Sungdong Kim, Seungone Kim, Sheikh Shafayat, and Minjoon Seo.
Langbridge: Multilingual reasoning without multilingual supervision.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 7502–7522, 2024.
- Yu et al. (2023)
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.
Metamath: Bootstrap your own mathematical questions for large language models.
*arXiv preprint arXiv:2309.12284*, 2023.
- Yuan et al. (2025)
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Boji Shan, Zeyuan Liu, Jia Deng, Huimin Chen, Ruobing Xie, et al.
Advancing llm reasoning generalists with preference trees.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Yue et al. (2023)
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.
Mammoth: Building math generalist models through hybrid instruction tuning.
*arXiv preprint arXiv:2309.05653*, 2023.
- Yue et al. (2025)
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang.
Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?
*arXiv preprint arXiv:2504.13837*, 2025.
- Zhang et al. (2025)
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao.
R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization.
*arXiv preprint arXiv:2503.12937*, 2025.
- Zhang et al. (2024)
Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin.
Chain of preference optimization: Improving chain-of-thought reasoning in llms.
*Advances in Neural Information Processing Systems*, 37:333–356, 2024.
- Zheng et al. (2024)
Yaowei Zheng, Richong Zhang, Junhao Zhang, YeYanhan YeYanhan, and Zheyan Luo.
Llamafactory: Unified efficient fine-tuning of 100+ language models.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)*, pp. 400–410, 2024.

## Appendix A Appendix

### A.1 Use of LLM

During the preparation of this work, large language models (e.g., ChatGPT) were used for English writing refinement and minor assistance in code debugging. All ideas, experiments, and analyses are solely by the authors.

### A.2 Experimental Settings

#### A.2.1 Prompts for generating translated GSM8K training datasets

Here is the prompts for generating translated GSM8K training datasets.

#### A.2.2 Prompts for generating translated LUFFY training datasets

Here is the prompts for generating translated LUFFY training datasets.

”

### A.3 Results details

Table: Table 7: Performance of base model, SFT, and RL tuning models on MGSM. Base denotes the orginal Qwen2.5-3B-Base model. SFT (zh) and RL (zh) mean we tune the base model in Chinese data through SFT and RL, respectively. We report the accuracy score on 10 linguistic settings. $\Delta$ (RL-SFT) represents the performance difference between RL and the corresponding SFT score. Each score represents the average accuracy over six measurements. Avg represents the average of the scores of 10 language settings and Gen represents the generalization score.

Table: Table 8: Performance of base model, SFT, and RL tuning models on MAIME2024. Base denotes the orginal Qwen2.5-3B-Base model. SFT (zh) and RL (zh) mean we tune the base model in Chinese data through SFT and RL, respectively. We report the Pass@16 score on 10 linguistic settings. $\Delta$ (RL-SFT) represents the performance difference between RL and the corresponding SFT score.

Figure: (a) A comparison of performance on MMath500.
Refer to caption: x5.png

Table: Table 9: Performance of models on MMath500. “RL (zh)” denotes the model trained with Reinforcement Learning on Chinese data. “+ Prompt” indicates the addition of language control prompts during both the training and the inference. “+ Consistency Reward” further incorporates a language consistency reward into the training objective. We report the accuracy score on 6 linguistic settings. We test 6 times and report the average accuracy scores and pass@k scores.

To complement Figure [4](https://arxiv.org/html/2509.23657v1#S4.F4), Table [10](https://arxiv.org/html/2509.23657v1#A1.T10) reports the quantitative measurements of representational movement under different RL configurations. Specifically, “Model Center Distance” denotes the distance between each model’s representation center and the base model center, while “Model Shift Distance” denotes the distance between the model’s shift center and the zero point. These measurements provide quantitative evidence supporting the representational patterns illustrated in the figure.

**Table 10: Numerical results corresponding to Figure [4](https://arxiv.org/html/2509.23657v1#S4.F4), reporting the model center distance and shift distance under different RL configurations.**
| Config | Model Center Distance | Model Shift Distance |
| --- | --- | --- |
| RL-En | 2.255 | 41.332 |
| RL-Zh | 1.815 | 41.294 |
| RL-De | 1.753 | 41.241 |
| RL-De+Prompt | 1.891 | 41.286 |
| RL-De+Prompt+Reward | 1.908 | 41.652 |

### A.4 Case Study

Table: Table 11: Case study on MMath500. We select the German question to show the case.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: See weaknesses above
soundness: 3
strengths: 1. This paper addresses an important question of how RL training affects cross-lingual generalization, and the finding that RL shows better generalization is very insightful for practitioners.

2. The paper is written very clearly, and it was easy to follow the content while reading.

3. The findings in Section 4.2., in that online optimization is also beneficial for cross-lingual generalization (in addition to "reasoning", which the community has most focused on) highlights the benefit of how the learning algorithms could affect generalization in a broader scope. The analysis shown in Figure 4 can also be adopted in future work.
summary: This paper investigates if RL could be beneficial for cross-lingual generalization compare to that of SFT. The authors train Qwen2.5-3B-Base on translated GSM8K and LUFFY datasets (into Chinese and German) and ablate training on SFT and RL. The models are evaluated across ten languages on 4 benchmarks (MGSM, MMath500, MMLU-ProX-Lite, and MGPQA-Diamond), where RL consistently outperforms SFT across different settings.
weaknesses: 1. Across Table 1,2,3,4, the performance of base model, SFT'ed model, and RL'ed model are shown. In practice, a lot of people also first apply SFT (referred to as cold-start) and then apply RL on top of that. Would the performance of such baseline outperform the only RL'ed variant or not? I think such cold-start is adopted a lot in practice, so there should be a baseline added.

2. Taking the French scores on MGSM (Table 1) for example, when training with SFT, the Chinese dataset yields higher performance than German dataset (56.1 vs 52.8). However, with RL, the German dataset yields higher performance than Chinese dataset (80.0 vs 76.1). What could be the reason why the transferability differs across training algorithms and how does that relate to the similarity between the language used for training and the target language that is evaluated? Although it might be difficult to answer this question holistically, I think since the motivation of this work is on cross-lingual generalization w.r.t different algorithms, there should be a minimal explanation related to this.

3. How would the performance differ when applying SFT or RL with English, Chinese, and German datasets combined? I think this is the typical set-up done for improving multilingual capabilities of LMs, hence it should be added to the results section. Does the performance drop compared to that of training on a single target language?

4. How is the language consistency reward defined in Equation 1, and how is it implmeneted/measured? I think there should be more explanation of this in the text body.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Prompt/consistency vs. accuracy inversion (Fig. 2 & Table 6): could you run a crossed intervention where the prompt explicitly requests mixed/other language while the reward penalizes similarity to the input language (an inconsistency reward)—i.e., the opposite of your current setting? This would help stress-test whether encouraged inconsistency further improves RL, or whether the benefit is emergent rather than enforced. 
2. Why is German so strong for RL? Please discuss whether advantages come from (a) dataset bias/translation quality in LUFFY/GSM8K translations, (b) pretraining distribution (German coverage/quality), or (c) intrinsic linguistic properties (morphology/compounding influencing tokenization and reasoning decomposability). Any measurements—e.g., token length, subword overlap with English/Chinese, or per-language perplexity/coverage—would help separate these hypotheses.
soundness: 4
strengths: 1. Intuitive central idea with clear, repeated gains over SFT across languages/datasets. 
2. Actionable takeaway for the community: train RL on non-English (esp. German/Chinese) to improve cross-lingual generalization. 
3. Solid experimental practice: MGSM/MMath500/MAIME2024 (6×/6×/16×); ~10 languages;
4. Insightful analysis:
- (a) “Trained with German, might not reason strictly in German”—documented via Table 5 examples and Table 6 consistency stats. 
- (b) Prompt/consistency reward decreases accuracy (inverse correlation) → linguistic flexibility appears beneficial. Figure 2 & Table 6. 
- (c) Sampling/online RL > SFT/RFT for exploration and alignment with the model’s distribution. 
- (d) Semantic-shift analysis: smaller representational drift (PCA diffs) aligns with stronger cross-lingual generalization.
summary: The paper investigates how reinforcement learning (RL) vs. supervised fine-tuning (SFT) affects cross-lingual reasoning when training small LLMs (Qwen2.5-3B-Base, plus a check on SmolLM3-3B-Base). The key findings are: (1) RL substantially outperforms SFT on multilingual benchmarks and transfers better across languages; (2) non-English RL training (notably German, also Chinese/Japanese/French) produces higher average accuracy and generalization than English RL, a pattern not observed for SFT. Mechanistic analyses point to (i) language inconsistency during reasoning (models trained with German instructions frequently reason in mixed/non-German), (ii) benefits of on-policy sampling/exploration beyond SFT/RFT, and (iii) smaller semantic drift from pretraining features correlating with better cross-lingual generalization.
weaknesses: 1. Scale limitation: results are shown on two 3B models; it’s unclear if the “non-English RL > English RL” effect holds at a larger model and dataset scale.
2. Domain breadth: math and knowledge-oriented reasoning are well covered; fewer open-ended or instruction-following multilingual tasks.
3. Mechanistic claims are suggestive (correlational): e.g., language inconsistency ↔ performance—great ablations, but causal proof remains preliminary.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1.The main conclusion is drawn only from experiments on small models (3B). How do you ensure that this phenomenon does not simply arise from limited model capacity and implicit English-centric priors?

2.How do you ensure data translation quality, particularly for low-resource languages?

3.What is the complete implementation setup of RFT, including sampling configuration, filtering criteria?
soundness: 3
strengths: 1. The paper is well-written and logically structured, making the content easy to follow.
2. Extensive experiments across multiple multilingual reasoning benchmarks provide solid and convincing results.
3. The analysis of model semantic feature shift offers interpretability and valuable insights into the underlying mechanisms.
summary: This paper systematically compared Reinforcement learning (RL) with supervised fine-tuning (SFT) for cross-lingual complex reasoning. Experiments on multilingual mathematical, commonsense, and scientific reasoning benchmarks show that RL yields higher accuracy and stronger cross-lingual generalization than SFT. RL trained on non-English data even generalizes better than RL trained on English data, a pattern not observed in SFT. Mechanistic analyses suggest that RL fosters more robust reasoning strategies.
weaknesses: 1. It would be valuable to investigate whether the observed phenomena also hold for larger models, such as Qwen2.5-7B. Verifying the trends at scale would make the findings more robust and convincing.

2. Prior studies have shown that RL enhances robust reasoning strategies across domains, but this work focuses mainly on multilingual math reasoning, which limits the generality of the conclusions.

3. The training data are obtained through machine translation from English using Qwen3-30B. Given that Qwen3-30B’s multilingual capabilities, this may significantly affect translation quality, especially for low-resource languages. In fact, MathOctopus [1] has already provided a validated multilingual GSM8K dataset(MGSM8K-Instruct).

4. The paper also introduces RFT as a key baseline to analyze the role of sampling, yet crucial experimental details are missing, including sampling size, selection strategy, and decoding temperature.

[1] Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses.
soundness: 2
strengths: - This paper presents a systematic investigation of the difference between RL and SFT in cross-lingual reasoning generalisation
- This paper shows that using non-English data for RL can more effectively enhance performance and cross-lingual generalisation, which might provide important insights and practice into future multilingual reasoning works
- The authors perform extensive experiments and analysis on several reasoning tasks
summary: This paper performs a systematic investigation into cross-lingual reasoning generalisation of reinforcement learning (RL) and supervised fine-tuning (SFT). Experimental results show that (1) models trained with RL outperform better than SFT and (2) using non-English data achieves better overall performance than using English data.
weaknesses: - The training dataset size (GSM8K with around 8k samples) is pretty small and simple, which might influence the findings and conlcusions
- Some details of the experimental setup are missing.   
(1)  Inference setting of the base model. It is unclear if the authors use zero-shot or few-shot prompting. Since this is a base model without instruction-tuning, it would be more fair to include few-shot prompting as a baseline;   
(2) Training setting of SFT and RL. The authors do not report the training epoch and stopping setting. Compared to the base model, the improvement of SFT is very limited in some languages, such as English, which is unexpected?
- As the results indicate that RL-tuned models exhibit a language-mixing pattern in their outputs, it would be interesting to include statistics on language usage in addition to the language (in)consistency analysis already presented in the paper. In other words, this can show which language the model prefers to use to think and reason, which could be used to analyse cross-lingual transfer and show why RL training on non-English data yields better performance


Others:
- Lines 231-239: The results discussed here are not clearly linked to any table or figure. Is it Table 2?


Decision: Accept (Poster)