Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoning
Authors: Guijin Son, Jiwoo Hong, Hyunwoo Ko, James Thorne
Accepted at: acl25

Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoning

Guijin Son1,2,4    Jiwoo Hong3    Hyunwoo Ko2,4    James Thorne3
Yonsei University1    OneLineAI2    KAIST AI3    MODULABS4
spthsrbwls123@yonsei.ac.kr

Abstract

Scaling pre-training compute has proven effective for achieving multilinguality, but does the same hold for test-time scaling? In this work, we introduce MCLM, a multilingual math benchmark featuring competition-level problems in 55 languages. We test three test-time scaling methods—Outcome Reward Modeling (ORM), Process Reward Modeling (PRM), and Budget Forcing (BF)—on both Qwen2.5-1.5B Math and MR1-1.5B, a multilingual LLM we trained for extended reasoning. Our experiments show that using Qwen2.5-1.5B Math with ORM achieves a score of 35.8 on MCLM, while BF on MR1-1.5B attains 35.2. Although “thinking LLMs” have recently garnered significant attention, we find that their performance is comparable to traditional scaling methods like best-of-N once constrained to similar levels of inference FLOPs. Moreover, while BF yields a 20-point improvement on English AIME, it provides only a 1.94-point average gain across other languages—a pattern consistent across the other test-time scaling methods we studied—highlighting that test-time scaling may not generalize as effectively to multilingual tasks. To foster further research, we release MCLM, MR1-1.5B, and evaluation results.1

1 Introduction

Large Language Models (LLMs) have achieved impressive gains across a wide range of tasks by scaling compute during pre-training (Thoppilan et al., 2022; Smith et al., 2022). Contrary to early concerns about a so-called “curse of multilinguality,” (Conneau et al., 2020; Pfeiffer et al., 2022) which suggested that training in diverse languages would degrade overall performance, sufficiently large decoder-only architectures have demonstrated strong multilingual capabilities (Dubey et al., 2024; Aryabumi et al., 2024b).

![Bar chart showing performance of Qwen2.5-1.5B-Math with different test-time scaling strategies](page_1012_370_367_246.png)

Figure 1: Performance of Qwen2.5-1.5B-Math with different test-time scaling strategies.—Once configured to use comparable inference FLOPs, all three methods (Outcome Reward Modeling, Process Reward Modeling, and Budget Forcing) achieve similar performance.

Yet as further scaling becomes increasingly difficult—due to data scarcity (Longpre et al., 2024), diminishing returns, or prohibitive costs (Achiam et al., 2023)—researchers have begun exploring test-time scaling methods that expand a model’s reasoning or generation capacity at test time. An intriguing question arises: Does test-time scaling confer the same cross-lingual benefits we see at train-time scaling during pre-training?

Early studies demonstrated that chain-of-thought prompting (Wei et al., 2022) and scratchpads (Nye et al., 2021) can significantly boost model performance—particularly in mathematics (Lewkowycz et al., 2022; Azerbayev et al., 2024) and code (Le et al., 2022; Chae et al., 2024). Building on this, recent work proposes “test-time scaling,” which further lengthens the chain-of-thought (Snell et al., 2025; Muennighoff et al., 2025). While such methods have proven effective for puzzles like Sudoku (Jellyfish042, 2024) and Hex (Jones, 2021), where action spaces are limited, mathematical reasoning remains relatively unexplored, largely due to its exponentially larger search space. To address this challenge, researchers have investigated external verifiers—such as best-of-N selection (Wang et al., 2022), Monte Carlo Tree Search (Guan et al., 2025; Tian et al., 2024; Wan et al., 2024), and
process/outcome reward modeling (Zhang et al., 2025; Liu et al., 2025). Meanwhile, state-of-the-art LLMs (OpenAI, 2024, 2025) are capable of self-correction—often referred to as “system 2” reasoning (Xiang et al., 2025)—without explicit external verification. While longer chains of reasoning provide more room for in-depth thinking, they may also amplify the risk of error propagation (Bengio et al., 2015; Arora et al., 2022; Holtzman et al., 2020), making them more susceptible to out-of-domain disturbances such as language variation (Zhao et al., 2023; Chen et al., 2024b). From this vein, it remains unclear whether these strategies robustly generalize to new questions (SRI_Lab, 2025), languages, or domains.

In this work, we investigate the linguistic generalizability of test-time scaling methods by proposing a fine-grained multilingual complex reasoning benchmark, showing that test-time scaling alone does not yield robust multilingual performance. We build MCLM (Multilingual Competition Level Math), a math reasoning dataset composed of four subsets varying source covering 55 languages.

We analyze three test-time scaling methods, outcome reward modeling (Wang et al., 2022, ORM), process reward modeling (Zhang et al., 2025, PRM), and budget forcing (Muenninghoff et al., 2025, BF). We examine (1) accuracy to determine whether models retain overall performance across languages and (2) consistency to observe whether models can solve the same questions in different languages. While ORM and PRM provide clear gains on relatively easy datasets, the improvements are marginal for challenging tasks and inconsistent across the languages. Meantime, BF delivers noticeable gains only in English for tougher questions, with minimal impact on other languages. These findings underscore that while test-time scaling can enhance accuracy under certain conditions, it does not guarantee robust or consistent performance across multiple languages.

Finally, we introduce MRI-1.5B, an open multilingual thinking LLM trained on Deepseek-R1-1.5B using 100k R1-distilled instances translated by GPT-4o. Despite having only 1.5B parameters, MRI achieves performance on par with GPT-4o-Mini in multilingual mathematical reasoning.

2 Multilingual Competition Level Math

In this section, we introduce Multilingual Competition Level Math (MCLM), a multilingual math reasoning benchmark with challenging competition-level questions in 55 languages.

<table>
  <tr>
    <th>Models</th>
    <th>MGSM</th>
  </tr>
  <tr>
    <td>Gemma2-9B</td>
    <td>78.37</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Instruct</td>
    <td>82.27</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>88.16</td>
  </tr>
  <tr>
    <td>Mistral-Large</td>
    <td>89.01</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>87.36</td>
  </tr>
  <tr>
    <td>o3-mini</td>
    <td><b>89.30</b></td>
  </tr>
</table>

Table 1: MGSM performance of different models. The 2025-01-31 version is used for o3-mini, remaining scores were sourced from the Yang et al. (2024b).

Going beyond math word problems  A translated version of GSM8K (Cobbe et al., 2021), MGSM (Shi et al., 2023), has been widely used to assess the mathematical reasoning skills of multilingual LLMs (Anil et al., 2023; Shao et al., 2024; Aryabumi et al., 2024a). However, in Table 1, we observe that recent LLMs saturate MGSM. This implies the limitations of simple math word problems in accurately assessing the math reasoning capabilities of LLMs and necessitates a higher degree of complexity in reasoning benchmarks.

Assessing complex reasoning capabilities  In this vein, recent studies have evaluated the reasoning capabilities of LLMs using competition-level math questions (MAA, 2024; Gao et al., 2025). While these benchmarks address limitations in simple math word problems, they are largely restricted to English and Chinese, limiting their ability to study multilingualism at scale.

2.1 Curating the MCLM benchmark

Machine-translated reasoning  We select AIME and MATH-500 (Lightman et al., 2024), two widely used mathematical benchmarks, as the main source of complex math questions. For 100 questions randomly sampled from MATH-500 and full AIME datasets, we translate both benchmarks with GPT-4o (OpenAI et al., 2024), as shown to be proficient in translating mathematical contexts (Chen et al., 2024a; Lai et al., 2023). We then verified that the answers and equations remained unchanged after translation, removing one sample from MATH500 due to translation inconsistencies. Both subsets consist of questions with numerical answers only. For further details on the machine translation and sampling process, see Appendix A.2.
<table>
  <tr>
    <th>Subset</th>
    <th>Source Benchmark</th>
    <th>Languages</th>
    <th>Sample Size per Language</th>
    <th>Evaluation Method</th>
  </tr>
  <tr>
    <td>MT-MATH100</td>
    <td>Math-500</td>
    <td>55</td>
    <td>100</td>
    <td>Rule-based verifier</td>
  </tr>
  <tr>
    <td>MT-AIME2024</td>
    <td>AIME 2024</td>
    <td>55</td>
    <td>30</td>
    <td>Rule-based verifier</td>
  </tr>
  <tr>
    <td>M-IMO</td>
    <td>IMO (2006, 2024)</td>
    <td>38</td>
    <td>22–27</td>
    <td>LLM-as-a-Judge</td>
  </tr>
  <tr>
    <td>M-MO</td>
    <td>Domestic/Regional Olympiads</td>
    <td>11</td>
    <td>28–31</td>
    <td>LLM-as-a-Judge</td>
  </tr>
</table>

Table 2: Overview of benchmark subsets: source benchmarks, language coverage (full lists in the appendix), sample sizes, and evaluation methods. Please see Appendix A.1 for the full list of languages.

Human-annotated reasoning To mitigate the potential biases in machine-translated data from translation artifacts (Plaza et al., 2024; Son et al., 2025), we also include human-translated or originally written questions.

First, we manually review 114 International Mathematical Olympiad (IMO, 2024, IMO) questions from 2006 to 2024 in English, excluding proof-based and image-heavy problems, resulting in a final set of 27. We then collect their official translations in 38 languages. Where official translations are unavailable, we do not substitute machine-generated versions, leaving those entries missing.

Second, we gather problems from various domestic and regional mathematical Olympiads worldwide. These contests originate in multiple languages, providing valuable data for multilingual mathematical reasoning. For English, Chinese, and Korean—where competition-level benchmarks already exist (He et al., 2024; Ko et al., 2025)—we incorporate existing datasets rather than recollecting data. While we exclude proof-based questions for simplicity, the final dataset still features a diverse range of answer formats (e.g., numerical, Boolean, descriptive) and spans 11 languages. We use GPT-4o-mini1 for evaluation. An overview is provided in Table 2, additional details are in Appendix A.3.

3 Experimental Settings

In this section, we provide an overview of the test-time scaling methods evaluated (Section 3.1), compare the inference budgets in terms of FLOPs across different scaling techniques (Section 3.2), and describe the evaluation metrics used to assess their performance (Section 3.3).

3.1 Baselines: Test-Time Scaling Strategies

In this work, we evaluate three test-time scaling strategies using Qwen2.5-1.5B and 7B instruct models (Yang et al., 2024a) as baselines (Figure 2). We selected these model sizes because they offer a balanced trade-off between reasoning capacity and computational efficiency. Models smaller than 1.5B lack the capacity to solve complex problems, while larger models can be prohibitively expensive to scale (Biderman et al., 2023).

Outcome Reward Modeling We generate \( N \) responses per instance and use Qwen2.5-Math-72B-RM (Yang et al., 2024b) to evaluate them, selecting the highest-scoring answer as the final output.

Process Reward Modeling In contrast to outcome reward modeling, this strategy integrates the reward model during inference to guide the generation process. We employ Qwen2.5-Math-72B-PRM (Zhang et al., 2025); the model generates \( c \) candidate continuations at each step and selects the best one. For both ORM and PRM, the generator and reward model are served on separate servers, thereby avoiding the overhead of repeatedly on- and off-loading model weights.

Budget Forcing Recent LLMs, such as R1 (Guo et al., 2025) and O1 (OpenAI, 2024), are designed to generate longer chain-of-thoughts with in-context exploration and correction, allowing them to naturally scale during inference. However, this approach lacks controllability. To mitigate this, we adopt the budget-forcing method proposed by Muenninghoff et al. (2025). In budget forcing, these thinking models are truncated and required to output an answer if they exceed a predefined budget. Conversely, if they fall short of the budget, they are prompted to generate additional reasoning steps,

![Comparison of different inference-time scaling strategies. Blue boxes represent selected outputs, while red boxes indicate rejected ones.](page_1012_393_388_181.png)

Figure 2: Comparison of different inference-time scaling strategies. Blue boxes represent selected outputs, while red boxes indicate rejected ones.
encouraging further exploration and correction.

3.2 Calculating Inference FLOPs
For our experiments, we first establish a unified inference budget based on two key estimates: the generator’s cost is approximated as \(2N_G D\) (Kaplan et al., 2020), where \(N_G\) is the number of parameters in the generation model and \(D\) is the total number of tokens generated per instance. The verifier’s (or reward model’s) cost is estimated as \(4N_V\), where \(N_V\) is the number of parameters in the verifier. Here, the multiplier of 4 for the reward model’s cost reflects a base cost of \(2N_V\) that is doubled to account for the additional overhead incurred when invoking the reward model during inference (Snell et al., 2025). In the ORM, we generate \(k\) responses per instance, leading to a total inference cost of

\[
k \times \left(2N_G D + 4N_V\right).
\]

In Figure 3, under greedy generation, we observe that models tend to generate longer responses once they produce an error, particularly on harder benchmarks. Additionally, the 7B model generally produces longer outputs than the 1.5B model. However, no systematic trends emerge across the 55 languages—there is no clear pattern, such as longer outputs for low-resource languages. Although token counts vary with configuration, these differences are negligible compared to the dominant effect of model size on inference FLOPs. Consequently, we use an average of 921 tokens per question to estimate cost.

Outcome Reward Modeling In ORM with \(k = 2\) responses, the inference cost is approximated as

\[
\text{ORM FLOPs} \approx 2\left(2N_G \times 921 + 4N_V\right).
\]

Assuming \(N_G = 1.5 \times 10^9\) and \(N_V = 72 \times 10^9\), this configuration results in an estimated cost of approximately \(6.10 \times 10^{12}\) FLOPs per instance.

Process Reward Modeling In PRM, at each generation step, the model produces \(c\) candidates, with each candidate generating \(x\) tokens (we fix \(x = 128\) in our experiments). The total inference cost over \(S\) steps is given by

\[
\text{PRM FLOPs} = S\ c\left(x \cdot 2N_G + 4N_V\right).
\]

For PRM configurations, \(S\) and \(c\), our preliminary experiments indicated that scaling one parameter in isolation produced suboptimal performance: a high \(S\) with a low \(c\) failed to explore sufficient alternatives, while an excessively low \(S\) prevented the generation process from completing. Therefore, we opted to proportionally scale both \(S\) and \(c\) to achieve a balanced search during generation.

Budget Forcing In contrast, BF relies solely on the generator, so its inference cost is given by

\[
\text{SC FLOPs} = 2N_G \cdot BF,
\]

where \(BF\) denotes the effective number of tokens that may be generated during the inference with budget forcing. In Table 3, we select the PRM parameters \(S\) and \(c\) and adjust the BF token budget \(BF\) so that the overall inference cost of each method matches that of ORM for \(k = 2, 4,\) and 8.

<table>
  <tr>
    <th>k</th>
    <th>\((S, c)\)</th>
    <th>BF</th>
  </tr>
  <tr>
    <td>2</td>
    <td>(3, 3)</td>
    <td>\(\approx 2048\) tokens</td>
  </tr>
  <tr>
    <td>4</td>
    <td>(4, 5)</td>
    <td>\(\approx 4096\) tokens</td>
  </tr>
  <tr>
    <td>8</td>
    <td>(5, 8)</td>
    <td>\(\approx 8192\) tokens</td>
  </tr>
</table>

Table 3: Selected configurations for PRM and BF. Each \(S\), \(c\), and \(BF\) is set so that the inference FLOPs match ORM.

3.3 Evaluation Metrics
We evaluate our models using two primary metrics that capture performance at multiple levels: (1) accuracy and (2) cross-lingual consistency.

Accuracy We measure accuracy at a surface level to determine whether a single model achieves comparable performance across different languages.

Cross-Lingual Consistency To examine whether the model tends to solve (or fail) the same questions across languages, we compute Fleiss’ kappa (Fleiss, 1971), which is originally designed to measure agreement among multiple annotators. In our
Figure 4: Gains of ORM compared to a greedy-decoding baseline. The semi-transparent “cloud” indicates the 2D data distribution via a KDE density plot, and the overlaid lines are third-order polynomial regressions modeling how each ORM setting scales with the baseline score.

setup, however, we treat each language as an “annotator”: for each problem, each “annotator” (i.e., each language version of the model) provides either a correct or incorrect label. We then define consistency through Fleiss’ kappa as:

\[
\kappa = \frac{\bar{P} - \bar{P}_e}{1 - \bar{P}_e}
\] (5)

\[
\bar{P} = \frac{1}{N} \sum_{i=1}^N \frac{1}{n(n-1)} \sum_{j=1}^k n_{ij}(n_{ij} - 1)
\] (6)

\[
\bar{P}_e = \sum_{j=1}^k p_j^2, \quad p_j = \frac{1}{Nn} \sum_{i=1}^N n_{ij},
\] (7)

where \( N \) is the number of problems, \( n \) is the number of languages, and \( n_{ij} \) is the count of how many times language \( j \) gives a particular label (correct or incorrect) for problem \( i \). In this formulation, \( \bar{P} \) is the observed agreement (i.e., the proportion of problems for which all languages concur on correctness or incorrectness), and \( \bar{P}_e \) is the expected agreement by chance. A high Fleiss’ kappa indicates that the model responds consistently across languages (solving the same problems), not merely achieving similar overall accuracy by chance.

4 Result 1: ORM and PRM

In this section, we assess the multilingual robustness of both ORM and PRM. We find that, while each approach can boost performance, these gains do not consistently generalize across different languages and levels of difficulty.

4.1 Outcome Reward Modeling

For ORM, we use Qwen2.5-Math-1.5B and 7B-Instruct models to generate \( K \) samples per query and then apply Qwen2.5-Math-72B to score each sample, selecting the one with the highest score.

Limited gains at scale in non-English settings
In Figure 4, we plot each model’s baseline performance (averaged across 55 languages) on the \( x \)-axis versus the relative gain of each ORM setting (with \( K \in \{2, 4, 8\} \)) on the \( y \)-axis. On the MT-MATH100 dataset, both the 1.5B and 7B models show consistent improvement as \( K \) increases. However, on the more challenging MT-AIME2024 dataset, the gains for different \( K \) values are largely indistinguishable and, in some cases, even negative. This trend is comparable to English, which shows steady improvements also on MT-AIME2024—for instance, the 1.5B model rises from 16.67 to 26.67 to 36.67 as \( K \) increases, while the 7B model goes from 20.00 to 26.67 to 36.67.

Overall, while ORM is a viable scaling strategy in English, it yields limited returns in many other languages—possibly due to the models’ difficulty in generating high-quality candidates. With few plausible options available, the reward model cannot effectively identify an improved solution.

4.2 Process Reward Modeling

Along with the configurations mentioned in Table 3, we experiment with additional setups to study how PRMs scale. Details are provided in Figure 5. In general, we evaluate two approaches: fixing \( S \) while increasing \( C \) (pink) and fixing \( C \) while scaling \( S \) (green). Additionally, we compare the efficacy of a 7B verifier against the original 72B. Due to cost constraints, these configurations are tested only on 14 languages of MT-MATH100 using Qwen2.5-Math-1.5B-Instruct.

No scalable gains for variance or consistency
Figure 6 shows that, in PRM, the average performance of Qwen2.5-Math-1.5B-Instruct increases
Figure 6: Inference FLOPs versus PRM performance and consistency. (Left) Second-degree polynomial regressions for average performance on 14 languages, comparing the 7B (blue) and 72B (green) reward models. (Right) Fleiss’ kappa (top) and standard deviation (bottom) plotted against the same FLOPs budget; the fitted curves reveal no clear monotonic trend.

steadily with the inference budget. Even though the 7B reward model provides a larger search space, the 72B reward model achieves better outcomes under comparable compute. From a hardware standpoint, it can be more effective to run fewer steps and rely on a larger verifier. However, no clear pattern emerges for Fleiss’ kappa or the standard deviation of individual scores, suggesting that adding more budget does not necessarily improve model consistency or reduce variance. In practical terms, while accuracy may scale with compute for PRM, cross-lingual consistency does not appear to follow, even for a relatively easier dataset like MT-MATH100.

4.3 ORM over PRM

As discussed earlier, both ORM and PRM exhibit unstable multilingual performance growth, with greater variance and lower Fleiss’ Kappa scores at higher inference FLOPs. However, despite this instability, as shown in Figure 7, ORM consistently outperforms PRM in average accuracy, suggesting that, in general, it may be the more reliable choice. This is especially true given that, despite being assigned the same inference FLOPs, PRM invokes the reward model more frequently, requiring iterative back-and-forth interactions between the generator and evaluator, leading to higher latency.

5 Result 2: Budget Forcing

LLMs with “system 2” reasoning (Xiang et al., 2025)—such as OpenAI’s newest O-series models (OpenAI, 2024), Google’s Gemini Thinking1, and Deepseek R1 (Guo et al., 2025)—are emerging as a test-time scaling approach. By generating and refining responses within a single inference (without external verifiers), these models seek to dramatically expand the inference budget for improved performance. In this section, we examine their effectiveness as a test-time scaling strategy. Because proprietary solutions remain largely opaque, we train our own LLMs with system 2 thinking. Here, we describe our training methods (Section 5.1), report performance on MCLM (Section 5.2) and scaling affects from budget forcing(Section 5.3).

Figure 7: Comparison of PRM vs. ORM performance on MATH (solid lines) and AIME (dashed lines). 1.5B models are shown with plus markers, 7B models with stars. Blue lines represent PRM, green lines represent ORM. White box annotations indicate the performance difference (ORM - PRM) at the highest compute setting for each line.

5.1 Inducing System 2 Thinking

A number of concurrent works propose diverse strategies for developing LLMs with long thinking. Broadly, these approaches fall into two main categories: (1) online reinforcement learning with variable outputs (Luo et al., 2025; Ye et al., 2025a), and (2) supervised fine-tuning on the "thinking trajectories" of proprietary LLMs (Muennighoff et al.,

1https://deepmind.google/technologies/gemini/flash-thinking/
<table>
  <tr>
    <th>Models</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>M-IIMO</th>
    <th>M-MO</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Qwen2.5-Math-1.5B-Instruct</td>
    <td>42.32 ± 8.61</td>
    <td>16.36 ± 6.89</td>
    <td>12.23 ± 6.02</td>
    <td>25.00 ± 19.10</td>
    <td>23.98</td>
  </tr>
  <tr>
    <td>Deepseek-R1-1.5B</td>
    <td>49.40 ± 8.84</td>
    <td>17.21 ± 6.69</td>
    <td>21.94 ± 6.75</td>
    <td>26.77 ± 19.83</td>
    <td>28.83</td>
  </tr>
  <tr>
    <td>GPT-4o-Mini</td>
    <td>70.30 ± 3.68</td>
    <td>20.18 ± 6.83</td>
    <td>13.33 ± 5.36</td>
    <td>30.81 ± 15.80</td>
    <td>33.66</td>
  </tr>
  <tr>
    <td>o3-Mini</td>
    <td><b>84.89 ± 2.80</b></td>
    <td><b>45.33 ± 5.35</b></td>
    <td><b>29.75 ± 6.86</b></td>
    <td><b>51.42 ± 16.94</b></td>
    <td><b>52.85</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-1.5B + SFT</td>
    <td>37.47 ± 7.56</td>
    <td>14.85 ± 6.69</td>
    <td>10.50 ± 5.16</td>
    <td>18.40 ± 14.92</td>
    <td>20.30</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-1.5B + MT-SFT</td>
    <td>42.02 ± 7.46</td>
    <td>16.67 ± 7.31</td>
    <td>10.52 ± 4.63</td>
    <td>19.92 ± 12.68</td>
    <td>22.28</td>
  </tr>
  <tr>
    <td>Deepseek-R1-1.5B + MT-SFT</td>
    <td><b>55.61 ± 10.93</b></td>
    <td><b>19.94 ± 8.10</b></td>
    <td><b>19.20 ± 6.24</b></td>
    <td><b>28.97 ± 16.64</b></td>
    <td><b>30.93</b></td>
  </tr>
</table>

Table 4: Model performance across MCLM. Best model highlighted in bold for each panel. For results per language see Appendix C.

2025; Ye et al., 2025b). However, Luo et al. (2025) reports requiring 3,800 A100 GPU hours to induce such behavior, making it prohibitively expensive for our setting. Instead, we opt for supervised fine-tuning using thinking trajectories distilled from R1. Below are three training configurations.

Qwen2.5-Math-1.5B + SFT Following Muenninghoff et al. (2025) and Huang et al. (2024), we randomly sample 50K thinking trajectories generated by R1 from the OpenR1-220K 1 dataset and fine-tune our model for three epochs.

Qwen2.5-Math-1.5B + SFT with Translated Data While training exclusively on English math problems has been shown to be effective and can generalize to new languages to some extent (Liu et al., 2024)—likely due to the universal nature of mathematical logic—we explore whether translating the data helps training. Following Ko et al. (2025); Zhang et al. (2023), we translate the problem and solution components into 14 languages2 using GPT-4o, while keeping the reasoning process in English to leverage the model’s strong proficiency in English-based logical reasoning. For further details on the training dataset, see Appendix B.

Deepseek-R1-1.5B + SFT with Translated Data Finally, we try initializing the translated SFT from Deepseek-R1-1.5B (hereafter MR1-1.5B). Since the model is already proficient in generating extended reasoning, we observe that longer training leads to performance degradation. To mitigate this, we terminate training at 0.5 epochs.

5.2 Performance of trained models

Translated data improves cross-lingual performance. Table 4 compares Qwen2.5-Math-1.5B and Deepseek-R1-1.5B under various fine-tuning regimes, revealing two key trends. First, incorporating translated data into Qwen2.5-Math-1.5B delivers a modest +1.98% improvement over an English-only setup, indicating that relying exclusively on English data is insufficient for robust cross-lingual performance (Liu et al., 2024). As shown in Figure 8, models trained on multilingual inputs begin with lower accuracy—likely due to increased entropy—but soon surpass their English-only counterparts and maintain lower variance across languages. Second, initiating fine-tuning from Deepseek-R1-1.5B, already adept at extended chain-of-thought reasoning, yields even greater gains on MT-AIME2024, M-IIMO, and M-MO, performing on par with GPT-4o-Mini. Notably, MR1-1.5B reaches an average score of 30.93 (+2.1% over its original baseline) with just 0.5 epochs of training, underscoring how a self-correcting model more readily benefits from multilingual data. Collectively, these results suggest that while incorporating translated data benefits a monolingual base, leveraging a model with established self-correction capabilities can amplify these gains in multilingual math reasoning.

Figure 8: Performance of Qwen2.5-Math-1.5B +SFT and + MT-SFT at each training checkpoint. Average score and error bars for each checkpoint are displayed. The shaded region is the mean ± standard deviation for MT-SFT.

5.3 Budget-Constrained Scaling

To better compare self-correction with other scaling methods (e.g., ORM and PRM), we examine
Figure 9: Performance of MRI on MT-AIME2024 at \( BF = \{2048, 4096, 8192\} \). Grey dots represent individual languages. Solid lines indicate average performance, while dashed lines highlight reference performances for selected languages.

its performance under fixed inference budgets. We apply the budget forcing approach introduced by Muenninghoff et al. (2025) to constrain the generation budget of MRI-1.5B. Following the budget settings in Table 3, during inference, if the model reaches 90% of its allocated budget, we truncate the output and append "The final answer is" to prompt a concise answer. Conversely, if the model completes generation before reaching the limit, we truncate at the last line break, append "Wait...", and prompt the model to continue generating.

R1-like LLMs offer no clear edge over ORM
Contrary to the recent surge of interest in “system 2” LLMs, that scale test-time compute by generating long reasoning traces, constraining these models to the same inference budgets reveals no clear advantage over test-time scaling methods such as ORM or PRM (see Figure 1). As shown in Figure 9, budget forcing yields nearly linear performance gains for English but provides limited benefits for most other languages. The distribution remains largely unchanged, achieving an overall average increase of only 1.9% as \( BF \) scales from 2048 to 8096. In some cases, such as Latvian and Romanian, performance even declines. Implying that there is scant evidence that the variance in performance diminishes or that coverage expands uniformly.

6 Related Works

Test-Time Scaling
As concerns grow that the benefits of scaling pre-training compute may be saturating (Longpre et al., 2024), research has shifted toward test-time scaling, which expands the notion of chain-of-thought reasoning (Wei et al., 2022).

Intuitively, the reasoning capacity of an LLM is limited by the number of tokens it can generate; hence, more challenging questions may require a longer chain of thought (Wu et al., 2025). An early example is self-consistency CoT (Wang et al., 2022), which generates multiple responses and selects the best via voting. This idea has since been developed into more cost-effective strategies for searching broader solution spaces (e.g., tree-of-thought methods (Yao et al., 2024), Monte Carlo Tree Search (Guan et al., 2025), and process supervision (Zhang et al., 2025; Luo et al., 2024)). Recently, models trained with online reinforcement learning (Shao et al., 2024) appear to exhibit an “aha moment,” (Guo et al., 2025) wherein they dynamically decide to generate longer sequences to iteratively explore, solve, and self-correct.

Mathematical Reasoning in Non-English
Early attempts at multilingual math reasoning involved supervised fine-tuning on translated datasets (Chen et al., 2024a; Lai and Nissim, 2024), but performance often deteriorated when models shifted away from their original language embeddings (Hong et al., 2025). To minimize such degradation, more recent work has increasingly relied on English as a pivot language. This approach can be implemented in various ways: either internally, by mapping multilingual inputs into an English-centric latent space (Yoon et al., 2024; Fan et al., 2025; Zhu et al., 2024; She et al., 2024), or externally, by translating non-English tasks into English and then back to the target language (Zhang et al., 2023; Ko et al., 2025). Although this strategy has reduced the performance gap between English and other languages, the stability of transfer under different training conditions remains underexplored. Moreover, many studies rely on the MGSM benchmark (Shi et al., 2023), which appears too easy for large-scale models or those enhanced by advanced reasoning techniques such as test-time scaling.

7 Conclusion
In this work, we examine the linguistic generalizability of three test-time scaling methodologies—Outcome Reward Modeling (ORM), Process Reward Modeling (PRM), and Budget Forcing (BF)—under budget-constrained settings. Using Qwen2.5-1.5B Math as a generator, ORM achieves a 35.84 score on our newly introduced multilingual math benchmark, MCLM, which spans 55
languages. With BF, MR1-1.5B—our multilingual LLM demonstrating extended reasoning—attains 35.23. Notably, once constrained to similar inference budgets, all three scaling methods exhibit comparable levels of improvement. Additionally, although these approaches appear promising in English (e.g., a 20-point improvement on AIME for the 1.5B model), we find that such gains do not consistently extend to other languages, where improvements average only 1.94 points—a pattern observed across all three methods. Moreover, increasing test-time compute often amplifies performance variance and reduces cross-linguistic consistency. To enable further study of these issues, we release both MCLM and MR1-1.5B.

Limitations

Although this work focuses solely on mathematical tasks, the lack of multilingual generalization we observe could be even more pronounced in areas requiring extensive cultural or domain-specific understanding; we leave this for future works. Additionally, due to budget constraints, this work primarily focuses on smaller-scale experiments (e.g., Qwen2.5-Math-1.5B-Instruct and occasionally Qwen2.5-Math-7B-Instruct). Although these parameter ranges are commonly used in both industry and academia, the observed lack of multilingual generalization for test-time scaling may not necessarily extend to significantly larger models (70B or more). Moreover, our test-time scaling experiments also remain on the smaller side; for instance, El-Kishky et al. (2025) scales to as many as 1162 candidates in their best-of-n setting. (It should still be noted that even experiments at this scale required over \( 2500 \text{ A100 GPU hours} \)) Given that the so-called “curse of multilinguality” (Conneau et al., 2020) naturally disappears as pre-training compute grows by several orders of magnitude (Aryabumi et al., 2024b), it is plausible that larger models may behave differently. Nevertheless, our findings at smaller scales—where test-time scaling shows no indication of fostering robust multilingualism—remain valuable, as they reveal potential boundaries for less resource-rich setups and highlight directions for future research.

Acknowledgments

This research was supported by Brian Impact Foundation, a non-profit organization dedicated to the advancement of science and technology for all, and Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2024-00398115, Technology research to ensure authenticity and consistency of results generated by AI) and (RS-2019-II190075, Artificial Intelligence Graduate School Program (KAIST)).

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Alean, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Chouette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielinski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Heyeontak Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Rainer Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Au-rko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023. Palm 2 technical report. Preprint, arXiv:2305.10403.

Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Cheung. 2022. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation. In Findings of the Association for Computational Linguistics: ACL.
2022, pages 700–710, Dublin, Ireland. Association for Computational Linguistics.

Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Aycr Locatelli, Julia Kreutzer, Nick Frosts, Aidan Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024a. Aya 23: Open weight releases to further multilingual progress. Preprint, arXiv:2405.15032.

Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, et al. 2024b. Aya 23: Open weight releases to further multilingual progress. arXiv preprint arXiv:2405.15032.

Axolotl AI. 2025. Axolotl: Scalable fine-tuning framework for llms. https://axolotl-ai-cloud.github.io/axolotl/. Github.

Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2024. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations.

Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28.

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Halahan, Mohammad Affah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR.

Hyungjoo Chae, Yeonghyeon Kim, Seungone Kim, Kai Tzu-iun Ong, Beong-woo Kwak, Moohyeon Kim, Sunghwan Kim, Taeyoon Kwon, Jiwan Chung, Youngjae Yu, and Jinyoung Yeo. 2024. Language models as compilers: Simulating pseudocode execution improves algorithmic reasoning in language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22471–22502, Miami, Florida, USA. Association for Computational Linguistics.

Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dongmei Zhang, and Jia Li. 2024a. Breaking language barriers in multilingual mathematical reasoning: Insights and observations. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7001–7016, Miami, Florida, USA. Association for Computational Linguistics.

Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. 2024b. Do not think that much for 2+ 3=? on the overthinking of ol-like llms. arXiv preprint arXiv:2412.21187.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Boris Minaev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, et al. 2025. Competitive programming with large reasoning models. arXiv preprint arXiv:2502.06807.

Yuchun Fan, Yongyu Mu, YiLin Wang, Lei Huang, Junhao Ruan, Bei Li, Tong Xiao, Shujian Huang, Xiaocheng Feng, and Jingbo Zhu. 2025. Slam: Towards efficient multilingual reasoning via selective language alignment. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9499–9515. Association for Computational Linguistics.

Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.

Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. 2025. Omni-MATH: A universal olympiad level mathematic benchmark for large language models. In The Thirteenth International Conference on Learning Representations.

Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. 2025. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
Chaoqun He, Renjie Luo, Yuzhou Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. Olympiad-Bench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3828–3850, Bangkok, Thailand. Association for Computational Linguistics.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.

Jiwoo Hong, Noah Lee, Rodrigo Martínez-Castaño, César Rodríguez, and James Thorne. 2025. Cross-lingual transfer of reward models in multilingual alignment. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 82–94, Albuquerque, New Mexico. Association for Computational Linguistics.

Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. 2024. Liger kernel: Efficient triton kernels for llm training. arXiv preprint arXiv:2410.10989.

Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. 2024. OI replication journey—part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.16489.

IMO. 2024. International Mathematical Olympiad Website. Accessed: 2024-01-31.

Jellyfish042. 2024. Sudoku-rwkv: A specialized rwkv model for solving sudoku puzzles. https://github.com/Jellyfish042/Sudoku-RWKV. Accessed: 2025-02-11.

Andy L. Jones. 2021. Scaling scaling laws with board games. arXiv preprint arXiv:2104.03113.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.

Hyunwoo Ko, Guijin Son, and Dasol Choi. 2025. Understand, solve and translate: Bridging the multilingual mathematical reasoning gap. arXiv preprint arXiv:2501.02448.

Bespoke Labs. 2025. Bespoke-stratos-17k dataset. Accessed: February 1, 2025.

Huiyuan Lai and Malvina Nissim. 2024. mCoT: Multilingual instruction tuning for reasoning consistency in language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12012–12026, Bangkok, Thailand. Association for Computational Linguistics.

Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Demontcourt, Ryan Rossi, and Thien Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327, Singapore. Association for Computational Linguistics.

Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314–21328.

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857.

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.

Pengfei Liu, Yiming Wang, Tianyu Gao, et al. 2025. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456.

Zihan Liu, Yang Chen, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Acemath: Advancing frontier math reasoning with post-training and reward modeling. arXiv preprint arXiv:2412.15084.

Shayne Longpre, Robert Mahari, Ariel N. Lee, Campbell S. Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole J Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhil Singh, Manuel Cherep, Ahmad Mustafa Anis, An Dinh, Caroline Shamiso Chitongo, Da Yin, Damien Sileo, Deividas Mataciunas, Diganta Misra, Emad A. Alghamdi, Enrico Shipple, Jianguo Zhang, Joanna Materzynska, Kun Qian, Kushagra Tiwary, Lester James Validad Miranda, Manan Dey, Minnie Liang, Mohammed Hamdy, Niklas Muenninghoff, Seonghyeon Ye, Seungone Kim, Shrestha Mohanty, Vipul Gupta, Vivek Sharma, Vu Minh Chien, Xuhui Zhou, Yizhi LI, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker,
Jad Kabbara, and Alex Pentland. 2024. Consent in crisis: The rapid decline of the AI data commons. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. 2024. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592.

Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. 2025. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl- https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1.5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2-Notion Blog.

MAA. 2024. American invitational mathematics examination - aime. In American Invitational Mathematics Examination - AIME 2024.

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannane Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. In Workshop on Reasoning and Planning for Large Language Models.

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2021. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114.

Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2024. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656.

OpenAI, ., Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoochian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevido, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huwen Chang, Hyung Won Chung, Ian Kivilichan, Ian O’Connell, Ian O’Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ika Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraijjeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kevin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondracik, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Afak, Maddie Simens, Madeleine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro,
Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatulin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethershim, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermant, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunningham, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. 2024. Gpt-4o system card. Preprint, arXiv:2410.21276.

OpenAI. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.

OpenAI. 2025. Openai o3-mini system card.

Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479–3495, Seattle, United States. Association for Computational Linguistics.

Irene Plaza, Nina Melero, Cristina del Pozo, Javier Conde, Pedro Reviriego, Marina Mayor-Rocher, and María Grandury. 2024. Spanish and llm benchmarks: is mmlu lost in translation? arXiv preprint arXiv:2406.17789.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseemath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.

Shuaijie She, Wei Zou, Shujian Huang, Wenhao Zhu, Xiang Liu, Xiang Geng, and Jiajun Chen. 2024. MAPO: Advancing multilingual reasoning through multilingual-alignment-as-preference optimization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10015–10027. Association for Computational Linguistics.

Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations.

Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. 2022. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990.

Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. Scaling test-time compute optimally can be more effective than scaling LLM parameters. In The Thirteenth International Conference on Learning Representations.

Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2025. KMMLU: Measuring massive multitask language understanding in Korean. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4076–4104, Albuquerque, New Mexico. Association for Computational Linguistics.

SRI_Lab. 2025. eth-sri/matharena. Original-date: 2025-02-12T19:06:14Z.

Qwen Team. 2024. Qwq: Reflect deeply on the boundaries of the unknown.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaxiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, De-hao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So-raker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamim, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201.08239.

Open Thoughts. 2025. Openthoughts-114k dataset. Accessed: February 1, 2025.

Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. 2024. Toward self-improvement of llms via imagination, searching, and criticizing. arXiv preprint arXiv:2404.12253.

Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. 2024. Alphazero-like tree-search can guide large language model decoding and training. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Yuyang Wu, Yifei Wang, Tianqi Du, Stefanie Jegelka, and Yisen Wang. 2025. When more is less: Understanding chain-of-thought length in LLMs. In Workshop on Reasoning and Planning for Large Language Models.

Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafaïlov, Nathan Lile, Dakota Mahan, Louis Castricato, Jan-Philipp Franken, Nick Haber, and Chelsea Finn. 2025. Towards system 2 reasoning in llms: Learning how to think with meta chain-of-thought. Preprint, arXiv:2501.04682.

Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et al. 2024. Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024a. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.

An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. 2024b. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36.

Guanghao Ye, Khiem Duc Pham, Xinzhi Zhang, Sivakanth Gopi, Baolin Peng, Beibin Li, Janardhan Kulkarni, and Huseyin A Inan. 2025a. On the emergence of thinking in llms i: Searching for the right intuition. arXiv preprint arXiv:2502.06773.

Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. 2025b. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387.

Dongkeun Yoon, Joel Jang, Sungdong Kim, Seungone Kim, Sheikh Shafayat, and Minjoon Seo. 2024. Langbridge: Multilingual reasoning without multilingual supervision. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7502–7522. Association for Computational Linguistics.

Qian Zhang, Wei Li, Hao Chen, Jun Wang, and Yang Liu. 2025. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301.

Zhihan Zhang, Dong-Ho Lee, Yuwei Fang, Wenhao Yu, Mengzhao Jia, Meng Jiang, and Francesco Barbieri. 2023. Plug: Leveraging pivot language in cross-lingual instruction tuning. arXiv preprint arXiv:2311.08711.

Zirui Zhao, Wee Sun Lee, and David Hsu. 2023. Large language models as commonsense knowledge for large-scale task planning. In NeurIPS 2023 Foundation Models for Decision Making Workshop.

Wenhao Zhu, Shujian Huang, Fei Yuan, Shuaijie She, Jiajun Chen, and Alexandra Birch. 2024. Question translation training for better multilingual reasoning. CoRR, abs/2401.07817.
A  Additional details on MCLM

In this section, we provide additional details on the MCLM benchmark, including the languages covered by each subset (Section A.1), the sampling process for MT-MATH100 (Section A.2), the sources for the M-IMO, and M-MO subset (Section A.3), the prompts used for GPT-4o and -mini (Section A.4), and contamination considerations (Section A.5).

A.1  Details in Language Coverage

We examine four groups of languages in this paper: (A) the 55 languages into which MATH500 and AIME2024 have been translated, (B) the 14 languages frequently sampled to reduce evaluation costs, (C) the languages covered in M-IMO, and (D) those in M-MO. The complete list for each group is provided in Table 5.

<table>
  <tr>
    <th>Lang. Group</th>
    <th>Languages (ISO Codes, Sorted Alphabetically)</th>
    <th># Lang.</th>
  </tr>
  <tr>
    <td>(A)</td>
    <td>af, ar, bg, bn, ca, cs, cy, da, de, el, en, es, et, fa, fi, fr, gu, he, hi, hr, hu, id, it, ja, kn, ko, lt, lv, mk, ml, mr, ne, nl, no, pa, pl, pt, ro, ru, sk, sl, so, sq, sv, sw, ta, te, th, tl, tr, uk, ur, vi, zh-cn, zh-tw</td>
    <td>55</td>
  </tr>
  <tr>
    <td>(B)</td>
    <td>af, ar, de, en, es, fr, he, id, it, ja, ko, tr, vi, zh-cn</td>
    <td>14</td>
  </tr>
  <tr>
    <td>(C)</td>
    <td>af, ar, bg, cs, da, de, el, en, et, es, fi, fr, he, hr, hu, id, it, ja, ko, lt, lv, mk, nl, no, pl, pt, ro, ru, sk, sl, sq, sv, th, tr, uk, vi, zh-cn, zh-tw</td>
    <td>38</td>
  </tr>
  <tr>
    <td>(D)</td>
    <td>cs, de, en, fr, ja, ko, nl, pl, ru, sk, zh-cn</td>
    <td>11</td>
  </tr>
</table>

Table 5: Full language lists for each dataset subset. MT-MATH100, MT-AIME2024, M-IMO, and M-MO cover 55, 38, and 11 ISO codes respectively.

Running evaluations on all 55 languages can be computationally intensive, particularly when testing for test-time scaling. To address this, we have created a downsampled group (B) consisting of 14 languages for our experiments. These languages were chosen to represent a broad spectrum of linguistic families and writing systems. In terms of language families, the selection includes representatives from Afro-Asiatic (Arabic, Hebrew), Austronesian (Indonesian), Japonic (Japanese), Koreanic (Korean), Turkic (Turkish), Austroasiatic (Vietnamese), and Sino-Tibetan (Chinese). Additionally, the chosen languages encompass diverse scripts: several, including Afrikaans, German, English, Spanish, French, and Italian, use the Latin alphabet; others use distinct writing systems—Arabic and Hebrew employ abjads (consonant-based scripts); Japanese combines logographic characters (Kanji) with syllabic scripts (Hiragana and Katakana); Korean is written in Hangul; Turkish uses a modified Latin alphabet; Vietnamese utilizes a Latin-based alphabet with diacritics; and Chinese is written using Chinese characters.

A.2  Sampling MATH100

Once creating MGSM, Shi et al. (2023) opted to randomly sample 250 questions from the GSM8K dataset for computational efficiency. Similarly, we sample 100 questions from MATH500 to keep evaluation costs manageable across multiple languages. Initially, we conduct random sampling. Before extending this approach to all 55 languages, we first apply it to the language group (B). For language group (B), we create both the MT-MATH100 and MT-MATH500 versions, where entire subsets are translated for the later. We then evaluate 10 models—each trained using different methods to enhance mathematical reasoning—to determine whether the sampled MATH100 subset reliably represents the full dataset.

In Table 6, we report the performance of the evaluated models. The score differences are relatively small, and even after accounting for minor variations, the ranking of the 10 models remains largely consistent—with only a few instances of rank switching. We conclude that the sampled version serves as an acceptable proxy for the full dataset and proceed accordingly.
<table>
  <tr>
    <th>Rank</th>
    <th>Model</th>
    <th>MATH-500</th>
    <th>MATH-100</th>
    <th>Score Diff.</th>
    <th>Rank Diff.</th>
  </tr>
  <tr>
    <td>1</td>
    <td>o3-mini</td>
    <td>85.00</td>
    <td>85.93</td>
    <td>0.93</td>
    <td>-</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Eurus-2-7B-PRIME</td>
    <td>73.76</td>
    <td>76.63</td>
    <td>2.86</td>
    <td>-</td>
  </tr>
  <tr>
    <td>3</td>
    <td>Qwen2.5-Math-7B-Instruct</td>
    <td>73.70</td>
    <td>75.98</td>
    <td>2.27</td>
    <td>-</td>
  </tr>
  <tr>
    <td>4</td>
    <td>DeepSeek-R1-Distill-Qwen-32B</td>
    <td>72.73</td>
    <td>75.98</td>
    <td>3.24</td>
    <td>-</td>
  </tr>
  <tr>
    <td>5</td>
    <td>DeepSeek-R1-Distill-Qwen-7B</td>
    <td>67.25</td>
    <td>68.69</td>
    <td>1.44</td>
    <td>1 ▲</td>
  </tr>
  <tr>
    <td>6</td>
    <td>AceMath-7B-Instruct</td>
    <td>65.90</td>
    <td>70.06</td>
    <td>4.16</td>
    <td>1 ▼</td>
  </tr>
  <tr>
    <td>7</td>
    <td>AceMath-1.5B-Instruct</td>
    <td>65.60</td>
    <td>68.19</td>
    <td>2.58</td>
    <td>-</td>
  </tr>
  <tr>
    <td>8</td>
    <td>DeepSeek-R1-Distill-Qwen-1.5B</td>
    <td>53.74</td>
    <td>56.78</td>
    <td>3.05</td>
    <td>-</td>
  </tr>
  <tr>
    <td>9</td>
    <td>Qwen2.5-Math-1.5B-Instruct</td>
    <td>51.80</td>
    <td>51.30</td>
    <td>0.51</td>
    <td>-</td>
  </tr>
  <tr>
    <td>10</td>
    <td>Qwen2.5-Math-1.5B-OREO</td>
    <td>39.92</td>
    <td>38.45</td>
    <td>1.47</td>
    <td>-</td>
  </tr>
</table>

Table 6: Model rankings and score comparison between MATH-500 and MATH-100. The score difference was computed as the absolute difference between the MATH-500 and MATH-100 scores. The rank difference indicates the change in ranking on MATH-100 relative to the performance on MATH-500.

A.3 Sourcing M-IMO and M-MO

Relying solely on machine-translated benchmarks (MT-MATH100 and MT-AIME2024) carries inherent risks. To mitigate this, we supplement our dataset with questions from the International Mathematical Olympiad (IMO) and various regional math olympiads. Figure 10 provides an overview of the IMO questions included from 2004 to 2024, while Table 7 lists the sources for regional olympiads. For English, Chinese, and Korean, we utilize existing datasets rather than recollecting questions (He et al., 2024; Ko et al., 2025).

<table>
  <tr>
    <th>Language</th>
    <th>Competition Links</th>
  </tr>
  <tr>
    <td>French</td>
    <td>https://euler.ac-versailles.fr/spip.php?rubrique207</td>
  </tr>
  <tr>
    <td>German</td>
    <td>DeMO</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>https://www.imojp.org/domestic/jmo_overview.html#Problems</td>
  </tr>
  <tr>
    <td>Dutch</td>
    <td>https://prime.ugent.be/activiteiten/puma/<br>https://wiskundeolympiade.nl/wedstrijdarchief/1e-ronde</td>
  </tr>
  <tr>
    <td>Czech</td>
    <td>https://www.matematicaolympiada.cz/mo-pro-ss/rocnik</td>
  </tr>
  <tr>
    <td>Polish</td>
    <td>https://iksko.org/problems.php</td>
  </tr>
  <tr>
    <td>Slovakian</td>
    <td>https://skmo.sk/dokumenty.php?rocnik=74<br>https://riesky.sk/archiv/</td>
  </tr>
  <tr>
    <td>Russian</td>
    <td>https://mmo.mccme.ru/</td>
  </tr>
</table>

Table 7: Link to mathematical competition links that has been included in M-MO subset.

A.4 Prompts

The prompts used for question translation (Figure 13), solution translation (Figure 14), and model judgment (Figure 15) are detailed accordingly.

A.5 Benchmark Contamination

As LLMs are trained on ever-growing datasets, concerns about benchmark contamination have emerged (Xu et al., 2024). Because most training data and procedures remain proprietary, it is virtually impossible to guarantee that a benchmark is entirely free of contamination. Existing detection methods remain unstable, and our MCLM dataset, particularly its M-IMO and M-MO subsets collected from the Internet, may be prone to exposure in various model-training corpora. For instance, we observe that Llama-3.1-3B-Instruct can produce correct answers without any visible reasoning, suggesting prior
familiarity with certain questions. However, lacking a robust decontamination method, we have chosen not to remove potentially contaminated samples. Despite this possibility, the overall low performance of most models on these subsets suggests that MCLM remains a challenging and valuable resource for evaluating multilingual, competition-level math. Furthermore, our primary focus is on assessing multilingual consistency. If a model correctly “guesses” a single language version of a question (possibly due to prior exposure), it does not necessarily indicate strong multilingual reasoning. Conversely, if the model can solve all language variations after seeing only a few, it may demonstrate a degree of cross-lingual robustness.

A.6 License
The machine-translated subsets of MCLM are released under the MIT License. The remaining subsets are provided under a CC BY-NC-ND license, although we may transition to a more permissive license pending further review of the original competition data policies.

B Additional details in training LLMs with system 2 thinking

In this section, we provide additional details on the dataset used to train self-correcting LLMs (Section B.1), ablation studies on MR1 training (Section B.2), and the training configurations.

B.1 Additional details on the training dataset

In training MR1 (i.e., Deepseek-R1-1.5B + SFT with translated data), we first collect thinking trajectories generated by R1 (Guo et al., 2025) from the Numina Math dataset in Be-Spoke Stratos (Labs, 2025) and OpenThoughts (Thoughts, 2025). To ensure high-quality reasoning supervision, we exclude any data distilled from smaller thinking-model variants (e.g., QwQ-32B-Preview (Team, 2024) or the R1-Distil series). We then employ GPT-4o as an LLM-based judge to filter out instances with incorrect answers.

Next, the question and solution for each remaining instance are translated into one of the 14 languages in Group (B). We opt for only 14 languages—as opposed to all 55—because our ablation studies indicate that oversampling languages can negatively impact overall performance. A rule-based parser validates that the question and answer content remains unchanged post-translation. Beyond this verification, no additional quality checks are performed. However, when we encounter elevated loss spikes during training, we backtrack through the dataset to identify and remove problematic instances. Following this process, we retain approximately 120K instances.

Proxying problem difficulty Table 8 compares Euler-Instruct with existing multilingual math datasets: MGSM8KInstruct (Chen et al., 2024a) and mCoT-MATH (Lai and Nissim, 2024). MGSM8KInstruct extends GSM8K (Lightman et al., 2024) by translating it into 10 languages, yielding a parallel dataset of approximately 8,000 questions per language. In contrast, mCoT-MATH sources 560,000 seed questions from MetaMathQA and MathInstruct and translates them into 10 languages.

<table>
  <tr>
    <th>Dataset</th>
    <th># Lang.</th>
    <th># Inst.</th>
    <th>Diff.</th>
  </tr>
  <tr>
    <td>MGSM8KInstruct</td>
    <td>10</td>
    <td>73.6k</td>
    <td>G.S</td>
  </tr>
  <tr>
    <td>mCoT-MATH</td>
    <td>10</td>
    <td>6.3M</td>
    <td>G.S</td>
  </tr>
  <tr>
    <td>Euler-Instruct (Ours)</td>
    <td>55</td>
    <td>250K</td>
    <td>C.L</td>
  </tr>
</table>

Table 8: Comparison of Multilingual Mathematical Reasoning Datasets. The Diff. column indicates difficulty level, where G.S represents grade school level and C.L represents competition level.

To estimate the difficulty level of each dataset, we randomly sample 1,000 questions and measure the solve rate of LLMs. Since MGSM8KInstruct and mCoT-MATH were published before Euler-Instruct, they may have been included in the training of existing LLMs, potentially making a direct comparison unfair. To address this, we first evaluate using OLMo2-{7, 13}B (OLMo et al., 2024); we use the base model instead of the instruct model since GSM8K is used during the instruction tuning phase. Since OLMo2-7B-base is not instruction-tuned, we provide three-shot examples and prompt it to solve the
questions in a structured format: "The answer is X." Additionally, we evaluate with Qwen2.5-7[7, 32, 72]B-Instruct (Yang et al., 2024b) in a zero-shot setting, using a parser to extract answers.

Figure 11 illustrates that, for both mCoT-MATH and MGSM8KInstruct, over half of the problems are solved by Qwen2.5-7B-Instruct, and Qwen2.5-72B-Instruct solves more than 70%. In contrast, the solve rate for EULER-INSTRUCT remains low across all models, with the 7B variants solving less than 20% of the problems and even the 72B variants achieving below 40%. Notably, the Verify subset exhibits a lower solve rate than previous datasets, indicating that its difficulty remains high even when restricted to numerical answers.

B.2 Training Ablations

Before training MR1 we meet the question: How little additional data is required to incorporate a new language into a self-correcting model? To address this, we train four models under a fixed total budget of 24,000 training instances. Table 9 details the language composition and the per-language instance allocation for each model.

<table>
  <tr>
    <th>Languages</th>
    <th># Lang.</th>
    <th># Instances</th>
  </tr>
  <tr>
    <td>ko</td>
    <td>1</td>
    <td>24k</td>
  </tr>
  <tr>
    <td>af, fr, ko</td>
    <td>3</td>
    <td>8k</td>
  </tr>
  <tr>
    <td>af, ar, fr, he, id, ko, tr</td>
    <td>7</td>
    <td>\( \approx 3.5k \)</td>
  </tr>
  <tr>
    <td>all 14 in EULER-INSTRUCT</td>
    <td>14</td>
    <td>\( \approx 1.7k \)</td>
  </tr>
</table>

Table 9: Details on trained models. All models are trained with a total of 24,000 instances. # Instances denote the number of instances used per language.

Figure 12 presents the performance gains over the base model (DeepSeek-R1-1.5B) as a function of the number of training instances. On MT-MATH100, performance rises sharply once the per-language budget reaches approximately 3.5k instances. By contrast, MT-AIME2024 shows more gradual improvements, suggesting that transferring self-correction capabilities to a challenging set of new-language questions requires a larger data allocation. Based on these findings, we use 14 languages within a total budget of 120K, ensuring that each language includes at least 8k instances.

B.3 Training Configurations and Logs

Axolotl (Axolotl AI, 2025) is used for the SFT training in Section 5. We train Qwen2.5-Math-1.5B with DeepSpeed-Zero1 (Rajbhandari et al., 2020) on 4 A100 80GB GPUs for 8 hours per run. Hsu et al. (2024) is used for optimization.

<table>
  <tr>
    <th>Category</th>
    <th>Section 5</th>
  </tr>
  <tr>
    <td>Sequence Length</td>
    <td>16,384</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>\( 2 \times 10^{-5} \)</td>
  </tr>
  <tr>
    <td>Global Batch (Effective)</td>
    <td>128</td>
  </tr>
  <tr>
    <td>Learning Rate Scheduler</td>
    <td>Cosine Decay</td>
  </tr>
  <tr>
    <td>Warmup Ratio</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>Training Epochs</td>
    <td>3</td>
  </tr>
</table>

Table 10: SFT configuration details for Section 5.

C Additional Results

From Tables 11 to 26, we report detailed evaluation results for 55 languages with varying models and test-time scaling methods.
<table>
  <tr>
    <th>Language</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>M-IMO</th>
    <th>M-MO</th>
  </tr>
  <tr><td>Afrikaans</td><td>47.47</td><td>20.00</td><td>11.11</td><td></td></tr>
  <tr><td>Albanian</td><td>45.45</td><td>10.00</td><td>4.00</td><td></td></tr>
  <tr><td>Arabic</td><td>38.38</td><td>30.00</td><td>11.11</td><td></td></tr>
  <tr><td>Bengali</td><td>37.37</td><td>3.33</td><td></td><td></td></tr>
  <tr><td>Bulgarian</td><td>39.39</td><td>13.33</td><td>7.41</td><td></td></tr>
  <tr><td>Catalan</td><td>50.51</td><td>23.33</td><td></td><td></td></tr>
  <tr><td>Chinese (Simplified)</td><td>63.64</td><td>26.67</td><td>18.52</td><td>40.00</td></tr>
  <tr><td>Chinese (Traditional)</td><td>61.62</td><td>20.00</td><td>18.52</td><td></td></tr>
  <tr><td>Croatian</td><td>49.49</td><td>20.00</td><td>7.41</td><td></td></tr>
  <tr><td>Czech</td><td>44.44</td><td>13.33</td><td>14.81</td><td>6.67</td></tr>
  <tr><td>Danish</td><td>53.54</td><td>16.67</td><td>22.22</td><td></td></tr>
  <tr><td>Dutch</td><td>50.51</td><td>36.67</td><td>11.11</td><td>20.00</td></tr>
  <tr><td>Estonian</td><td>39.39</td><td>10.00</td><td>4.00</td><td></td></tr>
  <tr><td>Finnish</td><td>41.41</td><td>16.67</td><td>8.00</td><td></td></tr>
  <tr><td>French</td><td>62.63</td><td>30.00</td><td>18.52</td><td>51.61</td></tr>
  <tr><td>German</td><td>47.47</td><td>26.67</td><td>11.11</td><td>10.00</td></tr>
  <tr><td>Greek</td><td>33.33</td><td>13.33</td><td>5.26</td><td></td></tr>
  <tr><td>Gujarati</td><td>39.39</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Hebrew</td><td>38.38</td><td>13.33</td><td>3.70</td><td></td></tr>
  <tr><td>Hindi</td><td>35.35</td><td>6.67</td><td></td><td></td></tr>
  <tr><td>Hungarian</td><td>51.52</td><td>10.00</td><td>8.00</td><td></td></tr>
  <tr><td>Indonesian</td><td>56.57</td><td>16.67</td><td>14.29</td><td></td></tr>
  <tr><td>Italian</td><td>51.52</td><td>20.00</td><td>20.00</td><td></td></tr>
  <tr><td>Japanese</td><td>56.57</td><td>16.67</td><td>8.00</td><td>0.00</td></tr>
  <tr><td>Kannada</td><td>37.37</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Korean</td><td>44.44</td><td>13.33</td><td>3.70</td><td>36.67</td></tr>
  <tr><td>Latvian</td><td>40.40</td><td>10.00</td><td>12.00</td><td></td></tr>
  <tr><td>Lithuanian</td><td>45.45</td><td>6.67</td><td>18.52</td><td></td></tr>
  <tr><td>Macedonian</td><td>43.43</td><td>10.00</td><td>11.11</td><td></td></tr>
  <tr><td>Malayalam</td><td>43.43</td><td>23.33</td><td></td><td></td></tr>
  <tr><td>Marathi</td><td>34.34</td><td>13.33</td><td></td><td></td></tr>
  <tr><td>Nepali</td><td>36.36</td><td>6.67</td><td></td><td></td></tr>
  <tr><td>Norwegian</td><td>53.54</td><td>23.33</td><td>11.11</td><td></td></tr>
  <tr><td>Persian</td><td>38.38</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Polish</td><td>54.55</td><td>26.67</td><td>14.81</td><td>26.67</td></tr>
  <tr><td>Portuguese</td><td>55.56</td><td>10.00</td><td>24.00</td><td></td></tr>
  <tr><td>Punjabi</td><td>37.37</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Romanian</td><td>49.49</td><td>13.33</td><td>25.93</td><td></td></tr>
  <tr><td>Russian</td><td>59.60</td><td>20.00</td><td>16.00</td><td>20.00</td></tr>
  <tr><td>Slovak</td><td>48.48</td><td>20.00</td><td>11.11</td><td>6.67</td></tr>
  <tr><td>Slovenian</td><td>49.49</td><td>10.00</td><td>14.81</td><td></td></tr>
  <tr><td>Somali</td><td>42.42</td><td>23.33</td><td></td><td></td></tr>
  <tr><td>Spanish</td><td>55.56</td><td>20.00</td><td>18.52</td><td></td></tr>
  <tr><td>Swahili</td><td>34.34</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Swedish</td><td>58.59</td><td>20.00</td><td>8.00</td><td></td></tr>
  <tr><td>Tagalog</td><td>46.46</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Tamil</td><td>38.38</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Telugu</td><td>39.39</td><td>6.67</td><td></td><td></td></tr>
  <tr><td>Thai</td><td>39.39</td><td>23.33</td><td>3.70</td><td></td></tr>
  <tr><td>Turkish</td><td>43.43</td><td>13.33</td><td>7.41</td><td></td></tr>
  <tr><td>Ukrainian</td><td>38.38</td><td>13.33</td><td>11.11</td><td></td></tr>
  <tr><td>Urdu</td><td>35.35</td><td>20.00</td><td></td><td></td></tr>
  <tr><td>Vietnamese</td><td>44.44</td><td>13.33</td><td>7.41</td><td></td></tr>
  <tr><td>Welsh</td><td>39.39</td><td>16.67</td><td></td><td></td></tr>
  <tr><th>English</th><th>67.68</th><th>20.00</th><th>18.52</th><th>56.67</th></tr>
  <tr><th>Average</th><th>46.01</th><th>16.36</th><th>12.23</th><th>25.00</th></tr>
  <tr><th>Standard Deviation</th><th>8.61</th><th>6.89</th><th>6.02</th><th>19.10</th></tr>
  <tr><th>Fleiss’ Kappa</th><th>0.56</th><th>0.68</th><th>0.24</th><th></th></tr>
</table>

Table 11: Evaluation results of Qwen2.5-Math-1.5B-Instruct with greedy decoding on MCLM.
<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="2">ORM (K=2)</th>
    <th colspan="2">ORM (K=4)</th>
    <th colspan="2">ORM (K=8)</th>
  </tr>
  <tr>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
  </tr>
  <tr><td>Afrikaans</td><td>53.54</td><td>23.33</td><td>56.57</td><td>16.67</td><td>60.61</td><td>23.33</td></tr>
  <tr><td>Albanian</td><td>52.53</td><td>10.00</td><td>50.51</td><td>10.00</td><td>47.47</td><td>13.33</td></tr>
  <tr><td>Arabic</td><td>43.43</td><td>20.00</td><td>46.46</td><td>13.33</td><td>51.52</td><td>16.67</td></tr>
  <tr><td>Bengali</td><td>41.41</td><td>10.00</td><td>40.40</td><td>10.00</td><td>41.41</td><td>13.33</td></tr>
  <tr><td>Bulgarian</td><td>45.45</td><td>26.67</td><td>46.46</td><td>20.00</td><td>51.52</td><td>16.67</td></tr>
  <tr><td>Catalan</td><td>59.60</td><td>33.33</td><td>63.64</td><td>33.33</td><td>61.62</td><td>26.67</td></tr>
  <tr><td>Chinese (Simplified)</td><td>69.70</td><td>36.67</td><td>76.77</td><td>30.00</td><td>78.79</td><td>26.67</td></tr>
  <tr><td>Chinese (Traditional)</td><td>68.69</td><td>13.33</td><td>70.71</td><td>20.00</td><td>74.75</td><td>26.67</td></tr>
  <tr><td>Croatian</td><td>51.52</td><td>16.67</td><td>59.60</td><td>23.33</td><td>58.59</td><td>30.00</td></tr>
  <tr><td>Czech</td><td>49.49</td><td>13.33</td><td>56.57</td><td>10.00</td><td>59.60</td><td>16.67</td></tr>
  <tr><td>Danish</td><td>53.54</td><td>23.33</td><td>56.57</td><td>20.00</td><td>59.60</td><td>26.67</td></tr>
  <tr><td>Dutch</td><td>51.52</td><td>30.00</td><td>57.58</td><td>26.67</td><td>63.64</td><td>23.33</td></tr>
  <tr><td>Estonian</td><td>46.46</td><td>13.33</td><td>48.48</td><td>13.33</td><td>50.51</td><td>13.33</td></tr>
  <tr><td>Finnish</td><td>41.41</td><td>13.33</td><td>48.48</td><td>20.00</td><td>53.54</td><td>20.00</td></tr>
  <tr><td>French</td><td>64.65</td><td>40.00</td><td>68.69</td><td>33.33</td><td>73.74</td><td>30.00</td></tr>
  <tr><td>German</td><td>54.55</td><td>23.33</td><td>63.64</td><td>23.33</td><td>64.65</td><td>30.00</td></tr>
  <tr><td>Greek</td><td>39.39</td><td>13.33</td><td>44.44</td><td>10.00</td><td>47.47</td><td>10.00</td></tr>
  <tr><td>Gujarati</td><td>44.44</td><td>10.00</td><td>43.43</td><td>16.67</td><td>47.47</td><td>13.33</td></tr>
  <tr><td>Hebrew</td><td>44.44</td><td>16.67</td><td>46.46</td><td>13.33</td><td>49.49</td><td>10.00</td></tr>
  <tr><td>Hindi</td><td>40.40</td><td>10.00</td><td>45.45</td><td>13.33</td><td>47.47</td><td>16.67</td></tr>
  <tr><td>Hungarian</td><td>53.54</td><td>10.00</td><td>57.58</td><td>10.00</td><td>63.64</td><td>16.67</td></tr>
  <tr><td>Indonesian</td><td>58.59</td><td>20.00</td><td>56.57</td><td>20.00</td><td>59.60</td><td>16.67</td></tr>
  <tr><td>Italian</td><td>57.58</td><td>26.67</td><td>60.61</td><td>26.67</td><td>69.70</td><td>16.67</td></tr>
  <tr><td>Japanese</td><td>59.60</td><td>16.67</td><td>66.67</td><td>23.33</td><td>70.71</td><td>26.67</td></tr>
  <tr><td>Kannada</td><td>45.45</td><td>10.00</td><td>47.47</td><td>16.67</td><td>52.53</td><td>13.33</td></tr>
  <tr><td>Korean</td><td>53.54</td><td>16.67</td><td>56.57</td><td>23.33</td><td>57.58</td><td>13.33</td></tr>
  <tr><td>Latvian</td><td>45.45</td><td>10.00</td><td>51.52</td><td>20.00</td><td>54.55</td><td>16.67</td></tr>
  <tr><td>Lithuanian</td><td>48.48</td><td>10.00</td><td>52.53</td><td>10.00</td><td>57.58</td><td>13.33</td></tr>
  <tr><td>Macedonian</td><td>50.51</td><td>13.33</td><td>51.52</td><td>13.33</td><td>50.51</td><td>10.00</td></tr>
  <tr><td>Malayalam</td><td>47.47</td><td>20.00</td><td>52.53</td><td>20.00</td><td>56.57</td><td>23.33</td></tr>
  <tr><td>Marathi</td><td>39.39</td><td>13.33</td><td>43.43</td><td>23.33</td><td>43.43</td><td>20.00</td></tr>
  <tr><td>Nepali</td><td>38.38</td><td>6.67</td><td>46.46</td><td>3.33</td><td>46.46</td><td>6.67</td></tr>
  <tr><td>Norwegian</td><td>59.60</td><td>26.67</td><td>61.62</td><td>16.67</td><td>65.66</td><td>23.33</td></tr>
  <tr><td>Persian</td><td>40.40</td><td>13.33</td><td>41.41</td><td>13.33</td><td>39.39</td><td>16.67</td></tr>
  <tr><td>Polish</td><td>54.55</td><td>16.67</td><td>57.58</td><td>16.67</td><td>64.65</td><td>16.67</td></tr>
  <tr><td>Portuguese</td><td>58.59</td><td>13.33</td><td>60.61</td><td>13.33</td><td>62.63</td><td>26.67</td></tr>
  <tr><td>Punjabi</td><td>41.41</td><td>16.67</td><td>43.43</td><td>20.00</td><td>42.42</td><td>16.67</td></tr>
  <tr><td>Romanian</td><td>51.52</td><td>23.33</td><td>54.55</td><td>23.33</td><td>56.57</td><td>20.00</td></tr>
  <tr><td>Russian</td><td>60.61</td><td>20.00</td><td>65.66</td><td>23.33</td><td>68.69</td><td>23.33</td></tr>
  <tr><td>Slovak</td><td>52.53</td><td>10.00</td><td>54.55</td><td>20.00</td><td>55.56</td><td>33.33</td></tr>
  <tr><td>Slovenian</td><td>47.47</td><td>16.67</td><td>51.52</td><td>20.00</td><td>54.55</td><td>30.00</td></tr>
  <tr><td>Somali</td><td>44.44</td><td>16.67</td><td>46.46</td><td>16.67</td><td>46.46</td><td>10.00</td></tr>
  <tr><td>Spanish</td><td>58.59</td><td>23.33</td><td>65.66</td><td>26.67</td><td>68.69</td><td>30.00</td></tr>
  <tr><td>Swahili</td><td>37.37</td><td>13.33</td><td>41.41</td><td>20.00</td><td>45.45</td><td>13.33</td></tr>
  <tr><td>Swedish</td><td>57.58</td><td>20.00</td><td>59.60</td><td>23.33</td><td>60.61</td><td>20.00</td></tr>
  <tr><td>Tagalog</td><td>50.51</td><td>16.67</td><td>55.56</td><td>20.00</td><td>57.58</td><td>23.33</td></tr>
  <tr><td>Tamil</td><td>41.41</td><td>16.67</td><td>44.44</td><td>16.67</td><td>47.47</td><td>16.67</td></tr>
  <tr><td>Telugu</td><td>42.42</td><td>13.33</td><td>46.46</td><td>20.00</td><td>48.48</td><td>20.00</td></tr>
  <tr><td>Thai</td><td>44.44</td><td>10.00</td><td>49.49</td><td>20.00</td><td>57.58</td><td>13.33</td></tr>
  <tr><td>Turkish</td><td>50.51</td><td>16.67</td><td>46.46</td><td>13.33</td><td>54.55</td><td>20.00</td></tr>
  <tr><td>Ukrainian</td><td>44.44</td><td>23.33</td><td>51.52</td><td>16.67</td><td>52.53</td><td>26.67</td></tr>
  <tr><td>Urdu</td><td>38.38</td><td>16.67</td><td>41.41</td><td>16.67</td><td>44.44</td><td>20.00</td></tr>
  <tr><td>Vietnamese</td><td>49.49</td><td>23.33</td><td>50.51</td><td>30.00</td><td>52.53</td><td>33.33</td></tr>
  <tr><td>Welsh</td><td>38.38</td><td>16.67</td><td>44.44</td><td>16.67</td><td>44.44</td><td>20.00</td></tr>
  <tr><th>English</th><th>71.72</th><th>16.67</th><th>73.74</th><th>26.67</th><th>76.77</th><th>36.67</th></tr>
  <tr><td>Average</td><td>50.01</td><td>17.64</td><td>53.50</td><td>18.85</td><td>56.25</td><td>20.12</td></tr>
  <tr><td>Standard Deviation</td><td>8.47</td><td>7.05</td><td>8.83</td><td>6.23</td><td>9.50</td><td>6.97</td></tr>
  <tr><td>Fleiss’ Kappa</td><td>0.57</td><td>0.66</td><td>0.60</td><td>0.64</td><td>0.61</td><td>0.63</td></tr>
</table>

Table 12: Evaluation results of Qwen2.5-Math-1.5B-Instruct with Best-of-N (\( K = 2, 4, 8 \)) using Qwen2.5-Math-RM-72B as ORM on MT-MATH100 and MT-AIME2024.
<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="2">PRM (S=3, c=3)</th>
    <th colspan="2">PRM (S=4, c=5)</th>
    <th colspan="2">PRM (S=5, c=8)</th>
  </tr>
  <tr>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>M-IIMO</th>
    <th>M-MO</th>
  </tr>
  <tr><td>Afrikaans</td><td>52.53</td><td>6.67</td><td>57.58</td><td>20.00</td><td>64.65</td><td>10.00</td><td>22.73</td></tr>
  <tr><td>Albanian</td><td>44.44</td><td>13.33</td><td>52.53</td><td>10.00</td><td>45.45</td><td>16.67</td><td>11.54</td></tr>
  <tr><td>Arabic</td><td>41.41</td><td>13.33</td><td>52.53</td><td>13.33</td><td>45.45</td><td>10.00</td><td>7.41</td></tr>
  <tr><td>Bengali</td><td>40.40</td><td>13.33</td><td>44.44</td><td>13.33</td><td>41.41</td><td>16.67</td><td></td></tr>
  <tr><td>Bulgarian</td><td>42.42</td><td>20.00</td><td>42.42</td><td>10.00</td><td>55.56</td><td>10.00</td><td>11.11</td></tr>
  <tr><td>Catalan</td><td>55.56</td><td>10.00</td><td>66.67</td><td>26.67</td><td>61.62</td><td>26.67</td><td></td></tr>
  <tr><td>Chinese (Simplified)</td><td>64.65</td><td>13.33</td><td>75.76</td><td>16.67</td><td>71.72</td><td>33.33</td><td>25.93</td><td>53.33</td></tr>
  <tr><td>Chinese (Traditional)</td><td>63.64</td><td>26.67</td><td>73.74</td><td>16.67</td><td>72.73</td><td>26.67</td><td>29.63</td><td></td></tr>
  <tr><td>Croatian</td><td>50.51</td><td>13.33</td><td>51.52</td><td>20.00</td><td>54.55</td><td>23.33</td><td>14.81</td></tr>
  <tr><td>Czech</td><td>50.51</td><td>10.00</td><td>52.53</td><td>16.67</td><td>58.59</td><td>20.00</td><td>14.81</td><td>10.00</td></tr>
  <tr><td>Danish</td><td>57.58</td><td>10.00</td><td>60.61</td><td>30.00</td><td>60.61</td><td>20.00</td><td>22.22</td></tr>
  <tr><td>Dutch</td><td>56.57</td><td>20.00</td><td>56.57</td><td>26.67</td><td>59.69</td><td>20.00</td><td>7.41</td><td>20.00</td></tr>
  <tr><td>Esperanto</td><td>47.47</td><td>13.33</td><td>51.52</td><td>3.33</td><td>49.49</td><td>10.00</td><td>11.54</td></tr>
  <tr><td>Finnish</td><td>41.41</td><td>10.00</td><td>43.43</td><td>6.67</td><td>49.49</td><td>10.00</td><td>15.38</td></tr>
  <tr><td>French</td><td>62.63</td><td>13.33</td><td>65.66</td><td>30.00</td><td>70.71</td><td>20.00</td><td>18.52</td><td>51.61</td></tr>
  <tr><td>German</td><td>54.55</td><td>40.00</td><td>62.63</td><td>30.00</td><td>58.59</td><td>23.33</td><td>22.22</td><td>16.67</td></tr>
  <tr><td>Greek</td><td>42.42</td><td>13.33</td><td>39.39</td><td>6.67</td><td>44.44</td><td>20.00</td><td>4.35</td></tr>
  <tr><td>Gujarati</td><td>42.42</td><td>6.67</td><td>39.39</td><td>13.33</td><td>41.41</td><td>13.33</td><td></td></tr>
  <tr><td>Hebrew</td><td>46.46</td><td>6.67</td><td>42.42</td><td>23.33</td><td>47.47</td><td>6.67</td><td>7.41</td></tr>
  <tr><td>Hindi</td><td>39.39</td><td>10.00</td><td>46.46</td><td>20.00</td><td>47.47</td><td>10.00</td><td></td></tr>
  <tr><td>Hungarian</td><td>57.58</td><td>26.67</td><td>61.62</td><td>10.00</td><td>57.58</td><td>3.33</td><td>19.23</td></tr>
  <tr><td>Indonesian</td><td>56.57</td><td>16.67</td><td>57.58</td><td>13.33</td><td>64.65</td><td>13.33</td><td>20.83</td></tr>
  <tr><td>Italian</td><td>61.62</td><td>13.33</td><td>61.62</td><td>20.00</td><td>67.68</td><td>23.33</td><td>23.08</td></tr>
  <tr><td>Japanese</td><td>64.65</td><td>20.00</td><td>66.67</td><td>26.67</td><td>66.67</td><td>16.67</td><td>15.38</td><td>7.14</td></tr>
  <tr><td>Kannada</td><td>44.44</td><td>23.33</td><td>42.42</td><td>13.33</td><td>47.47</td><td>13.33</td><td></td></tr>
  <tr><td>Korean</td><td>46.46</td><td>10.00</td><td>45.45</td><td>13.33</td><td>50.51</td><td>13.33</td><td>14.81</td><td>26.67</td></tr>
  <tr><td>Latvian</td><td>47.47</td><td>6.67</td><td>50.51</td><td>16.67</td><td>51.52</td><td>10.00</td><td>15.38</td></tr>
  <tr><td>Lithuanian</td><td>42.42</td><td>10.00</td><td>49.49</td><td>6.67</td><td>45.45</td><td>16.67</td><td>14.81</td></tr>
  <tr><td>Macedonian</td><td>41.41</td><td>13.33</td><td>47.47</td><td>16.67</td><td>48.48</td><td>23.33</td><td>11.11</td></tr>
  <tr><td>Malayalam</td><td>38.38</td><td>16.67</td><td>42.42</td><td>16.67</td><td>43.43</td><td>13.33</td><td></td></tr>
  <tr><td>Marathi</td><td>39.39</td><td>10.00</td><td>43.43</td><td>10.00</td><td>36.36</td><td>13.33</td><td></td></tr>
  <tr><td>Nepali</td><td>41.41</td><td>16.67</td><td>41.41</td><td>26.67</td><td>42.42</td><td>10.00</td><td></td></tr>
  <tr><td>Norwegian</td><td>59.60</td><td>23.33</td><td>65.66</td><td>30.00</td><td>59.60</td><td>26.67</td><td>18.52</td></tr>
  <tr><td>Persian</td><td>37.37</td><td>20.00</td><td>43.43</td><td>13.33</td><td>39.39</td><td>13.33</td><td></td></tr>
  <tr><td>Polish</td><td>49.49</td><td>23.33</td><td>58.59</td><td>23.33</td><td>62.63</td><td>20.00</td><td>25.93</td><td>36.67</td></tr>
  <tr><td>Portuguese</td><td>58.59</td><td>20.00</td><td>57.58</td><td>16.67</td><td>61.62</td><td>30.00</td><td>19.23</td></tr>
  <tr><td>Punjabi</td><td>39.39</td><td>20.00</td><td>40.40</td><td>13.33</td><td>49.49</td><td>6.67</td><td></td></tr>
  <tr><td>Romanian</td><td>57.58</td><td>16.67</td><td>55.56</td><td>13.33</td><td>57.58</td><td>10.00</td><td>22.22</td></tr>
  <tr><td>Russian</td><td>53.54</td><td>23.33</td><td>65.66</td><td>23.33</td><td>64.65</td><td>20.00</td><td>15.38</td><td>23.33</td></tr>
  <tr><td>Slovak</td><td>51.52</td><td>10.00</td><td>52.53</td><td>13.33</td><td>53.54</td><td>20.00</td><td>14.81</td></tr>
  <tr><td>Slovenian</td><td>44.44</td><td>23.33</td><td>47.47</td><td>16.67</td><td>45.45</td><td>26.67</td><td>11.11</td></tr>
  <tr><td>Somali</td><td>43.43</td><td>6.67</td><td>42.42</td><td>23.33</td><td>40.40</td><td>3.33</td><td></td></tr>
  <tr><td>Spanish</td><td>60.61</td><td>16.67</td><td>65.66</td><td>26.67</td><td>72.73</td><td>30.00</td><td>29.63</td></tr>
  <tr><td>Swahili</td><td>38.38</td><td>13.33</td><td>41.41</td><td>13.33</td><td>41.41</td><td>10.00</td><td></td></tr>
  <tr><td>Swedish</td><td>55.56</td><td>13.33</td><td>57.58</td><td>13.33</td><td>57.58</td><td>20.00</td><td>15.38</td></tr>
  <tr><td>Tagalog</td><td>47.47</td><td>20.00</td><td>51.52</td><td>10.00</td><td>55.56</td><td>10.00</td><td></td></tr>
  <tr><td>Tamil</td><td>41.41</td><td>10.00</td><td>45.45</td><td>16.67</td><td>45.45</td><td>16.67</td><td></td></tr>
  <tr><td>Telugu</td><td>42.42</td><td>6.67</td><td>45.45</td><td>13.33</td><td>48.48</td><td>16.67</td><td></td></tr>
  <tr><td>Thai</td><td>39.39</td><td>6.67</td><td>47.47</td><td>6.67</td><td>50.51</td><td>10.00</td><td>14.81</td></tr>
  <tr><td>Turkish</td><td>45.45</td><td>13.33</td><td>50.51</td><td>23.33</td><td>45.45</td><td>10.00</td><td>11.11</td></tr>
  <tr><td>Ukrainian</td><td>39.39</td><td>6.67</td><td>45.45</td><td>23.33</td><td>51.52</td><td>6.67</td><td>18.52</td></tr>
  <tr><td>Urdu</td><td>39.39</td><td>20.00</td><td>40.40</td><td>16.67</td><td>42.42</td><td>13.33</td><td></td></tr>
  <tr><td>Vietnamese</td><td>47.47</td><td>26.67</td><td>53.54</td><td>20.00</td><td>51.52</td><td>13.33</td><td>29.63</td></tr>
  <tr><td>Welsh</td><td>43.43</td><td>10.00</td><td>48.48</td><td>6.67</td><td>51.52</td><td>6.67</td><td></td></tr>
  <tr><th>English</th><th>73.74</th><th>26.67</th><th>79.80</th><th>23.33</th><th>72.73</th><th>23.33</th><th>29.63</th><th>60.00</th></tr>
  <tr><td>Average</td><td>48.87</td><td>15.33</td><td>52.54</td><td>17.15</td><td>53.54</td><td>16.00</td><td>17.31</td><td>30.54</td></tr>
  <tr><td>Standard Deviation</td><td>8.76</td><td>6.93</td><td>9.98</td><td>6.95</td><td>9.71</td><td>7.15</td><td>6.44</td><td>18.88</td></tr>
  <tr><td>Fleiss’ Kappa</td><td>0.57</td><td>0.78</td><td>0.58</td><td>0.61</td><td>0.60</td><td>0.62</td><td>0.43</td><td></td></tr>
</table>

Table 13: Evaluation results of Qwen2.5-Math-1.5B-Instruct using Qwen2.5-Math-PRM-72B as PRM on MCLM.
<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="3">MT-MATH100</th>
  </tr>
  <tr>
    <th>PRM (S=7, c=5)</th>
    <th>PRM (S=7, c=7)</th>
    <th>PRM (S=7, c=11)</th>
  </tr>
  <tr><td>Afrikaans</td><td>55.56</td><td>51.52</td><td>58.59</td></tr>
  <tr><td>Arabic</td><td>44.44</td><td>42.42</td><td>44.44</td></tr>
  <tr><td>Chinese (Simplified)</td><td>71.72</td><td>74.75</td><td>76.77</td></tr>
  <tr><td>French</td><td>64.65</td><td>72.73</td><td>69.70</td></tr>
  <tr><td>German</td><td>57.58</td><td>58.59</td><td>58.59</td></tr>
  <tr><td>Hebrew</td><td>46.46</td><td>39.39</td><td>44.44</td></tr>
  <tr><td>Indonesian</td><td>59.60</td><td>62.63</td><td>61.62</td></tr>
  <tr><td>Italian</td><td>60.61</td><td>60.61</td><td>58.59</td></tr>
  <tr><td>Japanese</td><td>67.68</td><td>67.68</td><td>63.64</td></tr>
  <tr><td>Korean</td><td>48.48</td><td>45.45</td><td>50.51</td></tr>
  <tr><td>Spanish</td><td>64.65</td><td>67.68</td><td>68.69</td></tr>
  <tr><td>Turkish</td><td>50.51</td><td>53.54</td><td>48.48</td></tr>
  <tr><td>Vietnamese</td><td>51.52</td><td>49.49</td><td>51.52</td></tr>
  <tr><th>English</th><th>75.76</th><th>79.80</th><th>74.75</th></tr>
  <tr><td>Average</td><td>58.51</td><td>59.02</td><td>59.31</td></tr>
  <tr><td>Standard Deviation</td><td>9.62</td><td>12.57</td><td>10.60</td></tr>
  <tr><td>Fleiss’ Kappa</td><td>0.56</td><td>0.57</td><td>0.56</td></tr>
</table>

Table 14: Evaluation results of Qwen2.5-Math-1.5B-Instruct using Qwen2.5-Math-PRM-72B as PRM with steps fixed at (\( S = 7 \)) on MT-MATH100.

<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="3">MT-MATH100</th>
  </tr>
  <tr>
    <th>PRM (S=3, c=8)</th>
    <th>PRM (S=6, c=8)</th>
    <th>PRM (S=9, c=8)</th>
  </tr>
  <tr><td>Afrikaans</td><td>54.55</td><td>55.56</td><td>60.61</td></tr>
  <tr><td>Arabic</td><td>41.41</td><td>44.44</td><td>52.53</td></tr>
  <tr><td>Chinese (Simplified)</td><td>71.72</td><td>71.72</td><td>70.71</td></tr>
  <tr><td>French</td><td>67.68</td><td>64.65</td><td>67.68</td></tr>
  <tr><td>German</td><td>56.57</td><td>57.58</td><td>66.67</td></tr>
  <tr><td>Hebrew</td><td>42.42</td><td>46.46</td><td>45.45</td></tr>
  <tr><td>Indonesian</td><td>60.61</td><td>59.60</td><td>62.63</td></tr>
  <tr><td>Italian</td><td>56.57</td><td>60.61</td><td>61.62</td></tr>
  <tr><td>Japanese</td><td>63.64</td><td>67.68</td><td>62.63</td></tr>
  <tr><td>Korean</td><td>47.47</td><td>48.48</td><td>48.48</td></tr>
  <tr><td>Spanish</td><td>65.66</td><td>64.65</td><td>72.73</td></tr>
  <tr><td>Turkish</td><td>53.54</td><td>50.51</td><td>49.49</td></tr>
  <tr><td>Vietnamese</td><td>57.58</td><td>51.52</td><td>57.58</td></tr>
  <tr><th>English</th><th>75.76</th><th>75.76</th><th>77.78</th></tr>
  <tr><td>Average</td><td>58.23</td><td>58.51</td><td>61.18</td></tr>
  <tr><td>Standard Deviation</td><td>10.22</td><td>9.62</td><td>9.65</td></tr>
  <tr><td>Fleiss’ Kappa</td><td>0.56</td><td>0.58</td><td>0.58</td></tr>
</table>

Table 15: Evaluation results of Qwen2.5-Math-1.5B-Instruct using Qwen2.5-Math-PRM-72B as PRM with the number of candidates fixed at 8, on MT-MATH100.
<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="3">MT-MATH100</th>
  </tr>
  <tr>
    <th>PRM (S=7, c=7)</th>
    <th>PRM (S=7, c=11)</th>
    <th>PRM (S=7, c=18)</th>
  </tr>
  <tr><td>Afrikaans</td><td>51.52</td><td>58.59</td><td>58.59</td></tr>
  <tr><td>Arabic</td><td>42.42</td><td>44.44</td><td>52.53</td></tr>
  <tr><td>Chinese (Simplified)</td><td>74.75</td><td>76.77</td><td>76.77</td></tr>
  <tr><td>French</td><td>72.73</td><td>69.70</td><td>71.72</td></tr>
  <tr><td>German</td><td>58.59</td><td>58.59</td><td>60.61</td></tr>
  <tr><td>Hebrew</td><td>39.39</td><td>44.44</td><td>41.41</td></tr>
  <tr><td>Indonesian</td><td>62.63</td><td>61.62</td><td>62.63</td></tr>
  <tr><td>Italian</td><td>60.61</td><td>58.59</td><td>64.65</td></tr>
  <tr><td>Japanese</td><td>67.68</td><td>63.64</td><td>61.62</td></tr>
  <tr><td>Korean</td><td>45.45</td><td>50.51</td><td>50.51</td></tr>
  <tr><td>Spanish</td><td>67.68</td><td>68.69</td><td>68.69</td></tr>
  <tr><td>Turkish</td><td>53.54</td><td>48.48</td><td>52.53</td></tr>
  <tr><td>Vietnamese</td><td>49.49</td><td>51.52</td><td>51.52</td></tr>
  <tr><th>English</th><th>79.80</th><th>74.75</th><th>70.71</th></tr>
  <tr><td>Average</td><td>59.02</td><td>59.31</td><td>60.32</td></tr>
  <tr><td>Standard Deviation</td><td>12.57</td><td>10.60</td><td>9.84</td></tr>
  <tr><td>Fleiss’ Kappa</td><td>0.52</td><td>0.55</td><td>0.54</td></tr>
</table>

Table 16: Evaluation results of Qwen2.5-Math-1.5B-Instruct using Qwen2.5-Math-PRM-7B as PRM with the number of candidates fixed at 7, on MT-MATH100.

<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="3">MT-MATH100</th>
  </tr>
  <tr>
    <th>PRM (S=3, c=13)</th>
    <th>PRM (S=6, c=13)</th>
    <th>PRM (S=9, c=13)</th>
  </tr>
  <tr><td>Afrikaans</td><td>55.56</td><td>59.60</td><td>54.55</td></tr>
  <tr><td>Arabic</td><td>44.44</td><td>45.45</td><td>44.44</td></tr>
  <tr><td>Chinese (Simplified)</td><td>75.76</td><td>70.71</td><td>79.80</td></tr>
  <tr><td>French</td><td>64.65</td><td>71.72</td><td>73.74</td></tr>
  <tr><td>German</td><td>55.56</td><td>63.64</td><td>61.62</td></tr>
  <tr><td>Hebrew</td><td>46.46</td><td>43.43</td><td>47.47</td></tr>
  <tr><td>Indonesian</td><td>56.57</td><td>58.59</td><td>61.62</td></tr>
  <tr><td>Italian</td><td>62.63</td><td>60.61</td><td>61.62</td></tr>
  <tr><td>Japanese</td><td>58.59</td><td>67.68</td><td>59.60</td></tr>
  <tr><td>Korean</td><td>49.49</td><td>48.48</td><td>51.52</td></tr>
  <tr><td>Spanish</td><td>60.61</td><td>73.74</td><td>64.65</td></tr>
  <tr><td>Turkish</td><td>49.49</td><td>50.51</td><td>49.49</td></tr>
  <tr><td>Vietnamese</td><td>52.53</td><td>48.48</td><td>45.45</td></tr>
  <tr><th>English</th><th>71.72</th><th>73.74</th><th>77.78</th></tr>
  <tr><td>Average</td><td>57.43</td><td>59.74</td><td>59.52</td></tr>
  <tr><td>Standard Deviation</td><td>9.10</td><td>10.90</td><td>11.59</td></tr>
  <tr><td>Fleiss’ Kappa</td><td>0.54</td><td>0.55</td><td>0.52</td></tr>
</table>

Table 17: Evaluation results of Qwen2.5-Math-1.5B-Instruct using Qwen2.5-Math-PRM-7B as PRM with the number of candidates fixed at 13, on MT-MATH100.
<table>
  <tr>
    <th>Language</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>M-IMO</th>
    <th>M-MO</th>
  </tr>
  <tr><td>Afrikaans</td><td>47.47</td><td>36.67</td><td>5.56</td><td></td></tr>
  <tr><td>Albanian</td><td>31.31</td><td>13.33</td><td>8.00</td><td></td></tr>
  <tr><td>Arabic</td><td>36.36</td><td>23.33</td><td>7.41</td><td></td></tr>
  <tr><td>Bengali</td><td>33.33</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Bulgarian</td><td>41.41</td><td>10.00</td><td>11.11</td><td></td></tr>
  <tr><td>Catalan</td><td>47.47</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Chinese (Simplified)</td><td>57.58</td><td>23.33</td><td>18.52</td><td>23.33</td></tr>
  <tr><td>Chinese (Traditional)</td><td>43.43</td><td>16.67</td><td>22.22</td><td></td></tr>
  <tr><td>Croatian</td><td>38.38</td><td>16.67</td><td>7.41</td><td></td></tr>
  <tr><td>Czech</td><td>33.33</td><td>30.00</td><td>3.70</td><td>3.33</td></tr>
  <tr><td>Danish</td><td>41.41</td><td>23.33</td><td>7.41</td><td></td></tr>
  <tr><td>Dutch</td><td>45.45</td><td>16.67</td><td>7.41</td><td>16.67</td></tr>
  <tr><td>Estonian</td><td>38.38</td><td>10.00</td><td>12.00</td><td></td></tr>
  <tr><td>Finnish</td><td>30.30</td><td>23.33</td><td>12.00</td><td></td></tr>
  <tr><td>French</td><td>39.39</td><td>6.67</td><td>7.41</td><td>35.48</td></tr>
  <tr><td>German</td><td>45.45</td><td>23.33</td><td>18.52</td><td>6.67</td></tr>
  <tr><td>Greek</td><td>30.30</td><td>16.67</td><td>0.00</td><td></td></tr>
  <tr><td>Gujarati</td><td>27.27</td><td>6.67</td><td></td><td></td></tr>
  <tr><td>Hebrew</td><td>36.36</td><td>16.67</td><td>7.41</td><td></td></tr>
  <tr><td>Hindi</td><td>36.36</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Hungarian</td><td>39.39</td><td>16.67</td><td>8.00</td><td></td></tr>
  <tr><td>Indonesian</td><td>37.37</td><td>13.33</td><td>4.76</td><td></td></tr>
  <tr><td>Italian</td><td>41.41</td><td>13.33</td><td>12.00</td><td></td></tr>
  <tr><td>Japanese</td><td>45.45</td><td>20.00</td><td>12.00</td><td>3.57</td></tr>
  <tr><td>Kannada</td><td>32.32</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Korean</td><td>39.39</td><td>16.67</td><td>14.81</td><td>16.67</td></tr>
  <tr><td>Latvian</td><td>30.30</td><td>6.67</td><td>4.00</td><td></td></tr>
  <tr><td>Lithuanian</td><td>31.31</td><td>6.67</td><td>14.81</td><td></td></tr>
  <tr><td>Macedonian</td><td>31.31</td><td>0.00</td><td>7.41</td><td></td></tr>
  <tr><td>Malayalam</td><td>27.27</td><td>13.33</td><td></td><td></td></tr>
  <tr><td>Marathi</td><td>33.33</td><td>13.33</td><td></td><td></td></tr>
  <tr><td>Nepali</td><td>35.35</td><td>13.33</td><td></td><td></td></tr>
  <tr><td>Norwegian</td><td>37.37</td><td>16.67</td><td>11.11</td><td></td></tr>
  <tr><td>Persian</td><td>29.29</td><td>20.00</td><td></td><td></td></tr>
  <tr><td>Polish</td><td>38.38</td><td>6.67</td><td>11.11</td><td>13.33</td></tr>
  <tr><td>Portuguese</td><td>47.47</td><td>20.00</td><td>8.00</td><td></td></tr>
  <tr><td>Punjabi</td><td>29.29</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Romanian</td><td>41.41</td><td>10.00</td><td>18.52</td><td></td></tr>
  <tr><td>Russian</td><td>46.46</td><td>16.67</td><td>12.00</td><td>20.00</td></tr>
  <tr><td>Slovak</td><td>35.35</td><td>16.67</td><td>11.11</td><td>10.00</td></tr>
  <tr><td>Slovenian</td><td>35.35</td><td>23.33</td><td>11.11</td><td></td></tr>
  <tr><td>Somali</td><td>26.26</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Spanish</td><td>46.46</td><td>16.67</td><td>11.11</td><td></td></tr>
  <tr><td>Swahili</td><td>36.36</td><td>6.67</td><td></td><td></td></tr>
  <tr><td>Swedish</td><td>39.39</td><td>13.33</td><td>8.00</td><td></td></tr>
  <tr><td>Tagalog</td><td>35.35</td><td>13.33</td><td></td><td></td></tr>
  <tr><td>Tamil</td><td>33.33</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Telugu</td><td>34.34</td><td>13.33</td><td></td><td></td></tr>
  <tr><td>Thai</td><td>30.30</td><td>10.00</td><td>7.41</td><td></td></tr>
  <tr><td>Turkish</td><td>42.42</td><td>6.67</td><td>11.11</td><td></td></tr>
  <tr><td>Ukrainian</td><td>35.35</td><td>3.33</td><td>11.11</td><td></td></tr>
  <tr><td>Urdu</td><td>28.28</td><td>13.33</td><td></td><td></td></tr>
  <tr><td>Vietnamese</td><td>31.31</td><td>10.00</td><td>7.41</td><td></td></tr>
  <tr><td>Welsh</td><td>30.30</td><td>23.33</td><td></td><td></td></tr>
  <tr><th>English</th><th>65.66</th><th>20.00</th><th>25.93</th><th>53.33</th></tr>
  <tr><th>Average</th><th>37.47</th><th>14.85</th><th>10.50</th><th>18.40</th></tr>
  <tr><th>Standard Deviation</th><th>7.56</th><th>6.69</th><th>5.16</th><th>14.92</th></tr>
  <tr><th>Fleiss’ Kappa</th><th>0.41</th><th>0.13</th><th>0.19</th><th></th></tr>
</table>

Table 18: Evaluation results of Qwen2.5-Math-1.5B-Instruct + SFT on MCLM.
<table>
  <tr>
    <th>Language</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>M-IMO</th>
    <th>M-MO</th>
  </tr>
  <tr><td>Afrikaans</td><td>39.39</td><td>10.00</td><td>13.64</td><td></td></tr>
  <tr><td>Albanian</td><td>39.39</td><td>16.67</td><td>7.69</td><td></td></tr>
  <tr><td>Arabic</td><td>41.41</td><td>16.67</td><td>14.81</td><td></td></tr>
  <tr><td>Bengali</td><td>39.39</td><td>30.00</td><td></td><td></td></tr>
  <tr><td>Bulgarian</td><td>42.42</td><td>10.00</td><td>11.11</td><td></td></tr>
  <tr><td>Catalan</td><td>51.52</td><td>26.67</td><td></td><td></td></tr>
  <tr><td>Chinese (Simplified)</td><td>50.51</td><td>23.33</td><td>7.41</td><td>13.33</td></tr>
  <tr><td>Chinese (Traditional)</td><td>52.53</td><td>20.00</td><td>11.11</td><td></td></tr>
  <tr><td>Croatian</td><td>38.38</td><td>13.33</td><td>11.11</td><td></td></tr>
  <tr><td>Czech</td><td>51.52</td><td>23.33</td><td>11.11</td><td>10.00</td></tr>
  <tr><td>Danish</td><td>40.40</td><td>6.67</td><td>3.70</td><td></td></tr>
  <tr><td>Dutch</td><td>48.48</td><td>20.00</td><td>11.11</td><td>20.00</td></tr>
  <tr><td>Estonian</td><td>37.37</td><td>23.33</td><td>15.38</td><td></td></tr>
  <tr><td>Finnish</td><td>40.40</td><td>20.00</td><td>7.69</td><td></td></tr>
  <tr><td>French</td><td>46.46</td><td>10.00</td><td>7.41</td><td>32.26</td></tr>
  <tr><td>German</td><td>49.49</td><td>10.00</td><td>7.41</td><td>3.33</td></tr>
  <tr><td>Greek</td><td>28.28</td><td>20.00</td><td>17.39</td><td></td></tr>
  <tr><td>Gujarati</td><td>42.42</td><td>13.33</td><td></td><td></td></tr>
  <tr><td>Hebrew</td><td>39.39</td><td>13.33</td><td>3.70</td><td></td></tr>
  <tr><td>Hindi</td><td>45.45</td><td>13.33</td><td></td><td></td></tr>
  <tr><td>Hungarian</td><td>43.43</td><td>40.00</td><td>11.54</td><td></td></tr>
  <tr><td>Indonesian</td><td>51.52</td><td>16.67</td><td>16.67</td><td></td></tr>
  <tr><td>Italian</td><td>48.48</td><td>13.33</td><td>11.54</td><td></td></tr>
  <tr><td>Japanese</td><td>50.51</td><td>6.67</td><td>11.54</td><td>3.57</td></tr>
  <tr><td>Kannada</td><td>32.32</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Korean</td><td>55.56</td><td>10.00</td><td>11.11</td><td>26.67</td></tr>
  <tr><td>Latvian</td><td>42.42</td><td>10.00</td><td>15.38</td><td></td></tr>
  <tr><td>Lithuanian</td><td>36.36</td><td>13.33</td><td>7.41</td><td></td></tr>
  <tr><td>Macedonian</td><td>39.39</td><td>13.33</td><td>18.52</td><td></td></tr>
  <tr><td>Malayalam</td><td>34.34</td><td>26.67</td><td></td><td></td></tr>
  <tr><td>Marathi</td><td>37.37</td><td>23.33</td><td></td><td></td></tr>
  <tr><td>Nepali</td><td>42.42</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Norwegian</td><td>42.42</td><td>10.00</td><td>3.70</td><td></td></tr>
  <tr><td>Persian</td><td>47.47</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Polish</td><td>38.38</td><td>10.00</td><td>14.81</td><td>20.00</td></tr>
  <tr><td>Portuguese</td><td>50.51</td><td>26.67</td><td>11.54</td><td></td></tr>
  <tr><td>Punjabi</td><td>29.29</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Romanian</td><td>45.45</td><td>6.67</td><td>11.11</td><td></td></tr>
  <tr><td>Russian</td><td>57.58</td><td>13.33</td><td>7.69</td><td>36.67</td></tr>
  <tr><td>Slovak</td><td>47.47</td><td>20.00</td><td>7.41</td><td></td></tr>
  <tr><td>Slovenian</td><td>39.39</td><td>23.33</td><td>18.52</td><td></td></tr>
  <tr><td>Somali</td><td>22.22</td><td>26.67</td><td></td><td></td></tr>
  <tr><td>Spanish</td><td>44.44</td><td>16.67</td><td>0.00</td><td></td></tr>
  <tr><td>Swahili</td><td>34.34</td><td>6.67</td><td></td><td></td></tr>
  <tr><td>Swedish</td><td>42.42</td><td>10.00</td><td>3.85</td><td></td></tr>
  <tr><td>Tagalog</td><td>35.35</td><td>6.67</td><td></td><td></td></tr>
  <tr><td>Tamil</td><td>36.36</td><td>23.33</td><td></td><td></td></tr>
  <tr><td>Telugu</td><td>36.36</td><td>13.33</td><td></td><td></td></tr>
  <tr><td>Thai</td><td>34.34</td><td>26.67</td><td>14.81</td><td></td></tr>
  <tr><td>Turkish</td><td>39.39</td><td>23.33</td><td>7.41</td><td></td></tr>
  <tr><td>Ukrainian</td><td>49.49</td><td>10.00</td><td>7.41</td><td></td></tr>
  <tr><td>Urdu</td><td>32.32</td><td>20.00</td><td></td><td></td></tr>
  <tr><td>Vietnamese</td><td>47.47</td><td>10.00</td><td>18.52</td><td></td></tr>
  <tr><td>Welsh</td><td>28.28</td><td>20.00</td><td></td><td></td></tr>
  <tr><th>English</th><th>51.52</th><th>26.67</th><th>7.41</th><th>40.00</th></tr>
  <tr><th>Average</th><th>42.02</th><th>16.67</th><th>10.52</th><th>20.58</th></tr>
  <tr><th>Standard Deviation</th><th>7.46</th><th>7.31</th><th>4.63</th><th>13.17</th></tr>
  <tr><th>Fleiss’ Kappa</th><th>0.40</th><th>0.13</th><th>0.25</th><th></th></tr>
</table>

Table 19: Evaluation results of Qwen2.5-Math-1.5B-Instruct + MT-SFT on MCLM.
<table>
  <tr>
    <th>Language</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>M-IMO</th>
    <th>M-MO</th>
  </tr>
  <tr><td>Afrikaans</td><td>58.59</td><td>20.00</td><td>11.11</td><td></td></tr>
  <tr><td>Albanian</td><td>46.46</td><td>30.00</td><td>16.00</td><td></td></tr>
  <tr><td>Arabic</td><td>51.52</td><td>20.00</td><td>18.52</td><td></td></tr>
  <tr><td>Bengali</td><td>56.57</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Bulgarian</td><td>57.58</td><td>16.67</td><td>11.11</td><td></td></tr>
  <tr><td>Catalan</td><td>64.65</td><td>30.00</td><td></td><td></td></tr>
  <tr><td>Chinese (Simplified)</td><td>69.70</td><td>16.67</td><td>25.93</td><td>33.33</td></tr>
  <tr><td>Chinese (Traditional)</td><td>67.68</td><td>20.00</td><td>18.52</td><td></td></tr>
  <tr><td>Croatian</td><td>59.60</td><td>36.67</td><td>18.52</td><td></td></tr>
  <tr><td>Czech</td><td>57.58</td><td>33.33</td><td>18.52</td><td>16.67</td></tr>
  <tr><td>Danish</td><td>56.57</td><td>16.67</td><td>14.81</td><td></td></tr>
  <tr><td>Dutch</td><td>64.65</td><td>30.00</td><td>22.22</td><td>23.33</td></tr>
  <tr><td>Estonian</td><td>39.39</td><td>6.67</td><td>12.00</td><td></td></tr>
  <tr><td>Finnish</td><td>52.53</td><td>16.67</td><td>20.00</td><td></td></tr>
  <tr><td>French</td><td>63.64</td><td>26.67</td><td>29.63</td><td>48.39</td></tr>
  <tr><td>German</td><td>63.64</td><td>16.67</td><td>25.93</td><td>26.67</td></tr>
  <tr><td>Greek</td><td>38.38</td><td>13.33</td><td>10.53</td><td></td></tr>
  <tr><td>Gujarati</td><td>47.47</td><td>3.33</td><td></td><td></td></tr>
  <tr><td>Hebrew</td><td>61.62</td><td>23.33</td><td>7.41</td><td></td></tr>
  <tr><td>Hindi</td><td>61.62</td><td>23.33</td><td></td><td></td></tr>
  <tr><td>Hungarian</td><td>55.56</td><td>26.67</td><td>24.00</td><td></td></tr>
  <tr><td>Indonesian</td><td>69.70</td><td>13.33</td><td>23.81</td><td></td></tr>
  <tr><td>Italian</td><td>69.70</td><td>36.67</td><td>28.00</td><td></td></tr>
  <tr><td>Japanese</td><td>62.63</td><td>16.67</td><td>12.00</td><td>3.57</td></tr>
  <tr><td>Kannada</td><td>42.42</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Korean</td><td>61.62</td><td>20.00</td><td>11.11</td><td>30.00</td></tr>
  <tr><td>Latvian</td><td>49.49</td><td>6.67</td><td>20.00</td><td></td></tr>
  <tr><td>Lithuanian</td><td>40.40</td><td>23.33</td><td>14.81</td><td></td></tr>
  <tr><td>Macedonian</td><td>59.60</td><td>23.33</td><td>25.93</td><td></td></tr>
  <tr><td>Malayalam</td><td>41.41</td><td>3.33</td><td></td><td></td></tr>
  <tr><td>Marathi</td><td>39.39</td><td>23.33</td><td></td><td></td></tr>
  <tr><td>Nepali</td><td>50.51</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Norwegian</td><td>67.68</td><td>13.33</td><td>18.52</td><td></td></tr>
  <tr><td>Persian</td><td>61.62</td><td>13.33</td><td></td><td></td></tr>
  <tr><td>Polish</td><td>62.63</td><td>16.67</td><td>22.22</td><td>23.33</td></tr>
  <tr><td>Portuguese</td><td>75.76</td><td>23.33</td><td>16.00</td><td></td></tr>
  <tr><td>Punjabi</td><td>42.42</td><td>13.33</td><td></td><td></td></tr>
  <tr><td>Romanian</td><td>58.59</td><td>26.67</td><td>22.22</td><td></td></tr>
  <tr><td>Russian</td><td>68.69</td><td>33.33</td><td>20.00</td><td>26.67</td></tr>
  <tr><td>Slovak</td><td>58.59</td><td>13.33</td><td>11.11</td><td>20.00</td></tr>
  <tr><td>Slovenian</td><td>56.57</td><td>30.00</td><td>14.81</td><td></td></tr>
  <tr><td>Somali</td><td>30.30</td><td>20.00</td><td></td><td></td></tr>
  <tr><td>Spanish</td><td>69.70</td><td>30.00</td><td>25.93</td><td></td></tr>
  <tr><td>Swahili</td><td>42.42</td><td>20.00</td><td></td><td></td></tr>
  <tr><td>Swedish</td><td>54.55</td><td>13.33</td><td>20.00</td><td></td></tr>
  <tr><td>Tagalog</td><td>47.47</td><td>23.33</td><td></td><td></td></tr>
  <tr><td>Tamil</td><td>40.40</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Telugu</td><td>36.36</td><td>23.33</td><td></td><td></td></tr>
  <tr><td>Thai</td><td>59.60</td><td>13.33</td><td>29.63</td><td></td></tr>
  <tr><td>Turkish</td><td>61.62</td><td>36.67</td><td>22.22</td><td></td></tr>
  <tr><td>Ukrainian</td><td>67.68</td><td>16.67</td><td>18.52</td><td></td></tr>
  <tr><td>Urdu</td><td>50.51</td><td>20.00</td><td></td><td></td></tr>
  <tr><td>Vietnamese</td><td>61.62</td><td>13.33</td><td>33.33</td><td></td></tr>
  <tr><td>Welsh</td><td>34.34</td><td>16.67</td><td></td><td></td></tr>
  <tr><th>English</th><th>67.68</th><th>20.00</th><th>14.81</th><th>66.67</th></tr>
  <tr><th>Average</th><th>55.61</th><th>19.94</th><th>19.20</th><th>28.97</th></tr>
  <tr><th>Standard Deviation</th><th>10.93</th><th>8.10</th><th>6.24</th><th>16.64</th></tr>
  <tr><th>Fleiss’ Kappa</th><th>0.47</th><th>0.30</th><th>0.19</th><th></th></tr>
</table>

Table 20: Evaluation results of DeepSeek-R1-1.5B + MT-SFT on MCLM.
<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="2">BF (N=2048)</th>
    <th colspan="2">BF (N=4096)</th>
    <th colspan="3">BF (N=8192)</th>
  </tr>
  <tr>
    <th>MT-AIME2024</th>
    <th>MT-AIME2024</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>M-IMO</th>
    <th>M-MO</th>
  </tr>
  <tr><td>Afrikaans</td><td>23.33</td><td>23.33</td><td>59.60</td><td>30.00</td><td>9.09</td></tr>
  <tr><td>Albanian</td><td>23.33</td><td>26.67</td><td>48.48</td><td>26.67</td><td>7.69</td></tr>
  <tr><td>Arabic</td><td>16.67</td><td>23.33</td><td>60.61</td><td>26.67</td><td>14.81</td></tr>
  <tr><td>Bengali</td><td>33.33</td><td>30.00</td><td>54.55</td><td>23.33</td></tr>
  <tr><td>Bulgarian</td><td>33.33</td><td>33.33</td><td>61.62</td><td>26.67</td><td>22.22</td></tr>
  <tr><td>Catalan</td><td>20.00</td><td>43.33</td><td>64.65</td><td>43.33</td></tr>
  <tr><td>Chinese (Simplified)</td><td>20.00</td><td>16.67</td><td>69.70</td><td>16.67</td><td>22.22</td><td>40.00</td></tr>
  <tr><td>Chinese (Traditional)</td><td>26.67</td><td>26.67</td><td>70.71</td><td>36.67</td><td>18.52</td></tr>
  <tr><td>Croatian</td><td>30.00</td><td>30.00</td><td>60.61</td><td>30.00</td><td>37.04</td></tr>
  <tr><td>Czech</td><td>40.00</td><td>20.00</td><td>62.63</td><td>20.00</td><td>29.63</td><td>33.33</td></tr>
  <tr><td>Danish</td><td>30.00</td><td>33.33</td><td>61.62</td><td>30.00</td><td>22.22</td></tr>
  <tr><td>Dutch</td><td>10.00</td><td>23.33</td><td>70.71</td><td>36.67</td><td>25.93</td><td>20.00</td></tr>
  <tr><td>Estonian</td><td>23.33</td><td>16.67</td><td>40.40</td><td>20.00</td><td>15.38</td></tr>
  <tr><td>Finnish</td><td>20.00</td><td>33.33</td><td>51.52</td><td>20.00</td><td>30.77</td></tr>
  <tr><td>French</td><td>16.67</td><td>23.33</td><td>72.73</td><td>16.67</td><td>25.93</td><td>51.61</td></tr>
  <tr><td>German</td><td>26.67</td><td>20.00</td><td>75.76</td><td>26.67</td><td>25.93</td><td>30.00</td></tr>
  <tr><td>Greek</td><td>6.67</td><td>13.33</td><td>42.42</td><td>16.67</td><td>21.74</td></tr>
  <tr><td>Gujarati</td><td>16.67</td><td>16.67</td><td>51.52</td><td>16.67</td></tr>
  <tr><td>Hebrew</td><td>33.33</td><td>23.33</td><td>60.61</td><td>16.67</td><td>14.81</td></tr>
  <tr><td>Hindi</td><td>26.67</td><td>10.00</td><td>61.62</td><td>20.00</td></tr>
  <tr><td>Hungarian</td><td>30.00</td><td>26.67</td><td>58.59</td><td>23.33</td><td>26.92</td></tr>
  <tr><td>Indonesian</td><td>10.00</td><td>30.00</td><td>73.74</td><td>30.00</td><td>25</td></tr>
  <tr><td>Italian</td><td>20.00</td><td>26.67</td><td>74.75</td><td>36.67</td><td>23.08</td></tr>
  <tr><td>Japanese</td><td>20.00</td><td>16.67</td><td>63.64</td><td>36.67</td><td>23.08</td><td>7.14</td></tr>
  <tr><td>Kannada</td><td>10.00</td><td>13.33</td><td>49.49</td><td>10.00</td></tr>
  <tr><td>Korean</td><td>16.67</td><td>23.33</td><td>64.65</td><td>20.00</td><td>11.11</td><td>40.00</td></tr>
  <tr><td>Latvian</td><td>30.00</td><td>20.00</td><td>52.53</td><td>10.00</td><td>23.08</td></tr>
  <tr><td>Lithuanian</td><td>10.00</td><td>6.67</td><td>46.46</td><td>26.67</td><td>18.52</td></tr>
  <tr><td>Macedonian</td><td>20.00</td><td>20.00</td><td>63.64</td><td>23.33</td><td>25.93</td></tr>
  <tr><td>Malayalam</td><td>10.00</td><td>13.33</td><td>51.52</td><td>13.33</td></tr>
  <tr><td>Marathi</td><td>20.00</td><td>26.67</td><td>51.52</td><td>23.33</td></tr>
  <tr><td>Nepali</td><td>30.00</td><td>13.33</td><td>54.55</td><td>20.00</td></tr>
  <tr><td>Norwegian</td><td>26.67</td><td>26.67</td><td>65.66</td><td>20.00</td><td>18.52</td></tr>
  <tr><td>Persian</td><td>26.67</td><td>23.33</td><td>62.63</td><td>36.67</td></tr>
  <tr><td>Polish</td><td>23.33</td><td>20.00</td><td>66.67</td><td>16.67</td><td>14.81</td><td>23.33</td></tr>
  <tr><td>Portuguese</td><td>20.00</td><td>26.67</td><td>79.80</td><td>20.00</td><td>15.38</td></tr>
  <tr><td>Punjabi</td><td>23.33</td><td>26.67</td><td>51.52</td><td>20.00</td></tr>
  <tr><td>Romanian</td><td>30.00</td><td>23.33</td><td>60.61</td><td>10.00</td><td>22.22</td></tr>
  <tr><td>Russian</td><td>36.67</td><td>30.00</td><td>72.73</td><td>30.00</td><td>23.08</td><td>30.00</td></tr>
  <tr><td>Slovak</td><td>40.00</td><td>23.33</td><td>66.67</td><td>30.00</td><td>25.93</td></tr>
  <tr><td>Slovenian</td><td>20.00</td><td>20.00</td><td>60.61</td><td>33.33</td><td>25.93</td></tr>
  <tr><td>Somali</td><td>20.00</td><td>16.67</td><td>35.35</td><td>16.67</td></tr>
  <tr><td>Spanish</td><td>30.00</td><td>30.00</td><td>71.72</td><td>40.00</td><td>18.52</td></tr>
  <tr><td>Swahili</td><td>13.33</td><td>13.33</td><td>41.41</td><td>30.00</td></tr>
  <tr><td>Swedish</td><td>13.33</td><td>16.67</td><td>62.63</td><td>23.33</td><td>19.23</td></tr>
  <tr><td>Tagalog</td><td>10.00</td><td>20.00</td><td>52.53</td><td>23.33</td></tr>
  <tr><td>Tamil</td><td>26.67</td><td>20.00</td><td>44.44</td><td>23.33</td></tr>
  <tr><td>Telugu</td><td>13.33</td><td>16.67</td><td>44.44</td><td>20.00</td></tr>
  <tr><td>Thai</td><td>26.67</td><td>13.33</td><td>64.65</td><td>23.33</td><td>11.11</td></tr>
  <tr><td>Turkish</td><td>20.00</td><td>16.67</td><td>61.62</td><td>16.67</td><td>33.33</td></tr>
  <tr><td>Ukrainian</td><td>30.00</td><td>26.67</td><td>73.74</td><td>23.33</td><td>22.22</td></tr>
  <tr><td>Urdu</td><td>23.33</td><td>20.00</td><td>46.46</td><td>20.00</td></tr>
  <tr><td>Vietnamese</td><td>20.00</td><td>26.67</td><td>62.63</td><td>40.00</td><td>25.93</td></tr>
  <tr><td>Welsh</td><td>20.00</td><td>16.67</td><td>42.42</td><td>13.33</td></tr>
  <tr><th>English</th><th>20.00</th><th>26.67</th><th>71.72</th><th>40.00</th><th>22.22</th><th>76.67</th></tr>
  <tr><td>Average</td><td>22.48</td><td>22.24</td><td>59.45</td><td>24.42</td><td>21.55</td><td>35.21</td></tr>
  <tr><td>Standard Deviation</td><td>7.94</td><td>6.85</td><td>10.52</td><td>8.32</td><td>6.44</td><td>19.01</td></tr>
  <tr><td>Fleiss' Kappa</td><td>0.33</td><td>0.37</td><td>0.44</td><td>0.32</td><td>0.19</td></tr>
</table>

Table 21: Evaluation results of Qwen2.5-Math-1.5B-Instruct with Budget Forcing (\( BF = 2048, 4096, 8192 \)).
<table>
  <tr>
    <th>Language</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>M-IMO</th>
    <th>M-MO</th>
  </tr>
  <tr><td>Afrikaans</td><td>72.73</td><td>13.33</td><td>27.78</td><td></td></tr>
  <tr><td>Albanian</td><td>60.61</td><td>16.67</td><td>20</td><td></td></tr>
  <tr><td>Arabic</td><td>76.77</td><td>13.33</td><td>14.81</td><td></td></tr>
  <tr><td>Bengali</td><td>72.73</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Bulgarian</td><td>72.73</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Catalan</td><td>73.74</td><td>20.00</td><td></td><td></td></tr>
  <tr><td>Chinese (Simplified)</td><td>77.78</td><td>20.00</td><td>7.41</td><td>56.67</td></tr>
  <tr><td>Chinese (Traditional)</td><td>73.74</td><td>23.33</td><td>11.11</td><td></td></tr>
  <tr><td>Croatian</td><td>73.74</td><td>30.00</td><td>22.22</td><td></td></tr>
  <tr><td>Czech</td><td>75.76</td><td>20.00</td><td>11.11</td><td>16.67</td></tr>
  <tr><td>Danish</td><td>72.73</td><td>23.33</td><td>18.52</td><td></td></tr>
  <tr><td>Dutch</td><td>77.78</td><td>16.67</td><td>18.52</td><td>23.33</td></tr>
  <tr><td>Estonian</td><td>57.58</td><td>13.33</td><td>20</td><td></td></tr>
  <tr><td>Finnish</td><td>70.71</td><td>20.00</td><td>16</td><td></td></tr>
  <tr><td>French</td><td>77.78</td><td>20.00</td><td>25.93</td><td>48.39</td></tr>
  <tr><td>German</td><td>76.77</td><td>23.33</td><td>25.93</td><td>26.67</td></tr>
  <tr><td>Greek</td><td>64.65</td><td>13.33</td><td>10.53</td><td></td></tr>
  <tr><td>Gujarati</td><td>55.56</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Hebrew</td><td>71.72</td><td>20.00</td><td>7.41</td><td></td></tr>
  <tr><td>Hindi</td><td>70.71</td><td>30.00</td><td></td><td></td></tr>
  <tr><td>Hungarian</td><td>71.72</td><td>26.67</td><td>20</td><td></td></tr>
  <tr><td>Indonesian</td><td>69.70</td><td>20.00</td><td>19.05</td><td></td></tr>
  <tr><td>Italian</td><td>78.79</td><td>23.33</td><td>12</td><td></td></tr>
  <tr><td>Japanese</td><td>76.77</td><td>23.33</td><td>16</td><td>3.57</td></tr>
  <tr><td>Kannada</td><td>57.58</td><td>20.00</td><td></td><td>40</td></tr>
  <tr><td>Korean</td><td>77.78</td><td>20.00</td><td>14.81</td><td></td></tr>
  <tr><td>Latvian</td><td>59.60</td><td>13.33</td><td>20</td><td></td></tr>
  <tr><td>Lithuanian</td><td>61.62</td><td>16.67</td><td>25.93</td><td></td></tr>
  <tr><td>Macedonian</td><td>77.78</td><td>16.67</td><td>22.22</td><td></td></tr>
  <tr><td>Malayalam</td><td>56.57</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Marathi</td><td>63.64</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Nepali</td><td>67.68</td><td>20.00</td><td></td><td></td></tr>
  <tr><td>Norwegian</td><td>73.74</td><td>23.33</td><td>22.22</td><td></td></tr>
  <tr><td>Persian</td><td>74.75</td><td>30.00</td><td></td><td></td></tr>
  <tr><td>Polish</td><td>71.72</td><td>16.67</td><td>22.22</td><td>26.67</td></tr>
  <tr><td>Portuguese</td><td>78.79</td><td>26.67</td><td>20</td><td></td></tr>
  <tr><td>Punjabi</td><td>58.59</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Romanian</td><td>76.77</td><td>23.33</td><td>14.81</td><td></td></tr>
  <tr><td>Russian</td><td>77.78</td><td>20.00</td><td>20</td><td>43.33</td></tr>
  <tr><td>Slovak</td><td>74.75</td><td>23.33</td><td>18.52</td><td>23.33</td></tr>
  <tr><td>Slovenian</td><td>71.72</td><td>23.33</td><td>14.81</td><td></td></tr>
  <tr><td>Somali</td><td>38.38</td><td>6.67</td><td></td><td></td></tr>
  <tr><td>Spanish</td><td>75.76</td><td>30.00</td><td>14.81</td><td></td></tr>
  <tr><td>Swahili</td><td>46.46</td><td>13.33</td><td></td><td></td></tr>
  <tr><td>Swedish</td><td>76.77</td><td>16.67</td><td>24</td><td></td></tr>
  <tr><td>Tagalog</td><td>60.61</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Tamil</td><td>54.55</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Telugu</td><td>60.61</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Thai</td><td>73.74</td><td>20.00</td><td>14.81</td><td></td></tr>
  <tr><td>Turkish</td><td>70.71</td><td>20.00</td><td>7.41</td><td></td></tr>
  <tr><td>Ukrainian</td><td>76.77</td><td>23.33</td><td>14.81</td><td></td></tr>
  <tr><td>Urdu</td><td>63.64</td><td>50.00</td><td></td><td></td></tr>
  <tr><td>Vietnamese</td><td>76.77</td><td>26.67</td><td>14.81</td><td></td></tr>
  <tr><td>Welsh</td><td>50.51</td><td>20.00</td><td></td><td></td></tr>
  <tr><th>English</th><th>83.84</th><th>20.00</th><th>22.22</th><th>46.67</th></tr>
  <tr><th>Average</th><th>69.33</th><th>20.12</th><th>17.64</th><th>32.30</th></tr>
  <tr><th>Standard Deviation</th><th>9.42</th><th>6.57</th><th>5.38</th><th>15.92</th></tr>
  <tr><th>Fleiss Kappa</th><th>0.61</th><th>0.51</th><th>0.38</th><th>15.81</th></tr>
</table>

Table 22: Evaluation results of Qwen2.5-Math-7B-Instruct with greedy decoding on MCLM.
<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="2">ORM (K=2)</th>
    <th colspan="2">ORM (K=4)</th>
    <th colspan="2">ORM (K=8)</th>
  </tr>
  <tr>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
  </tr>
  <tr><td>Afrikaans</td><td>74.75</td><td>16.67</td><td>73.74</td><td>26.67</td><td>76.77</td><td>33.33</td></tr>
  <tr><td>Albanian</td><td>68.69</td><td>20.00</td><td>65.66</td><td>26.67</td><td>68.69</td><td>26.67</td></tr>
  <tr><td>Arabic</td><td>76.77</td><td>13.33</td><td>82.83</td><td>23.33</td><td>83.84</td><td>20.00</td></tr>
  <tr><td>Bengali</td><td>69.70</td><td>16.67</td><td>75.76</td><td>16.67</td><td>74.75</td><td>16.67</td></tr>
  <tr><td>Bulgarian</td><td>73.74</td><td>16.67</td><td>77.78</td><td>20.00</td><td>79.80</td><td>16.67</td></tr>
  <tr><td>Catalan</td><td>75.76</td><td>26.67</td><td>77.78</td><td>20.00</td><td>76.77</td><td>30.00</td></tr>
  <tr><td>Chinese (Simplified)</td><td>77.78</td><td>20.00</td><td>81.82</td><td>26.67</td><td>82.83</td><td>26.67</td></tr>
  <tr><td>Chinese (Traditional)</td><td>77.78</td><td>23.33</td><td>81.82</td><td>23.33</td><td>81.82</td><td>23.33</td></tr>
  <tr><td>Croatian</td><td>75.76</td><td>30.00</td><td>78.79</td><td>33.33</td><td>78.79</td><td>33.33</td></tr>
  <tr><td>Czech</td><td>75.76</td><td>20.00</td><td>81.82</td><td>23.33</td><td>81.82</td><td>23.33</td></tr>
  <tr><td>Danish</td><td>73.74</td><td>26.67</td><td>72.73</td><td>43.33</td><td>74.75</td><td>43.33</td></tr>
  <tr><td>Dutch</td><td>76.77</td><td>20.00</td><td>78.79</td><td>26.67</td><td>81.82</td><td>40.00</td></tr>
  <tr><td>Estonian</td><td>62.63</td><td>16.67</td><td>64.65</td><td>23.33</td><td>65.66</td><td>30.00</td></tr>
  <tr><td>Finnish</td><td>73.74</td><td>23.33</td><td>77.78</td><td>33.33</td><td>75.76</td><td>33.33</td></tr>
  <tr><td>French</td><td>81.82</td><td>23.33</td><td>81.82</td><td>20.00</td><td>81.82</td><td>26.67</td></tr>
  <tr><td>German</td><td>78.79</td><td>33.33</td><td>81.82</td><td>40.00</td><td>83.84</td><td>40.00</td></tr>
  <tr><td>Greek</td><td>65.66</td><td>20.00</td><td>67.68</td><td>23.33</td><td>70.71</td><td>16.67</td></tr>
  <tr><td>Gujarati</td><td>58.59</td><td>13.33</td><td>59.60</td><td>20.00</td><td>64.65</td><td>16.67</td></tr>
  <tr><td>Hebrew</td><td>73.74</td><td>13.33</td><td>75.76</td><td>20.00</td><td>76.77</td><td>30.00</td></tr>
  <tr><td>Hindi</td><td>70.71</td><td>26.67</td><td>75.76</td><td>26.67</td><td>75.76</td><td>36.67</td></tr>
  <tr><td>Hungarian</td><td>73.74</td><td>26.67</td><td>76.77</td><td>20.00</td><td>76.77</td><td>23.33</td></tr>
  <tr><td>Indonesian</td><td>75.76</td><td>30.00</td><td>76.77</td><td>33.33</td><td>77.78</td><td>43.33</td></tr>
  <tr><td>Italian</td><td>79.80</td><td>26.67</td><td>79.80</td><td>26.67</td><td>82.83</td><td>33.33</td></tr>
  <tr><td>Japanese</td><td>78.79</td><td>23.33</td><td>79.80</td><td>30.00</td><td>80.81</td><td>23.33</td></tr>
  <tr><td>Kannada</td><td>55.56</td><td>13.33</td><td>57.58</td><td>13.33</td><td>59.60</td><td>20.00</td></tr>
  <tr><td>Korean</td><td>79.80</td><td>16.67</td><td>76.77</td><td>23.33</td><td>77.78</td><td>26.67</td></tr>
  <tr><td>Latvian</td><td>61.62</td><td>16.67</td><td>65.66</td><td>10.00</td><td>66.67</td><td>10.00</td></tr>
  <tr><td>Lithuanian</td><td>63.64</td><td>20.00</td><td>68.69</td><td>30.00</td><td>69.70</td><td>20.00</td></tr>
  <tr><td>Macedonian</td><td>76.77</td><td>16.67</td><td>80.81</td><td>20.00</td><td>79.80</td><td>23.33</td></tr>
  <tr><td>Malayalam</td><td>59.60</td><td>10.00</td><td>62.63</td><td>16.67</td><td>68.69</td><td>23.33</td></tr>
  <tr><td>Marathi</td><td>65.66</td><td>26.67</td><td>68.69</td><td>20.00</td><td>69.70</td><td>16.67</td></tr>
  <tr><td>Nepali</td><td>64.65</td><td>13.33</td><td>69.70</td><td>16.67</td><td>68.69</td><td>16.67</td></tr>
  <tr><td>Norwegian</td><td>72.73</td><td>26.67</td><td>74.75</td><td>30.00</td><td>76.77</td><td>33.33</td></tr>
  <tr><td>Persian</td><td>76.77</td><td>23.33</td><td>75.76</td><td>23.33</td><td>76.77</td><td>16.67</td></tr>
  <tr><td>Polish</td><td>77.78</td><td>10.00</td><td>78.79</td><td>10.00</td><td>78.79</td><td>16.67</td></tr>
  <tr><td>Portuguese</td><td>81.82</td><td>26.67</td><td>80.81</td><td>36.67</td><td>83.84</td><td>40.00</td></tr>
  <tr><td>Punjabi</td><td>58.59</td><td>20.00</td><td>59.60</td><td>16.67</td><td>62.63</td><td>26.67</td></tr>
  <tr><td>Romanian</td><td>79.80</td><td>23.33</td><td>81.82</td><td>26.67</td><td>79.80</td><td>30.00</td></tr>
  <tr><td>Russian</td><td>78.79</td><td>26.67</td><td>82.83</td><td>20.00</td><td>86.87</td><td>26.67</td></tr>
  <tr><td>Slovak</td><td>77.78</td><td>30.00</td><td>79.80</td><td>33.33</td><td>81.82</td><td>30.00</td></tr>
  <tr><td>Slovenian</td><td>73.74</td><td>13.33</td><td>78.79</td><td>20.00</td><td>78.79</td><td>23.33</td></tr>
  <tr><td>Somali</td><td>38.38</td><td>6.67</td><td>42.42</td><td>13.33</td><td>44.44</td><td>20.00</td></tr>
  <tr><td>Spanish</td><td>75.76</td><td>26.67</td><td>78.79</td><td>26.67</td><td>81.82</td><td>30.00</td></tr>
  <tr><td>Swahili</td><td>48.48</td><td>13.33</td><td>49.49</td><td>20.00</td><td>51.52</td><td>23.33</td></tr>
  <tr><td>Swedish</td><td>77.78</td><td>30.00</td><td>76.77</td><td>30.00</td><td>77.78</td><td>30.00</td></tr>
  <tr><td>Tagalog</td><td>58.59</td><td>13.33</td><td>65.66</td><td>10.00</td><td>66.67</td><td>16.67</td></tr>
  <tr><td>Tamil</td><td>59.60</td><td>16.67</td><td>65.66</td><td>10.00</td><td>62.63</td><td>10.00</td></tr>
  <tr><td>Telugu</td><td>61.62</td><td>20.00</td><td>63.64</td><td>23.33</td><td>62.63</td><td>16.67</td></tr>
  <tr><td>Thai</td><td>76.77</td><td>16.67</td><td>79.80</td><td>23.33</td><td>77.78</td><td>30.00</td></tr>
  <tr><td>Turkish</td><td>76.77</td><td>26.67</td><td>79.80</td><td>26.67</td><td>79.80</td><td>26.67</td></tr>
  <tr><td>Ukrainian</td><td>77.78</td><td>23.33</td><td>78.79</td><td>23.33</td><td>79.80</td><td>26.67</td></tr>
  <tr><td>Urdu</td><td>66.67</td><td>33.33</td><td>67.68</td><td>30.00</td><td>72.73</td><td>30.00</td></tr>
  <tr><td>Vietnamese</td><td>73.74</td><td>33.33</td><td>76.77</td><td>33.33</td><td>80.81</td><td>36.67</td></tr>
  <tr><td>Welsh</td><td>51.52</td><td>20.00</td><td>53.54</td><td>16.67</td><td>56.57</td><td>6.67</td></tr>
  <tr><th>English</th><th>84.85</th><th>26.67</th><th>84.85</th><th>30.00</th><th>86.87</th><th>26.67</th></tr>
  <tr><td>Average</td><td>70.98</td><td>21.21</td><td>73.35</td><td>23.82</td><td>74.62</td><td>25.76</td></tr>
  <tr><td>Standard Deviation</td><td>9.46</td><td>6.52</td><td>9.20</td><td>7.41</td><td>8.86</td><td>8.37</td></tr>
  <tr><td>Fleiss’ Kappa</td><td>0.62</td><td>0.55</td><td>0.65</td><td>0.57</td><td>0.67</td><td>0.57</td></tr>
</table>

Table 23: Evaluation results of Qwen2.5-Math-7B-Instruct with Best-of-N (\( K = 2, 4, 8 \)) using Qwen2.5-Math-RM-72B as ORM on MT-MATH100 and MT-AIME2024.
<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="2">PRM (S=3, c=3)</th>
    <th colspan="2">PRM (S=4, c=5)</th>
    <th colspan="2">PRM (S=5, c=8)</th>
  </tr>
  <tr>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
  </tr>
  <tr><td>Afrikaans</td><td>70.71</td><td>20.00</td><td>70.71</td><td>16.67</td><td>70.71</td><td>20.00</td></tr>
  <tr><td>Albanian</td><td>60.61</td><td>16.67</td><td>62.63</td><td>33.33</td><td>61.62</td><td>26.67</td></tr>
  <tr><td>Arabic</td><td>65.66</td><td>26.67</td><td>78.79</td><td>26.67</td><td>82.83</td><td>30.00</td></tr>
  <tr><td>Bengali</td><td>67.68</td><td>16.67</td><td>70.71</td><td>10.00</td><td>68.69</td><td>23.33</td></tr>
  <tr><td>Bulgarian</td><td>69.70</td><td>20.00</td><td>74.75</td><td>10.00</td><td>75.76</td><td>30.00</td></tr>
  <tr><td>Catalan</td><td>72.73</td><td>16.67</td><td>70.71</td><td>20.00</td><td>71.72</td><td>16.67</td></tr>
  <tr><td>Chinese (Simplified)</td><td>72.73</td><td>16.67</td><td>73.74</td><td>33.33</td><td>78.79</td><td>30.00</td></tr>
  <tr><td>Chinese (Traditional)</td><td>71.72</td><td>16.67</td><td>76.77</td><td>20.00</td><td>77.78</td><td>23.33</td></tr>
  <tr><td>Croatian</td><td>69.70</td><td>20.00</td><td>72.73</td><td>16.67</td><td>70.71</td><td>33.33</td></tr>
  <tr><td>Czech</td><td>69.70</td><td>16.67</td><td>77.78</td><td>10.00</td><td>73.74</td><td>30.00</td></tr>
  <tr><td>Danish</td><td>63.64</td><td>23.33</td><td>69.70</td><td>33.33</td><td>66.67</td><td>30.00</td></tr>
  <tr><td>Dutch</td><td>71.72</td><td>6.67</td><td>72.73</td><td>26.67</td><td>75.76</td><td>26.67</td></tr>
  <tr><td>Estonian</td><td>46.46</td><td>20.00</td><td>51.52</td><td>13.33</td><td>59.60</td><td>20.00</td></tr>
  <tr><td>Finnish</td><td>64.65</td><td>16.67</td><td>66.67</td><td>13.33</td><td>72.73</td><td>33.33</td></tr>
  <tr><td>French</td><td>73.74</td><td>20.00</td><td>72.73</td><td>16.67</td><td>76.77</td><td>26.67</td></tr>
  <tr><td>German</td><td>73.74</td><td>10.00</td><td>68.69</td><td>10.00</td><td>76.77</td><td>26.67</td></tr>
  <tr><td>Greek</td><td>63.64</td><td>16.67</td><td>64.65</td><td>13.33</td><td>67.68</td><td>13.33</td></tr>
  <tr><td>Gujarati</td><td>56.57</td><td>13.33</td><td>56.57</td><td>26.67</td><td>55.56</td><td>13.33</td></tr>
  <tr><td>Hebrew</td><td>66.67</td><td>10.00</td><td>68.69</td><td>20.00</td><td>75.76</td><td>26.67</td></tr>
  <tr><td>Hindi</td><td>58.59</td><td>16.67</td><td>63.64</td><td>20.00</td><td>72.73</td><td>13.33</td></tr>
  <tr><td>Hungarian</td><td>68.69</td><td>16.67</td><td>69.70</td><td>30.00</td><td>72.73</td><td>20.00</td></tr>
  <tr><td>Indonesian</td><td>69.70</td><td>26.67</td><td>68.69</td><td>20.00</td><td>72.73</td><td>10.00</td></tr>
  <tr><td>Italian</td><td>71.72</td><td>16.67</td><td>77.78</td><td>30.00</td><td>73.74</td><td>23.33</td></tr>
  <tr><td>Japanese</td><td>71.72</td><td>23.33</td><td>75.76</td><td>16.67</td><td>76.77</td><td>13.33</td></tr>
  <tr><td>Kannada</td><td>46.46</td><td>16.67</td><td>53.54</td><td>10.00</td><td>54.55</td><td>16.67</td></tr>
  <tr><td>Korean</td><td>69.70</td><td>16.67</td><td>72.73</td><td>13.33</td><td>74.75</td><td>16.67</td></tr>
  <tr><td>Latvian</td><td>59.60</td><td>10.00</td><td>63.64</td><td>13.33</td><td>63.64</td><td>16.67</td></tr>
  <tr><td>Lithuanian</td><td>55.56</td><td>20.00</td><td>62.63</td><td>13.33</td><td>65.66</td><td>16.67</td></tr>
  <tr><td>Macedonian</td><td>69.70</td><td>16.67</td><td>75.76</td><td>16.67</td><td>75.76</td><td>23.33</td></tr>
  <tr><td>Malayalam</td><td>49.49</td><td>20.00</td><td>57.58</td><td>23.33</td><td>52.53</td><td>20.00</td></tr>
  <tr><td>Marathi</td><td>56.57</td><td>20.00</td><td>55.56</td><td>23.33</td><td>57.58</td><td>23.33</td></tr>
  <tr><td>Nepali</td><td>51.52</td><td>16.67</td><td>61.62</td><td>20.00</td><td>52.53</td><td>23.33</td></tr>
  <tr><td>Norwegian</td><td>69.70</td><td>20.00</td><td>67.68</td><td>20.00</td><td>69.70</td><td>26.67</td></tr>
  <tr><td>Persian</td><td>71.72</td><td>26.67</td><td>71.72</td><td>16.67</td><td>72.73</td><td>23.33</td></tr>
  <tr><td>Polish</td><td>61.62</td><td>13.33</td><td>67.68</td><td>13.33</td><td>76.77</td><td>10.00</td></tr>
  <tr><td>Portuguese</td><td>72.73</td><td>10.00</td><td>71.72</td><td>26.67</td><td>79.80</td><td>26.67</td></tr>
  <tr><td>Punjabi</td><td>46.46</td><td>13.33</td><td>45.45</td><td>10.00</td><td>52.53</td><td>20.00</td></tr>
  <tr><td>Romanian</td><td>66.67</td><td>13.33</td><td>70.71</td><td>33.33</td><td>77.78</td><td>30.00</td></tr>
  <tr><td>Russian</td><td>75.76</td><td>16.67</td><td>76.77</td><td>16.67</td><td>76.77</td><td>33.33</td></tr>
  <tr><td>Slovak</td><td>70.71</td><td>23.33</td><td>75.76</td><td>26.67</td><td>70.71</td><td>13.33</td></tr>
  <tr><td>Slovenian</td><td>70.71</td><td>23.33</td><td>72.73</td><td>30.00</td><td>74.75</td><td>20.00</td></tr>
  <tr><td>Somali</td><td>40.40</td><td>3.33</td><td>42.42</td><td>10.00</td><td>42.42</td><td>6.67</td></tr>
  <tr><td>Spanish</td><td>71.72</td><td>13.33</td><td>77.78</td><td>20.00</td><td>80.81</td><td>23.33</td></tr>
  <tr><td>Swahili</td><td>48.48</td><td>6.67</td><td>42.42</td><td>10.00</td><td>44.44</td><td>16.67</td></tr>
  <tr><td>Swedish</td><td>70.71</td><td>16.67</td><td>76.77</td><td>36.67</td><td>71.72</td><td>26.67</td></tr>
  <tr><td>Tagalog</td><td>55.56</td><td>23.33</td><td>59.60</td><td>16.67</td><td>58.59</td><td>13.33</td></tr>
  <tr><td>Tamil</td><td>50.51</td><td>10.00</td><td>55.56</td><td>3.33</td><td>57.58</td><td>20.00</td></tr>
  <tr><td>Telugu</td><td>53.54</td><td>13.33</td><td>58.59</td><td>20.00</td><td>54.55</td><td>20.00</td></tr>
  <tr><td>Thai</td><td>67.68</td><td>10.00</td><td>71.72</td><td>16.67</td><td>71.72</td><td>26.67</td></tr>
  <tr><td>Turkish</td><td>63.64</td><td>20.00</td><td>71.72</td><td>20.00</td><td>64.65</td><td>16.67</td></tr>
  <tr><td>Ukrainian</td><td>75.76</td><td>20.00</td><td>77.78</td><td>26.67</td><td>79.80</td><td>20.00</td></tr>
  <tr><td>Urdu</td><td>57.58</td><td>26.67</td><td>62.63</td><td>20.00</td><td>66.67</td><td>23.33</td></tr>
  <tr><td>Vietnamese</td><td>72.73</td><td>23.33</td><td>73.74</td><td>13.33</td><td>73.74</td><td>33.33</td></tr>
  <tr><td>Welsh</td><td>50.51</td><td>20.00</td><td>43.43</td><td>13.33</td><td>45.45</td><td>20.00</td></tr>
  <tr><th>English</th><th>73.74</th><th>23.33</th><th>75.76</th><th>20.00</th><th>75.76</th><th>23.33</th></tr>
  <tr><td>Average</td><td>64.17</td><td>17.27</td><td>67.09</td><td>19.27</td><td>68.45</td><td>22.00</td></tr>
  <tr><td>Standard Deviation</td><td>9.25</td><td>5.33</td><td>9.65</td><td>7.61</td><td>10.02</td><td>6.56</td></tr>
  <tr><td>Fleiss’ Kappa</td><td>0.56</td><td>0.56</td><td>0.54</td><td>0.57</td><td>0.56</td><td>0.59</td></tr>
</table>

Table 24: Evaluation results of Qwen2.5-Math-7B-Instruct using Qwen2.5-Math-PRM-72B as PRM on MT-MATH100 and MT-AIME2024.
<table>
  <tr>
    <th>Language</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>M-IMO</th>
    <th>M-MO</th>
  </tr>
  <tr><td>Afrikaans</td><td>73.74</td><td>23.33</td><td>9.09</td><td></td></tr>
  <tr><td>Albanian</td><td>66.67</td><td>20.00</td><td>15.38</td><td></td></tr>
  <tr><td>Arabic</td><td>71.72</td><td>16.67</td><td>3.70</td><td></td></tr>
  <tr><td>Bengali</td><td>64.65</td><td>3.33</td><td></td><td></td></tr>
  <tr><td>Bulgarian</td><td>72.73</td><td>20.00</td><td>18.52</td><td></td></tr>
  <tr><td>Catalan</td><td>70.71</td><td>26.67</td><td></td><td></td></tr>
  <tr><td>Chinese (Simplified)</td><td>70.71</td><td>23.33</td><td>14.81</td><td>26.67</td></tr>
  <tr><td>Chinese (Traditional)</td><td>69.70</td><td>23.33</td><td>11.11</td><td></td></tr>
  <tr><td>Croatian</td><td>72.73</td><td>16.67</td><td>18.52</td><td></td></tr>
  <tr><td>Czech</td><td>71.72</td><td>33.33</td><td>11.11</td><td>36.67</td></tr>
  <tr><td>Danish</td><td>71.72</td><td>23.33</td><td>22.22</td><td></td></tr>
  <tr><td>Dutch</td><td>69.70</td><td>20.00</td><td>3.70</td><td>3.33</td></tr>
  <tr><td>Estonian</td><td>76.77</td><td>16.67</td><td>15.38</td><td></td></tr>
  <tr><td>Finnish</td><td>72.73</td><td>6.67</td><td>15.38</td><td></td></tr>
  <tr><td>French</td><td>70.71</td><td>23.33</td><td>14.81</td><td>48.39</td></tr>
  <tr><td>German</td><td>73.74</td><td>20.00</td><td>18.52</td><td>26.67</td></tr>
  <tr><td>Greek</td><td>71.72</td><td>10.00</td><td>13.04</td><td></td></tr>
  <tr><td>Gujarati</td><td>67.68</td><td>13.33</td><td></td><td></td></tr>
  <tr><td>Hebrew</td><td>71.72</td><td>10.00</td><td>7.41</td><td></td></tr>
  <tr><td>Hindi</td><td>70.71</td><td>6.67</td><td></td><td></td></tr>
  <tr><td>Hungarian</td><td>73.74</td><td>26.67</td><td>11.54</td><td></td></tr>
  <tr><td>Indonesian</td><td>68.69</td><td>13.33</td><td>16.67</td><td></td></tr>
  <tr><td>Italian</td><td>72.73</td><td>23.33</td><td>11.54</td><td></td></tr>
  <tr><td>Japanese</td><td>70.71</td><td>30.00</td><td>7.69</td><td>7.14</td></tr>
  <tr><td>Kannada</td><td>61.62</td><td>23.33</td><td></td><td></td></tr>
  <tr><td>Korean</td><td>72.73</td><td>26.67</td><td>22.22</td><td>36.67</td></tr>
  <tr><td>Latvian</td><td>69.70</td><td>20.00</td><td>7.69</td><td></td></tr>
  <tr><td>Lithuanian</td><td>68.69</td><td>16.67</td><td>7.41</td><td></td></tr>
  <tr><td>Macedonian</td><td>71.72</td><td>20.00</td><td>22.22</td><td></td></tr>
  <tr><td>Malayalam</td><td>62.63</td><td>23.33</td><td></td><td></td></tr>
  <tr><td>Marathi</td><td>63.64</td><td>20.00</td><td></td><td></td></tr>
  <tr><td>Nepali</td><td>67.68</td><td>10.00</td><td></td><td></td></tr>
  <tr><td>Norwegian</td><td>75.76</td><td>30.00</td><td>11.11</td><td></td></tr>
  <tr><td>Persian</td><td>66.67</td><td>26.67</td><td></td><td></td></tr>
  <tr><td>Polish</td><td>72.73</td><td>13.33</td><td>22.22</td><td>26.67</td></tr>
  <tr><td>Portuguese</td><td>70.71</td><td>26.67</td><td>7.69</td><td></td></tr>
  <tr><td>Punjabi</td><td>69.70</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Romanian</td><td>73.74</td><td>26.67</td><td>11.11</td><td></td></tr>
  <tr><td>Russian</td><td>73.74</td><td>23.33</td><td>15.38</td><td>50.00</td></tr>
  <tr><td>Slovak</td><td>72.73</td><td>20.00</td><td>18.52</td><td></td></tr>
  <tr><td>Slovenian</td><td>72.73</td><td>16.67</td><td>7.41</td><td></td></tr>
  <tr><td>Somali</td><td>57.58</td><td>20.00</td><td></td><td></td></tr>
  <tr><td>Spanish</td><td>71.72</td><td>26.67</td><td>14.81</td><td></td></tr>
  <tr><td>Swahili</td><td>65.66</td><td>23.33</td><td></td><td></td></tr>
  <tr><td>Swedish</td><td>72.73</td><td>23.33</td><td>23.08</td><td></td></tr>
  <tr><td>Tagalog</td><td>71.72</td><td>20.00</td><td></td><td></td></tr>
  <tr><td>Tamil</td><td>67.68</td><td>20.00</td><td></td><td></td></tr>
  <tr><td>Telugu</td><td>66.67</td><td>16.67</td><td></td><td></td></tr>
  <tr><td>Thai</td><td>70.71</td><td>26.67</td><td>7.41</td><td></td></tr>
  <tr><td>Turkish</td><td>71.72</td><td>10.00</td><td>11.11</td><td></td></tr>
  <tr><td>Ukrainian</td><td>73.74</td><td>23.33</td><td>14.81</td><td></td></tr>
  <tr><td>Urdu</td><td>68.69</td><td>23.33</td><td></td><td></td></tr>
  <tr><td>Vietnamese</td><td>71.72</td><td>6.67</td><td>14.81</td><td></td></tr>
  <tr><td>Welsh</td><td>65.66</td><td>26.67</td><td></td><td></td></tr>
  <tr><th>English</th><th>75.76</th><th>33.33</th><th>7.41</th><th>50.00</th></tr>
  <tr><td>Average</td><td>70.30</td><td>20.18</td><td>13.33</td><td>30.81</td></tr>
  <tr><td>Standard Deviation</td><td>3.68</td><td>6.83</td><td>5.36</td><td>15.80</td></tr>
  <tr><td>Fleiss’ Kappa</td><td>0.71</td><td>0.33</td><td>0.25</td><td></td></tr>
</table>

Table 25: Evaluation results of GPT-4O-MINI with greedy decoding on MCLM.
<table>
  <tr>
    <th>Language</th>
    <th>MT-MATH100</th>
    <th>MT-AIME2024</th>
    <th>M-IMO</th>
    <th>M-MO</th>
  </tr>
  <tr><td>Afrikaans</td><td>85.86</td><td>46.67</td><td>33.33</td><td></td></tr>
  <tr><td>Albanian</td><td>86.87</td><td>53.33</td><td>28.00</td><td></td></tr>
  <tr><td>Arabic</td><td>86.87</td><td>43.33</td><td>22.22</td><td></td></tr>
  <tr><td>Bengali</td><td>86.87</td><td>43.33</td><td></td><td></td></tr>
  <tr><td>Bulgarian</td><td>87.88</td><td>46.67</td><td>40.74</td><td></td></tr>
  <tr><td>Catalan</td><td>87.88</td><td>53.33</td><td></td><td></td></tr>
  <tr><td>Chinese (Simplified)</td><td>85.86</td><td>50</td><td>25.93</td><td>66.67</td></tr>
  <tr><td>Chinese (Traditional)</td><td>84.85</td><td>40</td><td>29.63</td><td></td></tr>
  <tr><td>Croatian</td><td>84.85</td><td>46.67</td><td>33.33</td><td></td></tr>
  <tr><td>Czech</td><td>84.85</td><td>36.67</td><td>29.63</td><td>53.33</td></tr>
  <tr><td>Danish</td><td>85.86</td><td>40</td><td>40.74</td><td></td></tr>
  <tr><td>Dutch</td><td>86.87</td><td>50</td><td>33.33</td><td>40.00</td></tr>
  <tr><td>Estonian</td><td>83.84</td><td>50</td><td>28.00</td><td></td></tr>
  <tr><td>Finnish</td><td>84.85</td><td>40</td><td>28.00</td><td></td></tr>
  <tr><td>French</td><td>86.87</td><td>43.33</td><td>29.63</td><td>67.74</td></tr>
  <tr><td>German</td><td>86.87</td><td>43.33</td><td>33.33</td><td>43.33</td></tr>
  <tr><td>Greek</td><td>87.88</td><td>56.67</td><td>21.05</td><td></td></tr>
  <tr><td>Gujarati</td><td>83.84</td><td>46.67</td><td></td><td></td></tr>
  <tr><td>Hebrew</td><td>81.82</td><td>40</td><td>7.41</td><td></td></tr>
  <tr><td>Hindi</td><td>83.84</td><td>43.33</td><td></td><td></td></tr>
  <tr><td>Hungarian</td><td>86.87</td><td>53.33</td><td>28.00</td><td></td></tr>
  <tr><td>Indonesian</td><td>84.85</td><td>43.33</td><td>33.33</td><td></td></tr>
  <tr><td>Italian</td><td>82.83</td><td>50</td><td>36.00</td><td></td></tr>
  <tr><td>Japanese</td><td>86.87</td><td>50</td><td>16.00</td><td>17.86</td></tr>
  <tr><td>Kannada</td><td>86.87</td><td>43.33</td><td></td><td></td></tr>
  <tr><td>Korean</td><td>77.78</td><td>46.67</td><td>25.93</td><td>60.00</td></tr>
  <tr><td>Latvian</td><td>87.88</td><td>46.67</td><td>32.00</td><td></td></tr>
  <tr><td>Lithuanian</td><td>85.86</td><td>46.67</td><td>33.33</td><td></td></tr>
  <tr><td>Macedonian</td><td>83.84</td><td>43.33</td><td>33.33</td><td></td></tr>
  <tr><td>Malayalam</td><td>85.86</td><td>46.67</td><td></td><td></td></tr>
  <tr><td>Marathi</td><td>83.84</td><td>36.67</td><td></td><td></td></tr>
  <tr><td>Nepali</td><td>79.8</td><td>46.67</td><td></td><td></td></tr>
  <tr><td>Norwegian</td><td>82.83</td><td>53.33</td><td>22.22</td><td></td></tr>
  <tr><td>Persian</td><td>87.88</td><td>53.33</td><td></td><td></td></tr>
  <tr><td>Polish</td><td>81.82</td><td>43.33</td><td>37.04</td><td>40.00</td></tr>
  <tr><td>Portuguese</td><td>82.83</td><td>36.67</td><td>36.00</td><td></td></tr>
  <tr><td>Punjabi</td><td>87.88</td><td>43.33</td><td></td><td></td></tr>
  <tr><td>Romanian</td><td>81.82</td><td>40</td><td>40.74</td><td></td></tr>
  <tr><td>Russian</td><td>85.86</td><td>56.67</td><td>20.00</td><td>50.00</td></tr>
  <tr><td>Slovak</td><td>87.88</td><td>46.67</td><td>33.33</td><td>46.67</td></tr>
  <tr><td>Slovenian</td><td>85.86</td><td>46.67</td><td>29.63</td><td></td></tr>
  <tr><td>Somali</td><td>87.88</td><td>50</td><td></td><td></td></tr>
  <tr><td>Spanish</td><td>72.73</td><td>50</td><td>29.63</td><td></td></tr>
  <tr><td>Swahili</td><td>86.87</td><td>43.33</td><td></td><td></td></tr>
  <tr><td>Swedish</td><td>79.8</td><td>43.33</td><td>28.00</td><td></td></tr>
  <tr><td>Tagalog</td><td>85.86</td><td>46.67</td><td></td><td></td></tr>
  <tr><td>Tamil</td><td>84.85</td><td>43.33</td><td></td><td></td></tr>
  <tr><td>Telugu</td><td>82.83</td><td>33.33</td><td></td><td></td></tr>
  <tr><td>Thai</td><td>84.85</td><td>40</td><td>22.22</td><td></td></tr>
  <tr><td>Turkish</td><td>84.85</td><td>40</td><td>33.33</td><td></td></tr>
  <tr><td>Ukrainian</td><td>84.85</td><td>50</td><td>29.63</td><td></td></tr>
  <tr><td>Urdu</td><td>84.85</td><td>36.67</td><td></td><td></td></tr>
  <tr><td>Vietnamese</td><td>85.86</td><td>46.67</td><td>37.04</td><td></td></tr>
  <tr><td>Welsh</td><td>85.86</td><td>46.67</td><td></td><td></td></tr>
  <tr><th>English</th><th>83.84</th><th>36.67</th><th>29.63</th><th>80.00</th></tr>
  <tr><th>Average</th><th>84.89</th><th>45.33</th><th>29.75</th><th>51.42</th></tr>
  <tr><th>Standard Deviation</th><th>2.80</th><th>5.35</th><th>6.86</th><th>16.94</th></tr>
  <tr><th>Fleiss’ Kappa</th><th>0.88</th><th>0.73</th><th>0.44</th><th></th></tr>
</table>

Table 26: Evaluation results of O3-MINI with greedy decoding on MCLM.
Figure 10: Heatmap representation of IMO problems from 2006 to 2024. Each row corresponds to a competition year, and each column represents a problem (Q1–Q6). Green cells indicate questions that have been included in the M-IMO subset, while gray cells represent problems that were not selected.
Figure 11: Solve rates (%) of different multilingual math datasets evaluated. For the OLMo2 series, we use the base models, while for the Qwen2.5 series, the instruct-tuned variants are used. EULER-INSTRUCT presents a significantly lower solve rate, indicating its greater difficulty.

Figure 12: Model Results from Table 9. Left shows accuracy on MT-MATH500 (entire translated subset for language group (B)), and right shows average performance of MT-AIME2024.
You will be given an English question in the following format.

[Question]
<question>
{..question...}
</question>

Your job is to return a translated version of the question.
* Translate to <language>.
* The translation must be fluent, easy to read by native speakers.
* Do not solve the prompt translate it.
* You must preserve all details including math notations (latex) and code.
* The math notations and code must not be translated, keep it as is.
* Return you translation in the following format.

[Translation]
<translation>
{..translated question...}
</translation>

---------------------------------------------
The following is the math problem for you task:

[Question]
<question>
<source_question>
</question>

Figure 13: Question Translation Template

You will be given an English solution in the following format.

[Solution]
<solution>
{..solution in English...}
</solution>

Your job is to rewrite the English solution to <language>.
* The solution must preserve the original structure and details.
* You must preserve all details including math notations (latex) and code.
* The math notations and code must not be translated, keep it as is.
* The solution must be natural, easy and polite for a native speaker to read.

[Translation]
<translation>
{..translated solution...}
</translation>

---------------------------------------------
The following is the math problem and solution for your task:

[Solution]
<solution>
<source_solution>
</solution>

Figure 14: Solution Translation Template
You will be given a math problem, the correct answer, and a solution generated by a language model. Your task is to determine whether the solution generated by the model is correct.

[Question]
<question>
{..math question...}
</question>

[Correct Answer]
<answer>
{..correct answer...}
</answer>

[Model Solution]
<solution>
{..model-generated solution...}
</solution>

Instructions:
* Compare the model's solution with the correct answer.
* If the model's solution is correct, output [[TRUE]].
* If the model's solution is incorrect, output [[FALSE]].
* You do not have to judge the solution process; there are numerous possible 'Gold' solutions, and the model solution does not have to be identical with the one provided. As long as the model reaches the correct answer, it is correct.
* Do not provide any explanations -- only return your judgment ONLY.

-------------------------------------------------------------
The following is the math problem and solution for your task:

[Question]
<question>
<math_question>
</question>

[Correct Answer]
<answer>
<correct_answer>
</answer>

[Model Solution]
<solution>
<model_solution>
</solution>

Metareview score: 3.5
Metareview: The authors introduce a multilingual benchmark with competition-level math problems in 55 languages, and use it to evaluate test-time scaling methods -- Outcome Reward Modeling, Process Reward Modeling, and Budget Forcing -- on small to mid-sized language models. They find that while these methods improve performance in English, their effectiveness in non-English languages is limited, inconsistent, and highly variable, especially under constrained compute budgets. They find that smaller models sometimes match or outperform larger ones when inference FLOPs are controlled. The work is released together with benchmark and accompanying resources, supporting possible future research in multilingual math reasoning.
Summary of reasons to publish: - Evaluation of the cross-lingual transferability of test-time scaling, providing a first analysis of this phenomenon in cross-lingual settings
- Valuable math evaluation dataset in multiple languages, which is mostly machine-translated but also includes human-written segments and incorporates a method to ensure consistency in mathematical expressions across languages
- Strong reproducibility and transparency, supported by the availability of an anonymous repository that includes prompts, consistency evaluation scripts, and model responses.
- The evaluation is conducted at a sufficient scale to support robust conclusions; I can not see a clear motivation for requests of some reviewers requiring the scaling of existing evaluations across further models