Are Your LLMs Capable of Stable Reasoning?
Authors: Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, Kai Chen
Accepted at: acl25

Are Your LLMs Capable of Stable Reasoning?

Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao
Wenwei Zhang, Songyang Zhang†, Kai Chen†
Shanghai Artificial Intelligence Laboratory
to.liujn@outlook.com, zhangsongyang@pjlab.org.cn

Abstract

The rapid advancement of large language models (LLMs) has shown remarkable progress in complex reasoning tasks. However, a significant disparity exists between benchmark performances and real-world applications. We attribute this gap primarily to current evaluation protocols and metrics, which inadequately capture the full spectrum of LLM capabilities, especially in complex reasoning tasks where both accuracy and consistency are essential. In this paper, we introduce G-Pass@k, a novel evaluation metric that continuously assesses model performance across multiple sampling attempts, quantifying both the model’s performance potential and its stability. Through extensive experiments on various public and newly constructed benchmarks, we employ G-Pass@k in conjunction with state-of-the-art large language models to provide comprehensive insights into their potential capabilities and operational consistency. Our findings reveal a significant opportunity to enhance the realistic reasoning abilities of LLMs, underscoring the necessity for more robust evaluation metrics.

1 Introduction

Since the emergence of large language models (LLMs), complex reasoning, particularly in mathematical problem-solving, has been regarded as the crown jewel of LLM capabilities (Beeching et al., 2024; Shao et al., 2024). Numerous models have demonstrated remarkable performance on mathematical tasks, from general-purpose models like GPT-4o (OpenAI, 2024a), and the LLaMA series (AI, 2024) to specialized models such as DeepSeek-Math (Shao et al., 2024) and Qwen-Math (Yang et al., 2024c), which excel in complex and high-difficulty mathematical reasoning. More recently, long-chain-of-thought (Long-CoT) models like OpenAI-o1 (OpenAI, 2024b), QwQ (Team, 2024a), and DeepSeek-R1 (DeepSeek-AI et al., 2024) have further advanced the state-of-the-art in mathematical problem-solving.

In real-world applications, LLMs typically employ sampling with predefined decoding parameters (e.g., temperature, top-k, top-p, and repetition penalty) to maintain response diversity. Users often regenerate responses or initiate new sessions until obtaining satisfactory answers to specific questions. However, conventional evaluation metrics for LLMs, such as Greedy Accuracy, Pass@k (Chen et al., 2021), Best-of-N (BoN), and Majority Voting, demonstrate significant limitations in measuring real-world performance, particularly regarding long-term consistency. While these metrics effectively capture either instantaneous accuracy or peak performance across multiple samples, they inadequately reflect output stability. Such instability poses significant challenges for applications requiring reliable and predictable outcomes, highlighting the need for evaluation metrics that effectively balance response diversity with consistent performance and comprehensive analysis of the reasoning stability of current LLMs.

To address these challenges, we introduce G-Pass@k, a novel evaluation metric that simultaneously assesses both reasoning capability and performance consistency. The core principle of G-Pass@k lies in evaluating model performance under varying thresholds of correctness, providing a nuanced understanding of model behavior across different stringency levels. By integrating measures of both stability and potential, G-Pass@k delivers a comprehensive assessment of an LLM’s capabilities, particularly valuable for complex reasoning tasks.

To thoroughly study the reasoning stability of current LLMs using G-Pass@k, we conduct comprehensive experiments to evaluate various models on multiple mathematical reasoning benchmarks, encompassing different categories of large
language models. These include general LLMs, mathematics-specialized models, and Chain-of-Thought (CoT) enhanced reasoning models, which are known for their step-by-step reasoning capabilities in complex mathematical problem-solving. We also introduce LiveMathBench, a challenging bilingual mathematics benchmark assembled from multiple competitions and examinations to mitigate the impact of data leakage.

Through systematic comparison between traditional metrics (Greedy Accuracy, Pass@k) and G-Pass@k, our findings reveal distinctive insights into model performance that conventional evaluation approaches may overlook and demonstrate significant limitations in traditional evaluation metrics regarding output stability assessment. We observe a substantial performance degradation across all models as the G-Pass@k threshold becomes more stringent, a pattern consistent across both established open-source benchmarks (MATH (Lightman et al., 2024) and AIME (AIME2024, 2024; AIME2025, 2025)) and our proposed LiveMathBench. Specifically, the key observations include:

• Instability in LLM Reasoning (Section 3.2, Section 3.5): Both closed-source and open-source models exhibit significant instability when applied to challenging reasoning tasks. Performance drops exceed 50% in many cases, with the worst instances showing declines of up to 90%. This issue is particularly significant in high-difficulty questions, underscoring the necessity for further enhancement of current LLMs’ ability to address complex questions.

• Inconsistency Between Expanded Model Scale and Enhanced Reasoning Ability (Section 3.2): Merely scaling up model size does not necessarily enhance stable reasoning capabilities. The expected improvements in performance and stability are not consistently observed.

• Discrepancy Between Potential and Stability (Section 3.2): There is a noticeable gap between the models’ potential capabilities, as measured by their G-Pass@\( k_{\tau \rightarrow 0} \), and their actual stability, reflected in G-Pass@\( k_{\tau} \). This disparity highlights the need for further research into developing methods that can better harness the theoretical capabilities of these models in practical, stable applications.

We also investigate the robustness of the newly proposed metric G-Pass@k, demonstrating that G-Pass@k consistently evaluates model performance across different hyperparameters, guaranteeing its reliability and applicability (Section 3.3, Section 3.4). Furthermore, we conduct a preliminary investigation into enhancing the reasoning stability of the model and identifying the reasons behind strong stability. Our investigation reveals two key findings: 1) merely relying on SFT does not achieve significant improvement in reasoning stability (Section 4.1), and 2) the incorporation of backtracking and self-reflection mechanisms in o1-like models improves the stability of their reasoning capabilities (Section 4.2).

2 Generalized Metric for LLM Reasoning

2.1 Preliminary: Pass@k

Pass@k was initially proposed to evaluate the functional correctness of code generated by models (Kulal et al., 2019; Chen et al., 2021). With the growing application of large language models across diverse reasoning tasks (Rajani et al., 2019; Imani et al., 2023; Giadikiaroglou et al., 2024), the Pass@k metric has gained increasing recognition (Luo et al., 2023; Yu et al., 2024). It provides an effective measure of a model’s potential performance in solving complex questions. Pass@k represents the probability of generating at least one correct solution within \( k \) attempts for each question, as defined by the formula:

\[
\text{Pass}@k = \mathbb{E}_{\text{Questions}} \left[ 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}} \right],
\]

where \( n \) denotes the total number of generations in the reasoning task, and \( c \) is the number of correct solutions among them. Intuitively, Equation (1) calculates the expected proportion of questions for which at least one correct solution is obtained. In practice, \( n \) is typically set equal to \( k \), primarily to minimize inference costs.

2.2 Generalized Metric: G-Pass@k

While Pass@k provides an indication of a model’s performance potential, it does not consider the stability of the model’s reasoning across diverse attempts. To evaluate both the potential and stability of a model, we propose a generalized metric called G-Pass@k. Simply put, G-Pass@k assesses the stability of a reasoning model by measuring the probability of consistently generating correct solutions across all attempts.
Definition of G-Pass@k and G-Pass@k_\tau. Let \( p^* \) denote the latent success probability of a model generating correct solutions. Assuming each generation is independent and identically distributed (i.i.d.), the probability of obtaining \( m \) correct solutions follows a binomial distribution:

\[
m \sim B(n, p^*).
\] (2)

Since \( p^* \) is typically inaccessible, we use a hyper-geometric distribution to approximate the binomial distribution:

\[
\lim_{n \to \infty} H(m; k, c, n) \to B(m; n, p^*).
\] (3)

Therefore, G-Pass@k can be defined as (Yao et al., 2024):

\[
\text{G-Pass}@k = \mathbb{E}_{\text{Questions}} \left[ \frac{\binom{c}{j}}{\binom{n}{k}} \right].
\] (4)

Here, \( n \) denotes the total number of generations per question, and \( c \) signifies the number of generations that yield correct solutions. Considering the stringent requirement in Equation (4), we draw inspiration from the mean Average Precision (mAP) metric (Everingham et al., 2010) in the object detection task to introduce a tolerance threshold \( \tau \in (0.0, 1.0] \), leading to the definition of G-Pass@k_\tau:

\[
\text{G-Pass}@k_\tau = \mathbb{E}_{\text{Questions}} \left[ \sum_{j=\lceil \tau \cdot k \rceil}^{c} \frac{\binom{c}{j} \cdot \binom{n-c}{k-j}}{\binom{n}{k}} \right],
\] (5)

where \( \lceil \tau \cdot k \rceil \) denotes the smallest integer greater than or equal to \( \tau \cdot k \). Conceptually, for \( \tau < 1.0 \), there is flexibility to allow up to \( k - \lceil \tau \cdot k \rceil \) incorrect solutions within the \( k \) generations. In conclusion, we leverage the hypergeometric distribution for sampling without replacement to approximate the binomial distribution for sampling with replacement. Such approximations tend to have smaller errors when \( n \) is sufficiently large (\( k \ll n \)). Further details are provided in Appendix B.

Pass@k as a Special Case of G-Pass@k_\tau. Pass@k can be regarded as a special case of G-Pass@k_\tau, as formalized in the following theorem:

Theorem 2.1. Pass@k is a special case of G-Pass@k as \( \tau \) approaches 0, formally expressed as:

\[
\lim_{\tau \to 0} \sum_{j=\lceil \tau \cdot k \rceil}^{c} \frac{\binom{c}{j} \cdot \binom{n-c}{k-j}}{\binom{n}{k}} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}.
\] (6)

The proof is provided in Appendix C.

Definition of mG-Pass@k. Recall that when the threshold \( \tau \) is low, G-Pass@k_\tau primarily reflects the model’s performance potential. Conversely, at higher \( \tau \) values, G-Pass@k_\tau evaluates the model’s stability, i.e., its level of mastery over the question. Thus, G-Pass@k_\tau enables the continuous assessment of both performance potential and stability. We further define mG-Pass@k as follows:

\[
\begin{align*}
\text{mG-Pass}@k &= 2 \int_{0.5}^{1.0} \text{G-Pass}@k_\tau d\tau \\
&= \frac{2}{k} \sum_{i=\lceil 0.5 \cdot k \rceil + 1}^{k} \text{G-Pass}@k_{\frac{i}{k}}.
\end{align*}
\] (7)

Intuitively, mG-Pass@k provides an interpolated estimate of the area under the curve of G-Pass@k_\tau for \( \tau \in [0.5, 1.0] \), serving as a comprehensive metric that integrates all G-Pass@k_\tau values within a certain range. For models that are both optimal and stable, the mG-Pass@k value should approach 1.

3 Stability of LLM Reasoning

In this section, we perform extensive experiments on the mathematical reasoning task to evaluate and analyze the reasoning stability of current LLMs, given the prevalence and ease of verification of the mathematical reasoning task.

3.1 Setup

3.1.1 LLMs
We evaluate various LLMs recognized for their strong mathematical reasoning capabilities, including three categories: general LLMs, mathematical LLMs, and o1-like LLMs. For the details, please refer to Appendix E.

3.1.2 Data
In all experiments, we involve several benchmarks, which include the newly constructed benchmark named LiveMathBench and publicly available benchmarks, MATH500-L5, AIME2024-45, and AIME2025. For more detailed information about benchmarks, please refer to Appendix F.

3.2 Main Performance
Table 1 demonstrates the performance on LiveMathBench and Table 2 demonstrates the performance on MATH500-L5 and AIME2024-45. More experimental results can be found in Appendix I.
Table 1: Performance of models on LiveMathBench. We perform 48 runs and report results of greedy accuracy, and G-Pass@16_{\{0.5,0.75,1.0\}} and mG-Pass@16. A more detailed performance can be found in Table 6 at Appendix I.1.

<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th rowspan="2">Greedy</th>
    <th colspan="4">G-Pass@16 (Equation (5)) / %</th>
  </tr>
  <tr>
    <th>G-Pass@16_{0.5}</th>
    <th>G-Pass@16_{0.75}</th>
    <th>G-Pass@16_{1.0}</th>
    <th>mG-Pass@16</th>
  </tr>
  <tr><th colspan="6">General LLMs</th></tr>
  <tr><td>Llama-3.1-8B-Instruct</td><td>24.0</td><td>18.2</td><td>11.3</td><td>4.5</td><td>10.4</td></tr>
  <tr><td>Yi-1.5-34B-Chat</td><td>24.8</td><td>18.6</td><td>11.3</td><td>6.0</td><td>11.0</td></tr>
  <tr><td>Gemma-2-27b-it</td><td>26.9</td><td>23.5</td><td>17.8</td><td>12.7</td><td>17.3</td></tr>
  <tr><td>Llama-3.1-70B-Instruct</td><td>29.8</td><td>30.0</td><td>22.2</td><td>12.5</td><td>20.8</td></tr>
  <tr><td>Qwen2.5-7B-Instruct</td><td>37.0</td><td>36.5</td><td>27.2</td><td>16.0</td><td>25.8</td></tr>
  <tr><td>DeepSeek-V2.5-1210</td><td>38.7</td><td>38.9</td><td>27.9</td><td>17.3</td><td>26.7</td></tr>
  <tr><td>Llama-3.3-70B-Instruct</td><td>40.3</td><td>36.2</td><td>28.9</td><td>19.1</td><td>27.5</td></tr>
  <tr><td>GPT-4o-2024-11-20 †</td><td>44.8</td><td>41.9</td><td>32.9</td><td>22.2</td><td>31.6</td></tr>
  <tr><td>InternLM3-8B-Instruct</td><td>44.5</td><td>43.0</td><td>35.4</td><td>23.0</td><td>33.6</td></tr>
  <tr><td>Claude-3.5-Sonnet ‡</td><td>46.7</td><td>44.1</td><td>36.2</td><td>26.6</td><td>35.3</td></tr>
  <tr><td>Mistral-Large-Instruct-2411</td><td>41.6</td><td>39.4</td><td>37.1</td><td>32.9</td><td>36.4</td></tr>
  <tr><td>Qwen2.5-72B-Instruct</td><td>51.7</td><td>47.3</td><td>39.6</td><td>29.0</td><td>37.8</td></tr>
  <tr><td>Qwen2.5-32B-Instruct</td><td>50.8</td><td>48.3</td><td>39.5</td><td>28.6</td><td>38.1</td></tr>
  <tr><td>Gemini-1.5-Pro-Latest †</td><td>59.1</td><td>55.9</td><td>47.3</td><td>31.0</td><td>44.3</td></tr>
  <tr><td>DeepSeek-V3.0-Chat †</td><td>55.0</td><td>59.5</td><td>49.9</td><td>35.0</td><td>47.9</td></tr>
  <tr><th colspan="6">Mathematical Reasoning LLMs</th></tr>
  <tr><td>DeepSeek-Math-7B-RL</td><td>23.5</td><td>19.8</td><td>14.0</td><td>9.7</td><td>13.7</td></tr>
  <tr><td>NuminaMath-72B-CoT</td><td>40.8</td><td>34.0</td><td>27.1</td><td>14.2</td><td>25.0</td></tr>
  <tr><td>Qwen2.5-Math-7B-Instruct</td><td>44.1</td><td>44.1</td><td>38.3</td><td>28.1</td><td>36.6</td></tr>
  <tr><td>Qwen2.5-Math-72B-Instruct</td><td>57.6</td><td>52.7</td><td>45.4</td><td>27.9</td><td>42.3</td></tr>
  <tr><th colspan="6">O1-like Reasoning LLMs</th></tr>
  <tr><td>Skywork-o1</td><td>45.4</td><td>39.3</td><td>31.9</td><td>21.7</td><td>30.4</td></tr>
  <tr><td>QwQ-32B-Preview</td><td>72.7</td><td>74.9</td><td>65.8</td><td>40.1</td><td>61.2</td></tr>
  <tr><td>OpenAI o1-mini †</td><td>74.1 ‡</td><td>76.3</td><td>67.3</td><td>48.3</td><td>64.8</td></tr>
  <tr><td>DeepSeek Distill Qwen-32B</td><td>67.7</td><td>81.2</td><td>72.3</td><td>54.5</td><td>69.7</td></tr>
  <tr><td>DeepSeek Distill LLama-70B</td><td>74.8</td><td>80.8</td><td>73.0</td><td>53.0</td><td>69.7</td></tr>
  <tr><td>DeepSeek R1 †</td><td>81.1</td><td>83.6</td><td>79.1</td><td>69.5</td><td>77.6</td></tr>
</table>

† API-based LLMs.
‡ OpenAI o1 series model does not provide an optional temperature parameter, so we chose the average accuracy of 20 generations as the proxy for greedy accuracy.

From the results, we derive the following observations:

1) Reasoning Ability Still Needs to be Properly Evaluated. While most models demonstrate relatively strong performance in terms of Greedy Accuracy and Pass@16, their effectiveness significantly declines when assessed using the G-Pass@k metric. Specifically, when \( \tau \) is set to 1.0, indicating that the model must generate accurate solutions in all 16 attempts, almost all models experience a drastic decline in performance. Notably, even the most robust model in Table 1, DeepSeek R1, displays a 14.3% decline, diminishing from 81.1% to 69.5%. Even when \( \tau \) is relaxed to 0.5, which requires only half of the generated solutions to be correct, General LLMs, Mathematical Reasoning LLMs, and o1-like Reasoning LLMs still experience a decrease in performance. This indicates that, under demanding conditions, most models struggle to maintain consistency in their reasoning abilities across multiple samples, irrespective of whether the criteria are strict or lenient. These findings underscore the necessity for more rigorous evaluations of models’ reasoning capabilities, particularly in scenarios that require consistent and reliable performance across multiple instances. The current evaluation metrics, often reliant on single-shot greedy decoding, may not fully capture the real robustness and stability of these models in real-world applications.

2) Increasing Model Size May Not Significantly Enhance Robustness. A comparison of models within the same series, such as Qwen2.5-32B-
Table 2: Performance of models on MATH500 and AIME2024. Aligning with experiments on LiveMathBench, we also perform 48 runs and report results of greedy accuracy, G-Pass@16\(_{0.5,0.75,1.0}\), and mG-Pass@16. More detailed results are available in Table 8 at Appendix I.3.

<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th rowspan="2">Greedy</th>
    <th colspan="4">G-Pass@16 (Equation (5)) / %</th>
  </tr>
  <tr>
    <th>G-Pass@16<sub>0.5</sub></th>
    <th>G-Pass@16<sub>0.75</sub></th>
    <th>G-Pass@16<sub>1.0</sub></th>
    <th>mG-Pass@16</th>
  </tr>
  <tr>
    <th colspan="6">MATH500-L5</th>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td style="background-color:#e6b9c4;">56.0</td>
    <td>54.8</td>
    <td>43.3</td>
    <td>28.0</td>
    <td style="background-color:#bce8f1;">41.5</td>
  </tr>
  <tr>
    <td>Llama-3-70B-Instruct</td>
    <td style="background-color:#e6b9c4;">54.5</td>
    <td>55.4</td>
    <td>49.5</td>
    <td>35.0</td>
    <td style="background-color:#bce8f1;">47.3</td>
  </tr>
  <tr>
    <td>Mistral-Large-Instruct-2411</td>
    <td style="background-color:#e6b9c4;">55.2</td>
    <td>52.3</td>
    <td>51.2</td>
    <td>45.6</td>
    <td style="background-color:#bce8f1;">50.1</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td style="background-color:#e6b9c4;">63.4</td>
    <td>62.5</td>
    <td>54.4</td>
    <td>44.9</td>
    <td style="background-color:#bce8f1;">53.1</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B-Instruct</td>
    <td style="background-color:#e6b9c4;">64.2</td>
    <td>66.6</td>
    <td>59.4</td>
    <td>41.0</td>
    <td style="background-color:#bce8f1;">55.6</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-Instruct</td>
    <td style="background-color:#e6b9c4;">65.7</td>
    <td>65.0</td>
    <td>62.2</td>
    <td>57.6</td>
    <td style="background-color:#bce8f1;">61.5</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-72B-Instruct</td>
    <td style="background-color:#e6b9c4;">71.6</td>
    <td>64.9</td>
    <td>59.4</td>
    <td>46.0</td>
    <td style="background-color:#bce8f1;">57.6</td>
  </tr>
  <tr>
    <td>Skywork-01</td>
    <td style="background-color:#e6b9c4;">61.2</td>
    <td>56.5</td>
    <td>52.2</td>
    <td>42.9</td>
    <td style="background-color:#bce8f1;">50.7</td>
  </tr>
  <tr>
    <td>QwQ-32B-Preview</td>
    <td style="background-color:#e6b9c4;">83.6</td>
    <td>87.2</td>
    <td>78.8</td>
    <td>57.4</td>
    <td style="background-color:#bce8f1;">75.6</td>
  </tr>
  <tr>
    <td>DeepSeek Distill Qwen-32B</td>
    <td style="background-color:#e6b9c4;">83.6</td>
    <td>89.9</td>
    <td>83.8</td>
    <td>70.4</td>
    <td style="background-color:#bce8f1;">81.9</td>
  </tr>
  <tr>
    <th colspan="6">AIME2024-45</th>
  </tr>
  <tr>
    <td>Qwen2.5-32B-Instruct</td>
    <td style="background-color:#e6b9c4;">11.1</td>
    <td>7.1</td>
    <td>3.4</td>
    <td>2.2</td>
    <td style="background-color:#bce8f1;">3.7</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td style="background-color:#e6b9c4;">11.1</td>
    <td>8.9</td>
    <td>8.1</td>
    <td>4.7</td>
    <td style="background-color:#bce8f1;">7.5</td>
  </tr>
  <tr>
    <td>Mistral-Large-Instruct-2411</td>
    <td style="background-color:#e6b9c4;">13.3</td>
    <td>10.4</td>
    <td>6.8</td>
    <td>2.4</td>
    <td style="background-color:#bce8f1;">6.1</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td style="background-color:#e6b9c4;">13.3</td>
    <td>13.7</td>
    <td>12.9</td>
    <td>7.5</td>
    <td style="background-color:#bce8f1;">11.7</td>
  </tr>
  <tr>
    <td>Llama-3-70B-Instruct</td>
    <td style="background-color:#e6b9c4;">22.2</td>
    <td>25.3</td>
    <td>18.2</td>
    <td>6.9</td>
    <td style="background-color:#bce8f1;">16.4</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-Instruct</td>
    <td style="background-color:#e6b9c4;">11.1</td>
    <td>4.6</td>
    <td>2.6</td>
    <td>2.2</td>
    <td style="background-color:#bce8f1;">2.8</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-72B-Instruct</td>
    <td style="background-color:#e6b9c4;">20.0</td>
    <td>18.7</td>
    <td>16.2</td>
    <td>6.7</td>
    <td style="background-color:#bce8f1;">14.1</td>
  </tr>
  <tr>
    <td>Skywork-01</td>
    <td style="background-color:#e6b9c4;">11.1</td>
    <td>11.2</td>
    <td>10.3</td>
    <td>1.5</td>
    <td style="background-color:#bce8f1;">8.2</td>
  </tr>
  <tr>
    <td>QwQ-32B-Preview</td>
    <td style="background-color:#e6b9c4;">44.4</td>
    <td>41.0</td>
    <td>28.6</td>
    <td>8.1</td>
    <td style="background-color:#bce8f1;">24.7</td>
  </tr>
  <tr>
    <td>OpenAI o1-mini †</td>
    <td style="background-color:#e6b9c4;">60.3 ‡</td>
    <td>62.2</td>
    <td>53.3</td>
    <td>15.6</td>
    <td style="background-color:#bce8f1;">43.1</td>
  </tr>
  <tr>
    <td>DeepSeek Distill Qwen-32B</td>
    <td style="background-color:#e6b9c4;">62.2</td>
    <td>77.0</td>
    <td>66.5</td>
    <td>31.3</td>
    <td style="background-color:#bce8f1;">59.3</td>
  </tr>
</table>

† API-based LLMs.
‡ OpenAI o1 series model does not provide an optional temperature parameter, so we chose the average accuracy of 20 generations as greedy accuracy.

Instruct and Qwen2.5-72B-Instruct, reveals that despite a more than twofold difference in model size, their performance is similar across various metrics and datasets. For example, on both LiveMathBench and existing open-source datasets, the difference in Greedy Accuracy and mG-Pass@k between these two models is within two percentage points. Additionally, in the larger LLMs like Mistral-Large-Instruct-2411 (123B), although the scale has increased further, performance and stability have actually declined compared to Qwen2.5-72B-Instruct. This suggests that for certain tasks, particularly those requiring deep understanding and logical reasoning, mere parameter expansion may not yield the expected gains in performance or stability. Conversely, the current base model still holds untapped potential, and improved training paradigms or test-time scaling methods should be proposed to enhance its reasoning performance and ability, rather than merely increasing the model size (Snell et al., 2024).

Figure 1: Illustration of G-Pass@k w.r.t. different values of k, where \( k = \{4, 8, 16\} \) on MATH500-L5.

3) Significant Gap Between Theoretical Performance Potential and Actual Stability. In evaluating model performance, we observe a notable gap between the theoretical upper limit (\( G-Pass@16_{\tau \rightarrow 0} \)), the actual performance (Greedy Accuracy), and the stability across multiple samples (\( G-Pass@16_{\tau=1.0} \)). As evident from the main performance, while models theoretically possess high potential performance, their actual performance in practical applications falls short of this optimal
Table 3: Performance on CCEE and WLPMC. The table shows the decreasing trend of Greedy w.r.t. Pass@16 and G-Pass@16_{1.0} w.r.t. Greedy, which are marked with colors of different transparency.

<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th colspan="3">CCEE</th>
    <th colspan="3">WLPMC</th>
  </tr>
  <tr>
    <th>G-Pass@16_{-0}</th>
    <th>Greedy</th>
    <th>G-Pass@16_{1.0}</th>
    <th>G-Pass@16_{-0}</th>
    <th>Greedy</th>
    <th>G-Pass@16_{1.0}</th>
  </tr>
  <tr>
    <td>Llama-3.3-70B-Instruct</td>
    <td>75.9</td>
    <td>59.0_{22.2}</td>
    <td>35.2_{40.3}</td>
    <td>42.1</td>
    <td>9.1_{78.3}</td>
    <td>~0.0_{100.0}</td>
  </tr>
  <tr>
    <td>Mistral-Large-Instruct-2411</td>
    <td>71.5</td>
    <td>63.6_{11.0}</td>
    <td>52.3_{17.7}</td>
    <td>18.2</td>
    <td>9.1_{49.8}</td>
    <td>6.1_{32.9}</td>
  </tr>
  <tr>
    <td>DeepSeek-V3.0-Chat</td>
    <td>84.6</td>
    <td>68.2_{19.3}</td>
    <td>53.3_{21.8}</td>
    <td>56.8</td>
    <td>18.2_{68.0}</td>
    <td>6.6_{63.7}</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>80.3</td>
    <td>72.7_{9.4}</td>
    <td>56.9_{21.7}</td>
    <td>50.4</td>
    <td>18.2_{63.9}</td>
    <td>4.1_{77.7}</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro-Latest</td>
    <td>81.8</td>
    <td>68.2_{16.6}</td>
    <td>53.1_{22.1}</td>
    <td>60.0</td>
    <td>36.4_{40.0}</td>
    <td>4.3_{88.2}</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>79.4</td>
    <td>61.7_{22.3}</td>
    <td>41.8_{32.3}</td>
    <td>29.9</td>
    <td>18.2_{39.1}</td>
    <td>4.0_{78.0}</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-Instruct</td>
    <td>80.5</td>
    <td>63.6_{21.0}</td>
    <td>49.3_{22.5}</td>
    <td>43.8</td>
    <td>18.2_{58.5}</td>
    <td>0.7_{96.2}</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-72B-Instruct</td>
    <td>84.1</td>
    <td>77.3_{8.0}</td>
    <td>53.8_{30.4}</td>
    <td>47.0</td>
    <td>27.3_{41.9}</td>
    <td>18.2_{33.3}</td>
  </tr>
  <tr>
    <td>QwQ-32B-Preview</td>
    <td>92.3</td>
    <td>86.4_{6.3}</td>
    <td>55.0_{36.3}</td>
    <td>88.4</td>
    <td>27.3_{69.1}</td>
    <td>11.7_{66.9}</td>
  </tr>
</table>

![Illustration of G-Pass@k w.r.t. different values of n for DeepSeek-Math-7b-RL and NuminaMath-72B-CoT on LiveMathBench.](page_184_682_563_312.png)

Figure 2: Illustration of G-Pass@k w.r.t. different values of n for DeepSeek-Math-7b-RL and NuminaMath-72B-CoT on LiveMathBench.

level, particularly in terms of output stability.

3.3 Performance w.r.t. G-Pass@k_{r} Settings

Performance w.r.t. k. Figure 1 presents the results of selected models for G-Pass@4, G-Pass@8, and G-Pass@16. From the experiments, G-Pass@k can achieve consistent evaluation results under different k values. In addition, for advanced reasoning models with strong performance, a larger value of k has better differentiation.

Performance w.r.t. n. As previously noted, the number of attempts n is crucial for the accuracy of the estimation. We select two models, DeepSeek-Math-7b-RL and NuminaMath-72B-CoT, to conduct experiments with \( n = \{16\} \times \{1, 2, 3, 5, 8, 15\} = \{16, 32, 48, 128, 240\} \), and report G-Pass@16_{r}. The results are illustrated in Figure 2. When n is small, the estimation deviation is large, as shown by the significant fluctuations in the G-Pass@16; values for both models. Conversely, for larger n, G-Pass@16_{r} tends to stabilize, indicating a more consistent and reliable performance. Empirically, we recommend making at least \( n = 3k \) generations when calculating G-Pass@k to ensure estimation accuracy.

![G-Pass@k_r performance of LLMs w.r.t. sampling parameters, i.e., top-p, top-k, and temperature.](page_1012_682_563_312.png)

Figure 3: G-Pass@k_{r} performance of LLMs w.r.t. sampling parameters, i.e., top-p, top-k, and temperature.

3.4 Impact of Sampling Parameters

The configuration of sampling parameters influences the diversity of model generation and, consequently, its reasoning stability. To investigate the effects of various sampling parameters, we evaluate models across different combinations of these parameters. Specifically, we focus on 3 key parameters: *temperature*, *top-p*, and *top-k*. For each evaluation, we adjust one parameter while holding the others constant. Figure 3 illustrates the experimental results for the Llama-3.3-70B-Instruct, Qwen2.5-72B-Instruct, and QwQ-32B-Preview models. Additional details can be found in Appendix I.4, Appendix I.5, and Appendix I.6. Our findings can be summarized as follows:
1) G-Pass@\( k_r \) Demonstrates Robustness Across Sampling Parameters. The proposed metric, G-Pass@\( k_r \), demonstrates stability across various models under different sampling parameters. This indicates that G-Pass@\( k_r \) serves as a robust metric, accurately reflecting model performance.

2) Sensitivity of Different Models to Sampling Parameters Varies Greatly. Experimental results show that some models, including Llama-3.3-70B-Instruct, Mistral-Large-Instruct-2411, and Qwen2.5-72B-Instruct, maintain stable performance across various sampling parameters. We hypothesize that models with more parameters exhibit greater resistance to perturbations after adequate pre-training. In contrast, certain models, such as Qwen2.5-Math-72B-Instruct, experience significant performance degradation under atypical sampling parameters. This degradation may be attributed to the effects of post-training, making these models more susceptible to perturbations.

3) Long CoT Can Help to Enhance the Robustness of the Model. An intriguing observation is that several o1-like models exhibit considerable robustness to sampled parameters. For instance, the performance of QwQ remains consistently stable across all experiments. We believe that long COT reasoning aids the model in rectifying errors introduced by random sampling. This finding underscores the importance of backtracking and self-reflection in addressing complex reasoning problems.

3.5 Impact of Question Difficulty
We also examine the performance of models with respect to questions of varying difficulty levels. We analyze CCEE and WLPMC splits from LiveMathBench. CCEE is a college entrance examination that primarily involves fundamental high school mathematics knowledge, whereas WLPMC is a prestigious collegiate mathematics competition that presents significantly greater challenges. Table 3 shows the experimental results. The findings indicate that models struggle more with generalizing this ability to challenging questions. The sharp decline in performance on WLPMC, compared to CCEE, suggests a higher level of difficulty models face when grasping these questions. For instance, on WLPMC, the Greedy Accuracy of QwQ-32B-Preview declines by 69.1% compared to its Pass@16 performance. In contrast, on CCEE, the Greedy Accuracy of QwQ-32B-Preview only declines by 6.3% compared to its Pass@16 performance.

From the observation, we conjecture that models tend to learn superficial patterns from training data, which is reflected in the marked improvement of the Pass@\( k \) metric. However, this increase does not necessarily translate into an enhancement of the model’s real reasoning capabilities.

4 Enhancing the Stability of Reasoning
In this section, we aim to explore factors that can improve the reasoning stability. Initially, we assess whether straightforward supervised fine-tuning on specific corpora enhances reasoning stability. Subsequently, we explore the reasons for the pronounced stability of o1-like models, positing that the characteristics of their reasoning paths contribute to this improved stability.

4.1 Can SFT Enhance Model Stability? An Empirical Analysis of Data Contamination
Currently, most large language models are trained on massive pre-training corpora which may lead to data contamination. Data contamination arises when the test data is mixed into training data, also referred to as data leakage (Dickson, 2024; Dong et al., 2024). Additionally, supervised fine-tuning on specific datasets is a common post-training method to enhance model performance. In this section, we explore whether training a model on a particular corpus significantly affects its reasoning stability.

To investigate the effect of SFT, we train the Qwen2.5-7B model on a previously unseen corpus and evaluate its reasoning performance and stability throughout the training process to ascertain whether mere SFT (which we also refer to as overfitting or contamination) significantly impacts reasoning stability. To be specific, the training process begins with a base set of 200,000 randomly sampled instructions from the Numina-Math-CoT corpus (LI et al., 2024), which serves as the uncontaminated training set. Subsequently, we introduce incremental rounds of data contamination, consisting of 0, 6, 8, 10, and 16 rounds, where a round of 0 indicates the absence of contamination, i.e., training exclusively on the original NuminaMath data. The model’s efficacy was assessed across these five conditions, as illustrated in Figure 4.

Despite the observed increase in greedy score
Figure 4: The data contamination experiment involves different contamination rounds, where #Replication represents the number of these rounds. The term Slope denotes the slope value of the G-Pass@16_ curve with respect to \( \tau \).

with escalating rounds of contamination, the stability, as quantified by the G-Pass@k metric, did not exhibit a corresponding enhancement. Specifically, in Figure 4 left part, the disparity between actual performance (Greedy Accuracy) and stability across multiple samples (G-Pass@k@16_{\tau=1.0}) at each contamination round—6, 8, 10, and 16—was 22, 20, 18, and 26, respectively. In contrast, this gap for a non-contaminated model was only 5, which indicates that the discrepancy between performance and stability in contaminated models is more than three times greater. Additionally, as shown in Figure 4 right part, the slope becomes increasingly steep as the rounds of contamination increase, showing a deterioration in model stability with each additional round of contamination. This phenomenon is particularly significant in certain downstream training scenarios where overfitting becomes necessary, such as in contexts characterized by data scarcity. In these cases, while an increase in greedy accuracy might be achieved, it often comes at the cost of reduced stability. Notably, the aforementioned performance-stability gap may not necessarily narrow proportionally with increased levels of contamination.

From the experimental results, we can contend that simple SFT does not significantly enhance the model’s reasoning stability, or that LLMs are more inclined to memorize during SFT, rather than improving genuine reasoning ability (Chu et al., 2025).

4.2 Mechanistic Analysis of Long CoT in Enhancing Model Stability

From the experimental results presented in Table 1 and Table 2, it is evident that o1-like LLMs demonstrate greater stability when compared to general LLMs (e.g., Llama-3.1-8B-Instruct & Skywork-o1, and Qwen2.5-32B-Instruct & QwQ-32B-Preview).

In this section, we explore the reasons behind the superior inference performance and stability of o1-like LLMs.

We select Qwen2.5-32B-Instruct and QwQ-32B-Preview as the foundational models. Our first observation is that o1-like LLMs significantly surpass general LLMs in terms of the length of reasoning paths. This leads us to hypothesize whether extending the reasoning path length of general LLMs can enhance their reasoning performance and stability. To verify this hypothesis, we designed the following experiments: 1) leverage Qwen2.5-32B-Instruct to perform parallel reasoning m times, treating m reasoning paths combined as a single reasoning process; and 2) leverage Qwen2.5-32B-Instruct to execute serial reasoning m times, inserting a break-link (e.g., Wait, let’s re-evaluate the solution steps.) between each reasoning path. We subsequently compared the performance of these two enhanced reasoning models with that of QwQ-32B-Preview, and the results are illustrated in Figure 5. We observe that: 1) while parallel reasoning in general LLMs, combined with majority voting, can improve reasoning performance and stability to some extent, this improvement is limited. Such performance and stability remain significantly inferior to those of o1-like LLMs when considering an equivalent number of reasoning tokens at the same scale. While parallel reasoning is anticipated to outperform certain models with more parameters as the number of tokens rises, it unavoidably incurs substantial reasoning overhead. 2) a robust backbone model possesses the capability to reflect and backtrack. Introducing specialized break-links can enhance this capability to some extent, thereby improving the performance and stability of reasoning. However, this capability requires further stimulation compared to models similar to o1-like LLMs.

Figure 5: Comparison between Qwen2.5-32B-Instruct and QwQ-32B-Preview.

Therefore, we propose a preliminary hypothesis: o1-like LLMs exhibit superior reasoning performance and stability due to their reflective and backtracking capabilities, which enable them to
correct errors in the reasoning path. The reasoning path can be conceptualized as a gradient descent process in which general LLMs may easily become trapped in local optima when optimizing in a single direction. In contrast, o1-like LLMs, through their backtracking and reflection abilities, can escape local optima and more effectively identify global optima.

To validate our hypothesis, we investigate the hidden states of LLMs during the reasoning process, which reflects the decision process of LLMs (Ren et al., 2023; Wang et al., 2024). Specifically, we utilize the CoE-Score to quantitatively analyze the variation tendencies of hidden states across different layers of LLMs. For detailed information, please refer to the original paper by Wang et al. (2024). We choose several questions from LiveMathBench and prompted Qwen2.5-32B-Instruct and QwQ-32B-Preview to perform multiple reasoning. Subsequently, we gather all candidate answers generated by Qwen2.5-32B-Instruct and QwQ-32B-Preview and calculate the CoE-Scores with respect to candidate answers at each position of the reasoning paths. The curves are depicted in Figure 11, Figure 12, Figure 13, and Figure 14. These figures reveal that o1-like LLMs exhibit multiple decision changes during the reasoning process, as indicated by frequent peaks and troughs in their CoE-Score curves. In contrast, the decision process of general LLMs is more linear, resulting in a notably flat CoE-Score curve. This linearity makes general LLMs more susceptible to noise and incorrect intermediate steps, thereby reducing their inference performance and stability.

5 Related Work

Stability of LLM Reasoning. Large language models (LLMs) exhibit remarkable performance in reasoning tasks, encompassing question answering, programming, and mathematical problem-solving. Despite their prowess, the output stability of LLMs poses a significant challenge, whereby the model’s outputs can vary for the same input due to random sampling or hallucinations, impacting the model’s reliability and predictability (Gupta et al., 2024; Xu et al., 2024; Atil et al., 2024; Zhuo et al., 2024). Atil et al. (2024) introduced two new metrics: TARr@N for the total agreement rate at N runs over raw output and TARa@N for total agreement over parsed-out answers. However, TARr@N and TARa@N focus solely on measuring output consistency, our work introduces a novel evaluation metric G-Pass@k for evaluating the mathematical reasoning proficiency of LLMs. This metric aims to assess the model’s true reasoning ability by not only considering output consistency but also emphasizing correctness. Additionally, Yao et al. (2024) propose pass’k, an approach to assess the reliability and consistency of real-world agent tasks, sharing the same formulation as G-Pass@k. We relax the overly restrictive conditions of G-Pass@k and further introduce G-Pass@k_τ and mG-Pass@k. In comparison to the original G-Pass@k and pass’k, our methods offer enhanced flexibility and more comprehensive evaluation capabilities.

Mathematical Reasoning Benchmarks for LLMs. The assessment of large language models (LLMs) in mathematical reasoning has led to the development of specialized benchmarks focusing on different aspects of an LLM’s mathematical proficiency. GSM8K (Cobbe et al., 2021) presents a dataset composed of elementary-level math word problems, segregated into training and testing sets, that demand multi-step reasoning and detailed solution paths. MATH (Hendrycks et al., 2021) encompasses 12,500 problems derived from high school math competitions, challenging LLMs with advanced topics like calculus and algebra, and providing step-by-step solutions to facilitate coherent training. MathBench (Liu et al., 2024) is a hierarchical benchmark that assesses both theoretical and applied mathematical abilities, consisting of 3,709 questions spanning basic arithmetic to university level, structured across five educational tiers. Omni-Math (Gao et al., 2024) focuses on Olympic-level mathematical reasoning, featuring 4,428 competition-level problems categorized into over 33 subfields and 10 difficulty levels, spanning from entry-level to professional international competitions.

6 Conclusion

In this work, we propose G-Pass@k_τ and mG-Pass@k, novel evaluation metrics that assess both the reasoning capability and performance consistency of LLMs across varying correctness thresholds. Through detailed evaluations conducted on mathematical reasoning benchmarks, we find that current LLMs struggle with consistent reasoning. Additionally, we demonstrate G-Pass@k’s robustness and preliminarily explore strategies for enhancing the models’ reasoning stability.
Limitations

In this study, we analyze the stability of large language models in reasoning tasks, propose a new metric, and conduct experiments on various mathematical reasoning benchmarks. Due to constraints in space and resources, our work has the following limitations:

• our experiments are not extended to broader reasoning tasks. Nevertheless, we assert that mathematical reasoning, as a representative task, aptly supports our experiments and conclusions. Future research will address a wider range of reasoning tasks.

• We attempted to run experiments on as many large language models (LLMs) as possible; however, due to hardware limitations, API call overhead, and the service instability of some APIs, a few experimental results were missing. However, we believe that the experimental results presented in this paper are sufficient to support our conclusions.

• The benchmarks in this paper include only English and Chinese, however, the conclusions are generally applicable to large language models of all languages.

Acknowledgements

This work was supported by National Key R&D Program of China 2022ZD0161600, and Shanghai Oriental Talents Project BJZH2024070.

References

Meta AI. 2024. Llama 3.3 - 70b parameters instruct: A large language model supporting multilinguality, coding, reasoning, and tool usage. https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct. Accessed: 2024-12-13.

AIME2024. 2024. Aime problems and solutions. https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions.

AIME2025. 2025. Aime problems and solutions. https://artofproblemsolving.com/wiki/index.php/2025_AIME_I.

Anthropic Inc. 2024. Claude 3.5 Sonnet: A large language model for advanced text generation. https://claude.ai/chats. Accessed: 2024-12-10.

Berk Atil, Alexa Chittams, Liseng Fu, Ferhan Ture, Lixinyu Xu, and Breck Baldwin. 2024. LLM stability: A detailed analysis with some surprises. CoRR, abs/2408.04667.

Edward Beeching, Shengyi Costa Huang, Albert Jiang, Jia Li, Benjamin Lipkin, Zihan Qina, Kashif Rasul, Zijiu Shen, Roman Soletskyi, and Lewis Tunstall. 2024. Numinamath 72b cot. https://huggingface.co/AI-MO/NuminaMath-72B-Cot.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidi Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzi, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. CoRR, abs/2107.03374.

Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. 2025. Sft memorizes: rl generalizes: A comparative study of foundation model post-training. CoRR, abs/2501.17161.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.14168.

OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass.

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihaq Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhong Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L.
Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Menge Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Quishi Du, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuan Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaqoing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xinyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zhihu Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhigeng Xu, Zhongyu Zhang, and Zhen Zhang. 2024. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Quishi Du, R. J. Chen, R. L. Jin, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuan Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, and Wangding Zeng. 2024. Deepseek-v3 technical report. CoRR, abs/2412.19437.

Ben Dickson. 2024. Why data contamination is a big issue for llms. https://api-docs.deepseck.com/news/news1120. Accessed: 2023-7-17.

Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. In ACL (Findings), pages 12039–12050. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravan Kumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Alionsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esibov, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nairl, Grégoire Milon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billcock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasudevan Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.

Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. 2010. The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis., 88(2):303–338.

Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. 2024. Omni-math: A universal olympiad level mathematic benchmark for large language models. CoRR, abs/2410.07985.

Panagiotis Giadikiaroglou, Maria Lymperaioi, Giorgos Filandrianos, and Giorgos Stamou. 2024. Puzzle
solving using reasoning of large language models: A survey. In EMNLP, pages 11574–11591. Association for Computational Linguistics.

Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. 2024. Changing answer order can decrease MMLU accuracy. CoRR, abs/2406.19470.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In NeurIPS Datasets and Benchmarks.

Hugging Face. 2024. InternLM 3-8B Instruct Model. https://huggingface.co/internlm/internlm3-8b-instruct/. Accessed: 2025-01-15.

Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. In ACL (industry), pages 37–42. Association for Computational Linguistics.

Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. 2019. Spoc: Search-based pseudocode to code. In NeurIPS, pages 11883–11894.

Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Zijiu Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lampe, and Stanislav Polu. 2024. Numinamath. https://huggingface.co/AI-MO/NuminaMath-CoT.

Yucheng Li, Yunhao Guo, Frank Guerin, and Chenghua Lin. 2024. An open-source data contamination report for large language models. In EMNLP (Findings), pages 528–541. Association for Computational Linguistics.

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In ICLR. OpenReview.net.

Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhu, Wenwei Zhang, Songyang Zhang, Dahua Lin, and Kai Chen. 2024. Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark. In ACL (Findings), pages 6884–6915. Association for Computational Linguistics.

Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. CoRR, abs/2308.09583.

Shiwen Ni, Xiangtao Kong, Chengming Li, Xiping Hu, Ruifeng Xu, Jia Zhu, and Min Yang. 2024. Training on the benchmark is not all you need. CoRR, abs/2409.01790.

Skywork ol Team. 2024. Skywork-ol open series. https://huggingface.co/Skywork.

OpenAI. 2024a. Gpt-4o: Advancements in text and vision integration. https://www.openai.com/blog/gpt-4o-integration/. Accessed: 2024-12-10.

OpenAI. 2024b. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/. Accessed: 2024-12-10.

OpenAI. 2025. Openai o3-mini. https://openai.com/index/openai-o3-mini/. Accessed: 2025-1-32.

Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. In ACL, pages 4932–4942. Association for Computational Linguistics.

Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J. Liu. 2023. Out-of-distribution detection and selective generation for conditional language models. In ICLR. OpenReview.net.

Google Research. 2024. Gemini 1.5 pro: A large language model by google. https://ai.google.dev/. Accessed: 2024-12-12.

Google Research. 2025. Gemini 2.0 flash exp: A large language model by google. https://ai.google.dev/. Accessed: 2025-2-12.

Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussentot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Patterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijayakumar, Dominika Rogozinska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucinska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju-yeong Ji, Kareem Mohamed, Karitikeya Badola, Kat Black, Katie Millican, Keelin
McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjössund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, and Lilly McNealus. 2024. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.00118.

Zhihong Shao, Peiyi Wang, Qihaq Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300.

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.03314.

Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, and Seunghyeok Hong. 2024. Llm-as-a-judge & reward model: What they can and cannot do. CoRR, abs/2409.11239.

Qwen Team. 2024a. Qwq: Reflect deeply on the boundaries of the unknown.

The Mistral AI Team. 2024b. Mistral-Large Instruct 2411: A large language model supporting advanced instructions. https://huggingface.co/mistralai/Mistral-Large-Instruct-2411. Accessed: 2024-12-13.

A. T. Vandermonde. 1772. Mém. Acad. Roy. Sciences Paris.

Yiming Wang, Pei Zhang, Baosong Yang, Derek F. Wong, and Rui Wang. 2024. Latent space chain-of-embedding enables output-free LLM self-evaluation. CoRR, abs/2410.13640.

Hanzi Xu, Renze Lou, Jiangshu Du, Vahid Mahzoon, Elmira Talebianaraki, Zhuoan Zhou, Elizabeth Garrison, Slobodan Vucetic, and Wenpeng Yin. 2024. LMs’ classification performance is overclaimed. CoRR, abs/2406.16203.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024a. Qwen2 technical report. CoRR, abs/2407.10671.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024b. Qwen2.5 technical report. CoRR, abs/2412.15115.

An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xie, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024c. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. CoRR, abs/2409.12122.

Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. CoRR, abs/2406.12045.

Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang, Zijian Wu, Shuaibin Li, Fengze Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen, and Dahua Lin. 2024. Interimmath: Open math large language models toward verifiable reasoning. CoRR, abs/2402.06332.

Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Weihao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. 2024. Yi: Open foundation models by 01.ai. CoRR, abs/2403.04652.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhengguo Li, Adrian Weller, and Weiyang Liu. 2024. Metamath: Bootstrap your own mathematical questions for large language models. In ICLR. OpenReview.net.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohuan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS.

Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. Don’t make your LLM an evaluation benchmark cheater. CoRR, abs/2311.01964.

Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. 2024. Prosa: Assessing and understanding the prompt sensitivity of llms. In EMNLP (Findings), pages 1950–1976. Association for Computational Linguistics.
A  Detailed Related Work

B  Estimation of G-Pass@k

To demonstrate the unbiasedness of Equation (5), we conduct the simulation experiment illustrated in Figure 6. Specifically, we assume the probability of a model providing the correct solution in a single run is \( p^* = 0.4 \). For each \( n \), we perform several random Bernoulli samplings to obtain different values of \( c \) to calculate G-Pass@\( k_{\tau} \), and then compute the mean and variance to generate the figure. From Figure 6, it can be observed that Equation (5) is an unbiased estimator, facilitating fair comparison across different values of \( n \).

C  Proof of Theorem 2.1

Proof. Since \( j \) starts iterating at the upward rounding of \( \lceil \tau \cdot k \rceil \) and \( \tau \in (0, 1] \), so we have:

\[
\lim_{\tau \to 0} \sum_{j=\lceil \tau \cdot k \rceil}^{c} \frac{\binom{c}{j} \cdot \binom{n-c}{k-j}}{\binom{n}{k}} = \sum_{j=1}^{c} \frac{\binom{c}{j} \cdot \binom{n-c}{k-j}}{\binom{n}{k}}. \tag{8}
\]

According to the Vandermonde’s Identity (Vandermonde, 1772), the numerator term on the right side of Equation (8) can be written as

\[
\begin{align*}
&\sum_{j=1}^{c} \binom{c}{j} \cdot \binom{n-c}{k-j} \\
&= \sum_{j=0}^{c} \binom{c}{j} \cdot \binom{n-c}{k-j} - \binom{n-c}{k} \\
&= \binom{n}{k} - \binom{n-c}{k}.
\end{align*}
\tag{9}
\]

So we conclude that:

\[
\lim_{\tau \to 0} \sum_{j=\lceil \tau \cdot k \rceil}^{c} \frac{\binom{c}{j} \cdot \binom{n-c}{k-j}}{\binom{n}{k}} = \sum_{j=1}^{c} \frac{\binom{c}{j} \cdot \binom{n-c}{k-j}}{\binom{n}{k}} \\
= \frac{\binom{n}{k} - \binom{n-c}{k}}{\binom{n}{k}} \\
= 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}. \tag{10}
\]

D  Pass@k v.s. G-Pass@k

To facilitate an intuitive comparison between Pass@k and G-Pass@k, Figure 7 presents the metric values for various \( c \) values with \( n = 80 \). The figure illustrates that, while Pass@k offers insights into the model’s capabilities, relying solely on it may lead to an overestimation of the model’s actual performance. For instance, as shown in the upper left of Figure 7, even when the model solves the question correctly only 8 times out of 80 runs, Pass@k produces a notably high score (Pass@k > 0.8 for \( k \geq 16 \)). Additionally, as \( c \) increases beyond a certain threshold, differentiating performance based on Pass@k becomes increasingly challenging.

In contrast, G-Pass@k provides a more accurate and nuanced evaluation of the model’s performance, as depicted in Figure 7. Across varying \( c \) values, G-Pass@k demonstrates clear distinctions between performance levels. Moreover, by adjusting the threshold parameter \( \tau \), G-Pass@k can highlight different aspects of the model’s performance: a lower threshold emphasizes the model’s potential, while a higher threshold underscores its stability and mastery of the problem.

In summary, G-Pass@k not only delivers a more comprehensive performance assessment compared to Pass@k, but also, through flexible threshold configurations, effectively balances the evaluation of the model’s potential and stability.

E  Evaluated LLMs

In this paper, we conduct experiments on LLMs including: InternLM3-8B-Instruct (Ying et al., 2024; Hugging Face, 2024), DeepSeek-Math-7b-RL (Shao et al., 2024), DeepSeek-V3.0-Chat (DeepSeek-AI et al., 2024), Gemma-2-27b-it (Rivière et al., 2024), Llama-3.1-8B-Instruct (Dubey et al., 2024), Llama-3.1-70B-Instruct (Dubey et al., 2024), Yi-1.5-34B-Chat (Young et al., 2024), Llama-3.3-70B-Instruct (Dubey et al., 2024; AI, 2024), NuminaMath-72B-CoT (Beeching et al., 2024), Mistral-Large-Instruct-2411 (Team, 2024b), Qwen2.5-7B-Instruct (Yang et al., 2024b), Qwen2.5-Math-7B-Instruct (Yang et al., 2024c), Qwen2.5-32B-Instruct (Yang et al., 2024b), Qwen2.5-72B-Instruct (Yang et al., 2024b), Qwen2.5-Math-72B-Instruct (Yang et al., 2024c), Qwen2.5-Max (Yang et al., 2024b), Claude-3.5-Sonnet (Anthropic Inc., 2024), Gemini-1.5-
Figure 6: Illustration of estimation and the true value of G-Pass@\( k_{\tau} \).

Pro (Research, 2024), Gemini-2.0-Flash-Exp (Research, 2025), and GPT-4o-2024-11-20 (OpenAI, 2024a). Additionally, we include several o1-like LLMs, such as QwQ-32B-Preview (Team, 2024a), Skywork-o1-Open-Llama-3.1-8B (o1 Team, 2024), DeepSeek-R1 series models (DeepSeek-AI et al., 2024), OpenAI o1-mini (OpenAI, 2024b), and OpenAI o3-mini (OpenAI, 2025).

F Data Details

F.1 LiveMathBench
To effectively analyze the reasoning stability of large language models, we construct a new benchmark named LiveMathBench. LiveMathBench consists of the latest complex mathematical questions from various examinations and competitions, ensuring minimal risk of data contamination. LiveMathBench will undergo ongoing updates with new questions to continuously evaluate the mathematical reasoning performance of models.

F.1.1 Benchmark Construction
LiveMathBench is specifically designed to include out-of-domain question sets with different difficulty spans from various mathematical exams and competitions, aiming to avoid data contamination issues in existing LLMs and public math benchmarks (Zhou et al., 2023; Li et al., 2024; Ni et al., 2024). LiveMathBench (version of 202412) incorporates the latest problems from the China National Mathematical Olympiad (CNMO), China’s College Entrance Examination (CCEE), American Mathematics Competition (AMC), and William Lowell Putnam Mathematical Competition (WLPMC). These datasets encompass diverse levels of difficulty and linguistic variations and have low overlap with publicly available datasets, ensuring a comprehensive evaluation of the generalization capabilities of LLMs across various mathematical scenarios.

F.1.2 Data Sources
LiveMathBench (version of 202412) is composed of 4 parts including CNMO, CCEE, AMC, and WLPMC.

CNMO. The CNMO section features curated questions from the latest Chinese National Mathematics Olympiad. To enhance the difficulty level, single-choice questions are transformed into problem-solving tasks by concealing answer options, necessitating models to reason independently and provide solutions.
Figure 7: Comparison of Pass@k and G-Pass@k. In our simulation setup, we set \( n = 10 \) and \( c = \{8, 16, 24, 32\} \), and then calculate Pass@k and G-Pass@k.

<table>
  <tr>
    <th>Dataset</th>
    <th>Language</th>
    <th>#Fill-In-the-Blank</th>
    <th>#Problem-Solving</th>
    <th>#Questions</th>
  </tr>
  <tr>
    <td>CNMO</td>
    <td>en & cn</td>
    <td>-</td>
    <td>18\( \times 2 \)</td>
    <td>18\( \times 2 \)</td>
  </tr>
  <tr>
    <td>CCEE</td>
    <td>en & cn</td>
    <td>13\( \times 2 \)</td>
    <td>31\( \times 2 \)</td>
    <td>44\( \times 2 \)</td>
  </tr>
  <tr>
    <td>AMC</td>
    <td>en & cn</td>
    <td>-</td>
    <td>46\( \times 2 \)</td>
    <td>46\( \times 2 \)</td>
  </tr>
  <tr>
    <td>WLPMC</td>
    <td>en & cn</td>
    <td>-</td>
    <td>11\( \times 2 \)</td>
    <td>11\( \times 2 \)</td>
  </tr>
  <tr>
    <td>ALL</td>
    <td>en & cn</td>
    <td>13\( \times 2 \)</td>
    <td>106\( \times 2 \)</td>
    <td>119\( \times 2 \)</td>
  </tr>
</table>

Table 4: Statistics of LiveMathBench

CCEE. In the CCEE segment, we have selected questions from recent mock exams of China’s College Entrance Examination, excluding multi-modal proof problems. We have excluded multiple-choice questions and converted single-choice items into problem-solving questions, removing provided answer choices to assess the models’ ability to generate solutions autonomously.

AMC. The AMC section includes questions from the latest American Mathematics Competition, where each original question typically offers five possible answers labeled A through E, with only one correct option. Consistent with our approach in other sections, we convert these single-choice questions into problem-solving cues, encouraging models to deduce solutions without the aid of provided options.

WLPMC. We also include questions from the latest William Lowell Putnam Mathematical Competition (WLPMC). Regarded as one of the most prestigious university-level mathematics competitions globally, the WLPMC challenges participants with problems that span a broad spectrum of mathematical disciplines. These include geometry, algebra, trigonometry, calculus, linear algebra, combinatorics, probability theory, number theory, complex numbers, and differential equations.
F.1.3 Benchmark Statistics
Table 4 presents comprehensive statistics for the LiveMathBench. In order to enhance benchmark diversity and assess the performance of LLMs in multilingual settings, both English and Chinese versions of the questions are included.

F.1.4 Data Samples
Here we provide some samples in LiveMathBench.

Example in CNMO
[Question]
设复数z,w满足\( z + w = 2 \). 求\( S = |z^2 - 2w| + |w^2 - 2z| \)的最小可能值。
[Answer]
\( 8\sqrt{5} - 16 \)
[Question Type]
问答

Example in CCEE
[Question]
函数\( f(x) = \frac{x^3 e^{3x} - 3lnx - 1}{x} (x > 0) \)的最小值是____
[Answer]
3
[Question Type]
填空

Example in AMC
[Question]
The graph of \( y = e^{x+1} + e^{-x} - 2 \) has an axis of symmetry. What is the reflection of the point \( (-1, \frac{1}{2}) \) over this axis?
[Answer]
\( (0, \frac{1}{2}) \)
[Question Type]
Problem-Solving

Example in WLPMC
[Question]
A sequence \( y_1, y_2, \ldots, y_k \) of real numbers is called zigzag if \( k = 1 \), or if \( y_2 - y_1, y_3 - y_2, \ldots, y_k - y_{k-1} \) are nonzero and alternate in sign. Let \( X_1, X_2, \ldots, X_n \) be chosen independently from the uniform distribution on \([0, 1]\). Let \( a(X_1, X_2, \ldots, X_n) \) be the largest value of \( k \) for which there exists an increasing sequence of integers \( i_1, i_2, dots, i_k \) such that \( X_{i_1}, X_{i_2}, \ldots, X_{i_k} \) is zigzag. Find the expected value of \( a(X_1, X_2, \ldots, X_n) \) for \( n \geq 2 \).
[Answer]
\( \frac{2n+2}{3} \)
[Question Type]
Problem-Solving

F.1.5 Hard Split
To achieve better differentiation, we also implemented a hard split for evaluating models with strong inference performance. Specifically, we selected subsets where QwQ-32B-Preview and OpenAI o1-mini models demonstrated lower accuracy, consisting of 21 questions in English and 24 questions in Chinese.

F.2 MATH500-L5
MATH500 (Lightman et al., 2024) dataset is a curated subset of a larger collection MATH (Hendrycks et al., 2021), intended to challenge LLMs with complex mathematical problems. It encompasses a variety of advanced questions from multiple domains including algebra, geometry, probability, and number theory, thereby providing a comprehensive assessment of a model’s proficiency in mathematical reasoning. We select all questions with difficulty 5, resulting in **MATH500-L5**, which contains 134 questions.

F.3 AIME2024-45
Tailored for evaluating LLM performance at the American Invitational Mathematics Examination (AIME) level, the AIME question set (AIME2024, 2024; Yang et al., 2024c) presents a series of intricate tasks that test logical thinking, abstract reasoning, and accurate calculation skills. This resource aims to push the boundaries of what LLMs can achieve in solving sophisticated mathematical problems. We combine the part 1 and the part 2 of the American Invitational Mathematics Examination 2024, resulting in 45 questions, called **AIME2024-45**.
F.4 AIME2025
We also incorporate Part 1 of the American Invitational Mathematics Examination 2025, referred to as AIME2025 (AIME2025, 2025), as our evaluation benchmark, comprising 15 questions.

G Judge Details

G.1 Configurations of Judge Model
Inspired by previous works (Zheng et al., 2023; Son et al., 2024), we leverage Qwen-2.5-72B-Instruct (Yang et al., 2024a) to judge if the answers generated by the models are consistent with the golden answers, consider the high inference cost of the closed source models such as OpenAI models. We set the temperature to 0.0, and maximum output tokens to 8, 192.

G.2 Prompt for Judge
We leverage the prompts shown in Box Chinese Version of Judge Prompt and Box English Version of Judge Prompt to judge the consistency between candidate answers and reference answers.

G.3 Evaluation of Judge Model
To evaluate the effectiveness of our judge model, we compared the agreement rate between Qwen2.5-72B-as-Judge and GPT4o-as-Judge (OpenAI, 2024a). Specifically, we randomly selected 300 samples from the generations of five different models and used the judgments from GPT4o as the ground truth. We then calculated the agreement rate between the judgments made by our model and those by GPT4o. Table 5 presents the results, demonstrating that Qwen2.5-72B-as-Judge achieves high consistency with GPT4o-as-Judge across different models. These findings validate the feasibility of Qwen2.5-72B-as-Judge.

H Implementation Details
In all experiments, we set the number of generations, \( n \), to \( 16 \times 3 = 48 \) and report the greedy accuracy, Pass@k (G-Pass@k\(_{>0}\)), and G-Pass@k values, where \( k \in \{4, 8, 16\} \) and \( \tau \in \{0.25, 0.5, 0.75, 1.0\} \). For the sampling parameters of open-source models, we configure the temperature to 1.0, top-p to 0.8, top-k to 50, and repetition-penalty to 1.0. For open-source models, the maximum number of tokens is set to 8, 192 for non-o1 LLMs and 32, 768 for o1-like LLMs. For closed-source models, due to constraints of inference costs, we configured the maximum completion tokens to 4, 096 for non-o1 LLMs and 8, 192 for o1-like LLMs. We use the OpenCompass (Contributors, 2023) platform to evaluate all LLMs.

Due to the diverse formats of the final answers produced by models in complex mathematical questions, we leverage Qwen-2.5-72B-Instruct (Yang et al., 2024a) to judge whether the content generated by the tested model aligns with the standard answer. In our judge pipeline, we provide the original question, reference answer, and model-generated answer, prompting Qwen-2.5-72B-Instruct to determine whether the candidate solution is consistent with the reference answer. The details of the judging process can be found in Appendix G.

All experiments were conducted on clusters equipped with NVIDIA A800 GPUs and Intel(R) Xeon(R) Platinum 8336C CPUs, requiring hundreds of GPU hours to complete.

I Additional Experimental Results

I.1 Full Performance on LiveMathBench
Table 6 presents the comprehensive performance results on LiveMathBench.

I.2 Full Performance on LiveMathBench Hard Split
Table 7 presents the comprehensive performance results on LiveMathBench hard split.

I.3 Full Performance on MATH500-L5 & AIME2024-45
Table 8, Table 9 presents the comprehensive performance results for MATH500-L5, AIME2024-45, and AIME2025.

I.4 Full Performance w.r.t. Different temperatures
Figure 8 demonstrates the performance of models across different sampling temperatures. The values for the other sampling parameters, namely top-p (0.8), top-k (50), and repetition-penalty (1.0), remain constant. It can be observed that most models exhibit relatively stable performance when the temperature values are set to \{0.3, 0.5, 0.7, 1.0\}. This indicates that the G-Pass@k\(_{\tau}\) metric is robust for commonly used temperature parameters. However, it is worth noting that certain models, such as Qwen2.5-Math-7B-Instruct and Qwen2.5-Math-72B-Instruct, display an unusual sensitivity
Chinese Version of Judge Prompt

请你作为一个数学阅卷专家，判断下面的答案是否与标准答案一致，即考生是否回答正确。下面是一些评判标准:
1. 有些答案可能包含多项内容，可能有单选题、多选题、填空题和问答题，只要答案与标准答案一致即可, 对于多选题和多个空的填空题，需要考生对应的选项或空都回答正确才算正确。
2. 有些答案可能通过不同的方式表达，比如有些答案可能是一个数学表达式，有些答案可能是一个文字描述，只要表达的意思一致即可。且有些公式通过不同的方式表达，但等价，也是正确的。
3. 你不需要重新计算问题答案，因为标准答案已经给出，只需要根据问题形式来判断考生的答案是否与标准答案一致，是否正确即可。
请你根据上述标准，判断下面的答案是否与标准答案一致，如果一致，请在最后输出\boxed{\{yes\}}, 否则输出\boxed{\{no\}}. 如果难以判断，请输出\boxed{\{no\}}.
原问题: {question}
标准答案: {reference_answer}
考生答案: {candidate_answer}
分析:

English Version of Judge Prompt

Please act as an expert in grading mathematics exam papers, and judge whether the following answers match the standard answers, i.e., whether the examinee answered correctly. Here are some evaluation criteria:
1. Some answers may contain multiple parts, such as single-choice questions, multiple-choice questions, fill-in-the-blank questions, and problem-solving questions. As long as the answer matches the standard answer, it is considered correct. For multiple-choice questions and fill-in-the-blank questions with multiple blanks, the examinee must answer all corresponding options or blanks correctly to be considered correct.
2. Some answers may be expressed in different ways; for example, some answers may be mathematical expressions, while others may be textual descriptions. As long as the meaning conveyed is consistent, it is considered correct. Additionally, some formulas may be expressed differently but are equivalent, which is also considered correct.
3. You do not need to recalculate the problem answers, as the standard answers are already provided. You only need to judge whether the examinee’s answer matches the standard answer based on the form of the question and whether it is correct.
Please judge whether the following answer matches the standard answer according to the above criteria. If they match, output \boxed{\{yes\}}, otherwise output \boxed{\{no\}}. If it is difficult to judge, also output \boxed{\{no\}}.
Original Question: {question}
Standard Answer: {reference_answer}
Examinee’s Answer: {candidate_answer}
Analysis:
Table 5: Agreement rates between Qwen2.5-72B-as-Judge and GPT4o-as-judge.

<table>
  <tr>
    <th>Models Need to Judge</th>
    <th>Agreement</th>
    <th>Disagreement</th>
    <th>Accuracy (%)</th>
  </tr>
  <tr>
    <td>Deepseek-Math-7B-RL</td>
    <td>296</td>
    <td>4</td>
    <td>98.7</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B-Instruct</td>
    <td>282</td>
    <td>18</td>
    <td>94.0</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-72B-Instruct</td>
    <td>287</td>
    <td>13</td>
    <td>95.7</td>
  </tr>
  <tr>
    <td>Mistral-Large-Instruct-2411</td>
    <td>285</td>
    <td>15</td>
    <td>95.0</td>
  </tr>
  <tr>
    <td>QwQ-32B-Preview</td>
    <td>290</td>
    <td>10</td>
    <td>96.7</td>
  </tr>
</table>

to larger temperature coefficients. This discrepancy may reflect the impact of different training strategies on the model distribution. Intriguingly, we discover that the model QwQ-32B-Preview shows exceptional stability when it comes to sampling temperatures. We speculate that models similar to o1, which rely on long COT and reflection, are capable of consistently identifying the correct answers through continuous self-examination and backtracking. This characteristic contributes to the high reasoning stability observed in these models.

I.5 Full Performance w.r.t. Different top-p

Figure 9 illustrates the performance of models across various sampling top-p parameters. The values for the other sampling parameters, namely, temperature (1.0), top-k (50), and repetition-penalty (1.0), remain constant. Similar phenomena are also observed in the experiments conducted with varying temperatures. Most models exhibit stable performance within the range of commonly used parameters, demonstrating the effectiveness of G-Pass@k+. Furthermore, QwQ-32B-Preview also exhibits stability w.r.t. different top-p values.

I.6 Full Performance w.r.t. Different top-k

Figure 10 illustrates the performance of models across various sampling top-k parameters. The values for the other sampling parameters, specifically, temperature (1.0), top-p (0.8), and repetition-penalty (1.0), remain constant. Our analysis indicates that all models exhibit stable performance, suggesting that the top-k parameter has a minimal effect on sampling compared to temperature and top-p.
Table 6: Full performance of models on LiveMathBench. We report results of greedy decoding, Pass@16 (G-Pass@16_{\rightarrow 0}), G-Pass@16_{\{0.25,0.5,0.75,1.0\}}, and mG-Pass@16.

<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th rowspan="2">Greedy</th>
    <th colspan="6">G-Pass@16 (Equation (5)) / %</th>
  </tr>
  <tr>
    <th>G-Pass@16_{\rightarrow 0}</th>
    <th>G-Pass@16_{0.25}</th>
    <th>G-Pass@16_{0.5}</th>
    <th>G-Pass@16_{0.75}</th>
    <th>G-Pass@16_{1.0}</th>
    <th>mG-Pass@16</th>
  </tr>
  <tr><th colspan="8">General LLMs</th></tr>
  <tr><td>Llama-3.1-8B-Instruct</td><td>24.0</td><td>53.3</td><td>29.9</td><td>18.2</td><td>11.3</td><td>4.5</td><td>10.4</td></tr>
  <tr><td>Yi-1.5-34B-Chat</td><td>24.8</td><td>58.7</td><td>31.4</td><td>18.6</td><td>11.3</td><td>6.0</td><td>11.0</td></tr>
  <tr><td>Gemma-2-27B-it</td><td>26.9</td><td>54.3</td><td>33.6</td><td>23.5</td><td>17.8</td><td>12.7</td><td>17.3</td></tr>
  <tr><td>Llama-3.1-70B-Instruct</td><td>29.8</td><td>59.2</td><td>38.6</td><td>30.0</td><td>22.2</td><td>12.5</td><td>20.8</td></tr>
  <tr><td>Qwen-5.7B-Instruct</td><td>37.0</td><td>66.5</td><td>47.3</td><td>36.5</td><td>27.2</td><td>16.0</td><td>25.8</td></tr>
  <tr><td>Llama-3.3-70B-Instruct</td><td>40.3</td><td>62.0</td><td>45.8</td><td>36.2</td><td>28.9</td><td>19.1</td><td>27.5</td></tr>
  <tr><td>GPT-4o-2024-11-20 †</td><td>44.8</td><td>70.8</td><td>54.6</td><td>41.9</td><td>32.9</td><td>22.2</td><td>31.6</td></tr>
  <tr><td>InternLM3-8B-Instruct</td><td>44.5</td><td>69.5</td><td>50.8</td><td>43.0</td><td>35.4</td><td>23.0</td><td>33.6</td></tr>
  <tr><td>Claude-3.5-Sonnet †</td><td>46.7</td><td>71.2</td><td>54.3</td><td>44.1</td><td>36.2</td><td>26.6</td><td>35.3</td></tr>
  <tr><td>Mistral-Large-Instruct-2411</td><td>41.6</td><td>47.3</td><td>42.2</td><td>39.4</td><td>37.1</td><td>32.9</td><td>36.4</td></tr>
  <tr><td>Qwen-5.72B-Instruct</td><td>51.7</td><td>69.6</td><td>55.6</td><td>47.3</td><td>39.6</td><td>29.0</td><td>37.8</td></tr>
  <tr><td>Qwen-5.32B-Instruct</td><td>50.8</td><td>72.0</td><td>57.6</td><td>48.3</td><td>39.5</td><td>28.6</td><td>38.1</td></tr>
  <tr><td>Qwen-5.5-Max †</td><td>52.9</td><td>74.8</td><td>62.8</td><td>52.7</td><td>44.3</td><td>31.1</td><td>42.2</td></tr>
  <tr><td>Gemini-1.5-Pro-Latest †</td><td>59.1</td><td>78.8</td><td>65.7</td><td>55.9</td><td>47.3</td><td>31.0</td><td>44.3</td></tr>
  <tr><td>DeepSeek-V3.0-Chat †</td><td>55.0</td><td>80.7</td><td>69.7</td><td>59.5</td><td>49.9</td><td>33.0</td><td>47.9</td></tr>
  <tr><th colspan="8">Mathematical Reasoning LLMs</th></tr>
  <tr><td>DeepSeek-Math-7B-RL</td><td>23.5</td><td>45.0</td><td>29.0</td><td>19.8</td><td>14.0</td><td>9.7</td><td>13.7</td></tr>
  <tr><td>NuminaMath-72B-CoT</td><td>40.8</td><td>63.3</td><td>43.5</td><td>34.0</td><td>27.1</td><td>14.2</td><td>25.0</td></tr>
  <tr><td>Qwen-5.5-Math-7B-Instruct</td><td>44.1</td><td>68.4</td><td>53.0</td><td>44.1</td><td>38.3</td><td>28.1</td><td>36.6</td></tr>
  <tr><td>Qwen-5.5-Math-72B-Instruct</td><td>57.6</td><td>74.2</td><td>60.4</td><td>52.7</td><td>45.4</td><td>27.9</td><td>42.3</td></tr>
  <tr><th colspan="8">O1-like Reasoning LLMs</th></tr>
  <tr><td>Skywork-o1</td><td>45.4</td><td>61.1</td><td>47.7</td><td>39.3</td><td>31.9</td><td>21.7</td><td>30.4</td></tr>
  <tr><td>QwQ-32B-Preview</td><td>72.7</td><td>89.0</td><td>81.8</td><td>74.9</td><td>65.8</td><td>40.1</td><td>61.2</td></tr>
  <tr><td>DeepSeek Distill Qwen-1.5B</td><td>42.4</td><td>83.0</td><td>71.7</td><td>61.9</td><td>48.8</td><td>25.6</td><td>45.1</td></tr>
  <tr><td>DeepSeek Distill Llama-8B</td><td>58.4</td><td>88.8</td><td>78.4</td><td>67.8</td><td>56.8</td><td>31.9</td><td>52.2</td></tr>
  <tr><td>DeepSeek Distill Qwen-7B</td><td>65.6</td><td>88.5</td><td>81.8</td><td>73.0</td><td>66.4</td><td>48.4</td><td>63.1</td></tr>
  <tr><td>OpenAI o1-mini ‡</td><td>74.1 ‡</td><td>89.5</td><td>82.4</td><td>76.3</td><td>67.3</td><td>48.3</td><td>64.8</td></tr>
  <tr><td>DeepSeek Distill Qwen-14B</td><td>69.8</td><td>90.5</td><td>85.9</td><td>79.9</td><td>71.2</td><td>51.6</td><td>68.0</td></tr>
  <tr><td>DeepSeek Distill Qwen-32B</td><td>67.7</td><td>90.1</td><td>86.3</td><td>81.2</td><td>72.3</td><td>54.5</td><td>69.7</td></tr>
  <tr><td>DeepSeek Distill Llama-70B</td><td>74.8</td><td>89.6</td><td>86.1</td><td>80.8</td><td>73.0</td><td>53.0</td><td>69.7</td></tr>
  <tr><td>OpenAI o3-mini ‡</td><td>84.7 ‡</td><td>94.3</td><td>90.2</td><td>85.7</td><td>78.8</td><td>65.3</td><td>76.8</td></tr>
  <tr><td>DeepSeek R1 †</td><td>81.1</td><td>91.8</td><td>86.9</td><td>83.6</td><td>79.1</td><td>69.5</td><td>77.6</td></tr>
</table>

† API-based close-source LLMs.
‡ OpenAI o1-like series model does not provide an optional temperature parameter, so we chose the average accuracy of 20 generations as greedy accuracy.
Table 7: Full performance of models on LiveMathBench hard split. We report results of greedy decoding, Pass@16 (G-Pass@16_{-0}), G-Pass@16_{\{0.25,0.5,0.75,1.0\}}, and mG-Pass@16.

<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th rowspan="2">Greedy</th>
    <th colspan="6">G-Pass@16 (Equation (5)) / %</th>
  </tr>
  <tr>
    <th>G-Pass@16_{-0}</th>
    <th>G-Pass@16_{0.25}</th>
    <th>G-Pass@16_{0.5}</th>
    <th>G-Pass@16_{0.75}</th>
    <th>G-Pass@16_{1.0}</th>
    <th>mG-Pass@16</th>
  </tr>
  <tr>
    <th colspan="8">General LLMs</th>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>2.2</td>
    <td>35.5</td>
    <td>11.5</td>
    <td>0.8</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>13.3</td>
    <td>23.6</td>
    <td>11.4</td>
    <td>6.2</td>
    <td>3.2</td>
    <td>2.2</td>
    <td>3.3</td>
  </tr>
  <tr>
    <td>Llama-3.3-70B-Instruct</td>
    <td>4.4</td>
    <td>23.1</td>
    <td>11.0</td>
    <td>8.0</td>
    <td>4.7</td>
    <td>2.3</td>
    <td>4.4</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct</td>
    <td>4.4</td>
    <td>25.0</td>
    <td>17.5</td>
    <td>12.2</td>
    <td>7.5</td>
    <td>2.7</td>
    <td>7.0</td>
  </tr>
  <tr>
    <td>InternLM3-8B-Instruct</td>
    <td>11.1</td>
    <td>40.0</td>
    <td>19.3</td>
    <td>10.7</td>
    <td>8.2</td>
    <td>2.7</td>
    <td>7.1</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B-Instruct</td>
    <td>13.3</td>
    <td>30.0</td>
    <td>19.1</td>
    <td>14.1</td>
    <td>10.5</td>
    <td>3.5</td>
    <td>9.1</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>17.8</td>
    <td>29.0</td>
    <td>18.1</td>
    <td>15.3</td>
    <td>11.3</td>
    <td>5.4</td>
    <td>10.5</td>
  </tr>
  <tr>
    <th colspan="8">Mathematical Reasoning LLMs</th>
  </tr>
  <tr>
    <td>DeepSeek-Math-7B-RL</td>
    <td>8.9</td>
    <td>23.4</td>
    <td>6.9</td>
    <td>3.9</td>
    <td>2.3</td>
    <td>0.6</td>
    <td>2.2</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-Instruct</td>
    <td>15.6</td>
    <td>21.3</td>
    <td>13.1</td>
    <td>8.2</td>
    <td>3.3</td>
    <td>2.2</td>
    <td>3.8</td>
  </tr>
  <tr>
    <td>NuminaMath-72B-CoT</td>
    <td>11.1</td>
    <td>28.9</td>
    <td>10.3</td>
    <td>8.8</td>
    <td>7.3</td>
    <td>5.9</td>
    <td>7.3</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-72B-Instruct</td>
    <td>11.1</td>
    <td>32.2</td>
    <td>23.2</td>
    <td>11.8</td>
    <td>7.9</td>
    <td>5.9</td>
    <td>7.9</td>
  </tr>
  <tr>
    <th colspan="8">O1-like Reasoning LLMs</th>
  </tr>
  <tr>
    <td>QwQ-32B-Preview</td>
    <td>15.6</td>
    <td>54.3</td>
    <td>22.1</td>
    <td>5.9</td>
    <td>4.4</td>
    <td>2.4</td>
    <td>4.0</td>
  </tr>
  <tr>
    <td>DeepSeek Distill Qwen-1.5B</td>
    <td>6.7</td>
    <td>33.9</td>
    <td>15.8</td>
    <td>9.8</td>
    <td>4.5</td>
    <td>2.2</td>
    <td>4.6</td>
  </tr>
  <tr>
    <td>DeepSeek Distill Llama-8B</td>
    <td>8.9</td>
    <td>56.5</td>
    <td>30.7</td>
    <td>16.1</td>
    <td>5.6</td>
    <td>2.4</td>
    <td>6.2</td>
  </tr>
  <tr>
    <td>DeepSeek Distill Qwen-7B</td>
    <td>17.8</td>
    <td>57.4</td>
    <td>32.9</td>
    <td>13.9</td>
    <td>8.8</td>
    <td>3.3</td>
    <td>8.1</td>
  </tr>
  <tr>
    <td>OpenAI o1-mini †</td>
    <td>18.4 ‡</td>
    <td>68.3</td>
    <td>45.6</td>
    <td>21.0</td>
    <td>10.1</td>
    <td>0.5</td>
    <td>8.5</td>
  </tr>
  <tr>
    <td>DeepSeek Distill Qwen-14B</td>
    <td>15.6</td>
    <td>59.8</td>
    <td>43.0</td>
    <td>26.9</td>
    <td>15.9</td>
    <td>8.1</td>
    <td>15.5</td>
  </tr>
  <tr>
    <td>DeepSeek Distill Owen-32B</td>
    <td>22.2</td>
    <td>59.1</td>
    <td>43.8</td>
    <td>29.9</td>
    <td>16.9</td>
    <td>3.3</td>
    <td>15.1</td>
  </tr>
  <tr>
    <td>DeepSeek Distill Llama-70B</td>
    <td>35.6</td>
    <td>61.2</td>
    <td>50.1</td>
    <td>33.1</td>
    <td>19.0</td>
    <td>5.8</td>
    <td>17.3</td>
  </tr>
  <tr>
    <td>OpenAI o3-mini †</td>
    <td>43.3 ‡</td>
    <td>72.3</td>
    <td>57.1</td>
    <td>47.4</td>
    <td>32.5</td>
    <td>7.7</td>
    <td>28.6</td>
  </tr>
  <tr>
    <td>DeepSeek R1 †</td>
    <td>42.2</td>
    <td>71.5</td>
    <td>55.9</td>
    <td>46.6</td>
    <td>33.6</td>
    <td>9.8</td>
    <td>29.6</td>
  </tr>
</table>

* API-based close-source LLMs.
† OpenAI o1-like series model does not provide an optional temperature parameter, so we chose the average accuracy of 20 generations as greedy accuracy.
Table 8: Full performance of models on MATH500-L5 and AIME2024-45. Results of greedy decoding, Pass@16 (G-Pass@16_{→0}), G-Pass@16_{\{0.25,0.5,0.75,1.0\}}, and mG-Pass@16 are reported.

<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th rowspan="2">Greedy</th>
    <th colspan="6">G-Pass@16 (Equation (5)) / %</th>
  </tr>
  <tr>
    <th>G-Pass@16_{→0}</th>
    <th>G-Pass@16_{0.25}</th>
    <th>G-Pass@16_{0.5}</th>
    <th>G-Pass@16_{0.75}</th>
    <th>G-Pass@16_{1.0}</th>
    <th>mG-Pass@16</th>
  </tr>
  <tr><th colspan="8">MATH500-L5</th></tr>
  <tr><td>Llama-3-1.8B-Instruct</td><td>26.1</td><td>62.2</td><td>29.9</td><td>17.8</td><td>10.7</td><td>3.5</td><td>9.7</td></tr>
  <tr><td>Yi-1.5-34B-Chat</td><td>26.1</td><td>66.4</td><td>38.8</td><td>25.7</td><td>16.7</td><td>9.5</td><td>16.2</td></tr>
  <tr><td>Gemma-2-27B-it</td><td>24.6</td><td>50.5</td><td>32.6</td><td>23.7</td><td>18.4</td><td>13.0</td><td>17.6</td></tr>
  <tr><td>Llama-3-1.70B-Instruct</td><td>39.6</td><td>73.8</td><td>53.7</td><td>41.8</td><td>32.1</td><td>16.1</td><td>29.3</td></tr>
  <tr><td>InternLM-3.8B-Instruct</td><td>51.5</td><td>72.3</td><td>56.8</td><td>49.9</td><td>40.3</td><td>26.9</td><td>38.3</td></tr>
  <tr><td>Qwen2.5-7B-Instruct</td><td>56.0</td><td>76.1</td><td>66.6</td><td>54.9</td><td>43.3</td><td>28.0</td><td>41.5</td></tr>
  <tr><td>Llama-3-3.70B-Instruct</td><td>54.5</td><td>73.1</td><td>63.1</td><td>55.4</td><td>49.5</td><td>35.0</td><td>47.3</td></tr>
  <tr><td>Mistral-Large-Instruct-2411</td><td>55.2</td><td>58.4</td><td>54.4</td><td>52.3</td><td>51.2</td><td>45.6</td><td>50.1</td></tr>
  <tr><td>Qwen2.5-7B-Instruct</td><td>63.4</td><td>78.9</td><td>69.5</td><td>62.5</td><td>54.4</td><td>44.9</td><td>53.1</td></tr>
  <tr><td>Qwen2.5-Max †</td><td>63.4</td><td>87.1</td><td>73.5</td><td>65.8</td><td>57.3</td><td>38.9</td><td>54.5</td></tr>
  <tr><td>Qwen2.5-32B-Instruct</td><td>64.2</td><td>79.9</td><td>71.1</td><td>66.6</td><td>59.4</td><td>41.0</td><td>55.6</td></tr>
  <tr><td>Gemini-1.5-Pro-Latest †</td><td>72.4</td><td>92.7</td><td>82.7</td><td>74.4</td><td>64.9</td><td>45.3</td><td>61.8</td></tr>
  <tr><td>DeepSeek-Math-7b-Rf</td><td>15.7</td><td>41.0</td><td>26.3</td><td>15.5</td><td>8.7</td><td>5.7</td><td>9.0</td></tr>
  <tr><td>NuminaMath-72B-CoT</td><td>41.0</td><td>67.3</td><td>51.1</td><td>36.8</td><td>26.8</td><td>16.8</td><td>25.6</td></tr>
  <tr><td>Qwen2.5-Math-72B-Instruct</td><td>71.6</td><td>77.8</td><td>71.1</td><td>64.9</td><td>59.4</td><td>46.0</td><td>57.4</td></tr>
  <tr><td>Qwen2.5-Math-7B-Instruct</td><td>65.7</td><td>78.9</td><td>71.4</td><td>65.0</td><td>62.2</td><td>57.6</td><td>61.5</td></tr>
  <tr><td>Skywork-01</td><td>61.2</td><td>70.9</td><td>60.0</td><td>56.5</td><td>52.2</td><td>42.9</td><td>50.7</td></tr>
  <tr><td>DeepSeek Distill Qwen-1.8B</td><td>53.0</td><td>89.5</td><td>82.3</td><td>72.1</td><td>62.1</td><td>34.5</td><td>57.0</td></tr>
  <tr><td>QwQ-32B-Preview</td><td>82.8</td><td>95.9</td><td>92.5</td><td>87.2</td><td>78.8</td><td>57.4</td><td>75.6</td></tr>
  <tr><td>DeepSeek Distill Llama-8B</td><td>65.7</td><td>92.2</td><td>86.6</td><td>79.5</td><td>70.0</td><td>39.5</td><td>64.5</td></tr>
  <tr><td>DeepSeek Distill Qwen-7B</td><td>78.4</td><td>96.8</td><td>94.2</td><td>87.9</td><td>80.5</td><td>62.6</td><td>77.6</td></tr>
  <tr><td>DeepSeek Distill Qwen-14B</td><td>76.1</td><td>97.1</td><td>93.4</td><td>91.1</td><td>85.9</td><td>67.7</td><td>82.6</td></tr>
  <tr><td>DeepSeek Distill Llama-70B</td><td>87.3</td><td>96.5</td><td>93.5</td><td>89.6</td><td>85.5</td><td>66.8</td><td>81.9</td></tr>
  <tr><td>DeepSeek Distill Qwen-32B</td><td>83.6</td><td>96.1</td><td>93.6</td><td>89.9</td><td>83.8</td><td>70.4</td><td>81.9</td></tr>
  <tr><th colspan="8">AIME2024-45</th></tr>
  <tr><td>Yi-1.5-34B-Chat</td><td>2.2</td><td>20.5</td><td>5.9</td><td>0.5</td><td>0.0</td><td>0.0</td><td>0.0</td></tr>
  <tr><td>Llama-3-1.8B-Instruct</td><td>4.4</td><td>28.1</td><td>4.9</td><td>2.2</td><td>1.6</td><td>0.0</td><td>1.2</td></tr>
  <tr><td>Gemma-2-27B-it</td><td>6.7</td><td>21.0</td><td>8.3</td><td>5.2</td><td>1.8</td><td>0.0</td><td>1.8</td></tr>
  <tr><td>InternLM-3.8B-Instruct</td><td>11.1</td><td>20.5</td><td>13.3</td><td>7.2</td><td>4.3</td><td>1.0</td><td>3.7</td></tr>
  <tr><td>Qwen2.5-32B-Instruct</td><td>11.1</td><td>32.0</td><td>14.9</td><td>7.1</td><td>3.4</td><td>2.2</td><td>3.7</td></tr>
  <tr><td>Mistral-Large-Instruct-2411</td><td>13.3</td><td>15.4</td><td>11.1</td><td>10.4</td><td>6.8</td><td>2.4</td><td>6.1</td></tr>
  <tr><td>Qwen2.5-7B-Instruct</td><td>11.1</td><td>26.3</td><td>11.2</td><td>8.9</td><td>8.1</td><td>4.7</td><td>7.5</td></tr>
  <tr><td>Llama-3-1.70B-Instruct</td><td>15.6</td><td>41.2</td><td>23.5</td><td>15.0</td><td>8.1</td><td>3.0</td><td>8.0</td></tr>
  <tr><td>Gemini-1.5-Pro-Latest †</td><td>13.3</td><td>45.7</td><td>26.5</td><td>16.8</td><td>8.7</td><td>2.5</td><td>8.1</td></tr>
  <tr><td>Qwen2.5-7B-Instruct</td><td>13.3</td><td>33.7</td><td>16.3</td><td>13.7</td><td>12.9</td><td>7.5</td><td>11.7</td></tr>
  <tr><td>Llama-3-3.70B-Instruct</td><td>22.2</td><td>37.1</td><td>28.7</td><td>25.3</td><td>18.2</td><td>6.9</td><td>16.4</td></tr>
  <tr><td>Qwen2.5-Max †</td><td>22.2</td><td>44.4</td><td>35.2</td><td>15.5</td><td>9.9</td><td>5.3</td><td>9.8</td></tr>
  <tr><td>DeepSeek-Math-7b-Rf</td><td>2.2</td><td>16.3</td><td>4.4</td><td>1.5</td><td>0.1</td><td>0.0</td><td>0.1</td></tr>
  <tr><td>NuminaMath-72B-CoT</td><td>2.2</td><td>21.3</td><td>4.8</td><td>2.9</td><td>2.2</td><td>0.1</td><td>1.6</td></tr>
  <tr><td>Qwen2.5-Math-72B-Instruct</td><td>11.1</td><td>20.8</td><td>8.5</td><td>4.6</td><td>2.6</td><td>2.2</td><td>2.8</td></tr>
  <tr><td>Qwen2.5-Math-7B-Instruct</td><td>20.0</td><td>35.2</td><td>24.8</td><td>18.7</td><td>16.2</td><td>6.7</td><td>14.1</td></tr>
  <tr><td>Skywork-01</td><td>11.1</td><td>22.1</td><td>13.6</td><td>11.2</td><td>10.3</td><td>1.5</td><td>8.2</td></tr>
  <tr><td>DeepSeek Distill Qwen-1.8B</td><td>17.8</td><td>68.7</td><td>41.6</td><td>23.9</td><td>14.9</td><td>2.8</td><td>13.5</td></tr>
  <tr><td>QwQ-32B-Preview</td><td>44.4</td><td>74.3</td><td>59.3</td><td>41.0</td><td>28.6</td><td>8.1</td><td>24.7</td></tr>
  <tr><td>DeepSeek Distill Llama-8B</td><td>44.4</td><td>82.1</td><td>72.6</td><td>53.9</td><td>30.4</td><td>9.0</td><td>28.0</td></tr>
  <tr><td>DeepSeek Distill Qwen-7B</td><td>44.4</td><td>79.6</td><td>73.9</td><td>56.3</td><td>35.4</td><td>17.5</td><td>33.8</td></tr>
  <tr><td>OpenAI o1-mini †</td><td>60.3 ‡</td><td>86.7</td><td>80.0</td><td>62.2</td><td>53.3</td><td>15.6</td><td>43.1</td></tr>
  <tr><td>DeepSeek Distill Qwen-14B</td><td>62.2</td><td>86.5</td><td>79.3</td><td>75.8</td><td>62.9</td><td>26.5</td><td>56.0</td></tr>
  <tr><td>DeepSeek Distill Llama-70B</td><td>62.2</td><td>84.4</td><td>76.9</td><td>72.9</td><td>63.4</td><td>32.2</td><td>57.6</td></tr>
  <tr><td>DeepSeek Distill Qwen-32B</td><td>62.2</td><td>86.3</td><td>79.7</td><td>77.0</td><td>66.5</td><td>31.3</td><td>59.3</td></tr>
</table>

† API-based LLMs.
‡ OpenAI o1 series model does not provide an optional temperature parameter, so we chose the average accuracy of 20 generations as greedy accuracy.
Table 9: Full performance of models on AIME2025. We report results of greedy decoding, Pass@16 (G-Pass@16_{\rightarrow 0}), G-Pass@16_{\{0.25,0.5,0.75,1.0\}}, and mG-Pass@16.

<table>
  <tr>
    <th rowspan="2">LLMs</th>
    <th rowspan="2">Greedy</th>
    <th colspan="6">G-Pass@16 (Equation (5)) / %</th>
  </tr>
  <tr>
    <th>G-Pass@16_{\rightarrow 0}</th>
    <th>G-Pass@16_{0.25}</th>
    <th>G-Pass@16_{0.5}</th>
    <th>G-Pass@16_{0.75}</th>
    <th>G-Pass@16_{1.0}</th>
    <th>mG-Pass@16</th>
  </tr>
  <tr>
    <th colspan="8">General LLMs</th>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>0.0</td>
    <td>8.9</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Gemma-2-27B-it</td>
    <td>0.0</td>
    <td>9.5</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B-Chat</td>
    <td>0.0</td>
    <td>14.8</td>
    <td>4.8</td>
    <td>0.1</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>GPT-4o-2024-11-20 †</td>
    <td>0.0</td>
    <td>25.5</td>
    <td>7.4</td>
    <td>0.1</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct</td>
    <td>6.7</td>
    <td>21.3</td>
    <td>8.3</td>
    <td>4.6</td>
    <td>0.2</td>
    <td>0.0</td>
    <td>0.7</td>
  </tr>
  <tr>
    <td>InternLM3-SB-Instruct</td>
    <td>13.3</td>
    <td>30.3</td>
    <td>16.2</td>
    <td>6.7</td>
    <td>0.1</td>
    <td>0.0</td>
    <td>0.8</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B-Instruct</td>
    <td>20.0</td>
    <td>33.3</td>
    <td>28.1</td>
    <td>11.5</td>
    <td>0.2</td>
    <td>0.0</td>
    <td>1.4</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet †</td>
    <td>13.3</td>
    <td>34.4</td>
    <td>16.9</td>
    <td>6.4</td>
    <td>1.2</td>
    <td>0.0</td>
    <td>1.7</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>6.7</td>
    <td>25.2</td>
    <td>13.5</td>
    <td>9.7</td>
    <td>6.2</td>
    <td>0.2</td>
    <td>4.7</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>20.0</td>
    <td>33.2</td>
    <td>23.4</td>
    <td>12.2</td>
    <td>5.8</td>
    <td>0.1</td>
    <td>4.9</td>
  </tr>
  <tr>
    <td>Llama-3.3-70B-Instruct</td>
    <td>6.7</td>
    <td>13.6</td>
    <td>6.7</td>
    <td>6.7</td>
    <td>6.6</td>
    <td>0.5</td>
    <td>5.0</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro-Latest †</td>
    <td>20.0</td>
    <td>40.9</td>
    <td>25.8</td>
    <td>10.8</td>
    <td>6.7</td>
    <td>4.4</td>
    <td>6.8</td>
  </tr>
  <tr>
    <td>Qwen2.5-Max</td>
    <td>13.3</td>
    <td>39.9</td>
    <td>24.0</td>
    <td>11.9</td>
    <td>6.8</td>
    <td>2.9</td>
    <td>6.8</td>
  </tr>
  <tr>
    <td>Mistral-Large-Instruct-2411</td>
    <td>13.3</td>
    <td>19.7</td>
    <td>14.4</td>
    <td>10.8</td>
    <td>6.8</td>
    <td>6.7</td>
    <td>7.2</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash-Exp †</td>
    <td>26.7</td>
    <td>44.7</td>
    <td>30.3</td>
    <td>26.5</td>
    <td>21.5</td>
    <td>14.0</td>
    <td>21.2</td>
  </tr>
  <tr>
    <th colspan="8">Mathematical Reasoning LLMs</th>
  </tr>
  <tr>
    <td>DeepSeek-Math-7B-RL</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>NuminaMath-72B-CoT</td>
    <td>0.0</td>
    <td>21.0</td>
    <td>10.3</td>
    <td>6.7</td>
    <td>6.7</td>
    <td>4.4</td>
    <td>6.4</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-Instruct</td>
    <td>20.0</td>
    <td>36.8</td>
    <td>16.2</td>
    <td>8.7</td>
    <td>6.7</td>
    <td>6.7</td>
    <td>6.8</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-72B-Instruct</td>
    <td>13.3</td>
    <td>30.1</td>
    <td>15.7</td>
    <td>13.3</td>
    <td>13.3</td>
    <td>13.3</td>
    <td>13.3</td>
  </tr>
  <tr>
    <th colspan="8">OI-like Reasoning LLMs</th>
  </tr>
  <tr>
    <td>Skywork-o1</td>
    <td>13.3</td>
    <td>31.2</td>
    <td>21.5</td>
    <td>15.3</td>
    <td>13.3</td>
    <td>7.2</td>
    <td>11.8</td>
  </tr>
  <tr>
    <td>DeepSeek Distill Qwen-1.3B</td>
    <td>26.7</td>
    <td>54.6</td>
    <td>38.1</td>
    <td>31.9</td>
    <td>23.8</td>
    <td>1.4</td>
    <td>18.6</td>
  </tr>
  <tr>
    <td>DeepSeek Distill Llama-8B</td>
    <td>40.0</td>
    <td>62.2</td>
    <td>55.3</td>
    <td>40.4</td>
    <td>21.2</td>
    <td>7.9</td>
    <td>21.0</td>
  </tr>
  <tr>
    <td>QwQ-32B-Preview</td>
    <td>26.7</td>
    <td>60.5</td>
    <td>43.5</td>
    <td>34.5</td>
    <td>32.4</td>
    <td>15.6</td>
    <td>28.1</td>
  </tr>
  <tr>
    <td>OpenAI o1-mini ‡</td>
    <td>46.7 ‡</td>
    <td>62.0</td>
    <td>41.0</td>
    <td>39.9</td>
    <td>32.5</td>
    <td>14.0</td>
    <td>28.4</td>
  </tr>
  <tr>
    <td>DeepSeek Distill Qwen-7B</td>
    <td>46.7</td>
    <td>62.1</td>
    <td>53.2</td>
    <td>46.6</td>
    <td>38.3</td>
    <td>22.7</td>
    <td>36.1</td>
  </tr>
  <tr>
    <td>DeepSeek Distill Qwen-14B</td>
    <td>46.7</td>
    <td>67.7</td>
    <td>60.1</td>
    <td>58.8</td>
    <td>41.1</td>
    <td>25.2</td>
    <td>40.8</td>
  </tr>
  <tr>
    <td>DeepSeek Distill LLama-70B</td>
    <td>46.7</td>
    <td>75.1</td>
    <td>54.9</td>
    <td>52.5</td>
    <td>38.6</td>
    <td>26.8</td>
    <td>37.4</td>
  </tr>
  <tr>
    <td>DeepSeek R1 †</td>
    <td>66.7</td>
    <td>68.9</td>
    <td>63.7</td>
    <td>52.6</td>
    <td>46.8</td>
    <td>24.3</td>
    <td>42.5</td>
  </tr>
  <tr>
    <td>OpenAI o3-mini †</td>
    <td>53.3 ‡</td>
    <td>80.0</td>
    <td>77.7</td>
    <td>59.0</td>
    <td>46.5</td>
    <td>29.4</td>
    <td>43.6</td>
  </tr>
  <tr>
    <td>DeepSeek Distill Qwen-32B</td>
    <td>46.7</td>
    <td>72.6</td>
    <td>63.6</td>
    <td>59.7</td>
    <td>50.2</td>
    <td>29.5</td>
    <td>47.3</td>
  </tr>
</table>

† API-based close-source LLMs.
‡ OpenAI o1-like series model does not provide an optional temperature parameter, so we chose the average accuracy of 20 generations as greedy accuracy.
Figure 8: G-Pass@\( k_r \) performance of LLMs w.r.t. different temperatures.
Figure 9: G-Pass@\( k_T \) performance of LLMs w.r.t. different top-p.
Figure 10: G-Pass@\( k_r \) performance of LLMs w.r.t. different top-k.
Question 1
In \( \triangle ABC \), given \( \cos C = \frac{\sin A + \cos A}{2} = \frac{\sin B + \cos B}{2} \), find the value of \( \cos C \).

QwQ-32B-Preview Response for Question 1
Let’s tackle this problem step by step. So, we have a triangle ABC, and we’re given that \( \cos C \) equals both \( (\sin A + \cos A)/2 \) and \( (\sin B + \cos B)/2 \). Our goal is to find the value of \( \cos C \).
First, let’s write down what we know:

\[
\cos C = \frac{\sin A + \cos A}{2} = \frac{\sin B + \cos B}{2}
\]

........
Therefore, \( \cos C = (\sqrt{7})/4 \).

Wait, but earlier steps led to \( \cos C = -\cos 2A \), and now \( \cos C = (\sqrt{7})/4 \).

Let me verify if this makes sense.
We have \( \cos C = (\sqrt{7})/4 \), and from \( \cos C = -\cos 2A \).
So, \( -\cos 2A = (\sqrt{7})/4 \rightarrow \cos 2A = -\sqrt{7}/4 \).

But \( \cos 2A = 2 \cos^2 A - 1 \).
Substitute \( \cos A = (\sqrt{7} - 1)/4 \):

\[
2 \left( \frac{\sqrt{7} - 1}{4} \right)^2 - 1 = 2 \left( \frac{7 - 2\sqrt{7} + 1}{16} \right) - 1 = 2 \left( \frac{8 - 2\sqrt{7}}{16} \right) - 1 = \frac{8 - 2\sqrt{7}}{8} - 1 = \frac{8 - 2\sqrt{7} - 8}{8} = -\frac{\sqrt{7}}{4}
\]

This matches \( \cos C = -\cos 2A = -(-\sqrt{7}/4) = \sqrt{7}/4 \).
Therefore, \( \cos C = \sqrt{7}/4 \).

Wait, but I thought \( \cos C = (\sqrt{7})/4 \), and \( \cos 2A = -\sqrt{7}/4 \), which satisfies \( \cos C = -\cos 2A \).
Therefore, the value of \( \cos C \) is \( \sqrt{7}/4 \).

So, the answer should be \( \frac{\sqrt{7}}{4} \).

Final Answer
\[
\frac{\sqrt{7}}{4}
\]
Given the equation in \( \triangle ABC \):

\[
\cos C = \frac{\sin A + \cos A}{2} = \frac{\sin B + \cos B}{2}
\]

We will denote the common value of these expressions by \( x \):

\[
\cos C = x \\
\frac{\sin A + \cos A}{2} = x \\
\frac{\sin B + \cos B}{2} = x
\]

From these equations, we derive:

\[
\sin A + \cos A = 2x \\
\sin B + \cos B = 2x
\]

1. **Square both sides of each equation:**

\[
(\sin A + \cos A)^2 = (2x)^2 \\
(\sin B + \cos B)^2 = (2x)^2
\]

2. **Expand the squares:**

\[
\sin^2 A + \cos^2 A + 2 \sin A \cos A = 4x^2 \\
\sin^2 B + \cos^2 B + 2 \sin B \cos B = 4x^2
\]

3. **Use the Pythagorean identity** \( \sin^2 \theta + \cos^2 \theta = 1 \):

\[
1 + 2 \sin A \cos A = 4x^2 \\
1 + 2 \sin B \cos B = 4x^2
\]

4. **Subtract 1 from both sides:**

\[
2 \sin A \cos A = 4x^2 - 1 \\
2 \sin B \cos B = 4x^2 - 1
\]

.............

16. **Since \( \sin A = \cos A \) when \( A = B \):**

\[
\sin A = \cos A \implies \sin A = \frac{\sqrt{2}}{2} \\
2 \sin^2 A = 1
\]

17. **Substitute back to find \( x \):**

\[
x = -1 + 1 = 0
\]

Thus, the value of \( \cos C \) is:

\[
\frac{\sqrt{2}}{2}
\]
Figure 11: CoE-Score of QwQ and Qwen on Question 1.

Question 2
The sequence \( \{a_n\} \) satisfies \( a_1 = 1 \), and for any positive integer \( n \), we have \( a_{n+1} = 10^n a_n^2 \). What is the general term formula for \( \{a_n\} \)?
Let’s tackle this problem step by step. We have a sequence defined by a recursive formula, and we need to find a general term for it. The sequence is given by \( a_1 = 1 \), and for any positive integer \( n \), \( a_{n+1} = 10^n a_n^2 \). Our goal is to find a formula for \( a_n \) in terms of \( n \).
First, I’ll try to compute the first few terms of the sequence to see if there’s a pattern.
Given:
\[
a_1 = 1
\]
For \( n = 1 \):
\[
a_2 = 10^1 \cdot a_1^2 = 10 \cdot (1)^2 = 10
\]
For \( n = 2 \):
\[
a_3 = 10^2 \cdot a_2^2 = 100 \cdot (10)^2 = 100 \cdot 100 = 10,000
\]
For \( n = 3 \):
\[
a_4 = 10^3 \cdot a_3^2 = 1,000 \cdot (10,000)^2 = 1,000 \cdot 100,000,000,000 = 10^{12}
\]
………
Wait a minute, let’s check this with the earlier terms to verify.
For \( n = 1 \):
\[
a_1 = 10^{2^1-1-1} = 10^{2-1-1} = 10^0 = 1
\]
Correct.
For \( n = 2 \):
\[
a_2 = 10^{2^2-2-1} = 10^{4-2-1} = 10^1 = 10
\]
Correct.
For \( n = 3 \):
\[
a_3 = 10^{2^3-3-1} = 10^{8-3-1} = 10^4 = 10,000
\]
Correct.
For \( n = 4 \):
\[
a_4 = 10^{2^4-4-1} = 10^{16-4-1} = 10^{11} = 100,000,000,000
\]
Wait, but earlier I calculated \( a_4 = 10^{12} \). Hmm, there’s a discrepancy here.
Wait, let me double-check my earlier calculation for \( a_4 \):
Given \( a_3 = 10^4 \), then:
\[
a_4 = 10^3 \cdot (10^4)^2 = 10^3 \cdot 10^8 = 10^{11}
\]
Ah, I made a mistake earlier. I thought \( a_4 = 10^{12} \), but actually, it’s \( 10^{11} \). So, my general formula \( a_n = 10^{2^n-n-1} \) gives \( a_4 = 10^{11} \), which matches the correct calculation. Great!
Therefore, the general term formula for the sequence \( \{a_n\} \) is:
\[
a_n = 10^{2^n-n-1}
\]
Final Answer
\[
a_n = 10^{2^n-n-1}
\]
To find the general term formula for the sequence \( \{a_n\} \) defined by \( a_1 = 1 \) and \( a_{n+1} = 10^n a_n^2 \) for any positive integer \( n \), we will proceed step by step.

1. Initial Condition: \( a_1 = 1 \)
2. First Iteration: \( a_2 = 10^1 a_1^2 = 10 \cdot 1^2 = 10 \)
3. Second Iteration: \( a_3 = 10^2 a_2^2 = 10^2 \cdot 10^2 = 10^4 \)
4. Third Iteration: \( a_4 = 10^3 a_3^2 = 10^3 \cdot (10^4)^2 = 10^3 \cdot 10^8 = 10^{11} \)

From these calculations, we observe a pattern in the exponents of 10. Let’s hypothesize the general form of \( a_n \) and then prove it by induction.

5. Hypothesis: Suppose \( a_n = 10^{b_n} \), where \( b_n \) is some function of \( n \).
6. Base Case: For \( n = 1 \),
\[
a_1 = 1 = 10^0 \implies b_1 = 0
\]
7. Inductive Step: Assume \( a_k = 10^{b_k} \) for some \( k \geq 1 \). We need to show that \( a_{k+1} = 10^{b_{k+1}} \).

By the recurrence relation,
\[
a_{k+1} = 10^k a_k^2 = 10^k (10^{b_k})^2 = 10^k \cdot 10^{2b_k} = 10^{k + 2b_k}
\]
Thus, if \( a_k = 10^{b_k} \), then \( a_{k+1} = 10^{b_{k+1}} \) with \( b_{k+1} = k + 2b_k \).

8. Solving the Recurrence for \( b_n \): We have the recurrence relation:
\[
b_{k+1} = k + 2b_k
\]
with the initial condition \( b_1 = 0 \).
To solve this, we will find a pattern by calculating the first few terms:
\[
b_1 = 0 \\
b_2 = 1 + 2b_1 = 1 + 2 \cdot 0 = 1 \\
b_3 = 2 + 2b_2 = 2 + 2 \cdot 1 = 4 \\
b_4 = 3 + 2b_3 = 3 + 2 \cdot 4 = 11
\]
Observing the pattern, we hypothesize:
\[
b_n = \frac{2^n - n - 2}{3}
\]
9. Verification: We verify by induction: - Base case \( n = 1 \):
\[
b_1 = \frac{2^1 - 1 - 2}{3} = \frac{2 - 1 - 2}{3} = \frac{-1}{3} = 0
\]
- Inductive step: Assume \( b_k = \frac{2^k - k - 2}{3} \). Then,
\[
b_{k+1} = k + 2b_k = k + 2 \left( \frac{2^k - k - 2}{3} \right) = \frac{3k + 2^{k+1} - 2k - 4}{3} = \frac{2^{k+1} + k - 4}{3}
\]
which matches our hypothesis.
Therefore, the general term of the sequence is:
\[
a_n = 10^{b_n} = 10^{\frac{2^n - n - 2}{3}}
\]
The final answer is:
\[
\boxed{10^{\frac{2^n - n - 2}{3}}}
\]
Figure 12: CoE-Score of QwQ and Qwen on Question 2.

Question 3

If the real part of the complex number \( z \) is greater than 0, and \( \overline{z}(z + 1) = \frac{20}{3+i} \), then \( z = ? \)
Let's tackle this problem step by step. So, we have a complex number \( z \) with a positive real part, and it satisfies the equation \( \overline{z}(z + 1) = \frac{20}{3+i} \). We need to find what \( z \) is.

First, let's recall that for any complex number \( z = a + bi \), where \( a \) and \( b \) are real numbers, the complex conjugate \( \overline{z} \) is \( a - bi \).

So, if \( z = a + bi \), then \( \overline{z} = a - bi \).

Given that, the equation becomes:
\[
(a - bi)((a + bi) + 1) = \frac{20}{3+i}
\]
Simplify the left side:
First, compute \( z + 1 = (a + bi) + 1 = (a + 1) + bi \).
Then, \( \overline{z}(z + 1) = (a - bi)((a + 1) + bi) \).
Let's multiply these two complex numbers:
\[
(a - bi)((a + 1) + bi) = a(a + 1) + a(bi) - bi(a + 1) - bi(bi)
= a(a + 1) + abi - bai - bi^2
\]
Wait, that seems messy. Maybe there's a better way to expand this.

Actually, let's use the distributive property more carefully:
\[
(a - bi)((a + 1) + bi) = a(a + 1) + a(bi) - bi(a + 1) - bi(bi)
\]
Now, compute each term:
\[
a(a + 1) = a^2 + a
\]
\[
a(bi) = abi
\]
\[
-bi(a + 1) = -bai - bi
\]
\[
-bi(bi) = -b^2i^2 = -b^2(-1) = b^2
\]
Now, combine all these:
\[
a^2 + a + abi - bai - bi + b^2
\]
Notice that \( abi - bai = 0 \), since they cancel each other out.
So, we're left with:
\[
a^2 + a + b^2 - bi
\]
Wait, but that doesn't seem right because the imaginary parts should cancel out completely since the right side is a real number (as we'll see when we simplify it).

Wait, actually, no. Let's check the right side:
\[
\frac{20}{3+i}
\]
\[
\frac{20}{3+i} \cdot \frac{3-i}{3-i} = \frac{60-20i}{9+1} = \frac{60-20i}{10} = 6-2i
\]

However, the problem states that the real part of \( z \) is greater than 0. So, \( a > 0 \).

Therefore, \( a = 1 \) (since \( a = -2 \) is less than 0 and doesn't satisfy the condition).
So, \( z = a + bi = 1 + 2i \).

Let me verify this solution by plugging back into the original equation.
Compute \( \overline{z}(z + 1) \):
\[
\overline{z} = 1 - 2i
\]
\[
z + 1 = (1 + 2i) + 1 = 2 + 2i
\]
Now, \( \overline{z}(z + 1) = (1 - 2i)(2 + 2i) \)
Compute this product:
\[
(1)(2) + (1)(2i) + (-2i)(2) + (-2i)(2i) = 2 + 2i - 4i - 4i^2 = 2 - 2i - 4(-1) = 2 - 2i + 4 = 6 - 2i
\]
And the right side is \( \frac{20}{3+i} = 6 - 2i \), as we calculated earlier.

So, both sides match, confirming that \( z = 1 + 2i \) is indeed the correct solution.

Final Answer

\[
1 + 2i
\]
To solve the problem, let’s first set up the equation and use the given conditions to find the complex number \( z \).
Given: \( \overline{z}(z + 1) = \frac{20}{3+i} \)
Let’s start by simplifying the right-hand side of the equation:

\[
\frac{20}{3+i} = \frac{20(3-i)}{(3+i)(3-i)} = \frac{20(3-i)}{3^2 - i^2} = \frac{20(3-i)}{9+1} = \frac{20(3-i)}{10} = 2(3-i) = 6-2i
\]

So, the equation becomes:

\[
\overline{z}(z + 1) = 6 - 2i
\]

Let \( z = x + yi \) where \( x, y \in \mathbb{R} \) and \( x > 0 \). Then, \( \overline{z} = x - yi \).
Substituting \( z \) and \( \overline{z} \) into the equation:

\[
(x - yi)((x + yi) + 1) = 6 - 2i
\]

Simplifying the left-hand side:

\[
(x - yi)(x + 1 + yi) = (x - yi)(x + 1 + yi)
= x(x+1) + xyi - yix - y i^2 + (-yi)(yi)
= x(x+1) + y^2 + xyi - xyi
\]

Since \( i^2 = -1 \), we get:

\[
x(x+1) + y^2 + xyi - xyi = x^2 + x + y^2
\]

So, the equation becomes:

\[
x^2 + x + y^2 = 6 - 2i
\]

Separating the real and imaginary parts, we have:

\[
x^2 + x + y^2 = 6 \quad \text{and} \quad 0 = -2i
\]

Since the imaginary part of the left-hand side is zero and the imaginary part of the right-hand side is \( -2i \), it must be that the imaginary part of \( z \) is such that it cancels out. This means \( y = -2 \).
Substituting \( y = -2 \) into the real part equation:

\[
x^2 + x + (-2)^2 = 6
\]
\[
x^2 + x + 4 = 6
\]
\[
x^2 + x - 2 = 0
\]

Solving the quadratic equation \( x^2 + x - 2 = 0 \):

\[
x = \frac{-1 \pm \sqrt{1+8}}{2} = \frac{-1 \pm 3}{2}
\]

So, \( x = 1 \) or \( x = -2 \). Since the real part of \( z \) is greater than 0, we have \( x = 1 \).
Thus, \( z = 1 - 2i \).
The solution is:

\[
\boxed{1 - 2i}
\]
![Four line plots showing CoE-Score for QwQ-32B-Preview/CoE-R-Score, QwQ-32B-Preview/CoE-C-Score, Qwen2.5-32B-Instruct/CoE-R-Score, and Qwen2.5-32B-Instruct/CoE-C-Score, with true answer and false answer lines.](page_186_120_1077_1042.png)

Figure 13: CoE-Score of QwQ and Qwen on Question 3.

Question 4

Integers \(a\) and \(b\) are randomly chosen without replacement from the set of integers with absolute value not exceeding 10. What is the probability that the polynomial \(x^3 + ax^2 + bx + 6\) has 3 distinct integer roots?
So I’ve got this probability problem here. It says that integers \(a\) and \(b\) are randomly chosen without replacement from the set of integers with absolute value not exceeding 10. So, first things first, I need to understand what that set is. If the absolute value of the integers doesn’t exceed 10, that means they can range from -10 to 10, inclusive. So the set is \(\{-10, -9, -8, \ldots, 9, 10\}\). That’s 21 integers in total.

Now, we’re choosing \(a\) and \(b\) without replacement from this set, meaning that \(a\) and \(b\) are distinct integers from this set. So the total number of possible pairs \((a, b)\) is \(21 \times 20 = 420\), since there are 21 choices for \(a\) and then 20 remaining choices for \(b\).

The problem is asking for the probability that the polynomial \(x^3 + ax^2 + bx + 6\) has 3 distinct integer roots. So, for the polynomial to have 3 distinct integer roots, it must be factorable into \((x - r)(x - s)(x - t)\), where \(r, s, t\) are distinct integers.

Let’s expand \((x - r)(x - s)(x - t)\) to see how the coefficients relate to \(a\) and \(b\):

\[
(x - r)(x - s)(x - t) = x^3 - (r + s + t)x^2 + (rs + rt + st)x - rst
\]

Comparing this to the given polynomial \(x^3 + ax^2 + bx + 6\), we can see that:

\[
\begin{cases}
-(r + s + t) = a \\
rs + rt + st = b \\
-rst = 6
\end{cases}
\]

All these calculations seem correct.

Now, is there any possibility that different sets of roots could lead to the same \((a, b)\) pair? For example, if two different triplets of roots result in the same \(a\) and \(b\), then that would mean that pair is counted only once in the favorable outcomes.

But in our case, each set of roots gives a distinct \((a, b)\) pair, as we can see:
- \((0, -7)\)
- \((-2, -5)\)
- \((-6, -1)\)
- \((-4, 1)\)

No overlaps here.

Therefore, the number of favorable outcomes is indeed 4.

Total possible outcomes are \(21 \times 20 = 420\).

Thus, the probability is \(\frac{4}{420} = \frac{1}{105}\).

Wait, but perhaps I should consider that \(a\) and \(b\) are chosen without replacement, meaning that \((a, b)\) and \((b, a)\) are different unless \(a = b\), which isn’t the case here. But in our favorable outcomes, each \((a, b)\) is unique and distinct from \((b, a)\), since none of the \(a\) and \(b\) pairs have \(a = b\). Therefore, no need to adjust for that.

Alternatively, if the problem considered \((a, b)\) and \((b, a)\) as the same, but since it’s about choosing without replacement and the polynomial has distinct roots, I think the ordered pairs are appropriate here.

So, I think \(\frac{1}{105}\) is the correct probability.

Final Answer

\[
\boxed{\frac{1}{105}}
\]
To solve this problem, we need to analyze the conditions under which the polynomial \( x^3 + ax^2 + bx + 6 \) has three distinct integer roots.

1. Identify the Set of Integers: The set of integers with absolute value not exceeding 10 is \( \{-10, -9, -8, \ldots, 9, 10\} \). There are 21 integers in this set.
2. Polynomial Roots: Let the roots of the polynomial \( x^3 + ax^2 + bx + 6 \) be \( r, s, \) and \( t \). By Vieta’s formulas, we know:

\[
r + s + t = -a, \quad rs + rt + st = b, \quad rst = -6
\]

3. Finding Integer Roots: The integer roots \( r, s, \) and \( t \) must satisfy \( rst = -6 \). We list the possible sets of integer roots:

\[
\{1, 2, -3\}, \{1, -2, 3\}, \{-1, 2, 3\}, \{-1, -2, -3\}
\]

4. Counting Valid Combinations: We need to count the number of ways to choose \( a \) and \( b \) such that the polynomial has these roots. Each set of roots \( \{r, s, t\} \) must be chosen from the set \( \{-10, -9, \ldots, 9, 10\} \) without replacement.

5. Calculate \( a \) and \( b \) for Each Set: For each set of roots, calculate \( a \) and \( b \):
- For \( \{1, 2, -3\} \):
\[
r + s + t = 1 + 2 - 3 = 0 \implies a = 0
\]
\[
rs + rt + st = 1 \cdot 2 + 1 \cdot (-3) + 2 \cdot (-3) = 2 - 3 - 6 = -7 \implies b = -7
\]
- For \( \{1, -2, 3\} \):
\[
r + s + t = 1 - 2 + 3 = 2 \implies a = -2
\]
\[
rs + rt + st = 1 \cdot (-2) + 1 \cdot 3 + (-2) \cdot 3 = -2 + 3 - 6 = -5 \implies b = -5
\]
- For \( \{-1, 2, 3\} \):
\[
r + s + t = -1 + 2 + 3 = 4 \implies a = -4
\]
\[
rs + rt + st = (-1) \cdot 2 + (-1) \cdot 3 + 2 \cdot 3 = -2 - 3 + 6 = 1 \implies b = 1
\]
- For \( \{-1, -2, -3\} \):
\[
r + s + t = -1 - 2 - 3 = -6 \implies a = 6
\]
\[
rs + rt + st = (-1) \cdot (-2) + (-1) \cdot (-3) + (-2) \cdot (-3) = 2 + 3 + 6 = 11 \implies b = 11
\]

6. Total Possible Choices for \( a \) and \( b \): The total number of ways to choose \( a \) and \( b \) from the set of integers \( \{-10, -9, \ldots, 9, 10\} \) without replacement is:
\[
\binom{21}{2} = \frac{21 \times 20}{2} = 210
\]

7. Probability Calculation: There are 4 valid combinations of \( (a, b) \) that make the polynomial have three distinct integer roots. Therefore, the probability is:
\[
\frac{4}{210} = \frac{2}{105}
\]
Thus, the probability that the polynomial \( x^3 + ax^2 + bx + 6 \) has three distinct integer roots is \( \frac{2}{105} \).
Figure 14: CoE-Score of QwQ and Qwen on Question 4.

Reviewer paper summaries: This paper addresses the gap between LLM benchmark performance and real-world application efficacy by proposing G-Pass@k, a metric evaluating both accuracy and consistency in complex reasoning. Through extensive experiments on mathematical tasks, the authors demonstrate that current LLMs exhibit significant instability (e.g., performance drops up to 90% under stringent thresholds), challenge the assumption that model scaling improves stability, and reveal a disparity between theoretical potential and practical consistency. The work introduces LiveMathBench, a bilingual benchmark to mitigate data leakage, and analyzes factors like sampling parameters and question difficulty.

This paper proposes a new evaluation metric, G-Pass@K_\tao, to quantify the stability of a model’s reasoning and better evaluate LLMs. G-Pass@K_\tao is defined as the expectation of the correctness percentage in generated K samples that exceeds threshold \tao. The traditional Pass@K can be treated as G-Pass@K_{->0}, i.e., it captures the probability of obtaining at least one correct sample out of K generations. The paper also introduces mG-Pass@K, which aggregates the G-Pass@K_\tau metric by computing the area under the curve as \tau varies from 0.5 to 1.

By using the proposed metrics, this paper finds that 1) LLMs performance is weaker when evaluated with G-Pass@K_\tao (\tao>=0.5) compared to their greedy accuracy; 2) increasing model size does not necessarily increase the model stability as measured by G-Pass@K_\tao; 3) there is a large gap between G-Pass@K_\tao (\tau=1) and Pass@K.

The paper at hand introduces G-Pass@$k$, a new evaluation metric designed to assess the performance consistency of LLMs. Similar to pass^$k$ ([Yao et al., 2024](https://doi.org/10.48550/arXiv.2406.12045)), G-Pass@$k$ captures the probability that all $k$ i.i.d. answers to a question are correct. Drawing inspiration from the mean Average Precision (mAP) metric, the authors extend G-Pass@$k$ by introducing a tolerance threshold, $\tau \in (0.0, 1.0]$, which allows interpolation between a model’s potential $(\tau = 0)$, captured via Pass@$k$ ([Chen et al., 2021](https://doi.org/10.48550/arXiv.2107.03374)), and its performance consistency $(\tau = 1)$, captured via pass^$k$. Evaluations of various LLMs using G-Pass@$k_{\tau}$ on math benchmarks such as MATH500-L5, AIME2024-45, 255 AIME2025, and LiveMathBench—a newly introduced bilingual math benchmark assembled from multiple competitions and examinations—show that most LLMs struggle with performance consistency.
Summary of strengths: 1. This paper introduces a new evaluation metric Pass@k to quantify both accuracy and stability.

2. This paper constructs a new benchmark, LiveMathBench, designed to reduce data contamination, which is valuable for future research.

3. Comprehensive experiments across 20+ LLMs (general, math-specific, CoT-enhanced) and multiple benchmarks (MATH, AIME, LiveMathBench) validate the metric’s robustness.

1.	Conduct extensive experiments with a wide range of LLMs.
2.	Conduct extensive experiments with sampling parameters.

The paper’s strengths are summarized below:

- By loosening the requirements of pass^$k$ ([Yao et al., 2024](https://doi.org/10.48550/arXiv.2406.12045)) and extending the metric with a tolerance threshold $\tau$ (Equation 5, ll.194–203), G-Pass@$k_{\tau}$ introduces a flexible metric that effectively generalizes Pass@$k$ ([Chen et al., 2021](https://www.notion.so/Are-Your-LLMs-Capable-of-Stable-Reasoning-1b9ecf6fd83d80bfb9bde3012e164a69?pvs=21)) and pass^$k$.
- In addition to the new evaluation metric G-Pass@$k_{\tau}$, the authors introduce LiveMathBench—a new bilingual math benchmark assembled from multiple competitions and examinations.
- The authors conduct a notable number of experiments, evaluating an extensive set of closed-source and open-access LLMs on a range of math reasoning benchmarks. While this large number of analyses results in a lack of experimental detail for many of these experiments (see *Summary of Weaknesses* for an elaboration), the effort involved in conducting such a broad array of analyses (e.g., different parameter setups, the impact of question difficulty, or the influence of fine-tuning) should be acknowledged.
Summary of weaknesses: 1. The discussion on related work regarding the evaluation of LLM stability is limited. There is a need for a more thorough survey of existing work in this area, and a comparison between the proposed metric and other related metrics such as TARr@N and TARa@N, highlighting their respective strengths and weaknesses.

2. While the paper identifies instability as a core issue (Lines 281-283), it lacks a systematic error analysis to diagnose root causes. Potential factors—decoding strategy biases (e.g., greedy vs. stochastic sampling), training data diversity, or architectural limitations—are not rigorously disentangled.

3. The proposed "improvements" (Section 4) are observational (e.g., "CoT helps") rather than prescriptive. No concrete methods are proposed to enhance stability.

1.	Unclear motivation: This paper did not elaborate well on why or in what scenarios we need G-Pass@K metric in addition to greedy accuracy (expected accuracy when we adopt the greedy decoding as the answer) or Pass@K (expected successful rate when we can have k runs to pass the system, such as in code generation as mentioned by the paper). Can we simply use the standard error/error bar of accuracy, that is widely reported in the existing literature, to measure the stability of LLMs?
2.	The data size of benchmarks is relatively limited: 200+ questions in LiveMathBench, 100+ in MATH500-L5, and only 45 in AIME2024-45.
3.	The insights are relatively limited:
•	The first key finding is that LLM performance drop when evaluated using G-Pass@K_\tau (\tau >= 0.5) compared to greedy accuracy. This is mathematically expected as G-Pass@K_\tau (\tau > 0.5) is clearly a stricter evaluation metric than greedy accuracy. Besides, greedy-coding generally have a better performance than sample-based decoding. For the G-Pass@K_0.5, the reported performance does not drop very significantly, especially given the relatively limited data size.
•	The second claim, that “increasing model size may not significantly enhance robustness” is also not surprising. It aligns with the broader observation that increasing model size does not always lead to large gains in performance, as seen in existing LLM benchmarks and prior literature (some of which the authors cite). The G-Pass@K_\tau appears to be correlated strongly with other performance metrics reported.
•	The performance gap measured by greedy accuracy and Pass@k (denoted in the paper as G-Pass@K_{->0}) is known, which explains the usage of Pass@K in addition to accuracy in the existing literature, while Pass@K_1 is clearly a much stricter metric. The harder question has a higher relative gain in Pass@k compared to accuracy is also expected from mathematical perspective.
•	Finally, the finding that performance stabilizes as the sampling size (n) increases is also expected and aligns with basic statistical principles.

A list of the paper’s weaknesses can be found below:

- Many experiments lack important methodological details in the main text, to such an extent that it is not possible to assess whether the presented findings are sound without consulting the Appendix. Examples include (not exhaustive): results presented in Section 3.3—specifically Figures 1 and 2—where it is unclear on which datasets the models are evaluated; experiments in Section 4.1, where no quantitative results are presented at all; and the analyses in Section 4.2, for which the entire methodology is relegated to the Appendix, without proper description in the main text of how the results are obtained.
- The authors’ statement: “The benchmarks in this paper include only English and Chinese, however, the conclusions are generally applicable to large language models of all languages” (ll. 579–582) is not sufficiently supported by experimental evidence. In general, the authors are advised to avoid claims such as “the conclusions are generally applicable to large language models of all languages,” as this is extremely difficult to prove.
- Some claims would benefit from supporting references:
    - “Since the emergence of large language models (LLMs), complex reasoning, particularly in mathematical problem-solving, has been regarded as the crown jewel of LLM capabilities.” (ll. 27-30).
    - “Users often regenerate responses or initiate new sessions until obtaining satisfactory answers to specific questions.” (ll. 47-50).
Comments, suggestions and typos: 1. Acronyms, such as LLM and SFT, are expanded or redefined several times in this article.

Please see above.

Below is a list of comments and questions:

- How is G-Pass@$k$, as defined in Equation 4, different from Pass^$k$ ([Yao et al., 2024](https://doi.org/10.48550/arXiv.2406.12045))? If it is the same, why did the authors decide to rename it?
- In lines 302–306, you state: “Additionally, in the larger MoE LLMs like Mistral-Large-Instruct-2411 (123B), although the scale has increased further, performance and stability have actually declined compared to Qwen2.5 72B-Instruct.” To the best of my knowledge, Mistral-Large-Instruct-2411 is not a mixture-of-experts model but a dense model.
- I would strongly advise moving parts of the experimental details into the main text. When this is difficult due to space constraints, consider shortening certain sections, such as lines 326–342, since you have already made these arguments in a previous passage.
- In Section 3.4 (2), you mention that some models like Qwen2.5-Math-72B-Instruct are highly sensitive to sampling parameters. How does this affect G-Pass@$k$?
Overall assessment scores: 3.5

2.5

3.5