A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility
Authors: Andreas Hochlehnert
Hardik Bhatnagar
Vishaal Udandarao
Samuel Albanie
Ameya Prabhu
Matthias Bethge
Venue: COLM 2025 (2025)
Keywords: data curation
rl
grpo
math
reasoning
TL;DR: we conduct a comprehensive empirical study and find that current mathematical reasoning benchmarks are highly sensitive to subtle implementation choices

Abstract: Reasoning has emerged as the next major frontier for language models (LMs), with rapid advances from both academic and industrial labs. However, this progress often outpaces methodological rigor, with many evaluations relying on benchmarking practices that lack transparency, robustness, or statistical grounding. In this work, we conduct a comprehensive empirical study and find that current mathematical reasoning benchmarks are highly sensitive to subtle implementation choices—including decoding parameters, random seeds, prompt formatting, and even hardware and software configurations. Performance gains reported in recent studies frequently hinge on unclear comparisons or unreported sources of variance. To address these issues, we propose a standardized evaluation framework with clearly defined best practices and reporting standards. Using this framework, we reassess recent methods and find that most reinforcement learning (RL) approaches yield only modest improvements—far below prior claims—and are prone to overfitting, especially on small-scale benchmarks like AIME’24. In contrast, supervised finetuning (SFT) methods show consistently stronger generalization in the settings we study. To foster reproducibility, we release all code, prompts, and model outputs, for reasoning benchmarks, establishing more rigorous foundations for future work.

Published as a conference paper at COLM 2025
A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility
Andreas Hochlehnert
Hardik Bhatnagar"
Vishaal Udandarao
Samuel Albanie
Ameya Prabhu"
Matthias Bethge"
TTbbingen Al Center, University of Tübingen
University of Cambridge
Leaderboard
Code
Eval Logs
Abstract
Reasoning has emerged as the next major frontier for language models (LMs), with rapid advances from both academic and industrial labs. How- ever, this progress often outpaces methodological rigor, with many evalua- tions relying on benchmarking practices that lack transparency, robustness, or statistical grounding. In this work, we conduct comprehensive em- pirical study and find that current mathematical reasoning benchmarks are highly sensitive to subtle implementation choices-inclle including decoding parameters, random seeds, prompt formatting, and even hardware and software configurations. Performance gains reported in recent studies fre- quently hinge on unclear comparisons or unreported sources of variance. To address these issues, propose standardized evaluation framework with clearly defined best practices and reporting standards. Using this framework, reassess recent methods and find that most reinforcement learning (RL) approaches yield only modest improvements far below prior claims aadms-anddai prone to overfitting, especially small-scale bench- marks like AIME'24. In contrast, supervised finetuning (SFT) methods show consistently stronger generalization in the settings study: To foster repro- ducibility, we release all code, prompts, and model outputs, for reasoning benchmarks, establishing more rigorous foundations for future work.
Reported vs Measured Results
Variance DS-R1-1,5B
50
-17.0%
-6.1%
46.7
-6.8%
40
43.1
43.1
Accuracy (%)
34.2
-2.2%
36.3
Range of Improvement
30
32.0
29.7
20
10
DeepScaleR-1.58 OpenR53-1.58
11-1.58
FastCuRL-1.58
Seed
Top
Temperature
Figure 1: The Sombre State of LM Reasoning for Math. (left) when re-evaluating recent 1.5B reasoning-enhancel models AIME-24 using standardized framework (see Section 3), we find substantial drops to reported results in the original papers, (right) the observed improvements from recent methods (gray highlighted area) fall entirely within the variance range (orange box plots) of DeepSeek-Ri 1.5B model performance. This suggests that these methods do not significantly outperform the base undeessoonnnninnggggggg the importance of rigorous, multi-seed evaluation protocols for obtaining reliable performance estimates.
"equal contribution, contributor, 'equal advising

---

Published as a conference paper at COLM 2025
1 Introduction
"The first principle is that you must not fool yourself, and you the easiest person to fool."
Richard Feynman
Reasoning has become central to recent advances in large language models (LLMs),, playing key role in nearly all frontier systems (Jaech et al., 2024; Anthropic, 2025; OpenAT, 2025a; xAI, 2025; Meta-AI, 2025; DeepMind, 2025). Recent months have seen surge of research focused on understanding and improving LLM reasoning, accompanied by several open-source tools and training strategies (see Li et al. (2025c) for a survey). This momentum has sparked optimism that building capable, competitive reasoning models may soon be within reach.
However, as evaluation practices shape the direction and perceived progress of the field (Liao et al., 2021), concerns around methodological rigor are growing. Non-reproduuible or inconclusive evaluations can distort scientific understanding, misguide adoption, and skew future research priorities (Henderson et al., 2018; Marie et al., 2021; Musgrave et al., 2020; Prabhu et al., 2020; Andrychowicz et al., 2020; Colas et al., 2018). In the fast-moving area of LLM reasoning, where rapid publication cycles and benchmarking races are common, methodological shortcuts can quietly undermine progress. While concerns about repro- ducibility in LLM evaluations are well-documented (Reuel et al., 2024; Biderman et al., 2024), their persistence -eppeiialccees in reasoning calls for renewed scrutiny and higher standards.
Motivated by a growing number of inconsistent empirical claims across the reasoning land- scape, we conduct a rigorous investigation into the current state of reasoning benchmarks, focusing specifically on mathematical reasoning -ne of the most widely used testbeds for evaluating algorithmic advances in this space (HuggingFaceH14, 2024; AT-MO).
Our main finding is that many recent empirical conclusions may be overly optimistic and fail to generalize under careful re-evaluation. We identify a surprising degree of sensitivity in LLM-based reasoning pipelines to seemingly minor design parameters, prompt formatting, and random seeds to the hardware and software stacks used during evaluation (see Table 1). Particularly concerning is the instability introduced by small benchmark sizes: for example, AIME'24 and AMC'23 each contain only 30-40 examples, making performance metrics highly volatile-where even one question can shift Pass@1 by over 3 percentage points. This leads to substantial variance across seeds, often resulting in double-digit performance swings that challenge the reliability of published results. In Section 2, we systematically analyze the root causes of this instability, including sampling variance, decoding configurations, evaluation frameworks, and hardware heterogeneity. We show that these factors can significantly distort conclusions if not carefully controlled.
from decoding
In Section 3, we propose a set of best practices aimed at improving reproducibility and rigor in reasoning benchmarks. We also re-evaluate recent techniques using standardized and re- producible evaluation stack. Our findings are sobering sobering-reinforeee reinfrrcemettt learning (RL) applied to distillation-based models such as DeepSeek-RR R1 yields little to no statistically significant gains. Some methods, such as OpenRS, show promising results in original reports, but fail to hold up under repeated evaluation. RL training on base models like Qwen2.5 Math does show stronger performance, but still often underperforms instruction-tuned counterparts. Furthermore, RL-trained models exhibit significant performance drops on newer bench- marks such as AIME'25, echoing patterns of test set overfitting or "hill-climbing" observed in prior work (Golchin & Surdeanu, 2023; Roberts et al., 2023; Dominguez-Olmeet 2024). In contrast, supervised fine-tuning (SFT) continues to deliver stable, generalizable improvements across benchmarks, underscoring its maturity as a training paradigm. These observations point to critical need for more reliable and standardized evaluation protocols.
et al.,
Taken together, in this work, we aim to provide not only a clearer assessment of where current methods stand, but also the tools and practices needed to make reasoning evaluation more transparent, robust, and reproducible. To this end, we open-source all code, prompts, and outputs to facilitate fair and accounaable progress in this increasingly important area.
We note that OpenReasoo is consistent exception, achieving competitive performance.
2

---

Published as a conference paper at COLM 2025
2 Exploring the Design Space of Reasoning: What Matters Most?
Recent reasoning-focte foccssd language models are evaluated under highly heterogeneous conditions-includii differences in evaluation frameworks and hardware, number of random seeds, temperature, and nucleus sampling parameters (top_p) (see Table 1). While prior work has examined the effect of sampling parameters in multiple-choice (Renze, 2024) and coding tasks (Arora et al., 2024), the influence of these choices remains underexplored for open-ended reasoning models- pareiccllal those trained with reinforcement learning. In this section, we systematically assess how these evaluation design choices affect reported performance, and highlight the sources of variance that most impact the reliability of results.
2.1 Experimental Setup
We adopt consistent experimental setup throughout this section, unless otherwise stated. Our analysis includes nine widely used models grouped into two commonly benchmarked size classes: 1.5B and 7B parameters. For the 1.5B class, we evaluate: PerpSeee-R1-Disttiin
1.5B (DeepSeek-AI, 2025), DeepScaleR-1.5B (Luo et al., 2025), 11-1.5B-Preview (Intelligent Internet, 2025), , OpenRS1-1.55, OpenRs2-1.5B, and OpenRS3-1.5B (Dang & Ngo, 2025). Note that DeepScaleR-1.5B Il-1.5--Preview, and the ÖpenRS models are all initialized from -eepSeek-RL-Disttil-l-155 and subsequently finetuned via reinforcement learning (e.g,, GRPO (Shao al., 2024)) enhance mathematical reasoning capabilities. For the 7B class, we evaluate: DeepSeek-R1-Distille 7B (Open Thoughts, 2025). Both S1.1-7B and OpenThinker-7BB are finetuned Qwen2.5-7B- Instruct models (Yang et al., 2024a), trained using supervised learning on reasoning traces derived from DeepSeek-R1. All models are benchmarked on three widely used datasets: AIME'24 (AI-MO), A3C (AT-MO, 2024), and MATH500 (Hendrycks et al., 2021), using the Pass@1 metric. Each result averaged over multiple seeds and obtained on a standardized software stack (throguh a Docker image), and hardware with the following configuration: one 40 GB A100 GPU, an AMD 7302 32-core CPU, and 1TB RAM. All experiments were run using lighteval (Fourrier et al., 2023) with the vllm backend (Kwon et 2023).
S1.1-7B (Muennighoff et al., 2025), and Open
To systematically compare the impact of sampling parameters on accuracy, all experiments in this section were performed with a standardized configuration: temperature=0.3 top_ 0009, and both max 32,768 tokens. This context length matches the limits of models such as OpenThiinkrr-78 and S1.1-7B, although certain models (e.g., DeepSeek) support longer sequences of up to 131,072 tokens. We chose this standardized evaluation length to ensure comparability, with a detailed analysis of the influence of completion length presented in Figure 6. Unless otherwise specified, results in this section are averaged over 10 random seeds for AIME'24 and AMC'23, and 3 seeds for MATH500, following the recommendations from Section 2.2.1.
Sampling Parameters:
max_mood _______oe_______ len and max_ _ew tokens set to
2.2 Seed Variance in Evaluation
AIME24
AMC23
0.86
0.5
0.4
033
0.9
0.4
0.8
0.80
0.88
0.3
0.86
0.2
0.4
082
DeepSosieR-1.5 D1-psll .B
OpenR81-1.8 5B
OpenRS3-1.55 1.50
Open Thinkkerr-
OperR82-1.55 5B
[|-.5B
S1.1-78
Figure 2: Accuracy varies significantly across random seeds. We find significantly high Pass@1 variation across 20 different random seeds for nine models on AIME'24, AMC'23, and MATH500. Variance is particularly high on AIME'24 (upto 15%) and AMC'23 (upto 13%) due to the small number of test samples, highlighting instability of single-seed evaluations.
3

---

Published as a conference paper at COLM 2025
Table 1: Taxonomy of current open-weight reasoning models. For each model, we report the base model it was post-trained from and the exact type of post-training algorithm applied (RL vs SFT). Further, we note the evaluation framework that the original paper uses for reporting results along with the exact temperature, generation sequence length, and top_p sampling parameters used for AIME-24 evaluation, with the number of generations used for computing Pass@1 (K). It is evident that there is no clear standardization across different models with respect to evaluation frameworks used and the sampling parameters. This motivates the need to closely scrutinize the evaluations of current reasoning models.
Model
Algorithhm
Base
Framework Temp Top__
Seq.
Len K
DeepSeek-RI-Distilll
SFT
Qwen2.5-Math-1155
0.6
0.95
32,768
64
DeepSeck-R1-Dee R1-Distill-7B
SFT
Swen2.5-Math-76
0.6
0.95
32,768
64
DeepSeek-R1 Dabbcc--R-Distill-11B
SFT
Qwen2.5-14B
0.6
0.95
32,768
64
DeepSeek-R1-Distill-ee
SFT
Owen22.5-32B
0.6
0.95
32,768
64
OpenThinker--21B
SFT
Swen2.5-32--Instruc
evalchemy
0.7
32,768
Respoke-Stratos-228
SFT
Dwen2.55-32--nsttruut
evalchemy
0.7
32,768
Bespoke-Stratos-78
SFT
Qwen2.5-7B-Ine
evalchemy
0.7
0.8
32,768
5
s1.1-7B
SFT
Qwen2.5-7B:
2an2.5-7B-Instruct
Lm-eval-harness
0
32,768
64
s1.1-32B
SFT
Bree2.5-32B-instaae Lm-eval-harness
0
32,768
64
LIMO
SFT
Qwen2.5-3225
Insttsssttttt math-eval-harnee
32,768
MiniMatt--R1115B
oumi-ai
Light-RI-7B
SFT
DeepSeck-KI-Distill777
verl 0.6
0.95
32,768
DeepScalek-1.B--Preveeew
RL DeepSeek-KL-Distill--115
verl
0.6
0.95
32,768
16
L1-Exact
RI. DeepSeek-KI-Distill11.1
verl
0.6
0.95
8,000
L1-Max
RI. DeepSeek-Rl-Distiil-t
verl
0.6
0.95
8,000
Open-RS1
RL DxepSeck-R1-Distill.
lighteval
0.6
0.95
32,768
32
Open-RS2
RL SxppSeck-R1-Disttile
lighteval
0.6
32,768
Open-RS3
RL
lighteval
0.6
0.95
32,768
32
J1Thouugh:1.5B-PPeview
RL DeepSeek<KI-Distill1..2
evalscope
0.6
0.95
32,768
64
Oat-Zero-1.5B
RL Qwen2.5-Math-1158
custom
0
3,000
Oat-Zero-7B
RI.
Qwen2.5-Math-7R
custom
0
3,000
STILL--3115B-prrview
RL DeepSeek-Ri
custom
0.6
32,768
FesiCurt-1.5--Preview
RL DeepSeek-RL-Distil--
verl
0.6
0.95
32,768
16
I.MRR
RI.
Qwen2.5-Math--T
custom
0.4
0.95
3,072
4
SimpleRI-Zoo-P7 Zoo-7
RL
Qwen2.5-7B
verl
1
0.95
16,000
32
SimpleRL-Zov Zoo-Math-77
RL
Owen2.5-Math-/
verl
1
0.95
16,000
32
OpenReasoner- Zero-7BB
RL
Qwen2.7-7B
Eurus2 Prime
RL
Qwen2.5-Math-7B
custom
4,096
We begin by analyzing the variance induced purely by the random seed used during evaluation-an aspect often neglected in benchmarking practices. While recent work calls for statistical rigor (e.g., using error bars and multiple runs) (Bowyer et al., 2025; Biderman et al., 2024; Madaan et al.), evaluations frequently rely on single-seed runs, obscuring potential variability, We assess the seed-induced variance across 20 independent evaluation runs for each of the nine models. Results are shown in Figure 2.
Key Insight. Pass@1 values show surprisingly high standard deviation-rangee maationn-aaggng from 5 to 15 percentage points across seeds. This issue is particularly severe for AIME'24 and AMC'23, which have only 30 and 40 test samples respectively. A change in just one question shifts Pass@1 by 2.5-3.3 percentage points.
Takeaway 1 Single-seed evaluations on small datasets are highly unstable. Accurate reporting requires averaging over multiple seeds.
Takeaway 2 Small datasets such as AIME24 (30 samples) make model comparisons
unreliable, as solving just one extra question already shifts pass@1 by 3%. Variance from sampling parameters or random seeds can easily cause fluctuations of 1-2 correct answers, leading to unstable rankings - especially when models cluster around 30% performance.

---

Published as a conference paper at COLM 2025
Variance of Means
DeepSeek-R1-Diie -1-issilllis
DeepScaleR-1.58
FastCuRL-1.5B
20
20
20
10
20
K 60
40
20 40
60
40
60
K
K
Figure 3: Bootstrapped seed averaging is reliable only beyond a threshold. We plot the variance of Mean Pass@1 scores on AIME'24 when averaging over K = 1 to K = 60 seed runs, finding that the variance is extremely high for small K and significantly reduced by K = 30. This suggests that using multi-seed evaluations (K 2 30) would yield more stable estimates. For results on AMC23 and MATH500 see Figures 12 and 14 respectively.
2.2.1 Can Bootstrapping Improve Mean Estimates?
To mitigate high variance, recent work has adopted bootstrapping- averaging multiple evaluation runs stabilize results. For example, DeepSeek reports Pass@1 over 64 runs, while DeepScaleR uses 16. We study the effectiveness of this approach by bootstrapping estimates for AIME'24 using 60 evaluation runs. Figure shows that while variance extreme for K = 1 and still large for K = 5, it reduces sharply for K 2 30. Further analysis of variance across additional datasets presented Figures 12 and 14.
Takeaway 3 Bootstrapping over 30 runs substantially stabilizes Pass@1 estimates and should be considered minimal standard for reliable evaluation.
2.2.2
Variance from Sampling Parameters: Temperature and top-p
AIME24
AMC23
MATH500
3.25
2.4
0/
3.755
8.800
0.3
01
DeepScaleR-1.05
OpenRS1-1.58
OpenRS3-1.5B
OpenThinker-7B
DeepSeek-Rf- DoopSee---f-iiil----
DeepSeckRR.On 1.5
OpenRS2-1.5B
S1.1-7B
Figure 4: Higher temperatures yield better accuracies. We find across all three datasets, higher temperatures produce better peak accuracy but introduce instability, revealing tradeoff between performance and reproducibility. Results obtained by varying temperature from 0 to 1 in increments of 0.1, while keeping top_p fixed at 0.9.
Reducing the temperature or increasing the nucleus sampling parameter (top_p) improves the accuracy of performance estimates without incurring additional computational cost. Figure 4 illustrate the impact of temperature and Figure show that of top_p across multiple models and datasets. Notably, more reproducible estimate associated with significant drops in measured performance, highlighting a consistent tradeoff between reproducibility and high performance. We recommend optimizing the temperature for performance, and comparing the best parameter per model.
Takeaway 4 Temperature and top_p can introduce substantial performance varia- tion especcalcll on small benchmarks and should be set to each model's optimal values to ensure fair and stable evaluation.

---

Published as a conference paper at COLM 2025
AIME24
AMC23
MATH500
0.60
03
1/4
04
0.255
02
DnepScalnR-1..
OpenRS1-1.58
OpenRS3-1.55
OpenThinkar-7B
DoepSock-FR Distill-8
ReepSeck-R RrepSeek-R--isiil1... Disi
OpenRS2-1.58
111.5B
S1.1-7B
Figure 5: Higher top_p values improve performance at no cost stability. Across datasets, we find that higher top_p values generally improve performance while preserving similar amounts of variance as lower top_p values. Results were obtained by varying top_p from 0 to in increments of 0.1, while holding the temperature constant at 0.8.
AIME24
AMC23
MATH500
0.6
0.9
0
T
0.95
0.5
Accuracy
0.4
0.85
0.3
0.2
DempScatehc1..
0.5
DeepSeck-R1 Dieiii-..5 58
0.75
0.4
4096
16:04 32/68 85536131072
4096
8192
32/68 65536 131072
4096
8192 16384
65536
Figure 6: Models are extremely sensitive output token lengths. We sweep across different max_new_ tokens (number of tokens that models are allowed to generate) for DeepScaleR- 1.5B and KhepSeek-R1-Distill-.. sensitive to output length limits, with premature truncation degrading the performance.
three datasets and find that they heavily
2.3 Effect of Prompt Format and Context Length
Maximum Output Tokens. Figure shows that reducing max_new_ tokens harms perfor- mance-especiall on long-form problems. This sensitivity varies by model and dataset. Although reducing this setting lowers cost, it may induce premature stopping, leading to incorrect answers.
PeepScaleR-1.. R-w.aale-------PPP -.-e-ee
DeepSeek-Re
DeepSeek- Braiieekk-R---ee -i-il
0.9
0.8
0.7
0.8
Accuracy
0.6
MATHSOO
AMEZ3
0.6
0.7
0.5
0.6
0.4
0.5
0.3
0.2
0.4
Template
Template
Figure 7: Using no prompt templates yields worse performance. We compare Pass@1 scores across three prompt formats: (1) math-specific prompt with chat template, (2) default chat template only, and (3) no template. Instruction-turee models perform best with structured prompts and templates; omitting templates leads consistent performance drops.
Prompt Format, Prompt formatting has a measurable impact on accuracy. As shown in Figure 7, models perform best when using math-specific prompts and their native chat templates. Omitting templates leads to performance drops, particularly for instruction- tuned models. We compare accuracy under three different prompt settings (see Table 5) (1)

---

Published as a conference paper at COLM 2025
math-specific prompt formatted using the model's chat template, (2) only the model's chat template with no additional prompt, and (3) no template at all, ie,, the question without any special tokens instructions. Interestingly, while base models like Qwen2.5-Math may benefit from prompt-free setups (Liu et al., 2025b), instruction-tuned models rely heavily on format alignment. Thus, maintaining consistent and format-aware prompting is essential for maximizing instruction-tune model performance.
Takeaway is critical to use large generation context lengths to avoid output truncation which can degrade performance; further, using correct prompt formats and chat templates is important for extracting best model performance.
3 Way Forward: Standardization in Evaluations
In this section, we standardize evaluation frameworks, propose best practices, and compre- hensively evaluate existing methods.
3.1 Standardization Procedure
We adopt a largely consistent experimental setup with prior work, with the key difference be- ing our use of publicly accessible cloud instances from Runpod?. Each instance is equipped with single A100 PCIe GPU, 8 vCPUs, and 128 GB of RAM. We evaluate all models listed in Table 2 across six benchmarks: AIME'24 (AT-MO), AIME'25 (Lin, 2025), AMC'23 (Knovel Engineering, 2025), MATH550 (HuggingFacehh4, 2024), Minerva (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024). All experiments are conducted using the LightEval framework (Fourrier et al., 2023) (0.8.1) with vLLM backend, repeated across ten random seeds for A4ME'24, AIME'25, AMC'23 and three random seeds for the rest. Depending on the base model architecture, we set the maximum number of new tokens (e.g., 4096 for Qwen -badee models), apply optimal hyperparameters, and use the standardized chat template except for base models. LightEval's LaTeX-based answer extraction and evaluation pipeline ensures reliable and consistent result parsing and correctness matching, similar to math-verife fy.
Statistical Significance Testing. To rigorously assess whether RL-trained models genuinely outperform their baselines, we employ paired sample-wise statistical tests rather than sim- ply comparing mean accuracies. For each model-baseline pair, we: (1) compute the average accuracy per problem instance across all random seeds, (2) align these sample-wise accura- cies between the RL/SFT model and the baseline from which it was trained, and (3) conduct one-sided paired tests to evaluate whether the RL model achieves systematically higher performance. We employ both parametric (paired t-test) and non-parametric (Wilcoxon signed-rank) tests to ensure robustness to distributional assumptions. The null hypothesis is that there is no improvement from RL/SFT training, with the alternative hypothesis that the RL/SFT model performs better than its baseline. We report results as statistically significant at two thresholds: < 0.01 (marked with ") and p < 0.001 (marked with **) This conservative approach ensures that reported improvements represent genuine performance gains rather than statistical artifacts from multiple testing or random variation.
3.2 A Sober Look: Results
We present experimental results in Table 2, and analyze different aspects of the results.
We evaluated several reinforcement learning (RL) approaches (e.g., GRPO) using the DeepSeek R1-Distilll115B model. We first observe that none of the L1 models (Aggarwal & Welleck, 2025) outperformed the original DeepSeek R1-Distill baseline - an expected outcome given that [11 training prioritized smaller output length over accuracy. OpenRS (Dang & Ngo, 2025) reported strong gains (10-15%) on AIME, AMC, and OlympiadBench. However, our replication showed no statistically significant
RL-training on R1-Distill
7

---

Published as a conference paper at COLM 2025
Model
AIME'24 AIME22 AMC'23 MATH500 Minerva Olympiad
Based on: Deepseek R1 Distill Qwen 1.5B (RL)
R1-Distill (DeepSeek-AI, 2025)
28.7148
22.3=52
71,5:39
84,9=03
30.5:10
52.4-04
L1-Exact (Aggarwal & Welleck, 2025)
24,4133
22.3-42
70.5
86.6108
31.5
52.5-13
L1-Max (Aggarwal & Welleck, 2025)
27.71112
21.0.5e
73.2160
84.7101
33.3109
52.3_06
Open-RSI (Dang & Ngo, 2025)
28,916.
21.3-12
75.0133
85,1
30.4102
53.2-19
Open-RS2 (Dang & Ngo, 2025)
31.3+77
22.7-56
73.0+57
84,1=02
29.2+1.1
53.7-06
Open-RS3 (Dang & Ngo, 2025)
29.7:44
24.7=66
69.2:55
84.2=111
28.6:23
51.8-08
STILL-3 (Min et al., 2024)
34.715.5
24.0.664
72.5154
86.6119
30.0106
53.9-15
ZR1-1.5B (Zyphra, 2025)
30.3116
24.0..38
78.811.
86.6109
32,110.9
56,4101
I1-Thought (Intelligent internet, 2025)
32.0+59
24.0-41
79.5+31"
86.6+06
31.7+06
54.9+n4'
DeepScaleR (Luo et al., 2025)
37.0+66
30.3+43"*
76.2+46
87.8-10"
31.0+15
55.5+11
FasiCuRL (Song el 2025)
36.3:46
26.5-37
76.9
87.5-10"
30.8:16
56.7:08"
DisCO-logL (Li et al., 2025a)
39.01455
31.7148*
78,8134"
87 37."
32.7113
58.6103
Nemotron-RA (Mingjie Liu, 2025)
48.3+63"
33.0+5.1
87.0+48"*
91.0-a5*
37.5+16"
62.8+a9*
Based Deepseek R1 Distill Qwen 7B (RL)
R1-Distill (DeepSeek-Al, 2025)
52.3+663
39.0=5s
91.5+27
94.1=03
40.1+04
67.3-01
Sky-T1 (mini) (NovaSky, 2025)
51.3+45
39.0-57
90.0+33
93.3-10
41,8+1.2
67.7-09
DisCO-logl. (I.i et al., 2025a)
51.3+39
41.3-45
92.2+22
94.0-13
44.7+1.1*
67.6-03
Skywork-OR1 (He et al., 2025)
60.7+47"
49.7+6.2
94.0+38
95.7+a0*
43.0-10"
73.6=09 **
Based on: Deepseek R1 Distill Qwen 7B (SFT)
R1-Distill (DeepSeek-AA, 2025)
52.3+6.
39.0-59
91.5+22
94.1+03
40.1+04
67.3-01
Light-RI (Wen et al., 2025b)
53.0148
41.0=35
90.0a331
93,5205
41.3+1.
68.0=12
Based on: Qwen2.5 Math 1.5B (RL)
Math (Base) (Yang et al., 2024b)
11,3+36
5.7-27
44.0+49
51,7+55
11,3+22
26.0-06
Oat-Zero (Liu et al., 2025a)
16.0a32
6.7=34
52.5429
73.5-17**
26.32005*
37.2+13*
Math (Instruct) (Yang et al., 2024b)
12.0+1.7
11.7=5.7*
54.8453
74.7=05**
26.7+18"*
37.9+02*
LUFFY-Zero (Yan et al., 2025a)
15.9+49
11.7=2a
54.5+34
77.6=12*
27.2+0.4*
42.3+a5*
Based on: Qwen2.5 Math 7B (RL.
Math (Base) (Yang et al., 2024b)
20.7
8.7=39
56.2+5.
64.3=05
17.3#19
29.0
SimpleRL-Zoo (Zeng et al., 2025b)
22.7+5.2
10.7=34
62.2+36
76.9-18*
30.1+25**
39.3+a6"*
LIMR (Li et al., 2025b)
30.7132
7.8133
62.2134
76.5_0.**
34.9113**
39. 39.3109*
Oat-Zero (Liu et al., 2025a)
28.013.1
8.8.25
66,2136
79.4-03*
34.411.1 43.8
Sky-T1 (NovaSky, 2025)
19.3+26
21.0=a9*
67.2+3.**
85.2-05*
34.7a111
52.1+0s"*
Math (Instruct) (Yang et al., 2024b)
15.7
10.7=38
67.0+39
82.9_a1"
35.0=06**
4
LUFFY-Zero (Yan et al., 2025a)
26.7166
23.315,7*
73.5139**
87.9_01*
34.1122** 54.9103**
Based on: Qwen2.5 1.5B (RI.)
Qwen (Base) (Yang et al., 2024a)
0.0
0.0.00
2.5125
3.3
1.8104
1.5105
SimpleRL-Zoo (Zeng al., 2025b)
0.311.1
0.3
13.2147
62.0.6s**
4.0124
4.2120**
Qwen (Instruct) (Yang et al., 2024a)
1.3.1.7
0.7
26.2118.*
58.1_11"
19.411.
20.4105**
Based on: Qwen2.5 7B (RL)
Qwen (Base) (Yang et al., 2024a)
8.0132
3.7137
35.8149
59.7103
21.4119
27.0_10
SimpleRL-Zoo (Zeng al., 2025b)
14.0121
4.3.27
58.011s"
77.9_os*
33.0102"
39.0101**
Open Reasoner Zero (Hu al,, 2025)
19.7129
15.7-27
59.5
83.9_11**
31.6113*
47.6111*
Qwen (Instruct) (Yang et al., 2024a)
12.3+32
7.3:34
52.8:4.**
07
34.9+10**
"
Based on: Qwen2.5 7B Instruct (SFT)
Qwen (Instruct) (Yang et al., 2024a)
12.3+32
7.3+34
52.8+48
77.3+08
34.9+1.
38.9-a9
Eurus2 Prime (Cui et 2025)
18.7+17
14.3=16
64.8+40
80.1-a1*
37.5+1.**
*
s1.1 (Muennighoff et al., 2025)
19.0132
21.0u55*
59.5
80,8:06"
37.5a1.
48.2:14"
Bespoke Stratos (Bespoke Labs, 2024)
20.3143
18.0_48
60.2149
84.7_os**
39.1_13"
51.9111*
Open' (Open Thoughts, 2025)
30.5+62**
26.0++44*
71.4+39**
88.3-14*
37.9+3s**
55.6+14
OpenR1 (Hugging Face, 2025)
4
3
86.0+45**
93.5-a7"
4 *7.4+13**
OpenThinker2 (Open Thoughts, 2025)
53.0+46"*
41.0=50** 87.0=a5"*
94.4-77* 42.0+15"*
70.6=10"*
Table 2: A Standardized and Sober Compilation of LM-Reasoning Results. We report Pass@1 accu- racy (mean i std) of all models across six math reasoning benchmarks under a standardized evaluation setup. RL- and SFT-based variants are evaluated relative to their respective base or instruction tuned models. Main takeaways are that (1) RI-ttrineed methods do not yield meaningful performance gains, (2) SFT on reasoning traces yields significant generalization. lization. Note that * statistically significant at < 0.01; ** statistically significant at p < 0.001 (paired t-test relative to base model).
improvements over the R1 Distill baseline. Same case held for Still-3 and Light-R1 model, which showed no significant improvement over the R1-Distill baseline. Ih-Thought yields modest improvements across benchmarks, especially over ATME'24 but the observed gains
8

---

Published as a conference paper at COLM 2025
Model
AIME'24 AIME'25 AMC MATH500 Minerva Olympiad
Based on: Qwen2.5 32B (RL)
Qwen (Base) (Yang et al., 2024a)
8.0
2.3122
36,5168
62.8:35
28.9=22
29.5:200
DAPO (Yu et al,, 2025)
43.0=40"*
32.3+6.1*
88.5+36"
91.0=a3*
40.0=06"*
60.9+111*
Qwen (Instruct) (Yang et al., 2024a)
15.7+47
13.0+46
60.8+57
81.6+05
40.3+1.3
46.7+13
Based Qwen QwQ 32B (RL)
QwQ-32B (Team, 2025)
76.3+33
69.0+45
96.2+24
97.5+06
49.0=a2
78.1+10
INTELLECT-2 (Prime Intellect, 2025)
75.2
66.3166
96.0127
97.010.
49.8
78.0.os
Based on: DeepSeek R1 Distill Qwen 32B (SFT)
Distill R1 (DeepSeek-AI, 2025)
67.0119
55.3157
96.8121
95.1107
45.1117
73.810s
IinyRI-Preview (Sun et al., 2025)
73.3+44
66.0-66"
97.8+22
97.2+n4*
48.5+0.'
76.3+1.2
Light-R1 DSS (Wen al., 2025a)
76.7+47*
68.7+5,7*
97.0+16
97.3+a1*
48.2+111
76.9+0**
Based on: Qwen2.5
32B Instruct (SFT)
Qwen (Instruct) (Yang et al., 21124a)
15.7+47
13.0+46
60.8+57
81.6+05
40.3+113
46,7+13
Sky-TI-Preview (NovaSky, 2025)
37.0as.1*
22.7
75.8
88 $8.6205**
39.8=12
56.310s
Bespoke-Stratos (Bespoke Labs, 2024)
37.7
26.3273"
80.7+39**
90.3s03**
43.3=15
61.5#17*
LIMO (Ye et al., 2025)
57.81477* 46.3155
93.0128**
94.6112*
45,2123*
70.5
s1.1-32B (Muennighoff et al., 2025)
61.3169**
49.0169**
94.213.**
94,8105**
46.2.08**
72.4107**
OpenThinker (Open Thoughts, 2025)
68.5167 50.3162 95.0129*
95.5203*
46.8112
71.8105
OpenThinker2 (Open Thoughts, 2025)
71.3+6. 61.7+53** 95.8+25**
96.1=04**
46.6+06
75.5+06
Standalone Models (No tools)
GPT-OSS-20B (medium) (0penAA,,0255) 75.7+30
72.7+54
97.0+33
96.5+08
45.1+1.1
77.0+07
Table 3: A Standardized and Sober Compilation Lerge-Scale (20B-32B) LM-Reasoning Results. We report Pass@1 accuracy (mean + std) of all 32B-scale models across six math reasoning benchmarks under a standardized evaluation setup. RL- and SFT-based variants are evaluated relative to their respective base or inttructtinntunne models. Note that statistically significant at p < 0.01; statistically significant at p 0.001 (paired t-test relative base model).
did not carry over significantly to AIME'25 indicating overfitting to existing benchmarks. Only DeepscaleR and FastCuRL demonstrated statistically significant improvements across many benchmarks. Notably, recent models like Nemotron-RR deliver the first recipes with robust improvements over the DeepSeek R1-Distill baseline.
Takeaway 1 Most RL-trained variants of the DeepSeek R1-Distill model do not yield meaningful performance improvements (except DeepscaleR and FastCuRL), suggesting that reliable and scalable RL training recipes were lacking, Nemotron-RR, in contrast, now provides robust improvements.
RL Training on Qwen2.5 Math and Base Models: We next analyze RL. training applied to the Qwen2.5 Base and Qwen2.5 Math Base models, trend trying to replicate gains by Deepseek-R1 Zero. Unlike the R1-Distill results, RL training with Oat-Zero, LIMR, and SimpleRL-Zoo consistently produced statistically significant gains over the base model, especially across Math500, Minerva and OlympiadBench benchmarks. This indicates that RL-based approaches can indeed offer substantial improvements given a base model instead of distilled R1 model. However, these gains are often roughly comparable and sometimes slightly higher than achieved via instruction tuning in the original Qwen papers, suggesting that instruction tuning with additional math data alone may be sufficient to far achieve the current gains from RI. methods in this setting. We also observed that the improvements on AIME'24 were also significant, but did not carry over to AIME'25 indicating a troubling overfitting trend.
Takeaway While RL-trained methods can often substantially improve base model per- formance, instruction tuning remains superior (except Open Reasoner Zero), suggesting again that reliable and scalable RL training recipes are still lacking.
Effectiveness of Supervised Finetuning, We assessed supervised finetuning methods like 1,.,, Eurus2 Prime, Bespoke Stratos, OpenR1 and OpenThinker models, which further

---

Published as a conference paper at COLM 2025
refine instruction-tuned models using reasoning traces. Supervised methods consistently outperformed the instruct-tuned baseline across all benchmarks (even Minerva) and gener- alized comparatively well to AIME'25. The performance improvements from OpenThinker series and OpenRl were especially notable, showcasing the promise of data curation for supervised finetuning. These results underscore the maturity and effectiveness of SFT when training recipes are scaled to large datasets.
Takeaway 3 Supervised finetuning reasoning traces from larger models yields sig- nificant, generalizable gains across benchmarks with progress over time successfully replicated - highlightttngg its robustness and maturity as training paradigm.
Scaling to Larger Parameters. We extend our analysis to 32B scale models in Table 3. Among R1-Distill variants, Light-R1 DS achieves significant performance improvements, demonstrating that supervised finetuning can further improve even strong distilled models. For Qwen2.5-32B base models, SFT approaches like OpenThinker series achieve strong gains (71.3% and 68.5% on ATME'24 respectively) over the instruct model, showcasing the promise of data curation for supervised finetuning. Notably, DAPO an RL approach, shows improvements over the base model, and showcases strong performance gains even relative to the Qwen Instruct model. The QwQ-32B models demonstrate state-of-the-an performance (76.3% on AIME'24) and INTELLECT-2 shows no meaningful improvement over QwQ-32B despite additional training. These results reinforce our earlier findings: (1) SFT reasoning traces remains the most reliable approach for performance gains, and (2) scaling to larger models amplifies these benefits, with 32B models achieving substantially higher absolute performance than their smaller counterparts.
Takeaway Our findings remain robust to model scale, with SFT methods continuing to dominate RL approaches in both absolute performance and robust gains over baselines.
Overfitting and Generalization We now examine the overfitting by comparing performance on AIME'24 versus the more challenging AIME'25. RL-trained models showed a pronounced performance drop between the two, indicating overfitting to the training distribution. In contrast, supervised fine-tuning (SFT) models maintained consistent improvements, suggesting better generalization. Openthinker2 showed significant degradation compared to Openthinker across benchmarks not provided in their blogpost, indicating overfitting via data-curation. This highlights a gap in current evaluation protocols, and need to assess out-of-distribution generalization for reasoning models.
Takeaway 5 Current RL-based approaches are very susceptible to overfitting, empha- sizing the need for more rigorous out-of-distribution benchmarks. By comparison, SFT models exhibit stronger generalization and resilience.
4
Conclusion
Our study shows that much of the perceived progress in LLM-based reasoning, particularly in mathematical benchmarks, rests on unstable and often non-reproducible foundations. We find that minor differences in sampling parameters, prompt formatting, hardware, and software configurations can lead to major shifts in reported performance-cante casting doubt many recent empirical claims. Reinforcement learning methods, while promising in theory, offer at best modest gains in practice and are prone to overfitting, especially on small benchmarks like 14. In contrast, supervised finetuning continues to deliver consistent, generalizable improvements across a wide range of benchmarks and model sizes. To address these challenges, we advocate for standardized, transparent evaluation protocols. Our open- sourced framework, complete with Dockerized environments, eeer-aver metrics, and robust answer matching, provides reproducible foundations for future research. We hope this work shifts the focus from leaderboard chasing to methodological aagor-ensuring that future claims of progress in reasoning are both meaningful and measurable.
10

---

Published as a conference paper at COLM 2025
Author Contributions
Andreas, Vishaal and Ameya conceived the project. Andreas and Hardik co-led the exper- iments, with Vishaal and Ameya advising the experimental design. The manuscript written by Andreas, Hardik, Vishaal and Ameya. Matthias and Samuel provided helpful feedback and advice throughout the project.
Acknowledgments
The authors would like to thank (in alphabetical order): Matteo Farina, Shyamgopal Karthik, Nikhil Parthasarathy, Shiven Sinha, Joschka Strüber, Thaddäus Wiedemer for helpful feed- back on the draft. AH acknowledges funding by the Federal Ministry of Education and Research (BMBF), FKZ: 011S24079A. HB has received funding from the Digital Europe Programme under grant agreement No 101195233 (OpenEuroLLM)) AH, HB and VU thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for support. VU also thanks the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program for support. VU was supported by a Google PhD Fellowship in Machine Intelli- gence. AP' and MB acknowledge ledge financial support by the Federal Ministry of Education and Research (BMBF), FKZ: 01 011524085B and Open Philanthropy Foundation funded by the Good Ventures Foundation. This work was supported by the Digital Europe Programme under grant agreement No 101195233 (OpenEuroLLM).
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29322 2021.
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long reasoning model thinks with reinforcement learning, ar Xiv preprint ar Xxie=2503.00467,,2000 2025.
AI-MO, AIMO Validation AIME Dataset.
AI-MO. AIMO Validation AMC Dataset. https://hugginnfaccco//ttttt
AI-MO a a waw.oooaaaiaainonnnnnaaa 2024. Accessed: 2025-03-29.
Alibaba ModelScope Community, Evalscope documentation. https://evalscope. thadocss / /t//////laattteee Accessed: 2025-03-29.
Marcin Andrychowicz, Anton Raichuk, Piotr Starczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters in on-policy reinforcement learning? large-scale empirical study, arxiv preprint ar Xit::2006.05000 2020.
Anthropic. Claude 3.7 Sonnet System Card, 2025. URL https: :/atsp:://aastsssss anthropic.. pdf. Accessed: 2025-03-29.
Chetan Arora, Ahnaf Ibn Sayeed, Sherlock Licorish, Fanyu Wang, and Christoph Treude. Optimizing large language model hyperparameters for code generation. arXiv preprint arXiv:2408.10577 2024.
David Balduzzi, Karl Tuyls, Julien Perolat, and Thore Graepel. Re-evaluating evaluation. Advances in Neural Information Processing Systems, 31, 2018
Bespoke Labs. Bespoke-stratos-7bb
https: /ols:///hugge hhwhhhhgggggcceeee bo/besooke labss/ Bespoke- Setiee-ttrtos---e Bespoke-Stratos-78 7B, 2024. Accessed: 2025-03-29.
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches on reproducible evaluation of language models. Xiv preprint ar Xitt:2405.14782 2024.
11

---

Published as a conference paper at COLM 2025
Sam Bowyer, Laurence Aitchison, and Desi R Ivanova. Position: Don't use the CLT in LLM evals with fewer than a few hundred datapoints. arXiv preprint (xXi:2503.00147, 2025.
Xin Cai. One framework to rule them all: Unifying rl-based and rl-free methods in rlhf. Xiv preprint xrXi::2503.1952,, 2025.
Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. With little power comes great responsibility. arXiv preprint ar Xrx::20000065955 2020.
Gavin C Cawley. Baseline methods for active learning, In Active Learning and Experimen- tal Design workshop In conjunction with AISTATS 2010, pp. 47-57. JMLR Workshop and Conference Proceedings, 2011.
Gavin C Cawley and Nicola LC Talbot. On over-fitting model selection and subsequent selection bias in performance evaluation. The Journal of Machine Learning Research, 11: 2079-2107, 2010.
Stephanie CY Chan, Samuel Fishman, John Canny, Anoop Korattikara, and Sergio Guadar- rama. Measuring the reliability of reinforcement learning algorithms. arXiv preprint arXio:1912.05663, 2019.
Liang Chen, Lei Li, Haozhe Zhao, and Yifan Song. Vinci, rl-v: Reinforcing super generaliza- tion ability in vision-language models with less than 3 dollars.
Cédric Colas, Olivier Sigaud, and Fierre-Yves Oudeyer. How many random seeds? sta- tistical power analysis in deep reinforcement learning experiments. arXiv preprint ar arXiv:1806.08295 2018
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint ar arx::25502.01455 2025.
Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small Ilms: What 16219.
works and what doesn't, 2025. URL //axxvv. org/
Xingyu Dang, Christina Baek, J Zico Kolter, and Aditi Raghunathan. Assessing diver- sity collapse in reasoning. In Scaling Self- lelfoiing Foundation Models without Human Superoision.
Google 2025. gettr Accessed: 2025-04-07.
DeepMind.
URL Gemini 2.5: Our most intelligent ai model,
woogle//eccooooggg//// /goggy//gogloooo google -el-ee--eeindl
DeepSeek-AL.
Learning, 2025. URL hti.re/KK....neiiiiiiinnnnnnenne
in LLMs via Reinforcement
Mostafa Dehghani, Yi Tay, Alexey A Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. arXiv preprint XXiv:2107.07002, 2021.
Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Open- vithinker: An early exploration to complex vision-language reasoning via iterative self- improvement. ar preprint ar XrXi::2033.17322 2025.
Ricardo Dominguez-Olmedo, Florian E Dorner, and Moritz Hardt. Training on the test task confounds evaluation and emergence. Xiv preprint XXxx:2477.0789) 2024.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. The hitchhiker's guide to testing statistical significance in natural language processing. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 1383-1392, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/vo /11/////11111111/ 1128. URL https: https///aaaaaaaaaaoorrr.oo.e 1128//
12

---

Published as a conference paper at COLM 2025
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning in milms. ar preprint (tXiv:2503.21776, 2025.
Clémentine Fourrier, Nathan Habib, Hynek Kydlfček, Thomas Wolf, and Lewis Tunstall. LightEval: A lightweight framework for LLM evaluation, 2023. URL hom/hugginggaann highhttth.t....
Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for Ilm reasoning. arviv preprint ar arXi::2410.15115 2024a.
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024b. URL https://zenol http://rrrrrrero/////0//55/
Adhiraj Ghosh, Sebastian Dziadzio, Ameya Prabhu, Vishaal Udandarao, Samuel Albanie, and Matthias Bethge. Onebench to test them all: Sample-level benchmarking over open- ended capabilities. ar Xiv preprint XrXi:24112.66745 2024.
Shahriar Golchin and Mihai Surdeanu. Time travel in Ilms: Tracing data contamination in large language models. Xiv preprint Xiv::2330 388999930... 2023.
Rihab Gorsane, Omayma Mahjoub, Ruan John de Kock, Roland Dubb, Siddarth Singh, and Arnu Pretorius. Towards a standardised performance evaluation protocol for cooperative marl. Advances in Neural Information Processing Systems, 35:5510-5521, 2022.
Etash Guha, Negin Raoof, Jean Mercat, Ryan Marten, Eric Frankel, Sedrick Keh, Sachin Grover, George Smyrnis, Trung Vu, Jon Saad-Falcon, Caroline Choi, Kushal Arora, Mike Merrill, Yichuan Deng, Ashima Suvarna, Hritik Bansal, Marianna Nezhurina, Yejin Choi, Reinhard Heckel, Seewong Oh, Tatsunori Hashimoto, Jenia Jitsev, Vaishaal Shankar, Alex Dimakis, Mahesh Sathiamoorthy, and Ludwig Schmidt, November 2024.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024. URL https: http:///aaaao.o//////////// 14008.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xi- aoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner series. httpss//carrrccoooss/yvoogggngg mottiooionn......i.o.......... site/ Saywork-Open-aee har.re/sssssseeeooooooeaeaaeeeaa 2025. Notion Blog.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Xiv preprint drXiv:2103.03874, 2021.
Jian Hu. Reinforce++: simple and efficient approach for aligning large language models. arxiv preprint arXiv:2501.032: 032662, 2025.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung-Yeung Shum Xi- angyu Zhang. Open-reasoner-zenoo An open source approach to scaling reinforce- ment learning on the base model. mat. Open-Reasoner- Zero, 2025.
Reasoner--eer---erreere -Zero/
13

---

Published as a conference paper at COLM 2025
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.067799 2025.
Hugging Face. Open rl: A fully open reproduction deepseek-rl, January 2025. URL https: www.tttww......eeeeeeee cww/ouggiiiicccccccccccoocoo.ooo ing aaac//ppppe
HuggingFacehH4. Math-500 dataset.
Hugg hates/ngaaaa//AHH---AAA
lob/ maii/README.... md, 2024. Accessed: 2025-03-29.
Ben Hutchinson, Negar Rostamzadeh, Christina Greer, Katherine Heller, and Vinodkumar Prabhakaran. Evaluation gaps in machine learning practice. In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, pp. 1859-1876, 2022.
Intelligent Internet. Il-Thought A Large-Scale, High-Quality Reasoning Dataset, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arviv preprint ar (rXiv:2412.167200 2024.
Piyush Jha, Prithwish Jana, Pranavkrishna Suresh, Arnav Arora, and Vijay Ganesh. Rlsf: Reinforcement learning via symbolic feedback. ar preprint arXiv:2405.1666) 2024.
Scott Jordan, Yash Chandak, Daniel Cohen, Mengxue Zhang, and Philip Thomas. Evaluating the performance reinforcement learning algorithms. In International Conference on Machine Learning, pp. 4962 9962-4973. PMLR, 2020.
Scott M Jordan, Adam White, Bruno Castro Da Silva, Martha White, and Philip S Thomas. Position: Benchmarking is limited in reinforcement learning research. ar preprint arXio:2406.16241, 2024.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. Infernational Conference on Machine Learning, pp. 15696-15707, PMLR, 2023.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv preprint arXfv:2410.01679, 2024.
Knovel Engineering. Amc-23 dataset, 2025,
URL https: https://hugginngaccecom
bottasets/knoveleegg/Atee AMCC/ 23.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quan- titative reasoning problems with language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 3843-3857. Curran Associates, Inc., mewpeeddddnnnnnneeessss........ neurips. f wtwp:/aallsaaaa////eeppeaaapaeee /2022/ 2022. URL httee//aabbeeeeeeeeeeffffsssseep happ://ssseeccfeeeee-----ee--rre file/
Paaereeee pdf.
Gang Li, Ming Lin, Tomer Galanti, Zhengzhong Tu, and Tianbao Yang. Disco: Reinforcing large reasoning models with discriminative constrained optimization, 2025a. URL https: //arxiv. a/ws 12366.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. LIMR: Less is More for RL Scaling. arXiv preprint ar Xix 11886, 2025b.
14

---

Published as a conference paper at COLM 2025
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. From system 1 to system 2: A survey of reasoning large language models. ar preprint arXii:2502.17419 2025c.
Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. Are we learning yet? A meta review of evaluation failures across machine learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Tweelfh International Conference on Learning Representations, 2023
Yen-Ting Lin. Aime 2025 dataset, 2025. URL https: https::/huggiiggfaeeeeeeee http://ggiggaaccc///aaaeeeaeaeea
yentinglin/an aainn/aime_aa 2025. Accessed: 2025-03-29.
Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the train-
ing of group relative policy optimization-base reasoning models. arXiv preprint arXiv:2503.22342, 2025a.
Zhiyu Lin, Yifei Gao, Xian Zhao, Yunfan Yang, and Jitao Sang. Mind with eyes: from language reasoning to multimodal reasoning. arxiv preprint Sttt::2503180771 2025b.
Zachary C Lipton and Jacob Steinhardt. Troubling trends in machine learning scholarship: Some ml papers suffer from flaws that could mislead the public and stymie future research. Queue, 17(1):45-77, 2019.
Jiawei Liu and Lingming Zhang. Code-rl: Reproducing r1 for code with reliable rewards. 2025.
Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, and Kai Chen. Are your lims capable of stable reasoning? arXiv preprint ar Xiv::2412.1314,, 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, and Min Lin. There may not be aha moment in r1-zero-like training pilot study. https:/ /top:://oatlli lomtlllm.o. notion.. late/oat-zers 2025a. Notion Blog.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, 2025b. URL https: //arxv//axiiv www..oor//bbbs...... 20783.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang- Visual-rft: Visual reinforcement fine-tuning. arXiv preprint drXiv:2503.01785, 2025c.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and lon Stoica. DeepScaleR: Surpassing O1-Previen with a 1.5B Model by Scaling RI, 2025. Notion Blog.
Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al. Exploring the limit outcome reward for learning mathematical reasoning. Xiv preprint ar arXio:2502.06788 2025.
Yan Ma, Steffi Chern, Xuyang Shen, Yiran Zhong, and Pengfei Liu. Rethinking rl scaling for vision language models: A transparent, from-scratch framework and comprehensive evaluation scheme. Xiv preprint ar drXio:2504.02587 2025.
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. ar Xiv preprint XrXio:2310.1299 2023.
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
15

---

Published as a conference paper at COLM 2025
Lovish Madaan, Aaditya K Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. Quantifying variance in evaluation benchmarks, 2024. URL https://uraio. orghbbs/240.11022)
Benjamin Marie, Atsushi Fujita, and Raphael Rubino. Scientific credibility of machine translation research: A meta-evaluation of 769 papers. arXiv preprint ar arXio:2106.15195
2021.
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. ar preprint arXim:2503.07365, 2025.
Meta-AI. The Ilama 4 herd: The beginning of new of natively multimodal ai innovation, 2025. Accessed: 2025-04-07.
URL https: /ai. meti.meta.cmmmmm cotttaog///b////////////
1lama-4mmu mul lomioollllneeelle--ee
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Imitate, explore, and self-improve: A reproduction report on slow- thinking reasoning systems, 2024. URL https::::// /aattps:/arxiv..ottt http://rxiv..orr/bb//22.......
Ximing Lu Jian Hu Xin Dong Yejin Choi Jan Kautz Yi Dong Mingjie Liu, Shizhe Diao. Prorl: Prolonged reinforcement learning expands reasoning boundaries large language models. arv preprint, 2025. URL https: http:s:///axxvo.oo/// org/ abs/ 2505 24864.
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and
Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arviv preprint ar XrXiv:241000522,, 2024.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL https: aa//i. org/ abs/ / 2501. 19393.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pp. 681-699. Springer, 2020.
Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti, and Jenia Jitsev. Alice in won- derland: Simple tasks showing complete reasoning breakdown state-of-the-art large language models. ar Xiv preprint ar Xi2:24400.00061, 2024.
NovaSky. Sky-tl: Train your own ol preview model within 450, https / /nvasky http://..io/poss////tt
ai.github.io/ posts/sky - 20,5 Accessed 2025 - 01 -9.
Open Thoughts. Open Thoughts. https://open-thouggee Janne January 2025.
OpenAl. OpenAl 03-mini System Card, January 2025a. URL hRRphttss///ccdoopeee
- 03 03-miní- seiin--sstee -etdest-eed----eeeaee pdf..
OpenAL. Introducing gpt-oss. https:/ htmps::/nnower http:////ooaayyiiigiuuubii/////e
August 2025b.
Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, and Shu Kong. The neglected tails in vasion-language models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12988-12997, 2024.
Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforcement learning. Journal of Machine Learning Research, 25(318) 2024.
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-r1: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. ar Xiv preprint arX11:2503.07536, 2025.
16

---

Published as a conference paper at COLM 2025
Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović, and Martin Vechev, Proof or bluff? evaluating llms on 2025 usa math olympiad. arviv preprint ar aro::5500 2025.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 524-540. Springer, 2020.
Ameya Prabhu, Shiven Sinha, Ponnurangam Kumaraguru, Philip HS Torr, Ozan Sener, and Puneet K Dokania. Randumb: A simple approach that questions the efficacy of continual representation learning. arXiv e-prints, pp. arXiv-2402, 2024a.
Ameya Prabhu, Vishaal Udandarao, Philip Torr, Matthias Bethge, Adel Bibi, and Samuel Albanie. Efficient lifelong model evaluation in an era of rapid progress. arxiv preprint
arXiv:2402.19472, 2024b..
Ori Press, Steffen Schneider, Matthias Kümmerer, and Matthias Bethge. Rdumb: A simple approach that questions our progress in continual test-time adaptation. Advances in Neural Information Processing Systems, 36:39915-39999 -593535, 2023.
Prime Intellect. Intellect-2: A reasoning model trained through globally decentralized rein- forcement learning. arXiv preprint ar 2xXiv:2505.072911 2025.
PyTorch Contributors. Reproducibility pytorch documentation. /oocs/stableenotes//eeee raapteeeeeerramnmnsseeeeee.m htmll, 2024. 2024. Accessed: 2025-04-09.
Matthew Renze. The effect of sampling temperature on problem solving in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 7346- 7356, 2024.
Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J Kochenderfer. BetterBench: Assessing AT Benchmarks, Uncovering Issues, and Establishing Best Practices. Xiv preprint arXiv:2411. 129900, 2024.
Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma, Akash Gupta, Samuel Roberts, Ioana Croitoru, Cemion-VVlad Bogolin, Jialu Tang, Florian Langer, Vyas Raina, et al. Zer- obench: An impossible visual benchmark for contemporary large multimodal models. arXiv preprint ar Xiv:2502.00966, 2025.
Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. To the cutoff... and beyond? a longitudinal perspective on Ilm data contamination. In The Twelffh International Conference Learning Representations, 2023.
2023.
Nicolas Le Roux, Marc G Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Eric Thibodeau-Laufer, Sándor Toth, and Sam Work. Tapered off-policy reinforce: Stable and efficient reinforcement learning for 1lms. arXiv preprint ar Xix::22003.144 14286, 2025.
Bellemare,Jonahhan
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https: /tttp:::/arxvv.org//aaa 2402. 03300.
Wei Shen, Guanlin Liu, Zheng Wu, Ruofei Zhu, Qingping Yang, Chao Xin, Yu Yue, and Lin Yan. Exploring data scaling trends and effects in reinforcement learning from human feedback. arXiv preprint arXiv:2503.222333 2025.
Shamus Sim and Tyrone Chen. Critique of impure reason: Unveiling the reasoning behaviour of medical large language models. Xiv preprint ar arXio:2412.15749 2024.
Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. FastCuRL: Curriculum Reinforcement Learning with Progressive Context Extension for Efficient Training R1-like Reasoning Models, 2025. URL http:: /ttp:///axxvvvo//e /2003. 17287.
17

---

Published as a conference paper at COLM 2025
Saurabh Srivastava, Anto PV, Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arxiv preprint arXio:2402.19450, 2024.
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Expanding with verifiable rewards across diverse domains. ar Xiv preprint arXi::2503.2382, 2025.
Lin Sun, Guangxiang Zhao, Xiaoqi Jian, Yuhan Wu, Weihong Lin, Yongfu Zhu, Linglin Zhang, Jinzhu Wu, Junfeng Ran, Sai-er Hu, et al. -abpr--eeeb-e Boosting accuracy with branch-merge distillation. ar arviv preprint ar Stt::5503008872 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint
250t:::5501129999 2025.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL httpss///went m. th... io b blogblllg//oo//o hggpogggwwagoooooooooooo 32bb/
Songjun Tu, Jiahao Lin, Xiangyu Tian, Qichao Zhang, Linjing Li, Yuqian Fu, Nan Xu, Wei He, Xiangyuan Lan, Dongmei Jiang, et al. Enhancing Ilm reasoning with iterative dpo: A comprehensive empirical investigation. arviv preprint ar httt:2503.128544 2025.
Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. No" zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. ar preprint arXie:22ff.142773 2022.
vLLM Contributors.
Inference reproducibility script.
https:/ g thub. com/ 2024. Accessed: 2025-04-09.
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. Thoughts are all over the place: On the underthinking of ol-like llms. ar preprint arXin:2501.1 18585, 2025.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. Light-r1: Curriculum sft, dpo and for long cot from scratch and beyond, 2025a.
URL htw.t: http///xxvvvvoo/// org/ abs/2500 10460.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond, 2025b.
URL wwwtthtts/sto.. //axivv//axvvvv .twpiii.orr/////o org. abs /5003. 10460.
xAI. Grok 3 beta - the age of reasoning agents. February 2025. URL Utthttps: / ai news/ grok-3. Accessed: 2025-03-29.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXio preprint ar XrXi::2502.14768, 2025.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance, 2025a. URL https: http:///axxvvoorr.eo abs/ 1504.14945
guidance
Kai Yan, Yufei Xu, Zhengyin Du, Xuesong Yao, Zheyu Wang, Xiaowen Guo, and Jiecao Chen. Recitation over reasoning: How cutting-edge language models can fail on elementary school- level reasoning problems? arxiv preprint ar arXio:2504.00500, 2025b.
18

---

Published as a conference paper at COLM 2025
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. aiv preprint arXiv:2412.15115, 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. ar preprint (tff::2409.121222 2024b.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXio:2502.03387, 2025.
Huimu Yu, Xing Wu, Weidong Yin, Debing Zhang, and Songlin Hu. Codepmp: Scalable prefer- ence model pretraining for large language model reasoning. arXiv preprint arXi::2410.02229, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint 7xXio:2503.14470 2025.
Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang, Ilia Kulikov, Kyunghyun Cho, Yuandong Tian, Jason E Weston, et al. Naturalreasoning: Reasoning in the wild with 2.8 m challenging questions. arXiv preprint arXio:2502.13124, 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? aiv preprint ar drvi:2200 003500.13333 13837, 2025a.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025b.
Thomas Zeng, Shuibai Zhang, Shutong Wu, Christian Classen, Daewon Chae, Ethan Ewer, Minjae Lee, Heeju Kim, Wonjun Kang, Jackson Kunde, et al. Versaprm: Multi-domain process reward model via synthetic reasoning data. arv preprint arXiv:2502.06737, 2025a.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. SimpleRL-Zoo Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild. arXiv preprint arXiv:2503. 18892, 2025b.
Sheng Zhang, Qianchu Liu, Guanghui Qin, Tristan Naumann, and Hoifung Poon. Med-rlvr: Emerging medical reasoning from a 3b base model via reinforcement learning. arXiv preprint arXio:2502.19655, 2025.
Zyphra. Zr1-1.5b: A small but powerful reasoning model for math and code, 2025.
19

---

Published as a conference paper at COLM 2025
Appendix
We now provide thorough details about the benchmark, baselines, and prompts.
Contents
A Recommendations: Which practices to adopt?
21
B Do Discovered Phenomena Replicate? A Detailed Analysís.
22
B.1 Are Incorrect Responses Longer?
22
B.2 Is There Diversity Collapse in Reasoning Training?
23
Extended Related Works
24
D Variance from Hardware Software Factors: A Critical Consideration
25
E Prompt Variants and Template Settings
27
Bootstrapping Results on Additional Datasets
28
G Temperature and top-p: Additional Analysis
31
H Effect of Output Length Limits
32
I Response Length vs. Accuracy Per-Model Breakdown
33
Diversity Collapse
36
K Optimal Decoding Parameters
37
20

---

Published as a conference paper at COLM 2025
A Recommendations: Which practices to adopt?
We propose a set of best practices informed by our experiments and guided with current research insights:
Hardware and Software Stack Standardization: To promote reproducibility and facilitate future work, we release all code within Docker container, along with step-by-step instructions for running experiments on Runpod's publicly accessible, on-demand GPU instances. This setup allows any researcher to replicate and extend our results under identical conditions.
For small benchmarks (e.g., AIME224), evaluations with at least ten random seeds. Report the mean and standard deviation to quantify uncertainty and assess the statistical significance of performance differences.
Variance Estimates:
Model-Specific Hyperparameter Optimization: Tune hyperparameters (such tem- perature and top_p) separately for each model, then fix them across tasks to ensure consistency and fair comparisons.
Context Length and Prompt Template Selection: Ensure the context length is sufficiently large-especial/. for models with long reasoning chains-to avoid premature truncation and under-reported accuracy. For Intrruction-tuned models, always use the appropriate chat template match the expected input format.
Robust Answer Matching: We strongly recommend using a resilient answer extraction pipeline that handles parsing issues and evaluates expression equivalence, rather than relying on exact string matching. This reduces the likelihood of spurious gains from formatting artifacts.
Transparent Evaluation Protocols: We recommend to release code, prompts, and model outputs, and clearly document the evaluation stack. Report uncertainties (e-g., via stan- dard deviations) and include both quantitative and qualitative analyses to enable thor- ough and reproducible comparisons.
21

---

Published as a conference paper at COLM 2025
B Do Discovered Phenomena Replicate? A Detailed Analysis.
We further investigate two recently noted phenomena to see if they replicate in our experi- ments: (1) how response length correlates with performance, and (2) the decline in response diversity following reasoning-fooee focussed training.
B.1
Are Incorrect Responses Longer?
Light-R1-78-DS
OpenThinker-78
030
Connet (1.0)
2500
Comre1 (18)
(1.0
1750
Ausegge Court Seed
1500
R30
1250
1000
1230
750
500
250
5300
10000
19000
20002
20000
20000
10000
15000
20000 20002
30000
Respora Langth
Raapprse Lergeh
Figure 8: Response Length Accuracy. Histogram of correct incorrect responses by response length, averaged over random seeds across AIME24, AIME25, AMC23, MATH50Ó, Minerva and BlynpiadBench benchmarks. Longer outputs tend to be more error-prone, even in complete responses not close to the maximum sequence length.
Recent research (Wang et al., 2025) suggests that incorrect answers often have dispropor- tionately long reasoning chains. We first verify whether this finding holds in our setting, and then we explore possible explanations behind the observed variations.
Do longer responses indicate a higher likelihood of an incorrect answer? We compare the distribution of response lengths for correct and incorrect answers across 6 datasets (AIME24, AIME25, AMC23, MATH500, Minerva and OlympiadBench) averaged across random seeds for each model. Figure shows histograms of the average number of responses per seed, binned by response length. A clear trend emerges: shorter responses are significantly more likely to be correct, while longer responses become progressively more error-prone. This pattern is consistent across all seeds and is especially pronounced for responses exceeding 10,000 tokens. We now address two questions:
Q1. Does this pattern hold for both RL- and SFT-trained models? Yes. We find the trend is consistent across both RL- and SFT-trained models (additional figures provided in Appendix figures 21 and 22 ). We consistently observe that the effect is more pronounced in RL-trained models (displayed on the left) than SFT-trained models (displayed on the right). As detailed in the Appendix, both the Qwen 2.5 Math base exhibit slight shift in length, though this shift is notably more evident R1-distill and subsequent RL-trained models.
Q2. Is this primarily because of truncated or incomplete responses? Although responses nearing the 32,000-token limit are almost always incorrect (due to limited context-length) this trend persists even for complete responses which are shorter- Longer responses are associated with a higher likelihood of being incorrect.
Takeaway Longer responses correlate with a greater chance of error, response length is practical heuristic for consensus@k, identifying low-confidence or failed generations.
22

---

Published as a conference paper at COLM 2025
AIME24
AMC23
MATH500
0.06
0.34
0.020
0.02
0.015
0.02
pasergk
0.00
pawo
0.005
-0.02
0.000
-0.09
-0.010
-0.08
0.04
0.015
DeepScaleRR1.5B
FastCuRL-1.5B
Figure 9: RL-trained models do show diversity collapse (Dang et al.) Across all bench- marks, RL-trained models (DeepScaleR-1 1.55 and FastCuRL-1.55) consistently underperform their baseline ReepStek-R1-Distill--. in Pass@k, with all APass@k values being negative. All models were evaluated using the decoding parameters Table 7.
B.2 Is There Diversity Collapse in Reasoning Training?
Dang et al. and Yue et al. (2025a) have reported counterintuitive phenomenon in reasoning models: improvements Pass@1 achieved through supervised fine-tuning RL can reduce Pass@k performance due to diminished output diversity- phenomenon termed diversity collapse. Theoretical analyses attribute this collapse to the model concentrating too much probability mass on single reasoning path, while current decoding strategies fail to recover the lost diversity.
To examine these claims, we compare the Pass@k performance (for k E {1,2,3,... 128) of two RI.trainee models (DeepScaleR-1155 and FastCuRL-1.5BB against their base model ReepSeek-IL--isttll---eeeee
across all datasets. Figures 9 and 23 show the delta in Pass@k relative to DeepSSek-Kt--Dstiil---we-e
Findings. We do observe minor diversity collapse. Gains in Pass@1 generally come with regression in Pass@k, though the magnitude of the decay varies.
Takeaway 7 Consistent with the diversity collapse hypothesis, the RL-trained models we tested show a regression in Pass@k accuracy.
23

---

Published as a conference paper at COLM 2025
C Extended Related Works
We contextualize our work in broader literature extensively in this section.
Language Model Reasoning (for Math). The recent releases of OpenAl-O1 (aaech et al., 2024) (in September 2024), OpenAl-O3 (OpenAl, 2025a) (in December 2024) and DeepSeek- R1 (DeepSeek-Al, 2025) (in January 2025), have spurred the language modelling community to work on improving the reasoning capabilites of language models. Several popular meth- ods for improving those capabilites have emerged with supervised fine-tuning (SFT) and reinforcement learning (RL) being the two primary methods of interest (Uesato et al., 2022; Lightman et al., 2023; Lyu et al., 2025; Open Thoughts, 2025). Recent works have built upon the DeepSeek-RI recipe by proposing newer RL algorithms, including LCPO (Aggarwal & Welleck, 2025), REINFÖRCE++ (Hu, 2025), DAPO (Yu et al., 2025), DPO-VP (Tu et al., 2025), VinePPO (Kazemnejad et al., 2024), CPPO (Lin et al., 2025a), VAPO (Yue et al., 2025b) and GRO (Cai, 2025). To gain a stronger understanding of how to induce mathematical capabilities, other works have conducted significant empirical studies exploring the design space of RL methods (Zeng et al., 2025b; Liu et al., 2025b; Team et al., 2025; Shao et al., 2024), including data scaling trends (Shen et al., 2025), curriculums (Wen et al., 2025a; Roux et al., 2025) and reward design (Gao et al., 2024a; Cui et al., 2025; Ma et al., 2023). Based on the success of these methods, there have also been recent efforts into scaling up reinforcement learning approaches to induce reasoning in domains beyond math, including code (Liu & Zhang, 2025; Xie et al., 2025; Jha et al., 2024; Yu et al., 2024), medicine (Zhang et al., 2025; Sim & Chen, 2024) and other sciences (Su et al., 2025; Yuan et al., 2025; Zeng et al., 2025a). Further, some works also explored scaling up RL-based approaches to modalities beyond just language, including vision (Ma et al., 2025; Meng et al., 2025; Huang et al., 2025; Peng et al., 2025; Chen et al.; Deng et al., 2025; Liu et al., 2025c; Feng et al., 2025; Lin et al., 2025b). In our work, we objectively re-evaluate the claims made by several of these recent works under standardized lens, and find that many of the reported gains do not hold up strongly when pitted on level-playing field against well-tuned baselines.
Sobering Studies on ML Progress. Machine learning is a field of rapid progress. Due to the lightning speed of papers coming out across the various sub-fields of machine learning, practitioners and researchers often fail to rigorously evaluate algorithmic progress (Hutchin- son et al., 2022; Dehghani et al., 2021; Machado et al., 2018; Ghosh et al., 2024; Balduzzi et al., 2018; Liao et al., 2021; Cawley & Talbot, 2010; Lipton & Steinhardt, 2019; Prabhu et al., 2024b; Card et al., 2020; Dror et al., 2018). This has led to several papers showing that simple well-tuned baselines outperform months of progress on a specific sub-field in machine learning, including in continual learning (Prabhu et al., 2024a; 2020), active learning (Cawley, 2011) and test-time adaptation (Press et al., 2023). With the rapid influx of reasoning-basee LMs, such statistically rigorous comparisons of models are ever more important- despite the heavy use of RL-algorithms for driving progress in reasoning, there is very little mention of how different methods standardize their evaluations across different factors of variability. RL-algorithms themselves are known to be quite fickle to extremely minor variations including random seeds (Agarwal et al., 2021; Gorsane et al., 2022; Chan et al., 2019; Jordan et al., 2020; Patterson et al., 2024). Some works have even gone as far suggesting that reliable benchmarking of RL-based methods is computationally infeasible (Jordan et al., 2024). Additionally, other works have demonstrated critical reliabil- ity issues in the generalization of frontier models to minor perturbations in the question inputs (Mirzadeh et al., 2024; Nezhurina et al., 2024; Srivastava et al., 2024), the type of tasks tested (Yan et al., 2025b; Petrov et al., 2025; Dominguez-Ommnn Olmedo et al., 2024; Roberts et al., 2025), metrics used (Liu et al., 2024) and in data-scarce scenarios (Udandarao et al., 2024; Kandpal et al., 2023; Parashar et al., 2024). Given such volatile landscape, in this work, we aim to level the playing field across recent LM-methods that have been released and provide objective look on the progress that the reasoning community has made. Our findings, which we discuss in the rest of the paper, are sobering at best.
24

---

Published as a conference paper at COLM 2025
D Variance from Hardware & Software Factors: A Critical Consideration
AMC23
AIME24
0.55
9.90
Gurer Dubie
0.50
0.85
0.89
0.35
0.30
0.25
0.20
Deepstend di Dinul 1.30
Deeprimd dartee
comntt A..3
Opeetty 1.18
Operitinte
$1.
Deepirate 59
Derprient 41 Oisad 50
Despo 41 Caata--2
open
qpentard
openttS sg
Comminten
51
$1.99
(a) AIME24. Significant differences are observed in model performance across compute clusters.
(b) AMC23. Similar variability is seen across hardware in AMC23 results.
Figure 10: Performance variation across compute clusters. Accuracy differences emerge when the same models are evaluated across compute clusters for both AIME24 and AMC23 datasets-these large differences in performance also persist when evaluating 7B models.
Performance can also vary due to non-obvious factors like hardware and evaluation frame- work-yet this is rarely acknowledged. Models are often tested on heterogeneous systems and evaluated using different toolchains. For example, S-1.1 (Muennighoff et al., 2025) uses (Gao et al., 2024b), the OpenRS model suite uses li Lighte- val (Fourrier et al., 2023), and 11-1.5--revieew uses evalscope (Alibaba ModelScope Community) for evaluation.
Lm-eva lua Laaaaattoaaaneeee
Hardware Variation. We evaluated the same model across five different compute clusters, each with varying GPU types and memory configurations. As shown in Figure 10, perfor- mance varied by up to 8% for OpenRS-1.5B and 6% for DeepSeek-Rl--isttee with similar trends observed on AMC'23. While known that inference engines such as vLLM can be sensitive to hardware differences (vLLM Contributors, 2024) that low-level optimizations PyTorch or CUDA (PyTorch Contributors, 2024) may introduce non-determinism-a our results demonstrate that these effects can measurably impact bench- mark accuracy, even when averaging over multiple seeds. In Figure 11, we show that similar discrepancies are observed on MATH500.
on AIME'24 24,
Can we alleviate this? We conducted extensive experiments to isolate and quantify hardware-induced variability, even when using identical GPU types and software stacks. Our investigation proceeded in three stages, each attempting to eliminate additional sources of variation:
Stage Standardized Docker across different A100 clusters. We first evaluated models using our standardized Docker container on A100 GPUs across two platforms: our internal cluster and Runpod (a cloud GPU service). Despite identical hardware specifications and containerized environments, we observed notable performance differences. For instance, Qwen2.5-Math- 1.5B achieved 3.8 on AIME'24 on our cluster versus 13.3%+3.6 on Runpod; a percentage point gap that exceeds the standard deviation.
Stage 2: Enforcing CUDA determinism. Suspecting non-determiiistic GPU operations, we en- forced strict determinism by setting torch. wew.uuusee______ dete wwwpiinnsttiiaaaaa thms (True), fixing CUDA seeds, and disabling cuDNN benchmarking, Comparing A100 and H100 clusters, variance decreased but persisted: OpenThinker222
scored 4.. on A100 versus 57.1 5.2 on H100 for AIME'24. Even with deterministic algorithms, hardware ar- chitectural differences and vLLM's backend variations (vLLM Contributors, 2024) continued to introduce discrepancies.
Stage 3: Identical hardware, updated stack. Finally, we updated all components (vLLM, LightE- val, math-verify) and ran multiple evaluations on identical hardware. Remarkably, even consecutive runs the same A100 cluster produced variations: Bespoke-Stratos-77 ranged
25

---

Published as a conference paper at COLM 2025
from 19.3% to 23.0% AIME'24 across runs. These persistent differences, despite trolling for hardware, software versions, and random seeds highlight deep-seated sources of non-determinism in modern inference stacks, potentially arising from dynamic kernel selection, memory allocation patterns, or lloatin-poiin accumulation order.
As shown in Figure 10, these effects are not merely theoretical but measurably impact benchmark accuracy even when averaging over multiple seeds. This underscores that reproducibility challenges extend beyond simple seed variance to fundamental architectural and systems-level factors.
Evaluation across different Python frameworks. Evaluation results can vary based the framework used, due to differences in prompt templates, inference engines (egg, vLLM (Kwon et al., 2023)), and response extraction strategies (e-g., Math Verify. For ex- ample: lighteval is used by OpenRS (Dang & Ngo, 2025), evalchem, (Guha et al., 2024) is used by models like Open and Bespoke-Strator other frameworks include munity).
(Gao et al., 2024b) and evalscope (Alibaba ModelScope Com-
To assess this impact, we compare lighteval and evalchemy, keeping all other variables fixed: model, dataset, hardware, decoding parameters, and random seeds (3 per model). For a fair comparison, we evaluated two models, DeepSeek-RI-Distill- default temper comperatur rature and top_ cop_P parameter values on a single GPU. We present results averaged over three seeds for higher robustness. As shown in Table 4, framework-induce differences are generally small (1-2pp) but can still affect model rankings in tightly clustered scenarios.
and $1.1-7B, at their
Overall, our findings underscore that significant per- formance variations can arise solely from differences in hardware and software configurations, emphasiz- ing the need to standardize for reliable evaluations.
Model
Lightevalevalchemy
R1-Distill-1.5B
26.6
26.6
S1.1-7B
22.2
17.7
Table 4: AIME24 across frameworks.
Takeaway 6 Hardware and software variations introduce irreducible noise into evalu-
ations, with performance differences of 2-5% persisting even under stringent controls. True reproducibility requires not just seed averaging but also explicit reporting of hard- ware configurations and acceptance that some variation is inherent to current inference systems.
MATH500
0.92
0.86
0.86
Deeginaket 1.50
Drepstval diatil- 73
openees Lu
preeeti Lu0
openiss. Li
$1.99
Crepiieat Dustii
Figure 11: Performance variation across compute clusters on MATH500. Differences in GPU type and environment lead to non-trivial shifts in performance, reinforcing the importance of hardware standardization.
26

---

Published as a conference paper at COLM 2025
E Prompt Variants and Template Settings
We provide the exact templates used for our three prompt settings in Table 5: Math, Default, and No Template. These formats are based on the DeepSeek tokenizer but adapted for each model's s specific chat template. Our results (in 2.3) indicate that instruction-tuned models are highly sensitive to prompt formatting, with performance degrading significantly when prompts deviate from their training-time structure.
Prompt Math
Example
>oovve the follow- ing math problem efficiently and clearly. The last line of your response should be of the following format: Thhreefore the final answer is: ANSSWERS I hope it is correct' (without quotes) where ANSWER is just the final number or expression that solves the problem. Think step by step before answering. \n Assis- (tni:x/ttiinnnnuuustiiiii tant { (Queston))))) AsAssis-
wew_sseeeeeeeeeee >< < O
Default
tant|><thinke>> No [emplate
Table 5: Prompt templates used in our evaluation. The inclusion exclusion of structured prompt tokens significantly impacts performance for instruction-tuned models.
27

---

Published as a conference paper at COLM 2025
F Bootstrapping Results on Additional Datasets
To complement our analysis in 2, we present bootstrapped variance results on two additional datasets: AMC'23 and MATH500. As shown in Figures 12 and 13, high variance in Pass@1 persists even when averaging over multiple seeds (K = 5), mirroring the trends observed on AIME'24. These results reinforce our conclusion that small benchmark sizes yield unstable estimates and that robust performance reporting requires multiple seed runs. Even for larger Benchmarks, like MATH500 (Figure 14), Minerva (Figure 15) and Olympiad Bench (Figure 16) the estimates remain volative, yet there magnitude is much smaller. An overview of the exact variance values can be found in Table
Variance of Means
DeepScaleR-1.58 D
FastCuRL-1.58
20
20
20
10
10
10
0
20
40
60
20
40
60
0
20
K
K
Figure 12: Variance of mean Pass@1 on AMC'23, Bootstrapped estimates show substantial variance even with K = 5 evaluation runs, highlighting the instability of single-seed evalua- tions.
Variance of Means
DeepSeek-RI-Distill1..555
DeepScaleR-1.58
FastCuRL-1.58
20
10
10
10
0
0
20
40
60
20
40
60
0
20
40
60
K
K
Figure 13: Variance of mean Pass@1 on AIME'25. Bootstrapped estimates show substantial variance even with K = evaluation runs, highlighting the instability of single-seed evalua- tions.
Variance of Means
DeepSeek-RL-Disttlll...5
DeepScaleR-1.58
FastCuRL-1.5B
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
10
20
30
10
20
10
20
Figure 14: Variance of mean Pass@1 MATH500. Similar to AIME'24 and AMC'23, the estimates remain volatile across seeds. However, since MATH500 is a larger dataset the variance is much smaller than for e.g. AIME'24.
28

---

Published as a conference paper at COLM 2025
Variance of Means
D
DeepScaleR-1.5B
FastCuRL-1.5B
2
2-
1
0
10
20
30
10
20
30
10
20
30
K
K
K
Figure 15: Variance mean Pass@1 on Minerva. Similar to AIME'24 and AMC'23, the estimates remain volatile across seeds. However, since Minerva is larger dataset the variance is much smaller than for egg. AIME'24.
Variance Means
DeepSeek-RL-Distilll...o 1.55B
DeepScaleR-1.5B
FastCuRL-1.5B
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
10
20
30
10
20
30
10
20
30
K
Figure 16: Variance of mean Pass@1 on Olympiad Bench. Similar to AIME'24 and AMC'23, the estimates remain volatile across seeds. However, since Olympiad Bench is a larger dataset the variance is much smaller than for e.g. AIME'24.
29

---

Published as a conference paper at COLM 2025
Table 6: Accuracy variance for different K values across models and datasets
Model
K = 1 = K = 3 K = 5 = 10 = 15 K = 20 = 30 K = 60
AIME'24
D
19.441
13.269
7.762
3.754
2.409
1.686
1.054
0.365
1ee5ScaleR-1.5B
27.452 13.477
9.117
5.342
2.495
1.587
1.149
0.705
0.246
FastCuRL-1.5B
28.807
14.647
9.488
5.658
2.711
1.709
1.247
0.757
0.256
AIME'25
7.759
4.568
2.199
1.419
1.017
0.628
0.209
DeepScaleR-1.5B
17.311
8.561
5.597
3.387
1.605
0.988
0.726
0.445
0.156
FastCuRL-1.5B
16.201
8.232
5.412
3.221
1.504
1.000
0.710
0.424
0.145
AMC'23
DeepseekkRt--Ditill---BB.....-.
11.241
7.301
4.288
2.093
1.315
0.942
0.585
0.202
DeepScaleR-1.5B
22.377
10.851
7.167
4.228
2.079
1.295
0.936
0.568
0.192
FasiCuRI-1.5B
23.671
11.841
7.649
4.605
2.149
1.404
0.980
0.594
0.211
MATH500
DeepSeek-RI-Disill-LL1.3
1.175
0.561
0.367
0.226
0.110
0.069
0.049
0.030
0.010
DeepScaleR-1.5B
1.050
0.513
0.351
0.198
0.099
0.062
0.044
0.027
0.009
FastCuRL-1.5B
0.939
0.447
0.305
0.179
0.085
0.055
0.040
0.024
0.008
Minverva
DeepSlek-RR-DDistll-..BB
3.485
1.741
1.143
0.665
0.326
0.206
0.149
0.091
0.031
DeepScaleR-1.5B
2.385
1.179
0.769
0.452
0.220
0.143
0.102
0.061
0.022
FastCuRL-1.5B
2.703
1.347
0.876
0.527
0.244
0.161
0.113
0.069
0.024
Olympiad Bench
DeepSeek-Rt-Distine
1.269
0.622
0.427
0.241
0.118
0.076
0.053
0.032
0.011
1eepScaleR-1.5B
1.018
0.505
0.334
0.195
0.093
0.059
0.043
0.026
0.009
FasiCuRI-1.5B
0.908
0.463
0.305
0.179
0.086
0.054
0.039
0.024
0.008
30

---

Published as a conference paper at COLM 2025
G Temperature and top-p: Additional Analysis
We additionally investigate the impact of the temperature and top_p hyperparameter as prior works often employ different temperature and top_p settings when comparing the same model. To isolate the impact of varying temperature and top_P, we averaged pass@1 across seeds and compute variation of this estimate across temperature and top_p in boxplot. Figure 17 and 18 show the performance variation. We see that temperature-induce and top_ sep_p-indeeed fluctuations not only affect performance estimates but also introduce substantial variability in performance itself, which can lead to unfair comparisons when evaluating the same model across different temperatures.
AIME24
AMC23
MATH500
25
2B
0.930
94
0.875
88:
02
85
6.200
DeepScaleR-1..m 18
OpenRS1-1.88
Open
OpenThinke-778
DeepSeak-Fi Baloek-RR-Diiie Diittil-77
DeepSockRR Ri-isll-----
OpenRS2-1.58
Figure 17: Accuracies vary significantly across temperature values. Across nine different models and three datasets, we observe consistently large variations in performance (upto 15%) induced by changing the temperature. Results were obtained by varying the tempera- ture from in increments of 0.1, while holding top_p constant at 0.9.
AIME24
AMC23
MATH500
0.94
0.50
0.52
0.45
0.90
3.40-
0.18
0.36
0.30
0.70
0.86
0.5
0.65
0.84
0.00
0.82
0.16
0.80
DeepScaleR11..
OpenRS1-1.68
OpenRS3-1.58
OpenThinkar-7B
DeepSock R1 Distil-7B
DeepSock-R1 Ri-itll----
OpenRS2-1.58
1-1.58
Figure 18: Accuracies vary significantly across top_p values. Across nine different models and three datasets, we observe consistently large variations in performance (upto 8%) induced by changing the top_p value. Results were obtained by varying top_p from 0 to in increments of 0.1, while holding the temperature constant at 0.8.
31

---

Published as a conference paper at COLM 2025
H Effect of Output Length Limits
We further explore how varying max_new tokens impacts model accuracy, Figures be- low compare OpenRS-series models (with 131,072-token context windows) and Open- Thinker /S11 models (with 32,768-token limits).
Figure 19 shows that OpenRS models are highly sensitive to this parameter--horteningg outputs results in clear accuracy drops, Similarly, Figure 20 reveals the same pattern for OpenThinner-77B and S1.1-7B, despite their smaller context lengths. In both cases, premature truncation leads to incomplete reasoning chains and incorrect answers, confirming the importance of setting appropriate generation limits.
AIME24
AMC23
MATH500
0.4
0.86
0.84
Accuracy
0.2
0.78
0.76
0penR511 1
OpenRS2 1. 58
0par853-1.58
4096 0192 16304 3276i
85536131072
4096 8192 16,064 32760
65536 151072
4096
8192
163864
32768
65536 131072
Figure 19: Impact of max_new _ww__eteeessseteeeeeeeeeeeeeeeee support (131,072 tokens) experience degraded performance when pax_new_tokene too low.
tokens on OpenRS models. Models with long context
AMC23
0.6
$1.177
7p
OpenThiier2
0.8
0.90
0.5
Accuracy
0.7-
0.85
0.3
0.2
0.4
0.1
0.70
40V6
5112
16584
32768
4016
8102
16384
32768
4016
8102
16584
32768
Max New Tokens
Max New Tokens
Max New Tokens
Figure 20: Impact of max_ _ew_ tokens on OpenThinker and S1.1 models. Despite shorter context limits (32,768 tokens), performance still degrades noticeably when output length is constrained.
32

---

Published as a conference paper at COLM 2025
I Response Length vs. Accuracy - Per-Model Breakdown
To supplement the aggregated results shown in Figure 8, we include detailed histograms for each individual model in the appendix. These plots show the distribution of correct and incorrect responses across response lengths, averaged over random seeds. Due to the number of models analyzed, we split the results into two figures for clarity.
Figures 21 and 22 reveal that the overall trend observed in the main paper holds consistently across nearly all models: incorrect responses tend to be longer than correct ones.
These results reinforce the idea that excessively long outputs often indicate failure modes such as hallucinated reasoning, verbose overthinking, degenerate loops. Importantly, this correlation persists well below the maximum sequence length, ruling out truncation as the sole cause. Across all models, longer responses are a consistent marker of incorrect outputs, making response length useful signal for detecting low-confidence or erroneous reasoning chains.
33

---

Published as a conference paper at COLM 2025
Owen2.5 2-22255-. M-tth--ah--aa0
Qwer225 5 4-Maath-1... 1.58-lintruct
M-Math-78
Incorea $40
150
106
1g
2000
2008
Owen25 Oeth25-ath--8 Math-7B Instruct
Owen2551..8
DoopBesk Distil Qawen-1.88
7000
20/02
190
1000
2000
2000
1000
3000
13000
12000
2000
30000
30000
1000
5000
1900
30000
30000
DaepSeak-RR Rt Daatl--Qwen-78
Quenn2. 5-1, -8--nnarurct
Qwan2.5 Sot--nnttuctt
2000
100
1000
19000
1900
20000
.000
10000
11000
2000
2400
A000
40000
19000
20000
LargoresLaggh
DeepScaloR 1.58 Preview
07
Open-R51
2008
Nog C0unroa 260
10000
1900
20000
20000
V050
1000
30000
1200
20000
3000
Opan-RS2
Opan RS3
2008
2000
Despore
Serporas
Figure 21: Response Length vs. Correctness - Models (1/2). Average number of correct and incorrect responses across response length bins for subset of models. Longer responses consistently correlate with incorrect predictions.
34

---

Published as a conference paper at COLM 2025
$1.1-78
B-Thoughe: Preview
Qwen2 wat.lh5-tttthhttt 5BB-Bttttt
3000
1300
1000
1908
20000
1000
19000
20000
Qwen2.5 -ww.5--MM-------t. Zerout-Zero
STILL-3-1 STILL-3-1.5B- p..eeie
Bespoko -tr atoo-78
2000
3000
1000
2002
1500
2000
2000
1000
10000
1900
12000
1000
30:00
1000
10000
10000
12000
3000
FastCulIL-11300 Prowleew
LIMR
Openitt-Qwan-18
20:00
17.00
A0g Oourte 566
1500
1240
V0000
19000
1800
1480
6000
13000
19000
10000
Qwron-2.5 Math-77 Stmppeel
Qwen-2.5.1.0 SimpleRL-Zoo
Qwon-2.5-8--Bin 20w.22.-8--SmmhRLL--0
6000
4000
2000
2000
3400
12000
14000
12000
L1-Qwee--1.ee Max
L1 Q.re-een.... Elantt
Open-Reaneee Soon-Raanonen-Zerr--T
6000
5000
5000
2000
2008
DesgorssLagg
SerporasLang
Langores
Figure 22: Response Length vs. Correctness - Models (2/2). Continuation model-wise response length analysis. The same trend holds across the remaining models, with incorrect answers being disproportionately long.
35

---

Published as a conference paper at COLM 2025
J Diversity Collapse
In addition to the diversity collapse discussed in section B.2 and Figure the results for additional datasets shown in Figure 23, For DeepSeek-RI-Distill-.e (SFT-trained) could not replicate diversity collapse shown in Figure 24.
AIME25
Minerva
Olympiad Bench
0.01
0.00
0.04
0.02
passgk
paow
0.01
0.02
0.00
0.02
-0.01
0.03
-0.04
-0.00
100
DeepScaleR-1.5B
FastCuRL-1.58
Figure 23: RL-trained models do show a diversity collapse (Dang et al.). We report the delta between Pass@k of RL-trained models (DeepScaleR-1.5B and FastCCRRL-1.5) and their corresponding baseline -i-SSeek----isiill--...- We observe a diversity collapse (Apass@k is below zero). All models were evaluated using the decoding parameters listed in Table 7.
A4ME24
AMC23
MATH500
0.36
0.30
0.25
paosgge
0.20
0.2
0.10
0.05
0.00
Figure 24: SFT-trained models do not exhibit diversity collapse. Across benchmarks, PeeSSee--R--Distil-..55. (SFT-trained) exceeds its baseline, Qwen2.5-Math-1155. in APass@k. All evaluations used the decoding parameters in Table 7.
36

---

Published as a conference paper at COLM 2025
K Optimal Decoding Parameters
Empirically, the decoding parameters in Table 7 consistently produced optimal performance for the models we evaluated.
Table 7: Optimal temperature and top-p settings for various models
Model Name
Temperature Top-p
1.0
0.7
bespokelbbb/Beeppke-SSttttt3
1.0
0.9
0.9
0.7
0.9
0.7
0.5
0.5
1.0
0.5
knoveleng:/Open-KS]
1.0
0.6
knovelengg/penn-R9
0.9
0.8
knoveleng:/penn-RS3
0.5
0.9
Glabb/L1-QWwen-115B--Xaaaa
0.5
0.7
Blabb/L1--ween-1BBBMMax
0.7
0.9
Niclyang/HessCCRRLLLLBBBePeeee
1.0
0.7
Operreesonerzzzm///peeeeeeeeeeeo
0.7
0.5
open-thoughtts/pppntliikee-B
0.8
0.95
[hhoo3300/LighhttR1-71BB1
0.7
1.0
RUC-AIIOOXSSTILLLLL1B--prreer
1.0
0.6
simplescaling//s1.177B
1.0
0.9
GAÍR/LIMR
0.6
0.6
open=1l/OpenR1--Owen-T7
0.8
0.9
Stvee//@wwnn255115--nnnttt
0.2
1.0
Oween/Qwee2.557B-Insstrr
0.4
0.95
Qwen/Qwent
1ween/QQwn2255Math-1333
0.7
0.5
Owenn/Qwenn..5-Mathh77
0.5
0.5
sail
0.6
0.6
0.6
0.6
37


Reviews:
[{"review_id": "vtGfA6Cmx8", "reviewer": "Reviewer_2SGm", "summary": "This paper studies the volatility of evaluation practices in reasoning research. Through a comprehensive investigation of open models, the authors diagnose significant variance in evaluation across different configurations of random seed, temperature, top-p, hardware specifications, and prompt format. They then propose a standard procedure for evaluation standardization, which leads to findings about the realistic effectiveness of RL and SFT algorithms.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The call for a more rigorous evaluation is important and timely. I do believe that the first part (section 2) of the investigation which highlights the numerous parameters that can affect evaluation is comprehensive.", "reasons_to_reject": "Section 3.3, which comprises a very important part of the paper narrative, is not easy to follow and understand. Table 3 is overwhelming with no markings to help digest it. It's hard to understand what numbers the text is trying to refer to while reading. The table's caption says, \"Best scores per benchmark are highlighted.\" But that's not the case. The text also says, \"Open Reasoner Zero 7B was the only RL-trained model to consistently outperform the instruct-tuned baselines across all benchmarks.\" But doesn't it underperform Qwen (Instruct) on Minerva? In general, RL and SFT systems need to be easier to tell apart in the table.\n\nAs another example of something that throws off the reader, line 232 says, \"Once again, improvements seen on AIME'24 did not generalize to AIME'25,\" which is completely out of place. It seems to belong to the next paragraph under *Overfitting and Generalization*. As it's hard for me to connect the table and the text, I also find it difficult to confirm if it corroborates the authors' claims.\n\nAlso, a dedicated \"Related Work\" section is missing. While the authors refer to related work throughout the text, given that similar sensitivities in evaluation have been studied elsewhere, a consolidated \"Related Work\" section would be helpful.", "questions_to_authors": "I'm not sure what Figure x on line 124 refers to.", "extra_scores": {}}, {"review_id": "tTD64eyi15", "reviewer": "Reviewer_ja19", "summary": "The authors measure performances of multiple LMs ranging from 1.5B ~ 7B parameter sizes and provide practical guidelines for reproducible evaluation. Specifically, the authors test the effect of (1) running the evaluation multiple times (bootstrapping), (2) setting different temperature and top p hyper-parameters, (3) using different hardware, (4) different prompt formats, and (5) different context lengths (max new tokens hyper-parameter).", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "While some might view that the findings are too trivial, I think that the authors pose an important message that are supported with practical guidelines that make this paper suitable to be presented at COLM.", "reasons_to_reject": "There are no reasons to reject this paper, yet, it would be great if the authors could answer the questions I have on the \"Questions to Authors\" tab.", "questions_to_authors": "1. Figure 1,2,3,4 gives important guidelines for practitioners to take into account when running evaluations, yet, is there a reason why the authors haven't checked whether the results are statistically significant? In addition to reporting that the variance tends to decrease (Figure 2) and whisker plots (Figure 1,3,4), maybe you could also run a paired t-test or report confidence intervals for ablating different configurations.\n\n2. While reporting the LM's performance with a single run is a bad practice, I think that the reason why it is hard to reproduce results in AIME24, AMC23, and MATH-500 also stems from the fact that the instances consisting the benchmark are too small. While it might be out of scope, could the authors also provide guidelines for benchmark creators as well?", "extra_scores": {}}, {"review_id": "ZA1dqv2Rnr", "reviewer": "Reviewer_xRby", "summary": "The paper assesses robustness in performance of both RL and SFT models in reasoning benchmarks, with a focus on maths. They investigate in particular the dependence on seeds, prompts, hardware, and sampling parameters.\n\nThey show that, also due to the small size of many benchmarks, results are brittle. In particular, several RL frameworks do not show consistent improvements over baseline models and can also overfit. \n\nThe authors provide a standardised framework, including optimized parameters per model, that allows fair and consistent, reproducible evaluation, including variance and statistical significance analysis.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "--- The paper is very systematic and its results are well-supported.\n--- Although some of the results (such as variation in small benchmarks, or seed dependence) are known in ML and statistics in general, it is valuable to be reminded of them in the LLM reasoning context.\n--- The paper investigates several aspects such as decoding parameters and hardware configurations that are less often reported\n--- The evaluation framework, if published, would greatly promote reproducibility and help to alleviate overclaims in other papers.", "reasons_to_reject": "--- Some of the results such as variation in small benchmarks are more or less to be expected. Here I also miss citation of papers such as Card et al (EMNLP 2020, with little power comes great responsibility) or Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of ACL. It seems that the NLP community needs to rediscover basic statistical facts on evaluation every 5 years or so. \n\n--- In line 2019 and following lines you say that for example FastCurl yields only modest improvements over the baseline on AMC 23, Math500 and OlympiadBench. I cannot see this from Table 3 where there are very clear significant gains of the model especially for Math500 and OlympiadBench with no overlapping confidence intervals.  I think here you need to be more nuanced. I am therefore not sure that one of your claims of the lack of progress via RL is completely true.\n\n--- Why are there more models in Table 3 than in Table 1? It would be good to see the information in table 1 for all models.", "questions_to_authors": "See above.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work investigates the stability of results of reasoning benchmarks, with a focus on math benchmarks. The authors investigate the robustness of results with respect to factors like random seeds, prompts, hardware, and decoding parameters. They then propose a standardized framework for how to perform more reliable evaluation of reasoning abilities.

All reviewers agreed that this was a valuable and very well executed study highlighting many of the shortcomings in evaluations of reasoning abilities.

While a lot of these findings in general (beyond the assessment of reasoning benchmarks) are not extremely new, they nicely highlight that the field should take better care in performing evaluations and comparisons between models. Furthermore, all reviewers found the proposed framework useful.

The reviewers made several valuable suggestions regarding presentation, especially regarding Section 3, which should be incorporated in the final version of this paper.
Decision: Accept