rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Authors: Xinyu Guan
Li Lyna Zhang
Yifei Liu
Ning Shang
Youran Sun
Yi Zhu
Fan Yang
Mao Yang
Venue: ICML 2025 oral (2025)
Keywords: LLM
Reasoning
Self-evolution
Primary area: deep_learning->large_language_models
TL;DR: We present rStar-Math to demonstrate that small language models (SLMs, 1.5B-7B) can rival or even surpass the math reasoning capability of OpenAI o1

Abstract: We present rStar-Math to demonstrate that small language models (SLMs) can rival or even surpass the math reasoning capability of  OpenAI o1, without distillation from superior models. rStar-Math achieves this by exercising ``deep thinking'' through Monte Carlo Tree Search (MCTS), where a math policy SLM performs test-time search guided by an SLM-based process reward model. rStar-Math introduces three innovations to tackle the challenges in training the two SLMs: (1) a novel code-augmented CoT data synthesis method, which performs extensive MCTS rollouts to generate step-by-step verified reasoning trajectories used to train the policy SLM; (2) a novel process reward model training method that avoids na\"ive step-level score annotation, yielding a more effective process preference model (PPM); (3) a self-evolution recipe in which the policy SLM and PPM are built from scratch and iteratively evolved to improve reasoning capabilities. Through 4 rounds of self-evolution with millions of synthesized solutions for 747k math problems, rStar-Math boosts SLMs' math reasoning to state-of-the-art levels. On MATH benchmark, it improves Qwen2.5-Math-7B from 58.8% to 90.0%, surpassing o1-preview by +4.5%. On the USA Math Olympiad (AIME), rStar-Math solves an average of 53.3% (8/15) of problems, ranking among the top 20% of the brightest high school math students. Code and data are available at https://github.com/microsoft/rStar.

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Xinyu Guan 12 Li Lyna Zhang" II
Yifei Liu 31 Ning Shang
Youran Sun 14
Yi Zhu'
Fan Yang"
Mao Yang'
Abstract
rStur-Math (Qwen-7B) (P'ii--mini)
OpenAl ol ul-preview
OpenAI ul-mini
GPT-4o DeepSeok-V3
(pase I Acc)
MATH
90.0
88.6
85.5
90.0
76.6
90.2
AIME 2024
53.3
46.7
44.6
56.7
9.3
39.2
We present rStar-Math to demonstrate that small language models (SLMs) can rival or even sur- pass the math reasoning capability of OpenAl ol, without distillation from superior models. rStar- Math achieves this by exercising "deep think- ing" through Monte Carlo Tree Search (MCTS), where math policy SLM performs test-time search guided by an SLM-based process reward model. rStar-Math introduces three innovations to tackle the challenges training the two SLMs: (1) a novel augeenu CoT data synthesis method, which performs extensive MCTS roll- outs generate step-by-step verified reasoning trajectories used to train the policy SLM; (2) novel process reward model training method that avoids naive step-level score annotation, yield- ing a more effective process preference model (PPM); (3) a self-evolution recipe in which the policy SLM and PPM are built from scratch and iteratively evolved to improve reasoning capabil- ities. Through rounds of self-evolution with millions of synthesized solutions for 747k math problems, rStar-Math boosts SLMs' math reason- ing to state-of-the-art levels. On MATH bench- mark, it improves Qwen2.5-Math-77 from 58.8% to 90.0%, surpassing ol -preview by +4.5%. On the USA Math Olympiad (AIME), rStar-Math solves average of 53.3% (8/15) of problems, ranking among the top 20% of the brightest high school math students. Code and data are available at ht https:/// //g thub. cww. mi crosoft/rStt t/Sarrr.
Olympiod Bench
College
Omnii Math
52.5
60.5
49.5
35.9
Table 1. rStar-Math enables frontier math reasoning in SLMSs via deep thinking over 64 trajectories.
1 Introduction
Recent studies have demonstrated that large language mod- els (LLMs) are capable of tackling mathematical prob- lems (Team, 2024; Yang et al., 2024; Liu et al., 2024). How- ever, the conventional approach having LLMs generate complete solutions in a single inference akin to System I thinking (Daniel, 2011) often yields fast but error-prone re- sults (Valmeekam al., 2023; OpenAl, 2023). In response, test-time compute scaling (Snell et al., 2024; Qi et al., 2024) suggests paradigm shift toward System 2-style think- ing, which emulates human reasoning through a slower and deeper thought process. In this paradigm, an LLM serves as policy model to generate multiple math reasoning steps, which are then evaluated by another LLM acting as reward model (OpenAI, 2024). The steps and solutions deemed more likely to be correct selected. The process repeats iteratively and ultimately derives the final answer.
In the test-time compute paradigm, the key is to train a pow- erful policy model that generates promising solution steps and a reliable reward model that accurately evaluates them, both of which depend on high--uallit training data. Unfortu- nately, it well-known that off-the-sheel high-quality math reasoning data is scarce, and synthesizing high-quality math data faces fundamental challenges. For the policy model, it is challenging to distinguish erroncous reasoning steps from the correct ones, complicating the elimination of low-quality data. It is worth noting that in math reasoning, a correct final answer does not ensure the correctness of the entire rea- soning trace (Lanham al., 2023). Incorrect intermediate steps significantly decrease data quality. As for the reward model, process reward modeling (PRM) shows great po- tential by providing fine-grained feedback on intermediate steps (Lightman et al., 2023). However, the training data is even scarcer in this regard: accurate step-by-ster feedback
'Equal contribution Microsofft Research Asia 'Peking Univer- sity Uniiersitty of Science and Technology of China singgau University: Xinyu Guan, Yifei Liu and Youran Sun did this work during the internship MSRA. Correspondence to: Lyna Zhang <lzhani @ microooott.cmm>
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

rStar-Math: Small LLMS Can Master Math Reasoning with Self-Evolved Deep Thinking
requires intense human labeling efforts and is impractical to scale, while those automatic annotation attempts show limited gains due to noisy reward scores (Luo et al., 2024; Chen et al., 2024). Due to the above challenges, existing distill-based data synthesis approaches to training policy models, e.g., scaling up GPT4-distilled CoT data (Tang et al., 2024; Huang et al., 2024), have shown diminishing returns and cannot exceed the capability of their teacher model; meanwhile, of today, training reliable PRMs for math reasoning remains open question.
Finally, a four-round self-evolution recipe that progressively builds both a frontier policy model and PPM from scratch. We begin by curating a dataset of 747k math word problems from publicly available sources. In each round, we use the latest policy model and PPM to perform MCTS, generat- ing increasingly high-quality training data using the above two methods to train a stronger policy model and PPM for next round. Each round achieves progressive refinement: (1) stronger policy SLM, (2) more reliable PPM, (3) generating better reasoning trajectories via PPM-augmented MCTS, and (4) improving training data coverage to tackle more challenging competition-level math problems.
In this work, we introduce rStar-Math, a self-evolvable System 2-style reasoning approach that achieves the state- oh-the-art math reasoning, rivaling and sometimes even surpassing OpenAl ol on challenging math competition benchmarks with a model size as small as billion. Un- like solutions relying superior LLMs for data synthe- sis, rStta-Math leverages smaller language models (SLMs) with Monte Carlo Tree Search (MCTS) to establish self- evolutionary process, iteratively generating higher-quality training data. To achieve self-evvlution, rStar-Math intro- duces three key innovations.
Extensive experiments across four SLMs (1.5B-7B) and seven math reasoning tasks demonstrate the effectiveness of rStar-Math. Remarkably, rStar-Math improves all four SLMs, matching even surpassing OpenAl ol on chal- lenging math benchmarks. On MATH benchmark, with 8 search trajectories, rStar-Math boosts Qwen2.5-Math- 7B from 58.8% to 89.4% and Qwen2.5-Math-1555 from 51.2% to 87.8%. With 64 trajectories, the scores rise to 90% and 88.4%, outperforming ol -p-perv by 4.5% and
2.6% and matching ol-mini's 90%. On the Olympiad-level AIME 2024, rStar-Math solves on average 53.3% (8/15) of the problems, exceeding ol ol-preview by 8.7% and all other open-sourced LLMs. We further conduct comprehensive experiments to verify the superiority of step-by-step verified reasoning trajectories over statc-of-the-uu data synthesis baselines, as well as the PPM's effectiveness compared to outcome reward models and Q value-based PRMs.
First, a novel code-augmented CoT data synthesis method, which performs extensive MCTS rollouts to generate step- by-step verified reasoning trajectories with self-annotated MCTS Q-values. Specifically, math problem-solving is de- composed into multi-step generation within MCTS. At each step, the SLM serving as the policy model samples candidate nodes, each generating a one-step CoT and the correspond- ing Python code. To verify the generation quality, only retained,
nodes with successful Python code execution thus mitigating errors in intermediate steps. Moreover, ex- tensive MCTS rollouts automatically assign Q-value to each intermediate step based on its contribution: steps con- tributing to more trajectories that lead to the correct answer are given higher Q-values and considered higher quality. This ensures that the reasoning trajectories generated by SLMs consist of correct, high-quality intermediate steps.
2 Related Works
Math Data Synthesis. Advancements in LLM math reason- ing have largely relied on curating high-quality CoT data, with most leading approaches being GPT-distilled, using frontier models like GPT-4 for synthesis (Wang et al., 2024b; Gou al., 2023; Luo et al., 2023). Notable works include NuminaMath (Jia LI & Polu, 2024a) and MetaMath (Yu et al., 2023b). While effective, this limits reasoning to the capabilities of the teacher LLM. Hard problems that the teacher LLM cannot solve are excluded in the training set. Even solvable problems may contain error-pron interme- diate steps, which are hard to detect. Although rejection sampling methods (Yuan et al., 2023; Brown et al., 2024) can improve data quality, they do not guarantee correct in- termediate steps. As a result, naively scaling up CoT data generation using frontier LLMs yields diminishing returns, with performance gains approaching saturation.
Second, a novel method that trains an SLM acting as process preference model, i.c.. a PPM to implement the de- sired PRM, that reliably predicts a reward label for each math reasoning step. The PPM leverages the fact that, al- though Q-values are still not precise enough to score each reasoning step despite using extensive MCTS rollouts, the Q-values can reliably distinguish positive (correct) steps from negative (irrelevant/incorree) ones. Thus the training method constructs preference pairs for each step based on Q-values and uses a pairwise ranking loss (Ouyang et al., 2022) to optimize PPM's s score prediction for each reason- ing step, achieving reliable labeling. This approach avoids conventional methods that directly use Q-values as reward labels (Luo et al., 2024; Chen et al., 2024), which are inher- ently noisy and imprecise in stepwise reward assignment.
Scaling Test-time Compute has introduced new scaling laws, allowing LLMs to improve performance across by generating multiple samples and using reward models for best-solution selection (Snell et al., 2024; Wu et al., 2024; Brown et al., 2024). Various search methods have been proposed (Kang et al., 2024; Wang et al., 2024a), including
2

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
MCTS driven deep thinking
question
Q-value filtering
PPM
0.8
6.7
Apply Verifiers (P PM/ pychon)
Step
Step
sinalstep
full solutions
(b) Construction of per-step preference pairs based on Q-values
One step
0.5
0.
6.6
Answer step (correct)
Terminal-guided
Teiddd
SLM-12
PPM-aupmente MCTS
5LM.1
MMaammmmetted L
MCTS
SLM-r1
MCTS
PPM-12
PPM-13
MCTS
PPM-
stswerstep (wrong)
Round
Round
Round
Round 4
(a) step step-by-ster step verified reasoning trajectory
(c) 4 rounds of selfevolutionn Figure I. The overview of rStar-Math.
random sampling (Wang et al., 2023) and tree-search meth- ods (Yao et al., 2024; Zhang et al., 2024a; Qi et al., 2024) like MCTS. However, open-source methods for scaling test- time compute have shown limited gains in math reasoning, often due policy LLM reward model limitations.
training data. Finally, we present a novel process reward model training approach that eliminates the need for pre- cise per-step reward annotations, yielding the more effective process preference model (PPM).
yieldinn
3.1 Step-by-Step Verified Reasoning Trajectory
Reward Models are crucial for effective System 2 reason- ing but are challenging to obtain. The major works include Outcome Reward Model (Yang et al., 2024; Yu et al., 2023a) and Process Reward Model (PRM) (Lightman et al., 2024). While PRMs offer promising dense, step-level reward sig- nals for complex reasoning (Luo al., 2024), collecting step-level annotations remains an obstacle. While (Kang et al., 2024; Wang et al., 2024a) rely costly human- annotated datasets like PRM800K (Lightman et al., 2024), recent approaches ( (Wang et al., 2024c; Luo et al., 2024) explore automated annotation via Monte Carlo Sampling or MCTS. However, they struggle to generate precise reward scores, limiting performance gains. rStar-Math introduces novel process preference reward (PPM) that eliminates the need for accurate step-level reward score annotation.
Given problem x and policy model M, we use standard MCTS to incrementally construct search tree for step-by- step solution exploration. As shown in Fig. ((a), the root node represents question æ, and child nodes correspond to intermediate steps generated by M. A root-to-leaf path ending terminal node sa forms trajectory
$1 t $2 i ... 8a, with each step 8; assigned Q-value ((+)). From the search tree T, we extract solution trajectories T {t' tt',t?... .t....,,t"! (n 2 1). Our goal is to select high- qugaituality trajectories from T to construct the training set. To achieve this, we introduce code-augmented CoT synthesis method to filter out low-quality generations and perform extensive rollouts to improve the reliability of Q-value accuracy.
Code-augmented CoT Generation. Prior MCTS ap- proaches primarily generate natural language (NL) CoTs (Qi et al., 2024; Zhang et al., 2024b). However, LLMs often suffer from hallucination, producing incorrect or irrelevant steps yet still arrive at the correct answer by chance (Lan- ham et al., 2023). These flawed steps are challenging to detect and eliminate. To address this, we propose novel code execution augmented COT. As shown in Fig. 2, the policy model generates one-step NI CoT alongside its corresponding Python code, where the NL CoT is embedded as a Python comment. Only generations with successfully exccuted Python code are retained as valid candidates.
3 Methodology
In our work, we explore using two 7B SLMs to generate higher-quality training data. However, self-generating data presents significant challenges for SLMs. SLMs frequently fail to generate correct solutions, and even when the final answer is correct, the intermediate steps are often flawed or of poor quality. Moreover, SLMs solve fewer challenging problems compared to advanced models like GPT-4
This section introduces our methodology, as illustrated in Fig. 1. To mitigate errors and low-quality intermediate steps, we introduce a code-augmented CoT synthetic method, which performs extensive MCTS rollouts to generate step- by-step verified reasoning trajectories, annotated with Q- values. To further improve SLM performance on chal- lenging problems, we introduce four-round self-evolution recipe. each round, both the policy SLM and the reward model are updated stronger versions, progressively tack- ling more difficult problems and generating higher-quality
Specifically, starting from the root node x, we perform multiple MCTS iterations through selection, expansion, simulation, and back-propagation. At step we collect the latest reasoning trajectory z C 81 θ 82 @ 8i-1 as the current state. Based on this state, we prompt (see Appendix A.4) the policy model to generate n candidates $ 8i Python code execution is then employed to filter valid nodes. As shown in Fig. 2, each generation
3

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Question: Bill walks 32 mile south, then HV Is mile east, and finally 32 & mile south. How many miles is he, in direct line, from his starting point? E Express your answer as decimal the nearest hundredth.
Pydhon code execution for step 1:
1*5tep 1: Calculete the total distance walked south
NL CoT as Python Comment
"Lola, Sonih
Step Calculate the total distance walked south lotal
*Stép 2: Calculate the total distance walked east
total, east= 33/4
To www. www. www.enttiooo....................
#Step 3: Use the Pythagorean theorem to find the direct distance from the starting point math
Pychon code execution for step 2:
direct, dislance = mathhsertitalll soubhrr2 + total, east")2)
Step Calculate the distance walked south total south 1/2 +1/2
*Step Round the direct distance to nearest hundreath
direct distance rounded - roundidirect distance, 2
@tep 2: Calculate the total distance walked east east=
From the result, can see that the direct distance from the starting isintis Vooxed jmiilsss
Figure 2. An example of Code-augmented COT. To avoid syntax errors, the code at step is executed by concatenating it with code from all previous steps. For example, step 2 includes code from step to ensure correct execution.
St is concatenated with the code from all previous steps, forming 81 & $2 @.... @ 8;-1 @ St.j. Candidates that execute successfully are retained as valid nodes and scored by the PPM, which assigns Q-value g(s;). Then, we use the well- known Upper Confidence bounds for Trees (UCT) (Koesis & Szepesvári, 2006) to select the best node among the n candidates. This selection is mathematically represented as:
when the PPM is unavailable or insufficiently accurate, we use serminal-guided annotation. Formally, let g(s:)* denote the q value for step 8; after back pack-propagation in the hth rollout, we score per-step Q-value as follows:
q(s:)* (s;))* + q(sa)";
where the initial q value (sx)) = in the first rollout. If this step frequently leads to a correct answer, its q value will increase; otherwise, it decreases. Terminal nodes (denoted as 8, in Eq. 2) are scored as ((aa) = 1 for correct answers and g(sa) = -1 otherwise, as shown in Fig. 1.
In Mparent(s) (CT(s) = =(a)= + +
q(*) (s))
(a))
(I)
N(+)
where (s) denotes the number of visits to node s, and (prra(()) is the visit count of s's 's parent node. The predicted reward (s)) is provided by the PPM and will be updated through nack-propagation. is constant that balances exploitation and exploration.
PPM-angmented annotation. Starting from the third round, we use PPM to score each step for more effective generation. Compared to cerminal-guided annotation. which requires multiple rollouts for meaningful g value, PPM directly predicts non-zero initial q value. PPM also helps the policy model to generate higher-quality steps, guiding solutions towards correct paths, Formally, for step 8; PPM predicts an initial g(s;)) value based on the partial trajectory:
We repeat the process until reaching a terminal node, either by arriving a final answer or hitting the maximum tree depth. This is referred to as a rollout. Based on whether the rollout reaches the correct answer, we perform back- proppagatonnnn to update the trajectory Q-value scores. We introduce the score annotation methods in next sections.
((s,) = PPM B $1 ① 2 T s ① s;)
(3)
Extensive Rollouts for Q-value Annotation. Accurate Q- value Q(s) annotation in Eq. 1 is crucial for guiding MCTS towards correct problem-solving paths and identifying high- quality steps within trajectories. Following AlphaGo (Silver et al., 2017) and rStar (Qi et al., 2024), we perform MCTS rollout to assign a Q-value to each step. However, insuf- ficient rollouts can lead to spurious Q-value assignments, such as overestimating suboptimal steps. To mitigate this issue, we draw inspiration from Go players, who refine their evaluations for each move through repeated gameplay. Anal- ogously, within each rollout, we update the Q-value of each step based on its contribution to achieving the correct final answer. Through extensive MCTS rollouts, steps consis- tently leading to correct answers achieve higher Q-values, occasional successes yield moderate Q- Q-values, and consis- tently incorrect steps receive low Q-values. Specifically, we introduce two self-annotatio methods to obtain these step-level Q-values. Fig. 1(c) shows the detailed setting in the four rounds of svll-evolution.
This q value will be updated based on terminal node's 's ((aa) value through MCTS back- pockppropagation in Eq. 2. For termi- nal node 8d, we do not use PPM for scoring during training data generation. Instead, we assign a more accurate score based on ground truth labels as cerminalgguidet rewarding.
3.2 Process Preference Model
Process reward models, which provide granular step-level reward signals, is highly desirable for solving challenging math problems. However, obtaining high-quality step-level training data remains an open challenge. Existing methods rely on human annotations (Lightman et al., 2023) or MCTS- generated scores (Zhang et al., 2024b; Chen et al., 2024) to label a score for each step. These scores then serve as training targets, with methods such as MSE loss (Chen et al., 2024) or pointwise loss (Wang et al., 2024c; Luo et al., 2024; Zhang et al., 2024b) used to minimize the difference between predicted and labeled scores. As a result, the precision of these annotated step-level scores determines the effectiveness of the final process reward model.
Terminal- guided annotation. During the first two rounds,
4

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Unfortunately, precise per-step scoring remains a unsolved challenge. Although our extensive MCTS rollouts improve the reliability of Q-values, precisely evaluating fine-grained step quality presents a major obstacle. For instance, among a set of correct steps, it is difficult to rank them as best, second-best, or average and then assign precise scores. Sim- ilarly, among incorrect steps, differentiating the worst from moderately poor steps poses analogous challenges. Even expert human annotation struggles with consistency, partic- ularly at scale, leading to inherent noise in training labels.
Table 2. Percentage of the 747k math problems correctly solved in each round. The first round uses DeepSeek Coder- as the
policy LLM, while later rounds use our one-ttned 7B policy SLM.
#
models in MCTS
GSM-level MATH-ljovl Olympina--evel
All
Round
DeepSeek-Coo Coder V2-Instract
96.61%
67.36%
20.99%
60.
Round 2
policy SL.M-1
97.88%
67.40%
56.04%
66 60%
Round
policy S1.M-2, PPM-12
98.15%
88 696
62.16%
27 86%
Round 4
policy SLM-13, PPM-13
98.15%
94.53%
80.58%
90.25%
Reasoning Trajectories Collection. Instead of using the original solutions in the 747k math dataset, we conduct ex- tensive MCTS rollouts (Sec. 3.1) to generate higher-quality step-by-step verified reasoning trajectories. In each self- evolution round, we perform 16 rollouts per math prob- lem, which leads to 16 reasoning trajectories. Problems are then categorized by difficulty based on the correct ratio of the generated trajectories: easy (all solutions are correct), medium (a mix of correct and incorrect solutions) and hard (all solutions are incorrect). For hard problems with no correct trajectories, an additional MCTS with 16 rollouts is performed. After that, all step-by-ste trajectories and their annotated Q-values are collected and filtered to train the policy SLM and process preference model.
We introduce a novel training method that trains a process preference model (PPM) by constructing step-level positive- negative preference pairs. As shown in Fig. I(b), instead of using Q-values as direct reward labels, we use them to select steps from MCTS tree for preference pair construction. For each step, we select two candidates with the highest Q- values as positive steps and two with the lowest as negative steps. Critically, the selected positive steps must lead to a correct final answer, while negative steps must lead to incorrect answers. For intermediate steps (except the final answer step), the positive and negative pairs share the same preceding steps. For the final answer step, where identical reasoning trajectories rarely yield different final answers, we relax this restriction. We select two correct trajectories with the highest average Q-values as positive examples and two incorrect trajectories with the lowest average Q-values as negative examples. Following (Ouyang et al., 2022), we define our loss function using the standard Bradley-Tenn Terry model with a pairwise ranking loss:
Self-evolution Recipe. Due to the weaker capabilities of SLMs, we perform four rounds of self-evolution to pro- gressively generate higher-quality data. Each round uses MCTS to generate step-by-step verified reasoning trajecto- ries, which are then used to train the new policy SLM and PPM. The new models are then used in next round. Fig. I(c) and Table 2 detail the models used for data generation in each round, along with the new trained models. Next, we outline the specific improvements targeted in each round.
Eypin (0) - Flr. of nes En) v²00))
+ Round I: Bootstrapping initial strong policy SLM-rI. To enable SLMs to self-generate reasonably good training data, we perform a bootstrap round to fine-tune an initial strong policy model. As shown in Table 2, we run MCTS with DeepSeek- -oder-VV--Issteee
u = $1 @ @ 8 @ s
when is not final answer step,
yi key 81 @ ... O 1 2.
(4)
re (zz, 31) denotes the PPM output, where is the problem and y is the trajectory from the first step to the tth step. o denotes the sigmoid function.
(236B) to collect the SFT data, With no available reward model in this round, we use terminal-guie -uideed annotation for Q-values and limit MCTS to 8 rollouts for efficiency. For correct solutions, the top-2 trajectories with the highest average Q-values are selected as SFT data, which are then used to SFT the first policy SLM, denoted as SLM-rl.
3.3 Sell-Evolved Deep Thinking
Math Problems Collection. We collect a large dataset of 747k math word problems with ground-truth answers, primarily from NuminaMath (Jia LI & Polu, 2024a) and MetaMath (Yu et al., 2023b). Notably, only competition- level problems (c.g., Olympiads and AIME/AMC) from NuminaMath are included, as we observe that grade-school- level problems do not significantly improve LLM complex math reasoning. To augment the limited competition-level problems, we follow (Li al., 2024) and use GPT-4 to synthesize new problems based on the seed problems in 7.5k MATH train set and 3.6k AMC-AIME training split. However, GPT-4 often generated unsolvable problems or incorrect solutions for challenging seed problems. To filter these, we prompt GPT-4 to generate 10 solutions per prob- lem, retaining only those with at least consistent solutions.
+ Round 2: Training a reliable PPM-r2. In this round, with the policy model updated to the 7B SL.Mrl, we con- duct extensive MCTS rollouts for more reliable Q-value annotation and train the first reliable reward model, PPM-12. Specifically, we perform 16 MCTS rollouts per problem. The resulting step-by-step verified reasoning trajectories show significant improvements in both quality and Q-value precision. As shown in Appendix Table 10, PPM-12 is no- tably more effecttiee than in the bootstrap round.
+ Round 3: PPM-augmented MCTS to significantly im-
prove data quality. With the reliable PPM-r2, we perform
5

---

rStar-Math: Small LLMS Can Master Math Reasoning with Self-Evolved Deep Thinking
PPM-augmented MCTS in this round to generate data, lead- ing to significantly higher-quality trajectories. The gener- ated reasoning trajectories and annotated Q-values are then used to train the new policy SLM-r3 and PPM-r3, both of which show significant improvements.
fier LLMs, including GPT-4o, the latest Claude, OpenAl ol-preview and ol-mini. We measure their accuracy on AMC 2023, Olympiad Bench, College Math, Gaokao and GSM8K, with accuracy numbers for other benchmarks are taken from public technical reports (Team, 2024). (ii) Open- sourced superior reasoning models; (iil) Both System I and System performance of the base models trained from the original models teams, including Instruct versions (e.g.. 1rren.5-Math--B. 1-Maaah---Instruuct and Best-of-N (e.g.. Qwen2.5- Math-72B- 11r7222-IInttttt-200 Swweoot+wwe22.Mattttt 3-MMah-RM--2BB) Notably, the reward model used for the three Qwen base models is a 72B ORM, significantly larger than our 7B PPM.
4 Round 4: Solving challenging problems. After the third round, while grade school and MATH problems achieve high success rates, only 12% of Olympiad-level problems included in the training set. This is NOT solely due to weak capabilities in our SLMs, as many Olympiad problems remain unsolved by GPT-4 or ol. To improve coverage, we adopt a straightforward strategy. For unsolved problems after 16 rollouts, we perform an additional 64 rollouts, and if needed, increase to 128. We also conduct multiple MCTS tree expansions with different random seeds. This boosts the success rate of Olympiad-leve problems to 80.58%
Evaluation Metric. We report Pass@1 accuracy for all baselines. For System 2 baselines, we use default evaluation settings, such as default thinking time for ol ol-mini and ol- preview. For Qwen models with Best-of-N, we re-evaluate MATH-500 and AIME/AMC accuracy; other benchmarks results are from their technical reports. For fair compari- son, rStar-Math run MCTS to generate the same number of solutions as Qwen. Specifically, we generate 16 trajectories for AIME/AMC and for other benchmarks, using PPM to select the best solution. We also report performance with in- creased test-time computation using 64 trajectories, denoted as rSthr-Math
After four rounds of self-evolution, 90.25% of the 747k math problems are covered in the training set, as shown in Table 2. Among the remaining unsolved problems, most are synthetic. A manual review of a random sample of 20 problems reveals that 19 are incorrectly labeled. Thus, we conclude that the unsolved problems are of low quality and stop the self-evolution at round 4.
4 Evaluation
4.2 Main Results
4.1 Setup
Results on diverse challenging math benchmarks. Ta- ble 3 shows the results of rStar-Math with comparing to of-te-of-the-art reasoning models. We highlight two key observations: (1) rStar-Math significantly improves SLMS math reasoning capabilities, achieving performance compa- rable to or surpassing OpenAl ol with substantially smaller model size (1.5B-7B). For example, 1yee2.5-MMah--1 originally at 58.8% accuracy on MATH, improved dramati- cally to 90.0% with rStar-Math, outperforming ol -prevvvew and Claude 3.5 Sonnet while matching ol-mini. On the College Math benchmark, rStar-Mate exceeds o1-mini by 2.7%. On AIME 2024, rStar-Math scored 53.3%, rank- ing just below ol-mini, with the 7B model solving 8/15 problems in both AIME I and II, placing in the top 20% of the brightest high school math students. Notably, 8 of the unsolved problems were geometry- requiring visual understanding. a capability rStar-Math currently does not support. (2) Despite using smaller policy models (I.SB- 7B) and reward models (7B), MSth significantly out- performs state-of-the-art System 2 baselines. Compared to Qwen Best-of-N baselines, which use the same base models (Qwen2-Math-7B, 0wen2.5-Math- 1.5B/7B) but a 10x larger reward model (Qwen2.5-Math Maan-5--Math--MM72B)) Math consistently improves the reasoning accuracy of all base models state-of-the-art levels. Even against Best- of-N with a 10x larger 2...----Maaa-72--nsttte policy model, Mttr surpasses it on all benchmarks except
Evaluation Datasets. We evaluate rStar-Math on diverse mathematical benchmarks. In addition to the widely-used GSM8K (Cobbe et al., 2021), we include challenging benchmarks from multiple domains: (i) competition and Olympiad-leve benchmarks, such as MATH-500 (Lightman et al., 2023), AIME 2024 (AI-MO, 2024a), AMC 2023 (AI- MO, 2024b) and Olympiad Bench (He et al., 2024). Specifi- cally, AIME is the exams designed to challenge the bright- est high school math students in America; (ii) college-level math problems from College Math (Tang et al., 2024) and (iii) out-of-domain math benchmark: GaoKao (Chinese Col- lege Entrance Exam) En 2023 (Liao et al., 2024).
Base Models and Setup. We use SLMs of differ- ent sizes as the base policy models: Qwen2.5-Math- 1.5B (Qwen, 2024b), Phi3-mini- Instrrt (3B) (Abdin al., 2024), Qwen2-Math-7B (Qwen, 2024a) and 2-ath.5-Math- 7B (Qwen. 2024c).Due to limited GPU resources, we per- formed 4 rounds of self-evolution exclusively on Qwen2.55 Math-7B, yielding 4 evolved policy SLMs (Table 9) and 4 PPMs (Table 10). For the other 3 policy models, we fine- tune them using step-by-step verified trajectories generated from 1een2.5-Math-11 's 4th round. PPM from this round is then used as the reward model for the 3 policy SLMs.
RM-72B), rStar-
Baselines We compare against three strong baselines repre- senting both System and System 2 approaches: (i) Fron-
6

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Table The results of rStar-Mart and other frontier LLMs on the most challenging math benchmarks. rStar-Math reports Pass I accuracy when searching 16 trajectories for AIME/AMC and 8 for others. rStar rStar-Math" shows results when sampling 64 trajectories.
Competition and College Level
OOD
Olympiad Bench
Model
Method
MATH
AIME 2024
AMC 2023
College Math
GSM8K
GaokanFin 2023
Frontier LLMs
GPT-4o
System 1
76.6
9.3
47.5
43.3
48.5
92.9
67.5
Claudc3.5-Sonnec
System
78.3
16.0
96.4
-PT-ol-preview
85.5
44.6
90.0
GPT-ol-mini
90.0
56.7
95.0
65.3
57.8
78.4
Open-Sourced Reasoning LLMs
CrepSock-Cooee-V2-BBl
System
75.3
13.3
57.5
37.6
46.2
94.9
64.7
NuminaMath-72D-Cor
System 1
64.0
3.3
70.0
32.6
39.7
90.8
58.4
LLaMA3. 1-0ttt-nttttt Qwen2.5-Moe 1...55-MMt----BBBB
System 1
65.4
23.3
50.0
27.7
42.5
94.1
54.0
System 1
85.6
30.0
70.0
49.0
49.5
95.9
71.9
Qwen2.5-Math- 1et--Mah--2B-Imssse 22B-Instruct+728 ORM
System 2
85.8
36.7
72.5
$4.5
50.6
96.4
76.9
General Base Model: Phi:-mini--nstrree (3.8B)
Phi3-mini-He
Instsrct--Instrctt Instruct (base model)
System
41.4
3.33
7.5
12.3
33.1
85.7
37.1
rStar-Math (3.8B SLM+7B PPM)
System
85.4
40.0
77.5
59.3
58.0
94.5
77.1
rStar-Math" (3.8B MLM+7B PPM)
System 86.4
43.3
80.0
60.3
59.1
77.7
Math-Specialized Base Model:
Swen2.5-Math-I1.
Qwen2.5-Maite 1.58 (base model)
System 1
51.2
0.0
22.5
16.7
38.4
74.6
46.5
Qwen2.5-Matte 1.5.5-Ma--11555 MaMMaah--.5B--nsse 1.5B-Instruct
System 1
60.0
10.0
60.0
38.1
47.7
84.8
65.5
Qwen2.5-Math- 1..-5-Mah---.SB-nstnueeee JSB-Instrrut+72BB ORM
System 2
83.4
20.0
72.5
47.3
50.2
94.1
73.0
rStar-Math (1.5B SLM+7B PPM)
System 2
87.8
46.7
80.0
63.5
59.0
94.3
77.7
rStar-Math (L.5B SLM+7B PPM)
System 2
88.6
46.7
85.0
64.6
59.3
94.8
79.5
Math-Specializer Base Model:
Qwen2-Math-78
Qwen2-Math-77 (base model)
System 1
3.3
25.0
17.3
39.4
47.3
1onn2-Matth-B--nssee
System 1
73.2
13.3
62.5
38.2
45.9
62.1
Owen2-Math-78B TB-Instrucc++22 ORM
System 2
83.4
23.3
62.5
47.6
47.9
95.1
71.9
rStar-Math (7B SLM+7B PPM)
System 2
88.2
43.3
80.0
63.1
58.4
94.6
78.2
rStar-Math (7B SLM+7B PPM)
System 2
88.6
46.7
85.0
63.4
59.3
94.8
79.2
Math-Specializee Base Model:
Qwen2.5
M-aahh-- (base model)
System
58.8
0.0
22.5
21.8
41.6
91.6
$1.7
Qwen2.5-Math-77 5-MMath7B--nstruue
System
82.6
6.0
62.5
41.6
46.8
95.2
66.8
1er.5--Math-----nsteee ORM
System 2
88.4
26.7
75.0
49.9
49.6
97.9
75.1
rStar-Math (7B SLM+7B PPM)
System 2
89.4
50.0
87.5
65.3
59.0
95.0
80.5
rStar-Math" 94 SLM+7B PPM)
System 2
90.0
53.3
87.5
65.6
60.5
81.3
01-praview
rStar-Math (7B SLM+7B PPM)
Qwen2.5 Best-of-N (72B LLM+72B ORM)
o1-mini
Qwen2.5 Best-of-N (7B SLM+72B ORM)
MATH
AIME 2024
Olympiad Bench
College Math
90
60
60
40
Accuracy
50
80
1
2
4
8
16
32
64
2
16
32
64
4
16
32
64
2
4
8
16
32
64
#Sampled Solutions
#Sampled Solutions
#Sampled Solutions
#Sampled Solutions
Figure 3. Reasoning performance under scaling up the test-time compute.

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
GSM8K, using the same number of sampled solutions.
where 32 trajectories are randomly sampled from the policy model, with high-quality solutions ranked by our trained ORM (appendix A.2). For faimess, we select two correct trajectories for each math problem in baseline (ii) and (iii). All SFT experiments use the same training recipe.
Scaling up test-time computation. By increasing test- time computation, rStar-Math can explore more trajectories, potentially improving performance. In Fig. 3, we show the impact of test-time compute scaling by comparing the accuracy of the official Qwen Best-of-N across different numbers of sampled trajectories on four challenging math benchmarks. Sampling only one trajectory corresponds to the policy LLM's Pass@1 I accuracy, indicating a fallback to System I reasoning. We highlight two key observations: (1) With only 4 trajectories, rStar-Math significantly out- performs Best-of-N baselines, exceeding ol o1-preview and approaching ol 01-mini, demonstrating its effectiveness. (2) Scaling test-time compute improves reasoning accuracy across all benchmarks, though with varying trends. On Math, AIME, and Olympiad Bench, rStar-Math shows sat- uration or slow improvement at 64 trajectories, while on College Math, performance continues to improve steadily.
Table 5. Ablation study the effectiveness of our step-by-ster ver- ified reasoning trajectories as the SFT dataset. We report the SFT accuracy of Qwen2.5-Math-- fine-tuned with different datasets.
Olympiad
College Math
Dataset
MATH
AIME
Bench
GPT-4o
76.6
9.3
43.3
48.5
GPT4-distilaaton
MeteMath
55.2
3,33
19.1
39,2
(Open-sourced)
Numina Math-Co
69.6
10.0
37.2
43,4
Self-gencration
policy SL.Ma3
Rejection sampling
Random sample
72.4
73.4
10.0
13.3
44.7
41.0
48.0
50.8
Step-by-stee verilied (ours)
78.4
26.7
47.1
52.5
Table 5 shows the math reasoning accuracy of Qwen2.5-
Math-7B fine-tuned on different datasets. We highlight two observations: (i) Fine-tuning with our step-by-step veri- fied trajectories significantly outperforms all other baselines. This is primarily due to our PPM-augmented MCTS for code-augmenter CoT synthesis, which provides denser veri- fication during math solution generation. It proves more ef- feetive than both random sampling, which lacks verification, and rejection sampling, where ORM provides only sparse verification. (ii) Even randomly sampled oudeauugeeted CoT solutions from our SLMM yields comparable or bet- ter performance than GPT-4 synthesized NuminaMath and MetaMath datasets. This indicates that our policy SLMs, af- ter rounds of self-evolution, can generate high high-quality math solutions. These results demonstrates the huge potential of our method to self-generate higher-quality reasoning data without relying on advanced LLM distillation.
4.3 Ablation Study and Analysis
Table The continuously improved math reasoning capabilities via rStar-Math self-evolved deep thinking. Starting from round 2, the 7B base model powered by -Star-Math surpasses GPT-4o.
Olympiad
College Math
Rounde
MATH AIME AMC
Bench
GSM8K Gaokao
4a
76.6
9.3
47.5
43.3
48.5
92.9
67.5
Base 7B model
58.8
0.0
22.5
21.8
41.6
91.6
51.7
MStar-Math Round
75.2
10.0
57.5
35.7
45.4
90.9
60.3
rStar Math Round
75.0
59.4
94.0
rStur Malh Round
87.0
46.7
80.0
61.6
56.5
94.2
77.1
rStar-Math Round 4 89.4
50.0
87.5
65.3
59.0
95.0
80.5
The effectiveness of self-evolution. The impressive results in Table 3 are achieved after 4 rounds of rStar-Math self- evolved deep thinking. Table details the performance across rounds, showing continuous accuracy improvements. In round 1, the main improvement comes from applying SFT to the base model. Round 2 brings significant boost with the application of a stronger PPM in MCTS, which unlocks the full potential of System 2 deep reasoning. Notably, start- ing from round 2, rStar-Math outperforms GPT-4o. Rounds 3 and 4 show further improvements, driven by stronger System 2 reasoning through better policy SLMs and PPMs.
The effectiveness of PPM. We train both strong ORM and Q-value score-based PRM (PQM) for comparison. To ensure a fair evaluation, we use the highest-quality train- ing data: the step-by-step verified trajectories generated in round 4, with selected math problems matching those used for PPM training. Similar to PPM, we use step-level Q- values as to select positive and negative trajectories for each math problem. The ORM is trained using a pairwise ranking loss (Ouyang al., 2022), while the PQM follows (Chen et al., 2024; Zhang et al., 2024b) to use Q-values as re- ward labels and optimize with MSE loss. Detailed training settings are provided in Appendix A.2.
The effectiveness of step-by-step verified reasoning tra- jectory. :Star-Math generates step-by-step verified reason- ing trajectories, which eliminate error intermediate steps and further expand training set with more challenging problems. To evaluate its effectiveness, we use the data generated from round 4 as SFT training data and compare it against three strong baselines: (i) GPT-ddstillationn which includes open- sourced CoT solutions synthesized using GPT-4 such as MetaMath (Yu et al., 2023b), NuminaMath-CoT (Jia LI & Polu, 2024b); (ii) Random sampling from self- gel-reener,,on, which use the same policy model (i.e., policy SLM-r3) to randomly generate trajectories; (iii) Rejection sampling,
Table 6. Ablation study on the reward model. Process reward models (PQM and PPM) outperfarm ORM, with PPM pushing the frontier of math reasoning capabilities.
RM
Inference
MATH
AIME
Olympiad Bench
College Math
ol-mini
90.0
56.7
65.3
53.6
ORM
Best-of-N
82.6
26.7
55.
55.5
PQM
MCTS
88.2
46.7
62.9
57.6
PPM
MCTS
89.4
50.0
65.3
59.0
8

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Table 6 compares the performance of ORM, PQM, and PPM for System 2 reasoning using our final round policy model. ORM provides reward signals only at the end of problem solving, so we use the Best-of-N, while PQM and PPM leverage MCTS-driven search. As shown in Table 6, both PQM and PPM outperform ORM by providing denser step- level reward signals, leading to higher accuracy on complex math reasoning tasks. However, PQM struggles on more challenging benchmarks, such MATH and Olympiad Bench, due to the inherent imprecision of Q-values. In con- trast, PPM constructs step-level preference data for training, enabling our 7B policy model to achieve comparable or superior performance to ol-mini across all benchmarks.
AI-MO. Ame 2023, 2024b. URL https::/huggiingggeeeeeeee
Brown, B., Juravsky. J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys:
Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.2178. 2024.
Chen, G., Liao, M., Li, C., and Fan, K. Alphamath almost zero: process supervision without process, 2024.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H.,
Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Conclusion
Daniel, K. Thinking, fast and slow. Macmillan, 2011.
In this work, we present rStar-Math, self-evolved Sys- tem 2 deep thinking approach that significantly boosts the math reasoning capabilities of small LLMs, achieving state- of-the-an OpenAl ol level performance. Extensive exper- iments across four different-sized SLMs and challenging math benchmarks demonstrate the superiority of rStar-Math, with achieving leading results while outperforming exist- ing math reasoning LLMs and Best-of-N baselines. We observe that there be further improvements with more challenging math problems, which we leave as future work.
Gou, Z., Shao, Z., Gong, Y., Yang. Y., Huang, M., Duan, N., Chen, W., al. Tora: A tool-integrated reasoning agent for mathematical problem solving, arXiv preprint arXiv:2309.0 /7452, 2023.
He, C., Luo, R., Bai, Y., Hu, S., Thai, L., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., al. Olympiadbench: challenging benchmark for promoting agi with olympiad- level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
Impact Statement
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. Large language mod- els cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023.
This paper presents work whose goal is to advance the field of deep learning. While our research focuses on enhancing the mathematical reasoning capabilities of large language models, we acknowledge potential societal implications such applications in education and automated problem- solving, There are many broader consequences of our work, none of which we feel must be specifically highlighted here.
Huang, Z., Zou, H., Li, X., Liu, Y., Zheng, Y., Chern, E., Xia, S., Qin, Y., Yuan, W., and Liu, P. O1 replication journey part 2: Surpassing ol oprpevew through simple distillation big progress bitter lesson? Github, 2024.
URL https: /gi thub. com wwt..////A-LL//OOOOOO--..
References
Jia LI, Edward Beeching, L. T. B. L. R. S. S. C. K. R. L. Y. J. Z. S. Z. Q B. D. Y. G. and Polu, S. Nu- minamath. [https://github wwtp:/ooeeocteeiiieco numina/ aineeppoggessspppzeee//ttttt///s com/
Inequality of arithmetic and geometric means. URL https://artofproblemnst
wttp://wwwno...oowiiii/iii..iie. com/wiki/index. php//AM-GMMInnequllity
Pythagorean theorem. URL
progettrwuttna//linnprrrersssppp main//reporttuuninn..tttteeet 2024a.
wikk//Pythagorenntteet
Shoelace theorem. URL htLLtttt/////rttopooolleesss com/wik com/wwikiinddx..p http://www...ppp/hehaaaeeeeeerre
Jia LI, Edward Beeching, L. T. B. L. R. S. S. C.
L. Y. A. J. Z. S. Z. Q B. D. L. Z. Y. L. and Polu, S. Numinamath cot, 2024b. URL https: http:///huggngcceee Litiiasstss/AI-Oo/Mmmmaaaaalal
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla. H., Bach, N., Bahree, A., Bakhtiari. A., Behl, H., al. Phi-3 technical report: A highly capable language model locally your phone. arXiv preprint arXiv:2404. 14219, 2024.
Kang, J., Li, X. Z., Chen, X., Kazemi, A., and Chen, B. Mindstar: Enhancing math reasoning in tre-trained Ilms at inference time. arXiv preprint arXiv:2405. 16265, 2024.
AI-MO. Aime 2024, 2024a. URL https::/huggingg/t co/datasets/AIIMOO
aime.
9

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Kocsis, L. and Szepesvári, C. Bandit based monte-carlo
OpenAI. Openai ol system card. preprint, 2024.
planning. volume 2006, pp. 282-293, 09 2006. ISBN
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
978-3-540-45375-8 doi: 10.
Kumar, A., Zhuang. V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., Iqbal, S., Bishop, C.,, Roelofs, R., et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917, 2024.
Qi, Z., Ma, M., Xu, J., Zhang, L. L.,, Yang, F., and Yang, M. Mutual reasoning makes smaller ilms stronger problem- solvers. arXiv preprint arXiv:2408.06195 2024.
Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Deni- son, C., Hemandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., et al. Measuring faithfulness in chain- of-thought reasoning. arXiv preprint arXiv:2307.13700 13702, 2023.
Qwen. Qwen2-math-7b, 2024a.
URL https://
huggingface hatt:i/ffacc..co//ee///eeeeaaaaa
Qwen. Qwen2.5-math-1.5b, 2024b.
URL https://
Li, C., Wang, W., Hu, J., Wei, Y., Zheng, N., Hu, H., Zhang, Z., and Peng. H. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706 2024.
Qwen. Qwen2.5-math-7b, 2024c. URL https://
Renze, M. and Guven, E. Self-reflection in Ilm agents:
Liao, M., Luo, W., Li, C., Wu, J., and Fan, K. Mario: Math reasoning with code interpreter output-a reproducible pipeline. arXiv preprint arXiv:2401.08/ 08190, 2024.
Effects on problem-solving performance. arXiv preprint prXiv:2405.06682 2024.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.20055 2023.
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Grae- pel, T., et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv: 1rXhv:1712.01815 2017.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J.. Sutskever, I., and Cobbe, K. Let's verify step by step. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: http:///pppeeieieeeeeeeeeee net/forum? id=v8L0pN6E01.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than seal- ing model parameters. arXiv preprint +xXiv:2408.03114 2024.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseck-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
Luo, H., Sun, Q. Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q. Chen, S., and Zhang, D. Wizard- math: Empowering mathematical reasoning for large lan- guage models via reinforced evol-instruct. arXiv preprint arXiv:2308. 09583, 2023.
Tang, Z., Zhang, X.,
Wan, B., and Wei, F. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.0288; 2024.
Team, Q. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL https:://wen.... gi thub. it/blogg//wwwpe//o////o 32b- --e---2bb--r-- wttp://ww-eppppeeiee...e
Luo, L., Liu, Y., Liu, R., Phatale, S., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng. L.. Sun, J., et al. Improve mathemati- cal reasoning in language models by automated process supervision. arXiv preprint arXiv:2406. 06592, 2024.
Valmeekam, K., Sreedharan, S., Marquez, M., Olmo, A., and Kambhampati, S. On the planning abilities of large language models (a critical investigation with proposed benchmark). arXiv preprint arXiv:2302.06706, 2023.
Noam Brown, I. A. and Lightman, H. Openai's noam brown, ilge akkaya and hunter lightman on ol and teaching 1lms to reason better, 2024. URL https://www.youttbbe youtube. comm/waatchuv=jPluXXppdo
Wang, C., Deng, Y., Lv, Z., Yan, S., and Bo, A. Q*: Im- proving multi-step reasoning for ulms with deliberative planning, 2024a.
OpenAl. Gpt-4 technical report. 2023.
10

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Wang, K., Ren, H., Zhou, A., Lu, Z.,, Luo, S., Shi, W., Zhang, R., Song, I.., Zhan, M., and Li, H. Math- coder: Seamless code integration in LLMs for enhanced mathematical reasoning. In The Twelfth International Conference on Learning Representations, 2024b. URL. met/forrumidd=28000t55.
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math techni- cal report: Toward mathematical expert model via self- improvement. arXiv preprint arXiv:2409.12122 2024.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Wang, P., Li, L.. Shao, Z., Xu, R. X., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations, 2024c.
Yu, F., Gao, A., and Wang, B. Dutcome-supervise verifiers for planning in mathematical reasoning. arXiv preprint arXiv:2311.0972 2023a.
Wang. X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H.. Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning lan- guage models. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: www..eoorrrnid-1PPLLLII...
Yu, L.,, Jiang, W., Shi, H., Yu,J. Liu, Z., Zhang, Y., Kwok, J. T., Li. Z., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309.1 2284, 2023b.
Weisstein, E. W. Fermat's little theorem, a. URL https: //mathwor ld. wolfram, com//eermatslittteeeeeerrr html.
Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning math- ematical reasoning with large language models. arXiv preprint arXiv:2308.00182. 2023.
Weisstein, W. Vieta's formulas, from mathworld- wolfram web resource, URL http://mathworld. wolfram. cwm/Treeeeeeeeeeeeee Tree. Trree.ehee html.
Zhang, D., Li, J., Huang, X., Zhou, D., Li, Y., and Ouyang, W. Accessing gpt-4 level mathematical olympiad solu- tions via monte carlo tree self-refine with llama-3 8b. arXiv preprint arXiv:2406. 07394, 2024a.
Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y. An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint (rXi::2408.00724, 2024.
Zhang, D., Zhoubian, S.. Hu, Z., Yue, Y., Dong, Y., and Tang, J. Rest-mets*: Llm self-training via process reward guided tree search. arXiv preprint arXiv:2406.0333 2024b.
Xin, H., Guo, D., Shao, Z., Ren, Z., Zhu, Q., Liu, B., Ruan, C., Li, W., and Liang. X. prrveeek-prover: Advancing theorem proving in Ilms through large-scale synthetic data. arXiv preprint arXiv:2405. 2024.

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
A Appendix
A.1 Findings and Discussions
Question: Given positive integers $x$ and $y$ such that $2x"2""3 +4y"3 = 149 + +3x22, what is the value of $x + ys?
<rode>
from sympy iyportsymolls, Eq. solve
Deffine the variables x y
Koyjebb-oyy
symbols
<end of steps
PPM score: 00.08
# DDffine the equation
equation = Eq + 4-y"33, 149 + 2
Intrinsic self-refiection:
sdlas
<end
step>
PPM score: -0 0.219
Thinking outside the box, find an easier solution!
#Solve the equation for in terms of solution solve(equation. y)
* Since we need positive integers, we need find the values of and that satisfy the equation
<end_ steps
PPM score: -0. 0.348
#W will check small intager values of and for x_ val in range(], 10):
for y_val in range(1, 10):
if if2x val**2y vai**3 +44y_vaai*s vai**
= 149 + +3% va
printif'' = & val] y 6 0_val3"
printift'x +y = x val +y_valX) break
# Now print the final answer print(solutionnyn
Wrong
PPM score: 0.620
<end_ code> coutput>
PPM score:-0.511
een of step>
# Now thent final answer
[13* 4-4)
44 4-4)
# The loop will find the first pair (x. y) and print +y <end of code> <output>
149/(2 + 000p://2312220003333111222222222 4] 149/22x*22 +
4))*411/3/2,, 210 +4))
*gr13))**33**222******** + +4492)2222 (1)3
x x33.y m2
<end of output>
x ++
M score: 0.698
cend of output>
<answer>
Wrong
<answer>
The value of 1ix +-ll Wboxed B <end_ of_ answer>
From the result, we can see the value of +
PPM score: -05599
Wboxed <end of answer>
Correct
PPM score: 0.835
Figure 4. An example of intrinsic self-reflection during rStar-Math deep thinking.
The emergence of intrinsic self-reflection capability. A key breakthrough in OpenAl ol is its intrinsic self-refiection capability. When the model makes an error, it recognizes the mistake and self-correct with a correct answer (Noam Brown & Lightman, 2024). Yet it has consistently been found to be largely ineffective in apen-sourced LLMs. Thus, the community has actively explored various approaches, including self-correction (Huang et al., 2023; Kumar et al., 2024), self-reflcction (Renze & Guven, 2024; Shinn et al., 2024), to explicitly train or prompt LLMs to develop such capability.
In our experiments, we unexpectedly observe that our MCTS- deep thinking exhibits self-reflection during problem- solving. As shown in Fig. 4, the model initially formalizes equation using SymPy in the first three steps, which would lead to incorrect answer (left branch). Interestingly, in the fourth step (right branch), the policy model recognizes the low quality of its earlier steps and refrains from continuing along the initial problem-solving path. Instead, it backtracks and resolves the problem using new, simpler approach, ultimately arriving at the correct answer. An additional example of self-correction is provided in AppendixA.3. Notably, no self-reflection training data or prompt was included, suggesting that advanced System reasoning can foster intrinsic self-reflection.
PPM shapes the reasoning boundary in System deep thinking. Both the policy and reward models are crucial for System 2 deep reasoning. Our experiments show that different policy models achieve similar performance in System 2 reasoning (see Appendix A.2 with the PPM largely determines the upper performance limit. Fig. 5 summarizes the accuracy of policy models of different sizes and the accuracy achieved after applying reward models. We conclude that, despite differences in Pass@1 1 accuracy due to variations in training methods, data, and model size, the reward model is the key factor in System 2 reasoning. For instance, although the SFT accuracy of rStar-Math-75 is lower than Qwen2.5-Matt 2ahh-722--sttuct pairing it with our 7B PPM allows -Star-Math to outperform the 72B policy model with Qwen 72B ORM. Moreover, despite varying @ I accuracy across our three policy SI.M sizes, the final reasoning accuracy converges after applying the PPM.
12

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Star Policy model
Qwen 72B Policy model
rStar /B PPM improvement
Owen /28 ORM mmpovement
MATH
AIME 2024
AMC 2023
Olympiad Bench
College Math
rSlar-Math (Qwen7B)
78.4 4
89
26 7
50
47
A7
47.1
1
65.3
52.5
59
rStar-Math (Qwen1.588
74,8
87.8
13.3
46.7
47.5
80
42.5
63.5
50.1
59
rStar-Math (Ph13.88)
68
85.4
10
40
37.5
77.5
36,6
39.3
49.5
50.6
Swwen.55-Math772.
85.6
85.8
30
36.7
72.5
49
54.5
50
20
40
0
50
25
50
0
25
50
Pass@1 1 accuracy (%)
Pass@1 1 accuracy (%)
Pass@1 1 accuracy (%)
Pass@1 1 accuracy (%)
Pass 1 accuracy (%)
Figure 5.
@ I accuracy of policy models and their accuracy after applying System reasoning with various reward models, shows that reward models primarily determine the final performance.
PPM spots theorem-appllcation steps. When solving challenging math problems, identifying and applying relevant theorems or key conclusions often form the cornerstone of successful problem-solving (Xin et al., 2024). our experiments, we find that during rStar-Math problem-solving, our PPM effectively identifies critical theorem- capremaappliittion intermediate steps within policy model's deep thinking process. These steps predicted with high reward scores, guiding the policy model to the correct solution. Appendix A.3 provides examples where the PPM successfully identifies key theorems such as Fermat's little theorem (Weisstein, a), Vieta's formulas (Weisstein, b), the AM-GM inequality (amg), the Pythagorean theorem (pyt), and the Shoelace Theorem (sho), etc.
Generalization to theorem proving. rStar-Math is generalizable to more challenging math tasks, such as theorem proving, though its current focus on word problems due to dataset limitations. Nonetheless, rStar-Math demonstrates the ability to prove mathematical statements. As shown Appendix A.3, it successfully proves an Olympiad-lever problem involving Fermat's 's Little Theorem, providing step-by-step correct proof through its deep reasoning process. This achievement is primarily due to the policy model's 's strong reasoning capabilities, the current PPM, which lacks step-level proof training, cannot yet reliably evaluate intermediate steps. Future work will focus on creating step-level proof datasets to enable the PPM to better solve theorem proving tasks.
A.2 Additional Experiments and Details
Data Generation Details. As detailed in Sec. 3.3, each round starts by self- seff-enneaaiing step-by-step verified trajectories for 747k math word problems. The maximum tree depth d is set to 16, with 16 MCTS rollouts conducted per problem by default. At each step, allow to explore 8 candidate nodes, and the constant c in Eq. is set to 2 to promote greater exploration. In the bootstrap round, due to the large size of the initial policy model (236B), we used smaller parameters: 8 rollouts and 5 candidate nodes per step. To improve the accuracy of solving challenging problems in round 4, we increase the number of candidate nodes to 16 and conduct MCTS tree expansions per problem using different random seeds. Detailed prompts are available in Appendix A.4.
Training Details. In each round, we collect step-by-step verified trajectories to fine-tune the policy LLM and train the PPM. To reduce noise in synthetic math problems (e.g., incorrect ground-truth answers labeled by GPT-4), we remove synthetic problems with trajectories achieving less than 50% accuracy. Based on our extensive experiments, the policy LLM is fine-tuned from the initial base model in each round, rather than training incrementally on the model from the previous round. All policy SLMs are trained for 2 epochs with a sequence length of 4096 tokens and a batch size of 128. We use AdamW W optimizer with a linear learning rate scheduler, setting the initial learning rate to 7e-6 for Qwen models, and a cosine scheduler with an initial learning rate of 5e-6 for Phi3-mini-Instrut The PPM is trained for epoch with a batch size of 512 and an initial learning rate of 7
Table 7. Training cost per round
GPUs
Training time
Policy PPM
8xMI300X
20 hours
MX300X
15 hours
13

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Training PPM. The PPM is initialized from the fine-tuned policy model, with its next-token prediction head replaced by scalar-value head consisting of a linear layer and tanh function constrain outputs to the range [-1, 1J. We filter out math problems where all solution trajectories are fully correct or incorrect. For problems with mixed outcomes, select two positive and two negative examples for each step based on Q-values, which are used as preference pairs for training data.
Training the ORM and PQM. The Outcome Reward Model (ORM) and the Q-value-based Process Reward Model (PQM) share the same model architecture and training parameters with and our PPM. For ORM training, we filter trajectories from math problems containing both correct and incorrect solutions. Specifically, the two trajectories with the highest average Q-values are selected as positive examples, while the two with the lowest are chosen as negative examples. Following Qwen2.5 (Yang al., 2024), adopt the pairwise ranking loss (Ouyang et al., 2022) optimize the ORM. To train the PQM, we follow (Chen et al., 2024) to use step-level Q-values as reward labels. Let x = x 3 82 BD....8. a be the trajectory, with annotated Q-values Q = (1)) (s1). (sx)) and predicted Q-values Q' = (Q' Q'(s1).... (s())) for each step. To stabilize PQM training, we treat each trajectory as a single training sample and predict Q-values for all steps simultaneously, rather than splitting it into individual per-step samples. Specifically, to predict the Q-value Q' (s,)) for step St. PQM takes the trajectory from the question up to step 8; (i.e., ar 6) 81 @ 82 @ ... @ 8;) as input and outputs a value between -1 and 1. We use a mean squared error (MSE) loss for PQM training:
pprm (x) = |Q - -Q112
Table Inference costs of rStar-Math. We show the average number of generated tokens required to generate trajectory for given
question.
MATH
AIME 2024
AMC 2023
Olympiad Bench
College Math
GSM8K
GaokaoEn 2023
5453
15693
14544
7889
4503
3299
6375
Table Pass@ accuracy of resulting policy SLM in each round, showing continuous improvement until surpassing the bootstrap
model.
Round##
MATH
AIME 2024
AMC 2023
Olympiad Bench
College Math
GSMSK GaokaoEn 2023
DeepSeek-Coder (2-oder-V-Innsrruct (bootstrap model)
75.3
13.3
57.5
37.6
46.2
94.9
64.7
Base
58.8
0.0
22.5
21.8
41.6
91.6
51.7
policy SLM-ri
69.6
3.3
30.0
34.7
44.3
88..4"
57.7
policy SLM-12
73.6
10.0
35.0
39.0
45.7
89.1
59.7
policy SLM-r3
75.8
16.7
45.0
44.1
49.6
89.3
62.8
policy SL.-r4
78.4
26.7
47.5
47.1
52.5
89.7
65.7
Table 10. The quality of PPM consistently improves across rounds. The policy model has been fixed with policy SLM-rl for fair
comparison.
Round#
MATH
AIME 2024
AMC 2023
Olympiad Bench
College Math
GSM8K
GaokaoEn 2023
PPM-rl
75.2
10.0
57.5
35.7
45,4
90.9
60.3
PPM-r2
84.1
26.7
75.0
52.7
54.2
93.3
73.0
PPM-r3
85.2
33.3
77.5
59.5
55.6
93.9
76.6
PPM-r4
87.0
43.3
77.5
61.5
56.8
94.2
77.8
Inference Setting. In our evaluation, we run multiple MCTS generate candidate solution trajectories. For each problem, generate 32 candidate nodes each step and use the PPM to score each node. Since the PPM is effective at providing step-level quality evaluations, we did not perform multiple MCTS rollouts to update per-step Q-values. After each MCTS finsh, we select the trajectory with highest PPM score as the final answer. Table 8 presents the average number of tokens generated to produce a trajectory in MCTS.
Self-evolution Inference Costs. In the initial bootstrap round, we use DeepSeek-Codent -n-Iruuttct (236B) as the policy model, using nodes of 8x80GB H100 GPUs with MCTS rollouts. This required approximately two weeks finish the
14

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
data generation. For rounds 2-4, using our fine-tuned 7B SLM the policy model, data generation was performed 15 nodes of 4x40GB A100 GPUs, with each round completed in three days. In the final round, to include more challenging problems, we increased the number of MCTS rollouts to 64, extending the data generation time to one week.
Table 11. @ass@ I accuracy of our fine-tuned policy models for Phi3-mini, Qwen2.5 5 Math-1.50 Qwen2 Qwen2-Math-a AB and Qwen2.5 Math- 7B,
Model
MATH
AIME 2024
AMC 2023
Olympiad Bench
College Math
GSM8K
GaokaoEn 2023
General Base Model: Phi3-mini Instrunttt (3.8B)
Phil-mini-lnstruct
41.4
3.33
7.5
12.3
33.1
85.7
37.1
Our policy model
68.0
10.0
37.5
36.6
48.7
87.9
53.2
Specialpei Base Model: Qwen2.5 3-Math-1.5B
Qwen2.5-Maile 5-Mathh--.3. 1.5B
51.2
0.0
22.5
16.7
74.6
Qwen2.5- M.th--155-IInsmmaa
60.0
10.0
60.0
38.1
47.7
Our policy model
74.8
13.3
47.5
42.5
50.1
83.1
58.7
Sath-Sppci Base Model: Qwen2 Owen2-Math-P2
Qwen2-Math-78 Qwen2- Matttre--nstruet
53.4
3.3
25.0
17.3
39.4
80.4
47.3
73.2
13.3
62.5
38.2
45.9
89.9
62.1
Our policy model
73.8
16.7
45.0
43,9
52.0
88.3
65.2
Math-S Sach-Sppcialized Base Model: Dwwen255-Math-75
Qwen2.5-Math--
58.8
0.0
22.5
21.8
41.6
91.6
51.7
Swen2.5-Math- 7B-Instruct
82.6
6.0
62.5
41.6
46.8
95.2
66.8
Our policy model
78.4
26.7
47.5
47.1
52.5
89.7
65.7
rStar-Qwen22 5-Math-7B
Owen2. --ttstt-ntttttt Olympiad Bench
AIME
MATH
College Math
95
80
65
60
70
90
40
60
55
85
20
50
50
2
8
32
64
2
16
32
2
8
32
64
2
16
32
64
#Sampled solutions
#Sampled solutions
#Sampled solutions
#Sampled solutions
Figure 6. @ass@ accuracy with random sampling from different policy models. Compared the official Qwen instruct version, our policy model exhibits a stronger ability to sample correct solutions.
Pass@ N. Table compares the math reasoning performance of our policy models with the instruct versions developed by the original model team. Our policy models do not consistently outperform the instruct versions. For example, on the Qwen2.5-Math-1 base model, Dwwe2.5--Math-7B-Iie
achieves 4.2% higher accuracy on the MATH benchmark. However, the pass@ @1 accuracy alone does not fully reflect the reasoning capabilities for the policy model in System 2 deep thinking paradigm.To provide more comprehensive evaluation, Fig.6 and Fig.7 present the pass @ N accuracy. In this metric, the policy model generates N solutions under two settings: random sampling (Fig.6) and PPM-augmented MCTS deep thinking (Fig.7). A problem is considered solved if one solution is correct. As shown Figure 6, our policy model performs similarly to Qwen2.5-Maiie --tth-- -B-ntrutttttt on the MATH benchmark's pass @ and significantly outperforms on others. This suggests that despite initial pass@1 accuracy differences, our policy model can generate correct solutions through multiple samples. Figure 7 further compares the pass @ N accuracy of our four policy models (different sizes) after MCTS deep thinking under the same PPM guidance. We can observe that after generating 64 trajectories, the @ accuracy of different policy models becomes comparable.
Ablation on the MCTS search parameters. We perform additional analysis on the number of candidate nodes per step in MCTS. As shown in Table 12, increasing the number of candidate nodes generally improves accuracy, but beyond 32 nodes, performance saturates.
15

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Saar--wwen2..-Mathhhe
SStar-Qwen2.b-e
rStar-Owen2-Math-77B
rStar-Phi3-mini
AIME
MATH
Olympiad Bench
College Math
70
95
50
65
90
65
60
85
60
30
2
8
16
32
64
2
4
16
32
64
2
8
16
32
64
2
4
16
32
64
#Sampled solutions
#Sampled solutions
#Sampled solutions
#Sampled solutions
Figure 7. Pass@1 accuracy with PPM-augmented MCTS. Under the same PPM guidance, the four policy models of varying sizes
demonstrate convergent capabilities in sampling correct solutions.
Table 12. Ablation study on the number candidate nodes MCTS.
MATH-500
AIME 2024
rollouts
64 rollouts
8 rollouts
64 rollouts
node=4
87.2
88.8
33.3
36.7
node=8
87.2
88.8
33.3
43.3
node=16
88.4
89.0
36.7
50.0
node=32
89.4
89.0
50.0
53.3
node=40
89.4
90.0
46.7
53.3
16

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
A.3
Examples of Reasoning Trajectories
Self-correction Example
Question: The coordinates of parallelogram are (5, 3), (6, 8), (7, 4) and (a, y) and æ > What is the value of +
Our trajectory:
code >
Import necessary libraries import numpy up of
i Define the coordinates of the parallelogram
ap array (15, 31)
B np. array (16, 8))
C up. array ([7. 4])
ofd.
Calculate the vector differences find missing coordinate # In parallelogram, the vector difference between two adjacent sides is equal to the vector difference between the other two adjacent sides We can use the formula: D = + (C B)
+ (C- B)
of step
# Since the coordinates provided are (5. 3), (6, 8), 4) and (x, can infer that the vector from B to C should be the same as the vector from A the missing point (x, y) This means: y, 3) = (I, because the from is 6, (1,-4)
# Solve for x and x
(-4)
end_of_ste _step >
Verify that the point (x. y) exists and satisfies the condition x Since not greater than we to correct formula that maintains the parallelogram property x
# Correcting the formula to find the other possible D [Self-correction]]
+ (B A)
&en_offstep
# Solve for and again
x 7 (6 5)
y (8 3)
end of step
# Verify that the point (x. exists and satisfies the condition 7
# Since = 8 greater than 7, which satisfies the condition given, we can now find the sum x and
-plus_y end.of >
# Now print the final answer
print plus y
# This should print the value of end_of Loade >
outpptt: 17-, end puttput
ansser > The value of x y 17
ond Lonswer >
17

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Math statement proof using Fermat's little theorem
Question: If pt+a is perfect square for some value of (t), then (a _ divisible by (p) (where prime number).
Our trajectory for proving:
code.
# Step I: Define the variables
from sympy import symbols, isprime, sqrt, inverse
if Define the variables
p, t, end of step
symbals ( ('p aa)
# Step 2: Assume p*t is perfect square # This meuns there exists integer such that u it Rearrange to find in terms of p. t, and k
# a m k2 p*1
end.of .step
Step Fermat's Theorem Fermat's Little Theorem that if p prime and is integer divisible by then a(p-1) = (mod p) # We need to show that a((x-1)/2) 1 is divisihle hy P end.of _step
# Step 4: Substitute = k2 ptt into Fermat's Little Theorem
# a((==1)/2) = (K2 p=t)((e-1)/2) (mod p)
Since pct is divisible by p. (k2 (xt-ptt)((p111)) 1)/2) = k(=-1) (mod p)
By Fermat's Little Theorem, k(p-1) =1 (mod p)
on
Step S: Therefore, (0)1)) (mod p) This implies ((p-1)/2). (mod p)
Hence, a(()-1)/2) is divisible by
end.of.she _step >
Now print the answer
The expression a((p-1)/2) 11/2) is divisible by p.")
end of code
output >
expression (2-1)) divisible
The statement is true.
end.of
ond_of-output Loutput >
18

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Problem-solving with upplication the Vieta's formulas and the AM-GM inequality
Question: Let be positive real number such that all the roots of
ax
Oz
real. Find the smallest possible value
Our trajectory for proving: code
# Step I: Define the polynomial equation from sympy import symbols, solve, Eq
Define the variable and the parameter symbols("x x a') end.of sstep >(PPM Score: -0.46430]
# Step 2:
Define the polynomial equation
polynomial = x + * a + end. of step (PPM Score: 0.53937]
If Step 3:
Use Vieta's formulas to relate the coefficients to the roots
Since the polynomial is monic (leading cnefficient I), the sum of roots (taken one at a time) -a
The of the products of the roots taken If The product of the roots is -1
time
Since all roots are real, we can denote them as r1, r2, r3
Vieta's formulas:
# f2 13
# rl #r2 + r2 r3*rl =a
r1*"2"r3=
send.of.step (PPM Score: 0.9989]
Step 4: Use the AM-GM inequality to minimum value of # By AM-GM, (r1*t2 + r2*r3 13*rl)/33 >= (1122333)/)))
Substituting the known values:
# #a/3 >= (-1)
#a/3 #a>=3
ondeo > (PPM Score: 0.9999]
Step S: Verify a
achievable
# a then rl + r2 + r3 = -3 and rl rl*r2 r2*r3 r3*rl =3 # We need to check if there exist real roots that satisfy these conditions Consider
case where
=3
it Then rl 12 = and r1* r2*13 r3*fl = 3, and r1*+2**3 =1
# Thus, a= 3 is achievable with real roots
end_of_step > (PPM Score: 1.0]
# Step 6: Print the final answer print(3)
eedd_o_codde output >3< end_of Loutput
ond_offoutput
answer > The smallest possible value of
cnd.oo.ansser Lanswer?
19

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Problem-solving with application Shoelace Theorem
Question: In diagram. and are the midpoints of AB respectively, Determine area of quadrilateral DBEF. [asy] Size(180); Faaaulpenniiewwidhh.ppt-- (pt+fontsize (Opt));
pair A. B, D, E.
A=(0,6);
B=(0,0);
C=(8.0):
D=(0,3);
E=(4,0);
F=(8/3,2);
draw(E-A---D::
draw((-.0)>(10,0), EndArrow);
draw((0-1))(0.8), EndArrow);
label("/10,,6" A, W);
label("B(O,, 0)", B, SW):
label("C(8, 0)", C);
labek" D. W);
label("E".E.. S);
label("/"", SW)::
"bbel("=".11..0" dir(O);;
label(""" y", (0,8), dir(90)); [/asy]
Our trajectory for proving: code
# Step I: Define the coordinates of points A, B, C, D, and
(0, 6)
B (0. 0)
C (8. 0)
3) Midpoint of
E. (4, 0) # Midpoint of BC
end of step >(PMM Score: -0, 0.766639
# Step 2: Define the coordinates of point F using the section formula (known to be (8/3, 2))
(8/3. 2)
end_of _step >(PM Score: 0.8429]
Step the Shoelace Theorem if Shoelace Theorem formula: Area 0.5 x1y2 x2y3 x3y4 x4yl (ylx2 y2x3 y3x4 y4x1)
area quadrilateral DBEF
# Coordinates of D. B, E, are (0, 3), (0, 0), (4, 0), (8/3, 2) respectively
= D x1, yl
x2.
B
x3.
E
x4.
y4
= F
area 0.5
(xl x2+y3 x3+y4 x4syl
(ylex2 + y2+x3 y3ax4 + y4xx1))
end of sstep >(PPM Score: 0.9972]
# Step 4: Print the area of quadrilateral DBEF print( a
end.on of code
output > 8.0 end.of Loutput >
answer
8
end of janswer
20

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
A.4
Prompt Examples
MCTS Prompt for DeepSeek- V2-Coder- Coder-Instruct (bootstrap round)
You are a powerful agent with broad math knowledge and great python programming skills. You need to use python interpreter to do accurate calculation on math equations.
!!! Remember:
code solve the problem step by step. The solution should include three parts: < code >, < output >, and < answer >.
2. All calculations should be done in python code. Provide concise reasoning and thinking in the comments of the code.
3. The most related python packages include 'math', 'sympy', 'scipy". and 'numpy".
Please use the following template:
Question: the input question
code >Construct the code step by step. Use <, end_of_step > indicate the end each step. Ensure your code can execute correttylcxcluding end of _step >) and print the answer. Avoid undefined variables (NameError), unimported packages. formatting errors (SyntaxFirror, TypeFrror).
the last step of the code, print the final answer and add comment: Now print the final
answer.<< end_of _code >>
output Execute the code using Python interpreter display printed results. < ond.of oouppt > The concise answer without verhose context, put your final answer's numerical part (without unit, only focus the numerical part if it's choice question) in boxed. < end of answer Now! It's your turn.
Question: [input}
The following are 2 demonstration examples: Question: Terrell usually lifts two 20 20-pound weights times. If uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?
code
Step I: Calculate
total weight lifted with two 20-pound weights
total_ weighteihht_20 22 20 * 12
end.of _step >>
Step 2: Calculate weight lifted per repetition with two 15-pound weights weight.per.rep.15. 15 =
15
&end.of.stpp >
Step 3: Calculate the number of repetitions needed to lift the same total weight with two 15 15-pound weights reps needed = total weight 20 weight per rep 15
eeddooffoo _step
# Now print the final answer printreps.needed) needed)
end of code
output >> 16.0 < end of output >> < answer >> From the result, we can that Terrel must lift the 15-pound weights
b6xed16 times to lift the same total weight. end.of Lonswer >,
Question: Find the value of that satisfics code
Express your answer fraction.
from sympy import symbols, Eq, solve, sqrt
# Define the variable
x symbols("x)) end of step
# Define the equation equation = Eq(sqit33*x 5) squ((** + 5), sqit(5) 3)
"end_of_stpp
# Solve the equation for x
solution solve(equation, x)
&endof_step
Now print the answer
print(solution)
end. of code
< outppt [
end of output < ansuee From the result, that the value of is
aad_of_answer
21

---

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
MCTS Prompt for our policy SLM
MCTS prompt for next step generation:
[uscr] >;
{input }uestion)
[assistond]
>: Let's think step by step and solve the problem with code. (trajectory of previous steps}
SFT Prompt Example: [user]
How many numbers can get by multiplying two more distinct members of the set
1, 3,5, 11 together?
[assistand] >: Let's think step by step and solve the problem with code.
code >
Step I: Create set numbers
numbers {I, 2, 3, 5, 11}
of
# Step 2: Define function to generate all possible products of two or more distinct members of the set from itertools import combinations
def generate products (numbers):
products ()t()
for in range len (numbers) 1):
# For combinations of 2, 3, 4, and 5 elements for combo in combinations (numbers, r):
product
for num in combo:
product *= num
products add (product) return products
sendLofstee _step >
Stipuee dhtsutmec prodieeratte tttpriuieerraeeppuuuutstssssss (numbers) ) end.of.step
Step 4: Calculate number of unique products
num _unique,,n.. products produccss = len unique.produet..
sstep
Now print the final
print( num wwumuunique_paar
on Loode _oode
output > 15- end_of ootput
>>
answer FFrom result, we can see there are
unique products.
ofd_of_answer Lonswer
22


Reviews:
[{"review_id": "SUjmixh56n", "reviewer": "Reviewer_F2Xg", "summary": "This work presents a methodology to improve the reasoning performance for math of small language models to levels competitive with the state-of-the-art models. Specifically, the rStar-Math approach uses MCTS along with a verifier by interpreting the reasoning steps as code, in order to train a policy model and a preference model over multiple rounds. The policy model, refined by the preference model, generates reasoning trajectories, and these reasoning trajectories are used to refine the preference model, eventually leading to a policy model capable of significantly enhanced reasoning performance. Among other things, ablations show that the math data thus generated is of inherently high quality even for pure SFT.\n\n## update after rebuttal\nI maintain my strong recommendation to accept this paper to the conference and thank the authors for the interesting discussion.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and positive feedback on our work. We sincerely appreciate your insights and your recognition of our contributions. Below, we address your specific comments.\n\n>Q1:  I strongly encourage the authors to share code, models and/or especially the data generated on the usual platforms the foundation model community uses if not done already.\n\n**Response**:  Thank you for your suggestion. We are committed to releasing the code and the generated data as soon as possible. \n\n>Q2: P3: “Candidates that execute successfully are retained as valid nodes and scored by the PPM, which assigns a Q-value” it is not clear to me why the PPM assigns the Q value as page 2 indicates that the Q value is generated separately from the number of trajectories leading to the correct answer for a given node, while the PPM is trained from the Q value; could you clarify that point (here and ideally either in page 2 or 3 depending on your answer)? Currently the flow of p2 and fig 1c makes it seem like the code-augmented component and Q value assignment of your pipeline happens already before (and is a prerequisite for) training the PPM.\n\n**Response**: Thank you for your question! We acknowledge that the current description may cause some ambiguity, and we will revise the paper to clarify this process.\n\nTo clarify: as shown in Fig. 1(c), in the first and second rounds of data generation, we did *not* have a PPM yet. Instead, we assigned Q-values to each candidate based on terminal-guided annotation. This Q-value-labeled data was then used to train the PPM.\nStarting from the third and fourth rounds, the PPM became available, enabling PPM-augmented Q-value annotation. Specifically, during MCTS, each newly generated node is initially assigned a Q-value predicted by the PPM (whereas in the first two rounds, the initial Q-value was set to 0). This Q-value is then refined through MCTS rollouts. We will update Page 3 to make this process clearer. \n\n>Q3: Fig 2: is it intended that the python code for step 2 repeats the python code for step 1 first?\n\n**Response**: Yes, this design is to ensure that the Python code executes correctly. Specifically, Step 2's code may depend on variables or functions defined in Step 1. If a node corresponding to Step 2 were to execute only its own Python code without including the preceding steps, it could lead to syntax or execution errors due to undefined references. \n\n>Q4: P5: “In each selfevolution round, we perform 16 rollouts per math problem, which leads to 16 reasoning trajectories. “ considering multiple steps and multiple future trajectories per step in rollouts, wouldn’t that lead to more than 16 reasoning trajectories? \n\n**Response**: Thank you for your question! Our definition of rollout follows prior works [1,2,3], where a full rollout refers to a complete reasoning trajectory from the root node (question) to a terminal answer node. Thus, each rollout consists of multiple reasoning steps and explores different paths, ensuring that performing 16 rollouts per math problem yields at least 16 full reasoning trajectories. We appreciate your careful reading and will revise the paragraph to make this clearer.\n\n[1] Reasoning with language model is planning with world model, emnlp 2023 \n\n[2] Mutual reasoning makes smaller LLMs stronger problem-solvers, iclr2025 \n\n[3] LLaMA-Berry: Pairwise optimization for olympiad-level mathematical reasoning via o1-like monte carlo tree search\n\n>Q5: Tab6: could PQM be underperforming due to PPM being more attuned to the policy model since it stems from several rounds of iterating?\n\n**Response**: Thank you for your insightful question! The primary reason PQM underperforms compared to PPM is that using score-based Q-values as direct labels for the process reward model inherently introduces imprecision. Although extensive MCTS rollouts help improve Q-value accuracy, these Q-values struggle to differentiate fine-grained quality levels. For example, distinguishing an optimal step (score=1.0) from  a near-optimal step (score=0.9) is inherently challenging, whether through MCTS automatic annotation or human annotation. This inevitably introduces noisy scores in PQM's training data, affecting its reliability.\n\nTo fully address your question, we conduct an additional experiment. We use the generated data from SLM-r1 (our first trained 7B policy SLM from round 1) to train both PQM and PPM. As shown in the table, PPM consistently outperforms PQM from the early stages of self-evolution. This empirically demonstrates that PPM's advantage stems from its superior design, which effectively mitigates the impact of noisy Q-value scores.\n\n||MATH|AIME 2024| AMC 2023| Olympiad Bench| College Math | GaokaoEn 2023|\n| :--:  | :--:   |  :--:   | :--:   | :--:   | :--:   |  :--:   | \n|SLM-r1+PQM (trained by SLM-r1 generated data)| 82.4 | 23.3|**75.0** |49.6|52.9|70.3|\n|SLM-r1+PPM (trained by SLM-r1 generated data)| **84.0**| **26.7**| **75.0**| **52.7**| **54.2**| **73.0**|", "claims_and_evidence": "I think the experiments support the claims made in this paper: the paper provides a recipe that does enable small models to be competitive with reasoning SotA of January 2025. See below for more.", "methods_and_evaluation_criteria": "The methods make sense and the intuition is well supported by past works. The metrics used to evaluate their models are standard and appropriate. Of course, human evaluations, and analysis of efficiency vs production models would have been great to have, but can’t be expected for obvious reasons.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I think the authors set up their experiments well and ran the ablations I most immediately would have wanted them to.", "relation_to_broader_scientific_literature": "I think this work connects well with the current interest of the community in reasoning, although see feedback below.", "essential_references_not_discussed": "There have been past works using code to verify steps of math reasoning before that are not mentioned/discussed in 3.1, e.g. LEVER, MathCoder (which you cite but do not discuss that aspect of it), etc.", "other_strengths_and_weaknesses": "While this is not required in the context of the reviews, I strongly encourage the authors to share code, models and/or especially the data generated on the usual platforms the foundation model community uses if not done already.", "other_comments_or_suggestions": "Typos:\n1. P1: “ranking among the top 20% the brightest high school math students.” missing word\n2. P3: “As a result, scaling up CoT data has diminishing returns, with gains nearing saturation.” I would slightly rewrite to make it clear that you are talking about scaling up by naively generating from frontier models, since after all your approach is also about scaling up CoT data, but in a more refined way.\n3. P4: “we perform MCTS rollut”\n4. P4: “we use terminal-guided annotation Formally” missing punctuation\n5. Eq2: I recommend reintroducing $s_d$ (can be done by adding “$s_d$” after “Terminal nodes” in the last sentence of the paragraph following the equation)\n6. P4: “PRM-augmented annotation.” should read PPM\n7. Eq 4: $\\sigma$ is not introduced, and if you are summing over the 4 combinations of pairwise choices of positive and negative $y_i$, it should be made clearer in the equation\n8. P5: “Problems are then categories by difficulty”\n9. P7: “Specifically, for AIME/AMC, we generate 16 trajectories for AIME/AMC and 8 for other benchmarks, using PPM to select the best solution.” sentencing, probably solved by removing the first “for AIME/AMC”\n10. P7: “srStar-Math”\n11. P7: “In Fig. ??” broken ref\n12. P8: “rStar-Math can achieve further improvements by collecting more challenging math problems, we leave this as future work.” phrasing (perhaps “We observe that [there can be further improvements with more data], which we leave as future work”) \n13. P13: “Training PPM” missing “the”\n\nSuggestions:\n1. P4: you evoke MCTS backpropagation. I suggest going into a bit more detail about that, perhaps in the appendix, as in my experience readers of practical LLM training methods may not be familiar with that.\n2. Tab3: “Specifically, for AIME/AMC, we generate 16 trajectories for AIME/AMC and 8 for other benchmarks, using PPM to select the best solution.” I recommend adding a version of that to the table’s caption", "questions_for_authors": "1. P3: “Candidates that execute successfully are retained as valid nodes and scored by the PPM, which assigns a Q-value” it is not clear to me why the PPM assigns the Q value as page 2 indicates that the Q value is generated separately from the number of trajectories leading to the correct answer for a given node, while the PPM is trained *from* the Q value; could you clarify that point (here and ideally either in page 2 or 3 depending on your answer)? Currently the flow of p2 and fig 1c makes it seem like the code-augmented component and Q value assignment of your pipeline happens already before (and is a prerequisite for) training the PPM. \n2. Fig 2: is it intended that the python code for step 2 repeats the python code for step 1 first?\n3. P5: “In each selfevolution round, we perform 16 rollouts per math problem, which leads to 16 reasoning trajectories. “ considering multiple steps and multiple future trajectories per step in rollouts, wouldn’t that lead to more than 16 reasoning trajectories? Unless you discard all but the highest scored one (which seems to be what is suggested later in the paper, in which case I’d clarify that in the paragraph).\n4. Tab6: could PQM be underperforming due to PPM being more attuned to the policy model since it stems from several rounds of iterating?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DVgPwJUsQq", "reviewer": "Reviewer_YH6P", "summary": "This paper shows that smaller language models ($\\leq7$ billion parameters) can learn to solve challenging math problems at a level comparable to much larger models (e.g., GPT-4 or o1 models). They achieve this by having these smaller models:\n - Generate and verify each step of a math solution (rather than producing it all at once).\n - Use Monte Carlo Tree Search (MCTS), guided by a specially trained “process reward model,” to explore different solution steps and pick the best paths.\n - Iteratively improve themselves (“self-evolution”) by generating new training data and refining both the small “policy” model (the one that proposes solution steps) and the reward model (the one that scores each intermediate step).\n\nThrough repeated rounds of this procedure, their 7B model eventually surpasses or matches some large commercial models (like OpenAI’s “o1-preview” or “o1-mini”) on difficult math benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: Clarification on the evaluation benchmarks: \"The authors’ experimental design generally appears sound, particularly in their ablation studies and comparisons on multiple benchmarks, but a closer look at smaller test sets, such as the 15-problem subset of AIME, raises concerns about cherry-picking or insufficient sample size.\"\n\n**Response**: Thank you for your thoughtful feedback! We would like to clarify that for AIME 2024, we  included both AIME I and II, resulting in 30 problems in total rather than 15. Moreover, our evaluation follows prior works [1,2,3] by selecting AIME 2024 and MATH-500 as key benchmarks, alongside six additional benchmarks, covering a total of 10,195 problems. The following table provides a detailed breakdown of the number of problems in each benchmark. Given this large-scale evaluation, we believe our results are robust and not based on cherry-picked subsets. We will revise the paper to clarify these details. \n\n|Total|MATH (500)|AIME 2024 (I&II)| AMC 2023| Olympiad Bench| College Math | GSM8K|GaokaoEn 2023| Omni-Math|\n| :--:  | :--:   |  :--:   | :--:   | :--:   | :--:   |  :--:   | :--:   | :--:   |  \n|10195 |500 | 30|40|675|2818|1319|385|4428|\n\n[1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\n\n[2] Kimi k1.5: Scaling Reinforcement Learning with LLMs\n\n[3] OpenAI o1, learning to reason with LLMs: https://openai.com/index/learning-to-reason-with-llms/\n\n\n\n>Q2:  the proposed code-augmented CoT data synthesis method has two main shortcomings: while it can detect computational and symbolic deduction errors, it cannot identify logical errors (since Python typically won’t produce an exception in those cases), and its verification is limited to tasks manageable via code execution rather than general math proofs, though this may still suffice for datasets like MATH.\n\n**Response**: Thank you for your insightful comment! We would like to clarify that the design of our code-augmented CoT is specifically focused on reducing computational and symbolic deduction errors, rather than logical errors. To address logical errors, we primarily rely on our proposed PPM (process preference model), which is designed to select the optimal step-level reasoning step at each stage, such that the final trajectory reaches the correct answer. \n\nRegarding math proofs,  as discussed in Appendix A.1 (Generalization to theorem proving), our method shows great potential for application to math proofs, as demonstrated by examples in Appendix A.3. The current limitation is step-level proof verification, which we believe can be addressed with a proof-capable PPM. To overcome this, our future work will focus on collecting step-level proof data to train a dedicated math proof PPM.\n\n\n>Q3: In many RL settings, methods that rely heavily on pre-trained or trainable critic models can be easily hacked by LMs. How does your approach mitigate these risks? Could you explain any strategies you employ to ensure the policy cannot simply exploit the critic’s learned features?\n\n**Response**: Thank you for your insightful question, which prompted deeper reflection and discussion on this important issue. Here are our thoughts:\n\n1. **MCTS-driven self-evolution adopts offline updates**: Reward hacking is a known challenge in the **general RL and RLHF**, when the policy is updated online[1,2,3]. In such cases, the policy can exploit flaws in the reward model, leading to increasing reward scores without real performance gains. Our approach uses offline updates for both the policy and reward model.  In each self-evolution round, we fix the policy model and  PPM (from the previous round) and use MCTS to generate training data offline. This decoupling ensures controlled (real high-quality) data selection and significantly reduces reward hacking.\n\n2. **Robust process-level rewards**: A key cause of reward hacking in general RL settings is the lack of reliably labeled response feedback, requiring a learned reward model for scoring. In contrast, math reasoning allows direct validation against ground truth, avoiding the need for a reward model to score final answers. To avoid potential unreliable reward label annotations during process-level reward, we incorporate: (i) MCTS Rollouts and Backpropagation: Initial step-level Q-values from the PPM are refined via MCTS rollouts, with backpropagation adjusting scores based on how often steps lead to correct answers. (ii) Process Preference Model:  Instead of using raw Q-values — often imprecise even with extensive rollouts — we construct preference pairs and train the PPM accordingly, reducing noise and improving robustness.\n\nExtensive experiments validate these strategies, showing our approach mitigates reward hacking while ensuring reliable process-level supervision. We welcome any further discussions!\n\n[1] Defining and Characterizing Reward Hacking \n\n[2] Scaling Laws for Reward Model Overoptimization \n\n[3] Reward Shaping to Mitigate Reward Hacking in RLFH", "claims_and_evidence": "The paper’s main claims—that small models can rival larger ones in mathematical reasoning through iterative self-evolution and process-level reward models—are largely backed by systematic experiments on multiple math benchmarks, ablation studies, and comparisons to existing baselines", "methods_and_evaluation_criteria": "Their proposed method is not as novel as stated, given that (1) the practice of augmenting solutions with comments or code already exists (e.g., program-aided language models), and (2) leveraging Monte Carlo search for data augmentation or self-evolution is a well-known technique rather than a fresh contribution. Finally, their AIME test set is very small (only 15 items), raising concerns that it could be a cherry-picked scenario.", "theoretical_claims": "They have no theoretical claims", "experimental_designs_or_analyses": "The authors’ experimental design generally appears sound, particularly in their ablation studies and comparisons on multiple benchmarks, but a closer look at smaller test sets, such as the 15-problem subset of AIME, raises concerns about cherry-picking or insufficient sample size.", "relation_to_broader_scientific_literature": "The methods presented—such as code augmentation, step-by-step reasoning, and Monte Carlo tree search for self-evolution—are already known in the broader literature; however, this paper combines them into a cohesive pipeline. Its primary contribution lies in showing that smaller models when using these techniques, can attain performance on par with larger LLMs like o1 for certain categories of math problems.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "A key strength is that the paper demonstrates even smaller language models can perform on par with larger ones (such as o1) when solving high school–level math problems. However, the proposed code-augmented CoT data synthesis method has two main shortcomings: while it can detect computational and symbolic deduction errors, it cannot identify logical errors (since Python typically won’t produce an exception in those cases), and its verification is limited to tasks manageable via code execution rather than general math proofs, though this may still suffice for datasets like MATH.", "other_comments_or_suggestions": "No other comments", "questions_for_authors": "In many RL settings, methods that rely heavily on pre-trained or trainable critic models can be easily hacked by LMs. How does your approach mitigate these risks? Could you explain any strategies you employ to ensure the policy cannot simply exploit the critic’s learned features?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rIS8uqSONy", "reviewer": "Reviewer_nzwK", "summary": "This paper aims to improve the mathematical reasoning capabilities of small LLMs through a self-evolved deep thinking framework, rStar-Math. The method involves three main contributions: (1) a code-augmented CoT data synthesis method; (2) a pairwise training method for the process preference model that avoids direct step-level reward annotations; and (3) a self-evolution recipe that iteratively improves the reasoning capabilities of the LLMs. Experimental results on math reasoning benchmarks show the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1:  The main issue with the paper is the overclaim regarding the \"Self-Evolved\" nature of the proposed method. The use of a 236B model (DeepSeek-Coder-V2-Instruct) in the initial round of self-evolution directly contradicts this claim. \n\n**Response**: We appreciate the reviewer' feedback regarding our claim of \"self-evolved deep thinking\". We clarify our approach and demonstrate that self-evolution remains the primary driver of improvement.  Notably, the effectiveness of self-evolution is acknowledged by all the other three reviewers.\n\n1. **Clarification of self-evolution**: Our method relies on iterative improvement through MCTS-driven deep thinking. While we use DeepSeek-Coder-V2-Instruct-236B in Round 1 for bootstrapping, **this does not constitute distillation**, as the main role is to provide *an initial dataset* and plays no role in later rounds. The key novelty lies in **Rounds2-4, where we progressively train stronger 7B policy and PPM to improve independently through self-evolution**. The two models did not rely on a superior model *during the learning*, which is the key characteristics that defines model distillation. This is similar to a person who self-evolves the capability through solving problems with known answers (the initial dataset) without other people's help.\n\n2. **Empirical evidence: performance gains from self-evolution**: Table 8 (Appendix A.2) details per-round performance.  We also provide the key results below for reference. The results show that: the policy model (round 1) did not match the performance of 236B, even with SFT data. Performance gains primarily occur in Rounds 2-4, driven by self-play and MCTS. The final 7B model outperforms the 236B model and approaches O1-mini's performance.\n\n||MATH| AIME| Olympiad Bench| College Math|\n| :--:  | :--:   | :--:   | :--:   | :--:   |\n|DeepSeek-Coder-V2-Instruct (bootstrap model)| 75.3| 13.3|37.6|46.2|\n|our policy Round1| 69.6| 3.3|  34.7| 44.5|\n|our policy Round4| **78.4**| **26.7**| **47.1**| **52.5**| \n| | | | | |\n|o1-mini | **90.0** |**56.7** |65.3 |57.8 |\n| our policy+PPM Round4| **90.0**|53.3 |**65.6** |**60.5** |\n\n3. **The role of round 1: code-augmented CoT format induction**: We use 236B in Round 1 primarily to ensure that our policy model can generate code-augmented CoT traces, not to transfer problem-solving ability. \n\n4. **Distinction from Distillation** : Distillation typically involves a student model mimicking a teacher model throughout training. In contrast: (i) we use 236B only once to generate the initial dataset, (ii) self-evolution proceeds without further reliance on 236B, and (iii) later performance improvements arise purely from self-play. **Our setup is analogous to AlphaGo**, where an initial policy network is required for stability, but key performance gains stem from iterative self-evolution.\n\nTo clarify, we will revise the paper to emphasize that Round 1 is for data formatting (code-augmented CoT), not capability transfer. We hope this addresses your concerns and welcome further suggestions.\n\n>Q2: The paper proposes a code-based CoT approach to enhance math reasoning. it raises concerns about domain-specificity.  The authors should address this limitation by discussing the potential generalizability of their method to other reasoning tasks beyond math. \n\n**Response**: **Clarification of Scope: focus on math reasoning**: Thank you for raising this point. We would like to emphasize that the primary focus of our paper is math reasoning, as clearly stated in the title. While generalizing to other domains is an interesting direction, exploring those is outside the scope of the current work. As you correctly point out, the potential generalizabililty of our method beyond math is definitely a worth exploring direction in our future work, as we elaborate next.\n\n1. **Code-augmented CoT for code reasoning**: Our code-augmented CoT approach could indeed be valuable for other reasoning tasks, such as code reasoning. Code reasoning shares similarities with math reasoning in terms of structured problem-solving, making this method applicable in such domains as well.\n\n2. **General Applicability of Self-Evolved Deep Thinking & PPM**: Beyond code-augmented CoT, our work’s core contributions — Self-Evolved Deep Thinking and the PPM — potentially offer a more general framework for various reasoning tasks. A key challenge in general reasoning lies in providing feedback to verify whether a trajectory reaches the desired outcome at the end of an MCTS rollout. In code reasoning, this could involve extensive test cases, while in other domains, feedback might come from human annotation or mutual verification with another LLM. Future work will explore these directions.\n\nThank you again for your valuable feedback and suggestions. We hope these responses address your concerns and clarify any confusion, and we kindly ask you to consider re-evaluating our work.", "claims_and_evidence": "There is a notable inconsistency that warrants attention. The authors claim that their method achieves **self-evolved** deep thinking **without distillation** from superior models, emphasizing the independence of their approach from larger or more advanced models. This claim is central to the paper's novelty and contribution. However, in Section 3.3, Round 1, the authors state that \"we run MCTS with DeepSeek-Coder-V2-Instruct (236B) to collect the SFT data\". This use of a much larger model directly contradicts the claim of not relying on superior models for data synthesis. The initial bootstrap round leverages the capabilities of a 236B model, which introduces a form of distillation from a more advanced model. This inconsistency suggests an overclaim in the introduction that is not fully aligned with the methodology described later in the paper.", "methods_and_evaluation_criteria": "- The proposed methods make sense for improving the mathematical reasoning capabilities of LLMs.\n- The used math reasoning benchmarks are appropriate for assessing the performance of LLMs.", "theoretical_claims": "The paper does not introduce new theoretical contributions.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound.", "relation_to_broader_scientific_literature": "The key contributions of the paper are related to the broader scientific literature on improving reasoning capabilities in language models.", "essential_references_not_discussed": "The paper has cited relevant prior work.", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper is well written and easy to understand.\n- The code-based cot is interesting.\n- The experimental results demonstrate clear performance gains.\n\nWeaknesses:\n\n- The main issue with the paper is the overclaim regarding the \"Self-Evolved\" nature of the proposed method. The authors claim that their approach does not rely on distillation from superior models. However, the use of a 236B model (DeepSeek-Coder-V2-Instruct) in the initial round of self-evolution directly contradicts this claim. This inconsistency undermines the core novelty of the method and needs to be clarified by the authors.\n- The paper proposes a code-based CoT approach to enhance math reasoning. While this method is effective for math problems, it raises concerns about domain-specificity. Specifically, the reliance on code execution and Python-based verification may limit the applicability of this approach to other domains where code-based reasoning is less relevant or feasible. The authors should address this limitation by discussing the potential generalizability of their method to other reasoning tasks beyond math.\n\nTypo:\n- In line 362, the figure number is missing.", "other_comments_or_suggestions": "See weaknesses.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mqBy8qFeVc", "reviewer": "Reviewer_xKZC", "summary": "This paper introduces rStar-Math, a novel approach demonstrating that small language models (SLMs) can achieve state-of-the-art mathematical reasoning capabilities without relying on knowledge distillation from larger models. The key innovation is a self-evolving deep thinking framework that enhances the reasoning ability of SLMs through Monte Carlo Tree Search (MCTS).\n\n1.rStar-Math uses SLM-based policy model and reward model to iterative improves problem-solving ability through multiple rounds of self-improvement.\n\n2.Major Innovations:\n\ta,Code-Augmented Chain-of-Thought Data Synthesis\n\tb,Process Preference Model (PPM)\n\tc,Four Rounds of Self-Evolution (solve increasingly difficult mathematical problems)", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: No sensitivity analysis of MCTS parameters—it is unclear whether performance saturates beyond 64 rollouts.\n\n**Response**: Thank you for your thoughtful review and for recognizing our contributions. \n\nWe sincerely appreciate your suggestions and have conducted additional analysis on MCTS parameters, specifically focusing on the number of candidate nodes per step and the number of rollouts, which we found to be the most influential factors.\n\n1) **number of candidate nodes per step**: To further analyze this parameter, we conducted additional experiments on Math-500 and AIME, testing different candidate node settings (4, 8, 16, 32, 40) under 8 and 64 rollouts. Notably, our paper adopts node=32. As shown in the table below, increasing the number of candidate nodes generally improves accuracy, but beyond 32 nodes, performance saturates.\n\n|MATH-500|8 rollouts| 64 rollouts|\n| :--:  | :--:   |  :--:   |\n| node=4| 87.2 | 88.8 |\n|node=8| 87.2 | 88.8 |\n|node=16| 88.4 | 89.0 |\n|node=32| 89.4 | 90.0 |\n|node=40| 89.4| 90.0|\n\n|AIME|8 rollouts| 64 rollouts|\n| :--:  | :--:   |  :--:   |\n| node=4| 33.3 | 36.7 |\n|node=8| 33.3 | 43.3 |\n|node=16| 36.7 | 50.0 |\n|node=32| 50.0 | 53.3 |\n|node=40|46.7|53.3|\n\n\n2) **number of mcts rollouts**: As mentioned in Section 4.2 (Scaling Up Test-Time Computation), different benchmarks exhibit different trends as rollout count increases. Specifically, Math, AIME, and Olympiad benchmarks saturate at 64 rollouts. For Gaokao and College Math, where performance showed signs of further improvement beyond 64 rollouts, we conducted additional 128-rollout experiments.\n\nAs shown in the following table, \nPass@N scores consistently improve with more rollouts, but gains become marginal beyond 64 rollouts compared to the doubled search costs. On Gaokao, increasing rollouts from 64 → 128 resulted in only a slight improvement (81.3 → 81.6 for pass@1). On College Math, performance fully saturated at 128 rollouts.\n\n\n|benchmark|1| 2| 4| 8| 16| 32| 64| 128|\n| :--:  | :--:   |  :--:   | :--:   | :--:   | :--:   |  :--:   | :--:   |  :--:   |\n|Gaokao En (pass@1) | 74.5| 78.4|79.0 |80.5| 80.5|81.0 | 81.3|81.6|\n|Gaokao En (pass@n) | 74.5| 78.4| 81.8| 83.4| 85.5| 86.8| 87.0| 87.5|\n| | | | | | ||||\n|College Math (pass@1) | 55.9| 57.2| 58.0 | 59.0| 59.6| 60.1| 60.5| 60.5  |\n|College Math (pass@n)|  55.9| 57.2| 60.9| 63.5| 64.9|66.4| 67.6|68.9 |\n\n\nWe appreciate the reviewer’s constructive feedback and will incorporate these analyses into our revision.\n\n\n>Q2: Computational efficiency of self-evolution is unclear—training costs (weeks of GPU time) may be prohibitive for broader adoption.\n\n**Response**: Thank you for your thoughtful feedback and for highlighting the importance of computational efficiency. We provide a detailed cost breakdown in the appendix and further clarify below.\n\nSelf-evolution primarily involves two stages: (1) Policy Model & PPM Training; (2) Training Data Generation via extensive MCTS rollouts.  As shown in the following tables, training is efficient. Each round completes within a day. \n\n| |GPUs|Training time|\n| :--:  | :--:   |  :--:   | \n|Policy model|8xMI300 | 20 hours |\n|PPM | 8xMI300 | 15 hours|  \n\n\nTraining data generation is the main cost, but it is scalable and affordable: (1) From Round 2 onward, since our policy model and PPM are both 7B, they can be served on a single 40GB A100. (2) To process 747K math problems efficiently, we used 15 groups of 4×40GB A100s, completing data generation in ~3 days. (3) Further speedup is feasible: This process scales linearly with more GPUs (i.e., reducing the number of problems assigned per GPU). Round 1 is the only costly stage, as it requires bootstrapping with DeepSeek-Coder-V2, which was done using 8×80GB H100s.\n\n\n|Round |GPUs|Data generation time|\n| :--:  | :--:   |  :--:   | \n|Round 1|5x8x80GB H100| 2 weeks|\n|Round 2| 15x4x40GB A100| 2-3 days |\n|Round 3| 15x4x40GB A100| 2-3 days |\n|Round 4| 15x4x40GB A100| 1 week |\n\nOverall, we believe the computational cost is reasonable and manageable, and the primary bottleneck—data generation—can be further optimized with additional GPUs. We will refine our explanation in the revision to provide a clearer analysis of efficiency and scalability.", "claims_and_evidence": "Questions for the claims:\nrStar-Math Exhibits Intrinsic Self-Reflection Without Explicit Training:\nSome examples (Figure 4) show that the model backtracks and corrects mistakes, but it is unclear if this behavior is a direct result of MCTS or an emergent property.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-aligned with the problem of improving math reasoning in SLMs. The combination of MCTS-guided deep thinking, PPM-based process rewards, and iterative self-evolution is novel and well-validated through comprehensive benchmarking. However, high computational costs and the lack of explicit theorem-proving evaluation remain areas for improvement.", "theoretical_claims": "This is not a theory paper.", "experimental_designs_or_analyses": "Minor issues:\n1.Limited theorem proving evaluation: The benchmarks focus primarily on word problems and algebraic reasoning, but lack an explicit evaluation on theorem proving, despite claims of generalization.\n\n2.Unclear computational costs.", "relation_to_broader_scientific_literature": "1.Unlike GPT-distilled datasets, rStar-Math bootstraps its own training data via MCTS, eliminating reliance on larger models.\n2.rStar-Math improves MCTS by integrating code execution validation and self-evolving data generation, leading to more reliable stepwise reasoning trajectories.\n3.Unlike prior PRMs, PPM avoids noisy absolute score annotations and instead uses pairwise ranking loss, improving reward signal quality.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Weaknesses mainly in computational efficiency and theorem proving evaluation.", "other_comments_or_suggestions": "None", "questions_for_authors": "1.No sensitivity analysis of MCTS parameters (e.g., search depth, exploration constant)—it is unclear whether performance saturates beyond 64 rollouts.\n\n2.Computational efficiency of self-evolution is unclear—training costs (weeks of GPU time) may be prohibitive for broader adoption.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces rStar-Math, a novel training pipeline that enables small language models (SLMs) to achieve math reasoning capabilities comparable to OpenAI’s o1. The key contributions of the proposed framework include:

1. A novel code-augmented CoT data synthesis method,
2. An innovative process reward model training method, and
3. A self-evolution recipe for iterative model improvement.

The experimental results are highly impressive. All reviewers acknowledged the significance of the paper’s contributions. While one reviewer initially raised concerns about potential overclaims, these were thoroughly addressed in subsequent discussions. The authors engaged constructively with the reviewers and resolving all questions.

Given the paper’s substantial contributions and the strong consensus among reviewers, I believe this work is highly relevant to current research interests in the community and will stimulate further discussion. Therefore, I recommend accepting this paper.
Decision: Accept (oral)