AlphaMath Almost Zero: Process Supervision without Process
Authors: Guoxin Chen
Minpeng Liao
Chengxi Li
Kai Fan
Venue: NeurIPS 2024 poster (2024)
Keywords: Mathematical Reasoning
Monte Carlo Tree Search
Question Answer
Primary area: natural_language_processing

Abstract: Although recent advancements in large language models (LLMs) have significantly improved their performance on various tasks, they still face challenges with complex and symbolic multi-step reasoning, particularly in mathematical reasoning. To bolster the mathematical reasoning capabilities of LLMs, most existing efforts concentrate on seeking assistance from either domain experts or GPT-4 for high-quality process-supervised data, which is not only expensive but also labor-intensive. In our study, we propose an innovative framework, AlphaMath, that bypasses the need for process annotations (from humans or GPTs) by leveraging Monte Carlo Tree Search (MCTS). This framework focuses on unleashing the potential of a well-pretrained LLM to autonomously enhance its mathematical reasoning. Specifically, we integrate a value model with the LLM, automatically generating both process supervision and step-level evaluation signals in MCTS. Furthermore, we propose an efficient inference strategy—step-level beam search, where the value model is crafted to assist the policy model (i.e., LLM) in navigating more effective reasoning paths, rather than solely relying on prior probabilities. The experimental results on both in-domain and out-of-domain datasets demonstrate that even without GPT-4 or human-annotated process supervision, our AlphaMath framework achieves comparable or superior results to previous state-of-the-art methods.

AlphaMath Almost Zero: Process Supervision without Process
Guoxin Chen', Minpeng Liao", Chengxi Li", Kai Fan*t
Tongyi Lab
chenguoxin220maille ucas ac.
fminpeng. 1mp, xiji. la..illc.... ftt.:kk.aaaaaaaaaaaaaaaaaa Caliabaaan..... com
Code: https://github. wwt.a/RIIccmMMRRRO00O. Math /to
Abstract
Although recent advancements in large language models (LLMs) have significantly improved their performance various tasks, they still face challenges with com- plex and symbolic multi-step reasoning, particularly mathematical reasoming. To bolster the mathematical reasoning capabilities of LLMs, most existing efforts con- centrate on seeking assistance from either domain experts or GPT-4 for high-quality srocess-supervised data, which is not only expensive but also labor-intensive. In our study, we propose an innovative framework, AlphaMath, that bypasses the need for process annotations (from humans or GPTs) by leveraging Monte Carlo Tree Search (MCTS). This framework focuses on unleashing the potential of a well-pretrained LLM to autonomously enhance its mathematical reasoning. Specif- ically, we integrate value model with the LLM,, automatically generating both process supervision and step-level evaluation signals in MCTS. Furthermore, propose an efficient inference strategy-step step-level beam search, where the value model is crafted to assist the policy model (i.e., LLM) in navigating more effecttvee reasoning paths, rather than solely relying on prior probabilities. The experimental results both in-domain and out-of-domain datasets demonstrate that even with- out GPT-4 or human-annotated process supervision, our AlphaMath framework achieves comparable or superior results to previous ttat--of-the-ar methods.
Introduction
Recent studies have extensively explored how improve mathemat-
Table 1: Annotation Cost
ical reasoning in large language models (LLMs) [27, 1, 37, 34, 2, 35]. effective approach [46, 38, 14, 21, 31, 25] to artificially inject on a substantial volume of high-quality, process-supervieee data (i.e,, solutions). As shown in Table 1, the annotation of high-quality solutions in current efforts primarily relies on domain experts or GPT-4 [27]. However, due to trillions of training tokens and billions of parameters, existing LLMs possess a vast reservoir of knowledge, which remains under- utilized in current finetuning-base approaches.
Annotation Source
Methods
external knowledge into LLMs through
Human
GPT-4
>
[21, 461
X
[14, 25, 38]
x
Ours
To more effectively harness the intrinsic knowledge of LLMs, advanced prompting techniques, such as Program-of Thought (PoT) [6] and Program- Aided Language (PAL) [13], have been developed, integrating the in-context learning proficiency with external tools such as code interpreter handle precise numerical and symbolic computation. However, these approaches have not fully unleashed
"equal contribution
t "oorespondiigg Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the potential of LLMs and often rely on self-consistent majority voting [39], which does not reflect the natural process by which humans solve mathematical problems. This discrepancy arises because both the PoT and PAI. frameworks pursue solution to its final answer regardless of the accuracy of intermediate steps. Unlike these approaches, humans tend to reassess and potentially alter their solution path upon encountering mistake or dead-end in the problem-solving process. In this manner, humans iteratively enhance their self-cognition and reinforce the utilization of knowledge.
In this research, we aspire for LLMs to possess the similar ability as humans to realize self-evolution and strengthen their utilization of knowledge autonomously. Notably, AlphaGo Zero [33] showcases how a neural network model can progressively evolve without human knowledge, autonomously producing the Go game training strategies. For the strategy (i.e., solution) of mathematical problems, both textual analysis [45] and code snippets [14] demand rigorous logical structuring. Consequently, most finetuning- based approaches concentrate on seeking assistance from domain experts or GPT-4 for annotated solutions, thereby overlooking the reservoir of knowledge inherent in LLMs.
Instead, we hypothesize that well pre-trained LLMS already possess the necessary mathematical knowledge to generate correct reasoning; however, they require appropriate dimulation-suu as an improved prompt or search strategy-to do so. In this work, solutions including both textual analysis and code snippets are autonomously generated by a well pre-trained LLM equipped with appropriate prompts and deliberately designed Monte Carlo Tree Search (MCTS) framework [4. 32]. Specifically, we integrate LLMs with the MCTS to strike a more effective balance between exploration and exploitation, enabling the generation of high-quality process- supervise solutions without professional human annotations. To enhance the efficiency of solution generation, we incorporate a value model into the same L.LM by appending a linear layer. This advancement removes the necessity for time- consuming rollouts for reward estimation. While the LLM learns to solve mathematical problems from its own annotated solutions, the value model simultaneously learns how to assess the quality of intermediate reasoning steps from the corresponding state values in MCTS, just like humans.
During the inference stage, with the value model, LLMS can perform MCTS inference, which significantly enhances their reasoning capabilities but limited by efficiency. Therefore, inspired by beam search algorithm [36], we propose a step-level beam search strategy, where the value model is crafted to assist the policy model (i.e., LLM) in navigating more effective solution paths, as opposed to relying solely on prior probabilities. Compared to the greedy or MCTS inference strategies, the step-level beam search significantly enhances the LLM's reasoning capability at a minimal cost.
Empirically, we build an iterative training framework as shown in Figure 1. Unlike in the game of Go, where the final board state directly indicates a win or loss, our methodology requires validation of the equivalence between predicted answers and actual ones. This is the fundamental reason why our training data necessarily consists of question statements and their final answers. Furthermore, we validate the applicability of our framework on three popular types of LLMs: domain speciiic pre- trained models [31], general-purge purpose pre-trained models [1 I], and supervised fine-tuned models [21]. Our contributions are summarized as follows:
We propose a novel approach that integrates pre-trained LLM with deliberately designed Monte Carlo Tree Search (MCTS) framework. This combination allows LLMs to autonomously generate high-quality mathematical reasoning solutions without the need for professional human annotations, leading to a more efficient utilization of their inherent knowledge.
To address the efficiency limitations of MCTS inference, we propose a step-level beam search strategy, which introduces a lightweight value model that works alongside the L.M,, enabling the simultaneous assessment of the quality of intermediate reasoning steps. This method parallels human problem-solving by allowing the LLM to learn from its own solutions while also evaluating the effectiveness of its reasoning strategy, thus enhancing the overall reasoning capabilities.
Extensive experiments demonstrate that our AlphaMath can effectively stimulate the internal knowledge of LLMs, achieving better or on par task performance on both in-domain and out-of- domain mathematical reasoning datasets, even without any process annotations.
2 Preliminary
We assume that, for any given input question q. the solution process can be broken into multiple reasoning steps (e.g., segmenting the solution based on distinct stages or simply on a period). From this perspective, we conceptualize mathematical problem solving within the context of reinforcement
2

---

Iterate K rounds
V(s;)
USM
seket
Droaneet Exponied mode
Ruciop
bed noss
LLM (TOLLy
Vay)
PVA
Vig)
</step>
filter
PHD
ViAD
(Q (Q.45
PIb
Sc-1
Training Data
MCTS
Solutions
self-evolutioo
palicy-value model
Our approach involves iterating through three distinct stages. (1) Collect mathematical dataset that comprises questions and their corresponding final answers. (2) Employ MCTS on the policy and the value model generate both correct and incorrect solution paths along with state values. (3) Optimize the policy and the value model with generated data from MCTS.
Figure 1:
learning. Concretely, consider a complete solution consisting of T reasoning steps. At a given time represent the partial solution the state 8,, and the subsequent reasoning step that might be taken as the action as a, For detailed definitions and examples of our reasoning step, please refer to Appendix C.I. In this scenario, the policy model is embodied by large language model, and the transition (5 (q, st) from one state to the next is deterministically accomplished through the concatenation operation.
(ayls)) LL.(s.)s:).
S
Cat(st, a.)
(I)
Our primary goal is to develop step-level value model, denoted as Va(s), which is capable of assessing the expected returns from the current partial solution and guiding the LLM to select more reasonable subsequent reasoning steps.
To train the value model, we first define the reward in the context of mathematical problem solving, by assigning the reward r
all non-terminal reasoning steps, and r #1 to a correct/incorret final answer. A common method to create the training signal is to employ Monte Carlo (MC) evaluation (s)
(8) ((s.) = き/ Σm= r (a.² (a. s(>ssss))))) where ar' and syst represent the actions and states in the
i-th simulation sampled by the policy model and the state transition function. r(-[s;) means the
reward of the final outcome in one simulation from state s,. Then, for any given partial solution s, we
can train the step-level value model Vo using a regression loss defined as follows:
Lv. (s) Vo(s)- v(s)
3 AlphaMath
In the above approach of MC evaluation, it requires multiple simulations from each state, which may be inefficient practice. We propose employing the Monte Carlo Tree Search (MCTS) algorithm, which has the potential to reuse simulations and update the estimated values in a principled manner.
3.1 MCTS Evaluation
As shown in Figure our approach employs iterative training. Before the (A + 1)-th round training, we have a value model Vo, and a LLM policy model iee which the same model but with different final layers in our paper. Using these models, we can construct an inference algorithm powered by MCTS. This algorithm starts with the initial state as its root, and through the synergistic use of the policy and value models, systematically grows the search tree by adding new nodes. These nodes correspond to the states deemed to have high potential based on the outcomes of simulated trajectories. Specifically within the context of mathematical problem-solving as shown in Figure 2, customize the four key operations of the MCTS algorithm follows:
Selection During the i-th simulation the MCTS, the process begins with So, representing the initial state containing the input question. The algorithm then proceeds to explore the tree Ty by selecting

---

Repeated N times
Selection
Expansion
Evaluation
Backpropagation
Tu, (s))
Va(ai)
or(s,)
Figure 2: An overview of the four key operations in MCTS
nodes according to a variant of the PUCT algorithm [30]. This selection process mathematically represented as:
parent a ((s,, a)
Q a) + CpuctTTA (als:)
arg max
where the state-action value ((s, a) and its visiting count ((, a) are stored in the tree and will be updated as the search progresses. (pprenn(a) represents the visiting count of the parent node of a. The action selection iterates until it encounters leaf node of the current search tree. In our case, the prior (als,) is defined as the exponential of averaged log- gee-rbbabiiity of all tokens in the step a, i.e. exp
log T(a)la=j,,s)
Expansion Back-tracing from the selected leaf node to the root forms a partial solution, serving as a prompt for further node expansions. In case, given that the LLM can theoretically generate unlimited number of potential actions (token sequence), we employ sampling generation with higher temperature to ensure diversity.
Evaluation Evaluation the leaf node or partial solution 8,, identified after the selection phase, conducted by weighted sum introduced in [32, 33].
(s,) (1 - A) (s:) + X 2 S 81
(4)
The intermediate value estimation V in MCTS differs from training signal V defined in preliminary section 2. The parameter À serves to balance the contribution of the value model's estimation with empirical reward obtained during the rollout.
In our case, we follow trade-off rollout strategy between AlphaGo [32] and AlphaGo Zero [33]. Because our tree depth much shallower than Go games (e.g. a maximum depth of 8) and expansions can easily reach terminal node, we set an indicator function = Iermmnal (Sz). If the expanded node terminal, the reward is returned; otherwise, the value is predicted by the model Vo.-
Backup We did not make any modifications to the backup. At the end of the i-th simulation, each edge (s, a) along the path from the leaf node to the root undergoes backward pass update. The updates to their state-action values and visiting counts are executed according to the following rules: N (ss, a) + (s, a) + and (s,, a)
[(s.n) Ej=1 ... (s:))
Value Estimation After running N simulations with the MCTS algorithm, we obtain the final tree Te. which stores the expanded nodes and their corresponding state-action values Q(s,, a). Considering that the transition function is deterministie, and assuming that Q(s a,) = r(st, a, + = (s$++) for non-terminal nodes". we can employ the Q values as training signals. This implies that we can directly fit the state-action value of non-terminal nodes
((s+1)) = (ss,
(5)
Reward is 0 for non- non-terminal node, and reward determined by the final answer in terminal node.

---

Algorithm Inference with MCTS
Require: Bi = question q (So). policy / value models nH, Va, simulations N, max depth T. 1: Build the complete tree T by running MCTSmn,Ve V (So, N, T).
2: C [so]. t
Initialize candidates b Max heap
and non-terminal path in do 4: Initialize priority queue Cz+1
3: while t <
5: for St in C do
6: for in Tehildren (St) do
Directly get children from tree
7:
St+-1 = Cat [Sr, a]
8:
Add (Se+1, (St, a)) to C++1
Directly get Q-value from tree
9: C - Top-B, of C++1
return Top-1 of
i Return top-1 as the final solution path
Algorithm 2 Step-level Beam Search
Require: Beam sizes B1, B2, question q (so). policy value models ne, Va. max steps T. Initialize candidates
I: C= [so] B1.
2: while f. <
and non-terminal path in do Initialize priority queue C++1
3:
Max heap
4: for St in C do
5:
Sample {a for = to B, b=1
~ re((s)))
LLM generates B2 samples in parallel.
6:
do St+1 Cat a()]
7:
8:
Add (sr+1, (o(S(+1)) to C++1
Vo($t+1) predicted by value model
9:
Top-BpB of C++1
return Top-1 of
Return top-1 as the final solution path
3.2 Iterative Training
Initialization Initially, our approach begins with pre-trained LLM as the policy model ne, We extend this model by adding an auxiliary linear layer with a tanh activation function, which works alongside the traditional softmax layer responsible for token prediction, as depicted in the rightmost panel of Figure 1. This design implies that these two models, parameters. The parameters of the linear layer associated with Va, are randomly initialized, leading to an initial tendency of the value head to predict a value close to 0 at the first (k: = 1) round of MCTS. However, as the simulations in the first round MCTS proceed, the rewards (#11) from terminal nodes are back-propagated to their parent nodes. As simulations N gradually increase, the estimated values Q of intermediate nodes converge towards the underlying true value within the range of [-1, 1].
and Vg, share the majority of their
Training Method From the T constructed from the k-th round of MCTS, can sample solution paths corresponding to terminal nodes with correct and incorrect predicted answers, denoted as x and x respectively, together with the value estimation of each node along these paths. We then apply multi-task loss function to update both the policy and value models.
T(x') Σ -
arg min log (0( + B- 8.0
Σ (Vo(s))-i(s.)
(6)
where the first term represents the negative log-likelihood loss for next-token prediction in correct solutions, and the second term within the big brackets captures the loss in value prediction for both correct and incorrect solutions, respectively. T(x) denotes the number of steps for solution path x. B tunable hyper- parammea control the weight of value loss. With the updated policy and value models o and Vor.,, we can advance to the next-round MCTS, iterating this training process to enhance our models further.
3.3 Inference
MCTS For MCTS inference, is necessary to set in the evaluation of Eq. (4). Unlike in board games, we cannot verify the correctness of a path during inference. Therefore, consistently
5

---

rely on the value model for node evaluation, including for terminal nodes. MCTS demands multiple simulations to update visiting counts and Q values, aiming to estimate robust policy distribution.
After the tree has been completely built, the algorithm iteratively selects the top- B1 steps (usually Bi = 1 in MCTS) from the root in a top-down manner. This selection is guided by the maximum Q-value stored in the child nodes of the tree. Subsequently, all child nodes from the previously selected B1 steps are collectively re-ranked based on their Q-values, and the top-B1 nodes from this ranking are retained for the next iteration. A summary of the algorithm can be found in Algorithm 1.
Step-level Beam Search However, MCTS is computationally intensive for simulations, making it less viable for use in production environments. To address this, we modify the MCTS inference process by eliminating the backup operation, introducing simplified method, which we refer to as Step-level Beam Search (SBS), detailed in Algorithm 2. This approach does not construct the entire tree; instead, it dynamically selects the best child node during node expansion.
There are two primary technical distinctions in SBS. First, since node expansion is required on the fly, we introduce new beam size, B2, to represent the maximum number of node expansions. Second, the selection criterion no longer relies on the Q-value converged after N simulations but instead uses the maximum value prediction directly from the value model. Importantly, with special case SBS B1 = 1 as a fast approximation of MCTS, it facilitates the sequential, streaming output of steps, rendering it more suitable for practical implementation in real-world production.
4 Experiments
4.1 Experimental Setup
In this study, we mainly investigate the math domain-specific language model, MeepSeekMath- Base-7B [31], pre-trained substantial math-related corpus without any supervised fine-tuning (SFT), which is believed to possess necessary mathematical knowledge to tackle a wide range of mathematical problems.
Training Data Generation via MCTS For the training sets, we exclusively extract question and answer pairs from GSM8K [7] and MATH [15], omitting the numan-annotated solution analysis. In total, our training set includes only 15k question-ansee answer pairs and 0 solution process.
In our setup, we utilize the MCTS framework to generate detailed solution processes equipped with the Python code interpreter. Initially, for the first round of MCTS, the prompt used for our solution generation adheres to the REACT [43] format, incorporating 2 demonstrations randomly selected from pool of 20 prepared examples. Starting from the second round, with an already fine-tuned model from the first round, we employ straightforward prompt in our SFT XML format without any demonstration. Two prompt examples are shown in Appendix F.
Specifically, we iteratively generate data and train our policy and value models through K = 3 rounds, continuing until the enhancement observed between any two consecutive rounds incremental. In every round, we build 10 trees for each question- question-answer answer pair and randomly sample at most 4 correct and incorrect solution processes. The ratio between positive and negative examples approximately 1:1, with the count of positive examples in each round varying between 57k and 59k.
Test Data We evaluate approach not only on GSM8K and MATH but also (OOD) datasets Gao33o2023 [21] and CCWCossees [19]. These two OOD datasets are even more challenging than MATH. Please refer Appendix C.5 for more details about the dataset statistics. To assess the accuracy of the predicted answers, we utilize the math evaluation toolkit [47].
Baselines We first compare approach with strong proprictary and open-source models, including OpenAl's ChatGPT and GPT-4 [27]. Llama2 [37]. Llemma [3]. By default, we report the results obtained using Chain of Thought (CoT) prompting [40], along with the prompting results of PAL [13], due to its enhanced performance in mathematical reasoning.
SFT models leverage high-quality seed data with process supervision derived from GPT-4 or humans to enhance their reasoning capabilities. To ensure fair comparison, we primarily contrast our approach with the highest-performing SFT models that utilize an external tool a Python code inter- preter. These include MAmmoTH [46], MathCoder [38], ToRA [14], MARIO [21], MathGenie [25], and DeepSeck-Maun Matss--tuutt----ttuuutt [31]. More implementation details can be found in Appendix C.

---

Table 2: Main results. The best results of open-sourced models are bold. For the methods with released model's 's outputs, performance metrics using the evaluation toolkit [47] are also provided in brackets. *Seed data refers to high-quality annotated (question, solution) pairs, typically annotated by humans or GPT-4. Unless otherwise specified, we set beam size B2 5 in SBS and number of simulations N 40 MCTS by default.
Model
Size
Seed Data
Data
Tool
Zero Shot
In-Domain
Annotation
Size
GSM8K
MATH
GK2023 OCW
Proprictary Models
GPT-4
X
92.0
42.5
GPT-4 (PAL)
94.2
69.7
43.6
30.1
ChatGP
X
80.8
35.5
ChatGPT (PAL)
X
78.6
38.7
Gemini-1
58.5
Claude-3.0 Sonnet
96.4
71.1
Open-Source Models
Llama
7B
X
13.3
4.1
3.7
CodeLLama
10.5
4.5
4.7
7B
27.1
17.2
Llemma
7B
X
36.4
18.0
7.7
(PAL)
21.5
DeepSockMath PALL
7B
66,9
31,4(33.2)
SFT Models
MAmmo TH-Coder
34B
GPT-4-Human
260k
72.7
43.6
25.2
14.0
MathCoder
GFT-
49k
81.7
(4.154888)
ToRA-Code
34B
GPT-4
16k
80.7
50.8(51.2)
31.7
5.5
MARIO
34B
GPT-4-uuman
27k
78.2
53.5
42.6
30,2
GPT4
Llama-2 SFT
Human
15k
41.3
7.2
IJama-2 RFT
7B
Human
15k
51.2
GPT-4-uman
260k
59,4
33.4
15.3
11.0
MathCoder
GPT4
49k
67.8
30,7(30.6)
ToRA
7B
GPT-4
16k
68.8
40.1
19.5
2.6
ToRA-Code
GPT-4
23.9
4.8
MARIO
7B
GPT-4-Human
27k
74.5
48.3
34.5
21.7
MathGenic
7B
GPT4
80k
76.0
48.3
DeepSockMath-Inee Instrruut
GPT-t-Human
776k
83.7
57.4(57.2)
43.9
18.0
SeepSeckMath
+our prompt 2-shot
59.7
33.2
21.9
9.2
MalphaMath 3)
73.5
53.6
40.5
26.1
SBS (B;
81.1
62.8
46.2
30.5
SBS (D: 3)
X
84,1
66.3
51.4
33.1
MCTS (B, 1)
X
83.2
64.0
48.4
33.8
4.2 Main Results
We report our in-domain and out-of-domain (OOD) results in Table Different from previous works [46, 38, 14. 21, 25], our proposed AlphaMath does not rely on high-quality solutions annotated by humans GPT-4, whether in the form of text analysis code snippets. Such solutions typically bolster the model's reasoning abilities but also entail substantial costs associated with annotation. Furthermore, our method differs from prior research by not incorporating any external datasets (e.g., new questions and solutions) beyond the GSM8K and MATH datasets. The last five rows of Table 2 present our principal findings.
First, we establish a baseline with the inherent mathematical reasoning ability of DeepSeekMath-BE using designed prompt in 2-shot setting. It's important to note that this outcome differs from the results reported for DeepSeekMath Base (PAL) in the original study, as it utilized prompts with 8-shot and 4-shot for the GSM8K and MATH datasets, respectively. Secondly, we only evaluate the policy model with greedy decoding. In comparison to our initial study, we record an enhancement of about 20 points for challenging problems in the MATH, GaoKao2023 (GK2023), and OCWCourses (OCW) datasets, and an improvement of more than 10 points for grade school math problems. Thirdly, we delve into the role of the value model in facilitating mathematical reasoning, utilizing computationally efficient step-level beam search (SBS) in Algorithm 2. When we increment B with default B₂ = and temperature of 1 1.0, corresponding gradual improvement in performance is observed. More discussion about the temperature in SBS can refer to Appendix 4.7. Ultimately, we evaluate our approach in Algorithm 1. In contrast to the training data generation, we construct single tree with 40 simulations, maximum of 5 child nodes, and temperature of 0.6. While MCTS
7

---

demonstrates improved performance more challenging datasets, attributed to its expansive search space, its substantial computational demands curtail practical upplicability in real-world scenarios.
In summary, our approach demonstrates that, even in the absence of high-quality GPT-4 or human- annotated solution processes, it remains competitive with or surpasses the performance of the state-of-the-art (SOTA) on 7B LLMs.
4.3 Analysis Performance of each round
We evaluate the problem-solee solving rate in the MATH training dataset, which categorizes each problem by difficulty level. As shown Figure 3, it becomes evident that MCTS achieves greater success in solving more challenging problems in subsequent rounds. In Figure 4, our findings show general increase in performance with additional rounds of training across all strategies, applicable both in-domain and out-of-domain test sets. Therefore, we can conclude that the quality of our self- generated training data improves incrementally with each round, and this enhancement reflected in the performance on the test set. More analysis can refer to Appendix B.3.
4.4 Analysis 2: Performance of different inference strategies
We explore the performance of our model under various inference strategies including greedy decoding, step-level beam search, and MCTS. The results of MATH and GaoKao2023 are illustrated in Figure 4, while the results of other datasets can be found in Appendix B. Specifically, for SBS, an enhancement in performance was observed with an increase in the beam size B, MCTS exhibited the higher performance than its approximation SBS (B, 1), but we previously noted its significant time consumption and computational inefficiency. Consequently, we provide a summary of the average problem-solving duration and the average number of intermediate steps taken the MATH dataset in Table 3. The results indicate that MCTS demands the longest solving time and the highest number of steps, attributable step-level beam search is more computationally friendly. Additionally, we observe an intriguing phenomenon: larger beam size B1 tends reduce the average problem-solving duration. This can be attributed to the decrease in the number of average steps required when larger B1 is employed.
our configuration of 40 simulations. To achieve similar accuracy,
Discussion of Majority Voting It is challeng- ing to directly compare maj@5 with step-level beam search due the inherent differences in their methodologies. Generally speaking, as Al- gorithm SBS will eventually return the top-1 final answer based on the value model, while maj@5 will generate all 5 possible final answers and vote the majority for evaluation.
Table 3: Analysis of Computational Efficiency on MATH dataset. # Sol. denotes the number of solutions obtained eventually.
Inference Strategy
Avg. time (s)
Avg. steps
Aec.
# Sol.
Greedy
53.62
1.6
3.10
Maj@5
61.84 (+8.22)
2.9
2.88
From the step-level perspective, maj@5 will maintain 5 candidates for the current step to generate another 5 candidates for the next step. In contrast, the SBS (e.g.. B1 = 1, B2 = 5) will always retain the t candidate, discarding the 4 others. This provides the advantage of step- by-step streaming output in real-world production, whereas maj @5 can only output the complete solution until the voting is finalized. To sum up, their specific mechanics of candidate selection and retention differ significantly.
SBS (B₁ = 1)
62.80 (+9.18)
3.1
3.01
SBS (B) = 2) 64.66 (+11.04)
2.4
2.36
2
SBS (B1 = 3)
66.30 (+12.68)
2.3
2.21
3
SBS (B, = 5)
65.98 (+12.37)
4.7
2.26
5
MCTS (B, 1)
64.02 (+10.40)
10.1
3.76
4.5 Analysis 3: Value model
In the left panel of Figure 5, we plot the fitted distribution of Q-values (as defined in Eq. (5)) on MATH training set for intermediate steps. For correct solutions, the distribution is markedly skewed towards value of In contrast, the distribution for incorrect solutions exhibits lower degree of skewness, albeit with the majority of the probability density leaning towards -1. This is because a correct final answer typically suggests that the entire solution process likely accurate, whereas an incorrect final answer may still encompass some correct intermediate steps. Thus, with the backup of MCTS, the Q-values of intermediate steps in incorrect solutions may also updated with a reward of I during simulations.
8

---

round
round
round
round
round
round
100
Salve Rate (50
10
Accurscy
Accuracy
Greedy
586
585
MGTs
Groedy
5B1
886
5B6
MGTS
Level
Level
Level
Level4 Level
(8: =
1)
[0:
21
(0s
(P: 11
(8s
13
m0 21
(8t
(8: =
Figure Solving Rate Different
(a) MATH (In-Domain)
(b) GaoKao2023 (Out-of of-oomiin
Levels on MATH Training Set
Figure Comparison of Different Inference Strategies.
In the right panel of Figure plot the Q- values distribution on the test set, including both intermediate and terminal steps. The distribution associated with correct solu- tions exhibits a shape similar that found in the training set. However, the distri- bution of incorrect solutions, which are the bad cases of the policy model, demon- strates bimodal pattern. (1) When the value model believes the incorrect solution predicted by the policy model be incor- rect, the Q-values cluster around -1. (2)
A Density
Correct Solution
Density
Correct Solution
Incorrect Solution
Incorrect Solution
Probabiling
Probability
Q-values
Q-values
Figure 5: (Left) Fitted distribution of Q-values of 3rd round MCTS on the training set. (Right) Fitted distribu- tion of Q-values via MCTS inference the test set.
Conversely, there are instances where the value model erroneously considers an incorrect solution as correct, resulting in another modal towards 1, which represents the bad cases of the value model.
4.6 Analysis 4: Self-evolution on General-purpose and SFT models
We further investigate the potential of other popular types of LLMs: general- purpose pre-trained models and SFT mod- els. These models represent the scenar- ios of lacking continual pre-training (CPT) in domain-specific data and supervised fine-tuning (SFT) on high-quality anno- tated domain data, respectively. We se- lect Llama3 [1]] and MARIO [21] as the base models and report the results in Ta- ble 4. For a fair comparison, the MARIO is rather
Table 4: Additional Results on Llama3 and MARIO. t DDeeSSeekatth -aatt--B---BBB... SOur designed prompt in shot setting.
In-Domain GSM8K MATH
OOD
Model
GK2023 OCW
Llama3-base
40.7
18.7
12.9
2.9
+ AlphaMath (K = 3)
59.4
36.8
27.1
6.6
+ SBS (B1 = 3)
71.8
41.9
31.4
10.7
DSM + MARTO data
78.4
56.1
41.6
25.0
+ AlphaMath (K 2)
80.2
58.8
48.1
31.3
+ SBS (Bi 3)
88.3
68.6
54.1
42.3
trained DeepSeckMath-Basse
than its original Llemma-77 7B [3]. First, although not proficient in mathematical reasoning, our Al- phaMath enhances Llama3's mathematical reasoning capabilities without any annotations, yielding average improvement of +20 points. Secondly, AlphaMath can significantly enhance the performance of existing SFT models, enabling MARIO to be competitive with and outperform GPT-4.
4.7 Analysis 5: The Effeets of Temperature on Step-level Beam Search
We further investigate the effects temperature during decoding the performance of inference algorithms. For the greedy strategy, the temperature is consistently main- tained at O, whereas step-level beam search (SBS) and Monte Carlo Tree Search(MCTS) are more significantly influenced by higher temperatures. Therefore, taking step- level beam search (B, = and B, = 3) as an example, obtained the results as illustrated in Figure
05
2-62
60-
54
groedy
0.1
0.6
0.8
1.2
First, under any temperature setting, the performance of step-level beam search significantly surpasses that of the
Figure 6: The Bffects of Temperature on the performance of SBS.

---

greedy strategy. This is attributed to the value model effectively assisting the policy model in identifying more effective reasoning paths. Secondly, at lower temperatures, the performance of step-level beam search is constrained due to the lack of diversity in the generated solutions. With elevated temperatures, the value model capable of discerning optimal paths within more diverse set of solutions, thereby effectively enhancing reasoning performance. Finally, with larger beam width, the model can explore more solutions. Therefore, the performance B1 always surpasses that of B1 =
Related Works
Solution Annotation in Math. Recent works [46, 38, 14, 20, 21, 31, 25, 16] on mathematical reasoning have made impressive progress empowered by process-supervised data. However, most existing efforts concentrate on seeking high-quality solutions from domain experts or formidable commercial models, such as GPT-4 [27], which hampers the scalability of methods and escalates the associated expenses. Unlike previous work, only with the help of question-ane -answerr pairs, we focus on activating the intrinsic knowledge within LLMs to realize iterative self-evolution and strengthen their utilization of knowledge autonomously, just like humans.
Value/Reward Model. Recent studies [7, 8, 22, 44, 49, 42, 41, 12] have demonstrated that pro- cess supervision can significantly enhance mathematical reasoning performance. Especially, value model [12, 23, 26] is incorporated into the decoding process, while reward model is the source of the training signal in reinforcement learning [28, 31]. However, these value/reward models require substantial annotated process-superve
data and introduce significant inference latency. In our
work, we consider the state values (s,) from MCTS as supervision signals, which are aligned with the solutions and eliminate the annotation costs. Furthermore, we integrate the value model into generative model to navigate more effective reasoning paths at minimal cost, thereby providing richer decoding strategies, such as step-level beam search or MCTS.
6 Conclusion
In this work, we introduce AlphaMath, simple iterative training paradigm for leveraging Monte Carlo Tree Search to unleash the potential of well pre-trained large language model autonomously enhance its mathematical reasoning capabilities. Furthermore, by applying step-level beam search, the value model can assist the policy model in selecting a more reasonable solution path, rather than solely relying on prior probabilities, which significantly enhances mathematical reasoning capabilities minimal cost. The experimental results on both in- in-domain and out-of-domain datasets demonstrate that even without GPT-4 or human-annen annotated process supervision, AlphaMath remains competitive with or surpasses the performance the state-of-the methods.
Acknowledgments
This work was supported by Alibaba Research Intern Program.
References
[1] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, Bailey, Z. Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
[2] Anthropic. Model card and evaluations for claude models. July 2023.
[3] Z. Azerbayev, H. Schoelkopf, K. Paster, M. D. Santos, S. McAleer, A. Q. Jiang, J. Deng, S. Biderman, and S. Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310. 10631, 2023.
[4] C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, Samothrakis, and Colton. survey of monte carlo tree search methods. IEEE Transactions Computational Intelligence and Al in games, 4(1):1-43,201
10

---

[5] G. Chen, K. Tang, C. Yang, F. Ye, Y. Qiao. and Y. Qian. SEER: Facilitating structured reasoning and explanation via reinforcement learning. In L.-.. Ku, Martins, and V. Sriku- mar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 5901-5921, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10. (/665////002..allloooggggo
URL
wr2g/222...2 hwtg/024.acc11oeee acl-long.3322 321.
[6] W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.1258. 2022.
[7] K. Cobbe, Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, arXiv preprint arXiv:2110. 14168, 2021.
[8] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
[9] T. Dao. Flashattention-2: 2: Faster attention with better parallelism and work partitioning, arXiv preprint
2023.
[10] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171-4186, 2019.
[11] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama herd of models. arXiv preprint arXiv:2407.21788, 2024.
[12] X. Feng, Z. Wan, M. Wen, Y. Wen, W. Zhang, and J. Wang. Alphazero tree-search can guide large language model decoding and training. arXiv preprint urXiv:2309.17177 2023.
[13] L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program- aided language models. International Conference on Machine Learning, pages 10764-10799. PMLR, 2023.
[14] Z. Gou, Z. Shao, Y. Gong, Y. Yang, M. Huang, N. Duan, W. Chen, et Tora: A tool-integrated reasoning agent for mathematical problem solving, arXiv preprint arXiv:2309. 2et.:23999.77..... 2023.
[15] D. Hendrycks, C. Burns, S. Kadavath, Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset, Advances in Neural Information Processing Systems, 2021.
[16] Y. Huang, X. Lin, Z. Liu, Q. Cao, H. Xin, H. Wang, Z. Li, L. Song, and X. Liang. Mustard: Mastering uniform synthesis of theorem and proof data. arXiv preprint arXiv:2402.08957 2024.
[17] Z. Ji, N. Lee, R. Frieske, Yu, Su, Y. Xu, E. Ishii, J. Bang, A. Madotto, and Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-33, 2023.
[18] W. Kwon, Z. Li, S. Zhuang. Y. Sheng, L. Zheng, C. H. Yu, E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles, 2023.
[19] A. Lewkowyez, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, Ramasesh, A. Slone, C. Anil, I. Schlag, T.. Gutman-Solo, et al. Solving quantitative reasoning problems with language models, Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
[20] C. Li, Z. Yuan, G. Dong, K. Lu, J. Wu, C. Tan, X. Wang, and C. Zhou. Query and re- sponse augmentation cannot help out-of-domain math reasoning generalization. arXiv preprint 200.52231. 2023.
11

---

[21] M. Liao, W. Luo, C. Li, J. Wu, and K. Fan. Mario: Math reasoning with code interpreter output-a a reproducible pipeline. arXiv preprint arXiv:2401.08/99 2024.
[22] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.2003 2023,
[23] J. Liu, A. Cohen, R. Pasunuru, Y. Choi, H. Hajishirzi, and A. Celikyilmaz. Don't throw away your value model! making ppo even better via value-guided monte-carlo tree search decoding. arXiv e-prints, pages arXiv-2309, 2023.
[24] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:171.001001. 2017.
[25] Z. Lu, A. Zhou, H. Ren, K. Wang. Shi, J. Pan, M. Zhan, and H. Li. Mathgenie: Generating synthetic data with question tonk-translation for enhancing mathematical reasoning of llms. arXiv preprint arXiv:2402. 16352, 2024.
[26] X. Mao, F.-L. Li, H. Xu, W. Zhang, and A. T. Luu. Don't forget your reward values: Language model alignment via value-based calibration. arXiv preprint wrx.v:2242.160335 2024.
[27] OpenAI. Gpt-4 technical report, 2023.
[28] I.. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,220 2022.
[29] S. Rajbhandari.. O. Ruwase, J. Rasley, Smith, and Y. He. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. In Proceedings of the international conference for high performance computing, networking, storage and analysis, pages 1-14, 2021.
[30] C. D. Rosin. Multi-armed bandits with episode context. Annals of Mathematics and Artificial Intelligence, 61(3):203 2000, 2011.
[31] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo. Deepseek- math: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033000 2024.
[32] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
[33] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. Nature, 550 (7676):354-355 2017.
[34] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 11805 2023.
[35] G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0 008953 2024.
[36] C. Tillmann and H. Ney. Word reordering and a dynamic programming beam search algorithm for statistical machine translation. Computational linguistics, 29(1):97-133, 2003.
[37] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[38] K. Wang, Ren, A. Zhou, Z. Lu, Luo, W. Shi, R. Zhang, I. Song, M. Zhan, and H. Li. Mathcoder: Seamless code integration in Ilms for enhanced mathematical reasoning, 2023.
12

---

[39] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. self-consisteney improves chain of thought reasoning language models. arXiv preprint arXiv:2203. 171, 2022.
[40] J. Wei, X. Wang. D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q Le, D. Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824- 24837, 2022.
[41] Y. Weng, M. Zhu, F. Xia, B. S. He, S. Liu, B. Sun, K. Liu, and Zhao. Large language models are better reasoners with self-verification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2550-2575, 2023.
[42] Y. Xie, K. Kawaguchi, Y. Zhao, X. Zhao, M.-Y. Kan, J. He, and Q. Xie. Decomposition enhances reasoning via self-evaluation guided decoding, 2023.
[43] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. The Eleventh International Conference on Learning Representations, 2022.
[44] F. Yu, A. Gao, and B. Wang. Outcome-supervised verifiers for planning in mathematical reasoning, arXiv preprint drXIv:2311.0972,,22 2023.
[45] L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
[46] X. Yue, X. Qu, G. Zhang, Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.0560 05653, 2023.
[47] B. Zhang, C. Li, and K. Fan. Mario eval: Evaluate your math 1lm with your math Ilm-a mathematical dataset evaluation toolkit. arXiv preprint UrXiv:2404.13922 2024.
[48] Y. Zheng, R. Zhang, J. Zhang, Y. Ye, Z. Luo, and Y. Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372 2024. URL http: /2b///42.... 13372.
/ //arxiv.org
[49] X. Zhu, J. Wang, L. Zhang, Y. Zhang. Y. Huang, R. Gan, J. Zhang, and Y. Yang. Solving math word problems via cooperative reasoning induced language models. In Proceedings of the 6lst Anmual Meeting of the Association for Computational Linguistics, pages 4471-4485, 2023.
13

---

Contents of Appendix
Dicussion
15
A.1 Limitation
15
A.2 Future Work
15
Supplementary Experiments and Analysis
15
B.1 More Results of Inference Strategies
15
B.2 More Analysis of Value Model
16
B.3 More Analysis of Problem Solving Rate of MCTS Each Round
16
B.4 Problem Solving Rate for Each L.M Training Set
17
C Implementation Details
18
C.1 Definitions of various elements in MCTS
18
C.2 Solution Filtering Algorithm
18
C,3 Parameter Details
20
C,4 Policy-Valie Value model Details
C.5 Datasets Details
23
C.6 Experiment Environments
23
Case Study
23
E Error Analysis
24
Prompts
27
F.1 Prompt Example of MCTS in Round
27
F.2 Prompt Example of MCTS after Round 1
28
14

---

A Dicussion
A.1 Limitation
Compared to previous works, our AlphaMath achieves comparable or even superior results without annotated high-quality, process-supervised data. However, unlike the game of Go, where the final board configuration directly reflects winning or losing, in mathematical reasoning, we rely on the actual answer the source of reward. This hinders us from "AlphaMath really from zero", unsupervised algorithm. However, compared to process-supervee answers is considerably more straightforward. For instance, existing question-ans datasets as well as questions from examinations typically encompass the answers, yet lack annotations for process supervision.
data, the acquisition of actual
A.2 Future Work
Directions for Future Work: Our research highlights several issues for further exploration:
Really from Zero: In this work, we have demonstrated that well pre-trained large language model can unleash its potential to identify correct mathematical reasoning processes through the AlpahMath framework, independent of GPT-4 or manually annotated process-supervised datasets. A challenging yet profoundly meaningful future direction is to identify an appropriate reward definition that allows AlphaMath to eliminate dependence on actual answers, thereby achieving really from zero. Notably, this process should avoid introducing additional annotation costs, such as training a reward model to replace the actual answers.
A closed-loop sell-evolution training framework:
With the question-answer pairs, our Al- phaMath framework can realize iterative self-evolution in complex reasoning scenarios, just like humans. In this study, as an initial attempt, we have maintained the same set of question-answer pairs (total only 15k pairs) in each iteration, which limits the potential of AlphaMath. In the future, we will explore how to automatically obtain such question -nswerr pairs from the Internet, which could facilitate the development of closed-loop self-evolution framework for AlphaMath. In this setup, the LLM automatically acquires question-answar pairs from the Internet and then autonomously enhances reasoning capabilities through our AlphaMath framework, thereby achieving complete independence from human intervention.
Explore beyond mathematical reasoning: Since mathematical reasoning tasks involve complex, symbolic multi-step reasoning, we primarily choose them as an example to investigate the effec- tiveness of AlphaMath. However, our proposed AlphaMath has the potential to be broadly applied to any task that can be evaluated against actual answers. In future work, we plan to expand its application broader range of tasks.
B Supplementary Experiments and Analysis
B.1 More Results of Inference Strategies
round
round 2
round 3
round 1
round 2
round 3
Accuracy
Accuracy(9%)
30
70
20
Greedy
SBS
SBS
SBS
MCTS
Greedy
SBS
SBS
SBS
MCTS
(B,
(B: 2)
3)
(8,
1)
(8)
(8) 3)
(a) GSM8K (In (n-Domain)
(b) OCWCourses (Out-of -ffoo--n--ain
Figure 7: Comparison of Different Inference Strategies (Other datasets).
15

---

In this experiment, we can draw similar conclusions as in Figure 4. With the progress of iteration, there a significant enhancement in the model's performance, especially between the first and second rounds, as shown in Figure 7. Furthermore, we observe that the performance of various inference strategies on the OCWCourses slightly differs from the other three datasets. This variation can be attributed to the fact that OCWCourses is mathematical dataset in the fields of physics and chemistry. Nonetheless, our method still significantly enhances the model's reasoning capabilities on such datasets overall.
B.2 More Analysis of Value Model
Incorrect final step
Incorrect final step
Incorrect inter. step
Incorrect inter. step
Probability Density
Probability Density
Correct final step
Correct final step
Correct inter. step
Correct inter.
-1.0
-0.5
0.5
1.0
-1.0
-0.5
0.5
value
value
Figure 8: (Left) Fitted distribution of value predictions of sampled solutions the training set. (Right) Fitted distribution of value predictions of sampled solutions on the test set. "Incorrect inter. step" denotes the state value V(s) of intermediate step within incorrect solution.
In addition to the discussion regarding the value model in Section 4.5, we have also specifically analyzed the overall distribution of the predicted state values V(s) by the value model for both the intermediate and final steps in correct/incorrect solutions, illustrated in Figure 8. "Final step" refers to the scoring of the entire solution the last step, representing the value model's overall assessment.
In the left panel of Figure 8, we plot the fitted distribution of the state values for both intermediate and final steps as predicted by the value model, during the process of solution generation on the training set. For correct and incorrect solutions, the value model's overall assessment is highly accurate, which is distinctly skewed towards and -1, respectively. Notably, "Incorrect inter. Step" represents the intermediate steps of an incorrect solution, rather than incorrect intermediate steps. Therefore. "Incorrect inter. Step" may also contain some correct processes, which explains why its distribution crosses over O. Overall, the distribution of the value model on the training set aligns very well with intuition, which aids in identifying higher-quality solutions MCTS.
In the right panel of Figure 8, we plot the distribution of state value V(s) predicted by the value model on the test set. It can be clearly correct and incorrect solutions, which explains why the performance of step-level beam search significantly surpasses that of greedy inference. The value model aids the policy model in navigating more efficient solutions, rather than relying solely on prior probabilities. Additionally, due to the fact that incorrect solutions may contain correct steps, their distribution is primarily concentrated near The intermediate steps of correct solutions exhibit bimodal distribution, with peaks concentrated near 0 and This can be attributed to the fact that even correct solution steps may contain some errors, such as coding mistakes. Therefore, in conjunction with the analysis in Section 4.5, we believe that our value model can effectively distinguish between correct and incorrect solutions, aiding the policy model in finding better solution paths.
that the value model accurately distinguished between
B.3 More Analysis of Problem Solving Rate of MCTS in Each Round
In this experiment, we evaluate the successful solving rate of MCTS across various rounds. Utilizing the MATH dataset, which categorizes each problem by difficulty level and subject type, we compute the problem-solving rate across different categories and difficulty levels. For our training set,
16

---

round 1
round
round 3
round
round 2
round
100
100
Solve Rate (%)
Solve Rate (%)
90
Agara
Goormity inlereditte Murber
Pradgetaa Procaiatt
Level
Level 2 Level 3 Level 4 Level
(a) Difficulty Level
(b) Subject Type
Figure 9: Problem Solving Rate on MATH Training Set
90
80
70
(96
70
Salve Rate (%)
60
60
50
50
40
40
20
10
Level4
Agate
&Podebilly
Agaar
Theory
Prodptt
(a) Difficulty Level
(b) Subject Type
Figure 10: Problem Solving Rate MATH Test Set
count the instances wherein problems are successfully solved along any the paths within the 10 constructed trees. As illustrated in Figure 9a, it becomes evident that MCTS achieves greater success in solving more challenging problems in subsequent rounds. Similarly, Figure indicates that, in later rounds, MCTS consistently demonstrates an improved capability to solve broader array of problems across different subjects.
For the test set depicted in Figure 10, include the results from round 0, which correspond to performance our "prompt 2-shot" in Table Unlike training set, we observe that improvement observed in round 3 is not consistent across different levels and subjects, even though the overall accuracy slightly increased. In fact, for easier problems, the performance in round actually declines. This is the reason we terminate our iterative training process after round 3.
B.4
Problem Solving Rate for Each LLM in Training Set
Table Solving Rate for Each LLM in Training Set. Since MARIO a SFT model and already possesses the capability to follow instructions, we opted to skip its first round.
Round 1
Round
Round 3
Model
GSM8K
MATH
GSM8K
MATH
GSMSK
MATH
MattsseekMtt-bas------ [31]
97.24%
83.93%
99.90%
93.73%
99.94%
95.61%
Hlama--base-Bl [11]
94.48%
78.42%
99.07%
89.77%
99.92%
94.50%
MARIO [21]
99.91%
94.51%
99.97%
94.79%
We further discuss the problem solving rates of different models in each round, as shown in Table 5. First, given the problems in the GSM8K dataset are relatively simple, the corresponding solution rates are higher, even in the first round. Despite the challenging nature of MATH, its problem-solving rate increases with more iterations, indicating continuous improvement in the model's performance. Secondly, there are also noticeable differences in the problem-solving rates between different models.
17

---

Since the SFT model (MARIO) is fine-tuned on high-quality data, it exhibits the best performance. Furthermore, the math domain sppeiicc pre-trained LLM DeepseeeMatt--bse-- outperforms general-purpose pre-trained LLM (Ilama3). This phenomenon is intuitive because domain-specific pre-trained L.MSs possess more specialized knowledge. Finally, we note that the third round, the solution rates of each model are quite similar. However, the final performance of the models differs significantly, as shown in Table 4. This discrepancy can be attributed to variations in the quality of solutions generated by different models. Generally speaking, the more relevant knowledge embedded within LLMs, the higher the quality of problem-solving solutions that will be generated. This explains why domain-specifer pretrained models significantly outperform general- purpose pretrained models. However, the advantage of our proposed AlphaMath lies in its ability to significantly enhance the performance of existing models without relying high-quality data annotated by GPT-4 or humans. Even with a weaker general-purpose pre-trained model, AlphaMath achieves remarkable +20 points improvement, shown in Table 4. Furthermore, AlphaMath enables a domain-specific pre-trained model to achieve comparable or even superior results compared to oate-offthe-art SFT models.
significantly
C Implementation Details
C.1
Definitions of various elements in MCTS
State The state s, is defined as a partial solution, consisting of the initial prompt (question) and all actions taken along the Monte Carlo tree search path from the root node to the current node, as shown in Eq.
Node Nodes are used to record information, such as the action (step) a, the value predicted by the value model Va, the state-action value Q from MCTS, depth, visiting counts, and etc. Each node defined to only contain single step.
Action (Steps) Following Liao al. [21], we define two types of actions (steps) a,: C-steps and A-steps. Each node contains only one type of action, and A-steps typically appear at the end, as shown in Figure C-step represents code execution, which is composed textual analysis, code snippets, and execution results. The textual analysis and code snippets are generated by the policy model (LLM), while the exccution results are the outputs returned by the Python code interpreter. A-step represents the summary of the answer, which is composed of text analysis and predicted answers. Both the text analysis and predicted answers are generated by the policy model. We organize these two steps in the following XML format:
C-step
<scep-lncp-ln(teettual amalypussllccpphhcoooelooooooo
auippptssun/dooeeuuppputccooooo output>Nur/p>An<//ttp>
A-step
<step>lnepslu((eextual amalyyaisVVccplllexxxxiinnn Answer: : (predicted answer)Jnc/pplue/sto
C.2
Solution Filtering Algorithm
After solution generation via MCTS, we randomly sample the correct and incorrect solutions of each question for training. During this process, we found that the generated solutions might suffer from issues such as hallucinations [17]. Hence, propose solution filtering algorithm to optimize the solution selection.
Algorithm 3 outlines the process of our solution filtering algorithm. We initially deduplicate and remove the solutions where code errors are present across steps (Lines 3-5). As indicated in Figure 11, the solutions that present code errors in all steps yet yield correct final answer
18

---

Algorithm 3 Solution Filtering
Require: Sampled solutions S.
Ensure: Candidate correct solutions Se: candidate incorrect solutions S..
1: S. = =1]. S.
> Initialization
2: for in do
3: if' not in S. or not in Se then if Code Errors persist across All Steps in then continue
De-duplication
4:
5:
> Eliminating solutions where errors permeate all steps.
6:
if s incorrect solution then Add to Se
7:
InCorrect Solution
8:
else
9:
flag <- False
for each output of code o in 8 do if is equiv(o, predict answer in s) then is is_equiv is the evaluation toolkit [47].
11:
12:
flag +- True Break
13:
14:
flag then Add to S.
15:
Level Correct Solution
16:
else
if code is correct at every step in then Add 8 to S.
18:
b Level Correct Solution
else
20:
L Add to S.
> Level Correct Solution
evidently hallucinations. To preserve the diversity of incorrect solutions, we refrain from excessive processing, which aids the value model in being exposed wide variety of incorrect solutions (Lines 6-7). Furthermore. to filter out higher-quality yet diverse correct solutions, we categorize them into different levels. We consider the solutions where the predicted answer is consistent with the code output as level 1 correct solutions because the model has generated code and understood the output correctly (Lines 9-15). Then, we observe that for some questions, the quality of solution cannot be solely gauged by the consistency between the code's 's output and the predicted answer (for which we provide an example in Figure 13). Therefore, we view the solutions with entirely correct code as level 2 correct solutions (Lines 17-18). Generating correct code to some extent proves that the model has a correct understanding of this question. Finally, we consider the remaining correct solutions as level 3 solutions (Line 20).
In general, Algorithm offers a method for swiftly filtering solutions. By categorizing solutions into different levels, we prioritize higher-level solutions for each problem. We provide several examples intuitively illustrate the necessity and rationality of Algorithm as shown Figure 11 13.
(a) An example of code errors persist across steps:
<question> What is the of all positive integers $'nu$ for which $\mathop(1em/ ['muu, 20]
=60$?
</question>
<step>
<p>
To find the sum of all positive integers $'nu$ for which d'mathoo(lcm) [mmu, =60$, we need to find the values of S'muš that satiefy the equation. We can start by simplifying the equation by dividing both sides by 20, which gives us $ (mm) ('nu, Anu,1]=36, -$$. Since $1mathop(1cm) [1,1]-1$, we can see that $'muš must be equal to 3. Therefore, the of all positive integers $'muš for which (mmathop(1cm) ['nu,, 20] -608 3.
</p>
<code>
python from sympy import lcm
19

---

Define the equation equation = 1cm (nu, 20) == 60 Solve the equation solution solve(equation, nu) Find the sum of the solutions sum_of_solution = sumouuuioo)) sooluttinn)
</code>
<output>
AttributeError: 'int' object has no attribute _s__cmmutatiie"
</output>
</step>
<step>
<p>
From the result, can that there is an error. This is because the "solve" function is not able solve the equation with the "lem" function. We need to find different way to solve the equation.
</p>
<code>
python
from sympy import factorint, lambdify
Define the equation
equation = lcm(nu, 20) == 60
# Factorize the numbers in the equation
factorized_equatioon squation.subs (1cm(((, 20), lcm(factorint (nu),
factorint (20)))
Simplify the equation
simplified_equatiine simpliiff(fatttrizee.eeuuuu
Evaluate the equation
evaluated_equationn eval (saliiimliiiiedeeeuuaa
Solve the equation
solution = # Find the of the solutions
equuttoon, nu)
ofslutions sum(soution)
(rimt soluutioos)
</code>
<output>
AttributeError: 'bool' object has attribute 'auba'
</output>
</step>
similar error repeated in subsequent steps...
<step>
<p>
From the result, we can see that there is an error. This is because the "subs" function is not able to substitute the "lem" function with the 'factorint" function. We need to find different way to factorize the numbers in the equation.
</p> <p>
Final Answer: $126$
</p>
</step>
Figure 11: An example of code errors persists across all steps. In this example, while the model was adjusting the code rectify coding errors, each step of the solution generated an error. Nevertheless, in the final step, the model still produced the correct answer. This is a classic hallucination. Hence, we discard such solutions Algorithm 3.
C33 Parameter Details
For the solution generation via MCTS, we set Cpuct = 1.25, set the temperature between 0.6 to 1.2, limit the maximum tree depth to 8, and allow each node to expand 5 child nodes (B2 5),
20

---

(b) An example of level Correct Solution
<question>
What is the value of $525-2 475-2$?
</question>
<step>
<p>
To find the value of $525 475-2$, can use the difference of squares
formula: $a b- (a b) (a b)$.
</p>
<code>
"python
525
475
Use the difference of equares formula value (a + b) (a b)
print
</code>
<output>
50000
</output>
</atep>
<step>
<p>
The result from the Python code gives us the value of $525-2 475-2$ $50000$
</p>
<p>
Final Answer: $50000$
</p>
</step>
An example of the level correct solution. In this example, the model generates the correct code and provides correct answer based on the output of this code. We regard such solutions as dependable and categorize this type of solution the level correct solutions.
Figure 12:
and a maximum of 40 simulations (N 40). Since MARIO [21] already has the ability to follow instructions, set K 2 for MARIO. For Math [31] and Llama3 [11], set K = 3. each round, we build 10 trees for each question-answe pair and randomly sample at most 4 correct and 4 incorrect solution processes for training. In this setting, the ratio of positive negative examples is approximately 1:1, and the count of positive examples varies between 57k to 59k for each round.
For supervised fine-tuning, we set the learning rate of 4e-5, batch size of 1024, the weight of the value loss 0.01 0.0005 (for Llama3 [1I]), and train the model for 10 epochs. We employ the AdamW W optimizer [24] and the cosine learning rate scheduler with the warmup rate set to 0.03. More hyperparameter details can be found in Table
For baselines, the results recorded in Table 2 come from corresponding published papers.
C.4 Policy-Value model Details
Unlike previous work [49, 44, 12, 5]. the value model is trained separately to assist the policy model. In this study, we integrate the value model into the policy model by appending linear layer, as illustrated in Figure 1. Since most of the existing LLMs adhere to decode-only architecture, we utilize the last token the representation of the entire reasoning step, similar to the role of "[CLS]" token in BERT [10]. In our case, it is typically "</step>", which ensures that the representation of the reasoning step will not be affected by the last token itself. The value model and the policy model share the majority of parameters, that is, they share the understanding of the reasoning steps. The value model assesses expected returns based on the current reasoning step, while the policy model generates the next token.
21

---

(c) An example of level 2 Correct Solution
<question>
How many $yS-intercepts does the graph of the parabola $x 2y 3y 7$ have?
</question>
<step>
<p>
To find the Sys-intercepts of a parabola, we need to set $x 08 and solve for Syg. In this case, we have $x 2y 3y + 7$, so we need to solve the quadratic equation $2y-2 3y 7 O$.
</p>
<code>
python
from sympy import solve, symbols
# define variables
y symbols('y')
solve the quadratic equation
solution = solve(2 * y**2 3 * + 7, y)
print
</code>
<output>
[3/4 sqrt (47))*1,,, 3/4 (qr(
</output>
</step>
<step>
<p>
The quadratic equation $2y 3y + 7 = 0$ has real solutions, which means
the parabola has no $y8-intercepts. We also this from the complex solutions for $y$ previous step.
</p>
<p>
Final Answer: $08
</p>
</step>
Figure 13: An example of the level 2 correct solution. In this example, the consistency between the code's output and the answer does not adequately measure the quality of the solution. Therefore, categorize solutions that are entirely correct in terms of the code as the level 2 correct solutions.
Table Key hyperparameters of AlphaMath
Hyperparameter
Value
Cpect
1.25
3 or 2 (for MARIO [21])
Weight of value loss B
0.1 or 0.0005 (for Llama3 [11]
B,
(1.3}
B2
5
Simulations
40
Temperature
(0.6, 1.0, 1.2)
max depth (max steps) T
8
Batch size
1024
Optimizer type
Adam [24]
Learning rate
4e-5
Ir scheduler type
cosine
Warmup ratio
0.03
Epochs
10
Weight decay
0.
22

---

C.5 Datasets Details
Table 7: Datasets Statistics
Dataset
OOD?
Training
# Test
GSM8K [7]
In-Domain
7473
1319
MATH [15]
In-Domain
7500
5000
GaoKao2023 [21]
OOD
385
OCWCourses [19]
OOD
272
Table 7 describes the statistics of datasets in detail. The division of the training and test set follows the previous work 17, 15]. GSM8K [7] is multi-step mathematical reasoning dataset compris- ing high-quality, diverse grade school math word problems, created by human problem writers. MATH [15] is dataset of challenging competitive mathematics problems. GaoKao2023 [21] is collection of mathematics problems from the 2023 Chinese National College Entrance Examination, the 2023 American Mathematics Competitions, and the 2023 American College Testing, while OCWCourses [19] comprises a collection of 272 STEM problems aimed at the undergraduate level, requiring multi-step reasoning for most questions.
C.6 Experiment Environments
All experiments were conducted Ubuntu 22.04 equipped with 8 * NVIDIA A100 GPUs. Our code mainly depends on Python 3. 1 and PyTorch 2. 2.1.2 We use our customized Llama Factory [48] as the training framework and customized vIJ.M [18] as the inference framework We trained all models with DeepSpeed ZeRO Stage2 [29] and Flash-Attention [9]. The pre-trained language models are derived from HuggingFace'.
D Case Study
Solution Generation in Round 1 Figure 14 illustrates an example of solution generation on the MATH dataset via MCTS in round 1. We guide the pretrained model, such as DeepseekMath- Base [31], to generate solutions in the form of ThThhhhh/Actiio/Acctoo only illustrate subset of the nodes of the original Monte Carlo tree in Figure 14. As shown in Figure 14, the path (a) + (c) + (f) represents a correct solution, whereas the other solutions contain enrors to some degree. Node (b) attempts to solve the problem in a single step and proposes a correct thought. However, minor errors in the coding process 7 #42 3843" was mistakenly written "k 7 3**3") led to mistakes in all subsequent steps. Node (d) attempts a different approach from node (c), specifically trying to solve for a first, then proceeding solve for a². Although this process is nonetheless a correct approach. However, in subsequent steps, we encountered errors of various forms. Firstly, within the node (g), the model mistakenly treats the output for (d) as equivalent a² leading directly to an output. At node (h), the model opts to calculate a relying on its capabilities: however, this results numerical error. From holistic perspective, we observe that, aided by MCTS, the pretrained model attempts to solve the problem through various approaches. During this process, we naturally excavate the knowledge embedded within the model, thereby reinforeing the model's understanding and application of this knowledge in subsequent training iterations. Furthermore, we collect the Q-values along the path aid the model in enhancing its judgment of the correctness of the reasoning process.
Inpu/Obserrvtton, as shown
in Sec. F.1. For clarity of presentation,
redundant compared to that of node (c), it is
Solution Generation in Round 3 Figure 15 illustrates an example of solution generation via MCTS in round 3. Compared to round 1, the quality of the model in round 3 has significantly improved, resulting in more correct solutions. We observe that the code quality in round 3 is superior, containing more comprehensive annotations. This signifies enhancement in the model's understanding of the question and its coding proficiency. Furthermore, as illustrated by node (b), the model attempts to resolve the question in a single step, yielding solutions that are both shorter and more efficient.
htttpp://ww.ppython.. org/
org/ 'We have released our customized framework in our Github Repository. https://hugginggace. co/
23

---

fores
Suppose that 5an2s varles inversely with 90"3s. If sa-75 when 5h-3s, find the value of $a-25 when Sb-bs, //qusstion
<step>
(R)
ogp
upp
given sa"2s varies inversely
$a"25 when tt-65,
st-ss. neans
that $a"2s reieaal st-as, We find constant
that sa"2 (tfracliHb"315
$a475 90-25,
the lonal ity. Then, we the constant of proportionality Lity to find the value of sa"2s when sbats,
this informat fon find the value skŞ.
ecedes
</pa
pyythn a
sicods
python
3ee3
printiki
</codes
</codes so
coutout>
1323
- 0.94
0.875
081
</output>
0.30
s/outpuc>
== -1.67
s/stepo
dsstep
(6)
(d)
(e)
found that 13239,
Fron the reselt, have şk 13234. Therefore, the equa tion for $a725
From the result, can that the va vul of $a"2s when Sbab5 is 58, $8.8755.
Find the value of $a"25 when sbrbs.
sen2 Vitrac (trr1) (bnd3s, Me can nov use this equation to fand the $a'9s $bnB5.
</pe
ecodes
<gp> Final 58,8755
python
088
1323
0-
11323 (0xx33) printia)
printlaz)
</eade> soutoutH
</code>
6.125
P- D91
soutpur
2.4748
F- oas
</output>>
Q 080
</output>
0 -- -0.5t
e/stepo
(1)
(g)
(h)
Fron the result,,
the value of 5a"25 is approximatell 1255.
that when sbabs,
resuls, sa"2 2.47485. Therefore, when shabs, the value $2"29 approxinately equal to 2.47.
Therefore,
approxinately $2.4748"2 6. 122936255.
sa"2s when Shibs
2,47485.
</pp-
Finat Answers $6.1255
</Step>
</pp
<ps
p- 005
Q- 1.3
Final Answer: $2.475
</p>
p- butti Q--10
</p>
ago
Final Answer: 38.122936255
Q -100
-- 807
Figure 14: Example of Solution Generation via MCTS in Round 1. The green and red areas represent correct and incorrect nodes, respectively. The red text segment indicates specific error in the solution. round 1, due the value head being randomly initialized, the estimated values are not accurate; therefore, we have not presented the estimated value here. For the sake of clarity in our demonstration, we only display a subset the original Monte Carlo tree and present each node in XML format (as detailed in Appendix F.2), even though the format utilized in round I was Thought/ Action//cooonnnnii Input/Observation (as detailed in Appendix F.I).
E Error Analysis
In this section. we conducted a comprehensive error analysis on the MATH dataset to foster future work. We randomly select 100 error solutions for analysis. We find the following three types of error..
(1) Numerical Errors (53%). This is of the main sources of errors. Within this category of eиrors, the model usually possesses the correct thought to solve the problem, but an eтror occurs during the numerical computation. The most common types of numerical computation errors arise from incorrect code generated by the model, which we will not discuss in detail. We have identified errors caused by numerical precision issues in Python, which we refer to as "numerical precision errors". Here is an example:
Question: The first two terms of a sequence are $a_1 1$ and $a_2 \frac (racc(1\asrrt3)S. For $n'ge 1$, \begin(align+) a_fn + 2} = \frac {a_n a_fn 1}11 a_ in 1}}. \end(aligntWWhat $2_0209)]
24

---

coupections
Supocso sar25 inversely Sii2s, Find the value of su'25 wher Stnds. </questiore
To solve this prooler, we need to estab 11st cormuttion aks equaction vardationn Since sor2s
Let sks be the constont of variation, Since 59'2s varies Lnversely with $3135, have the equation so"2 Woraadese wiich irplies S6 = 702 Avtlees 2r3s, Therefore, the coustion for $4"2s terms 925 sand Wiree Vitimes 3n3):00)3s.
imverssly by with sorts, we have lart
Witreckk (b-3))) solve for
gaven values Sants
scodes
pytton
an2 varies inversely with bnn tolue an2 when
beflnc the variables
wher
My sp.syetala an1
cssabllish
Deflne
variation
Toug w 3ex3
7462 3Sa33
the equesion for and it terms
c/ccades 1323
Find the valuer of and shen bnd
J- 031
spppeee __ x.xgaredd.uussib, 6)
v/outaute
3.67
4R I
cahen 3.1
(6)
(9)
<stea>
Topo
oge varcattoo
Fron the result,, no have sk 13236.
Now ve can use this value to find the
Now ve can use this volue to find
$4"25 when Sbmbs.
(o)
ecedes
find the value
pythhn
bud
fand the value of an2 when bu6 Luaaurred 1323 Band
From value (1)
50-65,
Laquared 1323 deed
printio squareel
e/eaden
stoedee 0.125
Final Annaees (1)
P 6.6
soutautt>
2.0
P- 0,99
Q-20
putu
srnutpato
centepo
ug- value Sw'25 Sben. Steds
eptepp
eptepp
1851.
result that when sbrts, the value of S419s is $5.1755.
</3>
$a"25 when Sbrbg 5,225.
Fanal Arsvvrr: $6.1255
P- 035
0 1.0
Faral Answor: 26, 1293
9-
</p>
Finat Answers 16,
0 100
007
Figure 15: Example of Solution Generation via MCTS in Round 3.
The actual answer:
The thought of LLM: To find $a_ (2009)$$ need to calculate the first 2009 terms of the sequence. However, the sequence is defined recursively, so we need find way to calculate the next term based on the previous two.
The code written by LLM:
import math a1 1
a2 1 / math. (3)
for i range 2009):
an = a2 an_ 1
a2 = (an + an_1) / (1 * an_1)
a1 an print (a2)
The output of above code: -2. 72.767039150000735
In this example, the model possesses the correct thought of problem-solving and generates the correct Python code. But an incorrect answer was still produced. To ensure numerical precision, we found that the correct implementation should simplify the equation using "sympy", shown below:
import sympy as sp a1
a2 / sp. wqrt (3)
25

---

for 1 in range (2, 2009):
an an_1 a1 a2 sp. sp.simmlify((an an_1) (11aa=ann_1)
a1 = an print (a2)
The output of above code:
(2) Logical Errors (45%). This is another major source of errors. In this type of error, the model usually ignores some details in the question, resulting in incorrect problem-solving approaches. Indeed, humans can also make similar mistakes due to carelessness. Here is an example:
Question: If $x$ and $y are numbers such that $(x (y-5) 0$, what is the smallest possible value of $x"2+ yr2$?
The thought of LLM: To calculate the value of 2+ yr2$, we need to solve the equation $(x 3) (y-5) 0$. After finding the solutions for $x$ and $y$, we need to calculate 2+ y by substituting these values into the expression $x y-2$.
The code written by LLK: from sympy import symbols, solve
x, y symbols y')
eq1 = Ex + 3, 0)
eq2 Eq 5, 0)
solution1 solve(eq1, x)
solution2 solve(eq2, y)
x_value solution1 [0]
y- value = solution2[0] [0]
result x valuu== + y_
print (result)
In this example, the model overlooked the requirement the question to find the smallest possible value, resulting in the incorrect code and final answer.
(3) Other Errors (2%). Aside from the two common types of errors mentioned above, we have also discovered other errors, which may stem from the ambiguity in question statements or inaccuracies in the actual answer. Here is an example:
Question: For which values of $k$ does the quadratic $ 3kx + 4k 7
have real roots?
The actual answer: (-4,0)
The thought of LLM: To find the values of $k$ for which the quadratic 0$ has real roots, need to consider the discriminant of the equation, which is $b^2 4ac$ If the discriminant is positive, the equation has distinct real roots, and if it is zero, the equation has one real root.
equation $kx-2 3kx + 4k
The code written by LLM: from sympy import symbols, Eq, solve k = symbols('k))
discriminant 1+3+k)++2 4+k* (4*k + 7)
print (discriminant)
The output of above code: 9**++2 4+k* (4*k + 7)
The thought of LLM: The discriminant is 29 4k 7) To find the values of $k$ which the discriminant non-negative we need to solve the inequality $9k-2 4k(4x 7) \geq O$.
26

---

The code written by LLM: from sympy import symbols, Eq, solve symbols('k')
inequality = 9*k**2 4*k* (4+k + 7) >= 0
solutions solve solve (inequality, k)
print (solutioon))
The output of above code: (-4 <= k) & (k <= 0)
The predicted answer by LLM: $[-4, 0]$
In this example, the model correctly solved the problem. However, due to the inaccuracy of the actual answer, the evaluation tool mistakenly deemed the model's predicted answer as incorrect.
Prompts
We provide instructions and examples for generating solutions and performing reasoning. For the solution generation through MCTS in round 1, the pre-trained models such as DeepseekMath- base [31] may not adhere instructions. Therefore, we employ few-shot learning to guide the model towards producing outputs in the format of Though//Actio//A Acttinnn///ninnn Input/Observation. Subsequently, we train the model in our SFT XML format and proceed with both the solution generation and reasoning in this format. We provide specific details regarding two formats of prompts as follows:
F.1 Prompt Example of MCTS in Round 1
We employ few-shot learning to steer pre-trained models toward performing mathematical reasoning, ensuring the output conforms the format of Thouuhh/Action/AAciie following example shows, the text in black is prompt, and the text in red is model generation.
Input/Observation. As the
You are a powerful agent with broad math knowledge and great Python programming skills, but not good at accurate calculation Answer the math questions as best you can. You have access to the following tool:
math equations.
A Python shell. Use this to execute python commands.
!! Reeember::
1. When solving math problem, you should think step by step, where each step includes mini-steps
2. If some step requires accurate calculation (including but not limited to numerical calculation and symbolic deduction), you should write Python code and execute for accurate result.
3. The most related Python packages include 'math', "sympy", "scipy" and "numpy".
4. Please use the following template.
Question: the input question
Thought: the text analysis, and list math equations necessary
Action: the action to take, should be "Python jyython_inteppreter". or None
Action Input: the Python Code in markdown format (if Action None, is None), e.g.,
python
import math
theta math.pi some_var math, cos (theta)
27

---

prine(ssmevvar
Observation: the result the action
(this Thought/Action/Actiinn Input/Observation can repeat times)
Thought: the final analysis of the answer
Final Answer: the concise answer without verbose context
Here are some examples:
<examples here>
Now! It's your turn.
Question: Ben works 8-hour shifts in a furniture shop. It takes him 5 hours to build rocking chair. How many chairs can build in 10 days?
Thought: calculate the number of chairs to first determine how many hours he would work in 10 days and then use the given information calculate number of chairs he can build in those hours.
can build
days, we need
Action: bython_interpree
Action Input: ... "'python
hours_per_day day = 8
total_days 10
hours_in_10_days = hours_per_day * total_days
chairs_per_hove hour 1 /
thairs_in_i0_d6 10 hours_in, ii__iinn11________ days chairs_pers pour in
Observation: 16.0
Thought: have calculated the number of chairs Ben can build in 10 days. The number of chairs 16.
Final Answer: 16
F.2
Prompt Example MCTS after Round
To train the SFT model in executing mathematical reasoning, utilize an XML format alongside zero-shot learning. This approach adopted because the math-related pre-training corpora are predominantly harvested from the Internet, where HTML tags serve distinguish various types of content, including text, equations, and code snippets. As the following example shows, the text in black is prompt, and the text in red is model generation.
inches every year. If she is currently 20 inches tall, what will be her height after 10 years7</question
Hneesti grows at the rate
<step>
<p>
To calculate Haley's height after years, need to add 10 times the growth rate of 3 inches to her current height.
</p>
<code>
pyytonn
current_heii _eight 20
28

---

growth_rate rate 3
years
future future_height = current_height + (growth_ratas rate * years)
print (aattfffueeiightte
</code>
<output>
50
</output>
</step>
<step>
<p>
I have calculated Haley's height after 10 years. Haley will be 50 inches tall
after 10 years.
</p> <p>
Final Answer: $50$
</p>
</step>
29

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect the paper's 's contributions and scope. The paper clearly states its objectives, methodology, and results in the abstract and introduction sections.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Due space constraints, discuss the limitations of this work and future work in detail in Appendix A.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
30

---

Answer: [Yes]
Justification:
We provide detailed theoretical assumptions in Section 2.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of our algorithms,, experimental setup, and information about hyperparameters, data preprocessing steps, and on, ensure that other researchers can reproduce our main experimental results. You can find these details in Section 4.1 "Experimental Setup" and Appendix C "Implementation Details".
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
31

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: All code and data have been submitted in the supplementary materials with sufficient documentation. We have ensured that all experimental results can be faithfully reproduced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: //iip..ccoo.m httpps.ccppbbbiccc/c wwwp///gg///////eee/eeee/oe/ /CodeSubmii ssi onsooicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We thoroughly specify all the training and test details our paper, including how we divided the data into training and testing sets, the settings of hyperparameters, and the types of optimizers deployed the model. These details are crucial for understanding and replicating results, and please refer to Section 4.1 "Experimental Setup" and Appendix C "Implementation Details" for this information.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have provided detailed parameter settings and conducted error analysis, reporting the average experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
32

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide a detailed description of our experimental environment in Ap- pendix C.6. We also discuss in detail the running times of various algorithms our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't it make it into the paper).
paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics https: //te::///eeuee http//nerrip...e wwtpp//wc/ppppc..ccccococcccccce httpbl/cc//cccccccieeeeeeeeeeeee
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification: We have ensured that the research conducted in our paper complies with the NeurlPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
33

---

Justification: In Appendix A, we discuss in detail the potential positive and negative social impacts of work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The work we have done in our paper does not involve such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual- use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have strictly adhered the licensing and usage terms of the data and models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
34

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: For all new assets introduced the paper, such as data, code, and models, we provide detailed documentation in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
35

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "bpWxZAFlP0", "reviewer": "Reviewer_2HCk", "summary": "This paper utilizes Monte Carlo Tree Search (MCTS) to sample high-quality process-supervised data for iterative training, effectively reducing the dependency on GPT-4 and thus lowering the associated costs. Additionally, the authors propose an inference strategy, step-level beam search, which leverages value models to enhance the efficiency and effectiveness of the tree search process.", "questions": "1. As mentioned in the weaknesses, what are the results for best-of-n or weighted voting? Additionally, how does the accuracy change as n increases during majority voting?\n2. How will the performance change when separating the policy model and value model as two totally different models?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written, with a clear introduction to MCTS and the authors' proposed method, step-level beam search (SBS).\n2. Their results effectively demonstrate the superiority of SBS over MCTS.\n3. The model they trained shows a significant improvement over the baseline model, with an increase in performance from 33.2 to 53.6.", "weaknesses": "1. The comparison in the experimental results appears to be rather unfair. Generally, more computation leads to higher performance. In Table 2, the authors compare their fine-tuned model using SBS with greedy decoding for other open-sourced models. In Table 3, they compare SBS with majority voting. It is possible that the improvement stems from the reward model rather than the SBS algorithm itself. The authors should also report the results of best-of-n or weighted majority voting using their value model to demonstrate the effectiveness of SBS.\n2. The experimental results are not detailed enough in Tables 2 and 3. The authors should also include major@n with different n values. Perhaps they can plot figures with $B_1$ and $n$ as the x-axis to provide a more comprehensive view.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 2HCk\n\nWe sincerely thank you for your thorough evaluation and for acknowledging the clarity of our paper, as well as the effectiveness and significant performance improvements we've demonstrated. Due to word limit, we apologize for omitting your question and replacing them with numbers and key words. Below, we address each of your comments in detail.\n\n> W1: Unfair comparison. Relationship between reward model and SBS, try best-of-n.\n\nWe appreciate the reviewers' feedback and the opportunity to clarify experimental comparisons.\n\n1. In `Line 223-224`, we mentioned the greedy decoding results are included in Table 2. To clarify, it is marked as **AlphaMath (K=3)**, where $K$ means the training iterations. Actually, due to our novel experimental setups, the experimental comparison puts our results at a disadvantage relative to previous works. \n\n  - Our training method is categorized as **almost unsupervised / weakly-supervised** learning, whereas previous works primarily employ **supervised learning**. If we were to exclude the human or GPT-4 annotated solution processes from the training data in previous works, and train their models using only question-answer pairs, we hypothesize that their performance would drop drastically.\n  - The dataset for supervised training in previous SOTA DeepSeek-Math-Instuct includes **776K question-process-answer** tuples. In contrast, our method only relies on the **15K question-answer** pairs.\n  - As shown in Table 2, in greedy decoding setup, AlphaMath achieves a performance score of 53.6, which is comparable or better than all previous supervised methods except DeepSeek-Math-Instuct's 57.2. The relatively small gap between an **almost unsupervised** method and a **supervised** method is noteworthy and underscores the potential of our approach.\n\n2. As illustrated by the SBS (step-level beam search) inference in Algorithm 2, our results are already presented as **best-of-n**, but on a step-level basis. The value loss (the 2nd and 3rd rightside terms in Eq. (6)) indicates that the value model is trained at the step-level. During inference, for example, in a setup of $B_1/B_2=1/5$ the SBS needs to choose the best out of 5 generated outputs at each step, using the step-level score prediction provided by the value model. In summary, the value model and SBS inference algorithm are highly integrated to improve the performance, by selecting step-level **best-of-n**.\n\n> W2: More n for maj@n.\n\nThanks for the suggestions to make our experiments in a more comprehensive view.\n\nFirst, We want to clarify that majority voting is not the focus of our work. Our training algorithm is fundamentally different from previous methods, which naturally leads to a distinct SBS inference algorithm compared to the majority voting used in earlier approaches. We propose an autonomous step-level process annotation method by leveraging the cooperation between policy and value models within the framework of MCTS. This policy-value reinforcement learning framework naturally favors the SBS inference algorithm.\n\nSecond, Table 2 is primarily used to compare our **almost unsupervised / weakly-supervised** results with previous methods in traditional **supervised learning**. Table 3 presents an analysis experiment of our own model to explore the trade-off between performance and efficiency using different inference algorithms. \n\nThird, as suggested, we would like to include the performance of maj@n with varying n values. From the table, our SBS is significantly better than major@n. More ressults can refer to `Table 6 in PDF file`.\n\n|n|Maj@n|Avg time s/q|SBS ($B_1=n$)|Avg time s/q|\n|--|:--:|:--:|:--:|:--:|\n|1|||62.32||\n|2|||64.66||\n|3|56.82||65.74|2.3|\n|5|61.84|2.9|65.98|4.7|\n|10|62.58||||\n|15|63.56||||\n|20|64.16|15.9|||\n\nWe hope the above explanations address your concerns and help you better understand our work.\n> Q1\n\nWe have addressed the relevant issue in the response for Weakness 1 and 2. \n\n> Q2: Performance of separating policy and value models.\n\nIn the realm of reinforcement learning (RL), whether or how to share networks between policy and value models is a long-debated design issue, [r1,r2] have noted that using two separate networks is usually easier to implement and more stable to train. However, this design is inefficient optimization as it doesn't allow for the sharing of internal representations. On the other hand, shared networks are challenging to train because the different losses generate gradients in various directions and scales, requiring careful tuning of the loss weights.\n\nMaking the training of shared networks stable is actually one of the contributions in our work as a RL method. Beyond this consideration, our insistence on tuning the shared architecture is driven by computational cost. Using two LLMs during inference nearly doubles the computational expense. Sharing networks had also been adopted in some famous RL works, such as AlphaZero.\n\nFor the response purpose, we'd like to include the new experimental results with two separate models. We set the temperature to 0.6, $B_1/B_2=1/5$, and compare our AlphaMath (joint training) with two separate models, as shown in the following table:\n\n|$B_1=1$|GSM8K|MATH|GK2023|OCW|avg. time (s/q) & GPU usage|\n|--|:--:|:--:|:--:|:--:|:--:|\n|two separate models|82.4|61.4|46.4|29.9|4.3 / $2\\times$A100-80G|\n|AlphaMath (Ours)|81.1|62.8|46.2|30.5|3.1 / $1\\times$A100-80G|\n\nFrom the table, our AlphaMath has similar overall performance to the two separate models, but two separate models requires more inference time and computing resources, which is what we want to avoid.\n\n[r1] Sutton & Barto's Reinforcement Learning: An Introduction\n\n[r2] The UC Berkeley course: Deep Reinforcement Learning (CS285)\n\nWe hope the above clarifications will help you better understand our work. We are happy to communicate with you further if you have any questions to enhance our manuscript.\n\nSincerely,\n\nAuthors of 6802", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dXst9KN7Z4", "reviewer": "Reviewer_EVyZ", "summary": "The paper introduces a novel approach leveraging the Monte Carlo Tree Search (MCTS) framework to generate process supervision and step-level evaluation signals automatically, thus enhancing the mathematical reasoning capabilities of LLMs. This method bypasses the need for costly and labor-intensive manual annotations by allowing the models to train iteratively on both policy and value aspects, with an efficient inference strategy called step-level beam search. This approach demonstrates significant improvements on various datasets, showing comparable or even superior results to existing state-of-the-art methods without relying on external process annotations.", "questions": "No question besides the one raised in weakness.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper leverages a Monte Carlo Tree Search (MCTS) framework to automatically generate both the process supervision and step-level evaluation signals necessary for training LLMs. This approach avoids reliance on human annotated data by generating necessary training signals through interactions within the MCTS environment. As a result, the model achieves comparable or even superior performance to previous state-of-the-art methods on both in-domain and out-of-domain datasets.", "weaknesses": "1. Duplicate citations, such as [6], [7]\n2. Appendix Figure 9, the reason why the performance of the third round model in MATH at Level 1 and Level 3, Counting & Probability, Number Theory and other categories has declined. Whether it will continue to decline after more rounds. \n3. The method integrates CoT and Code for cross-use. How does the performance of the model change if only CoT is used?\n4. Evaluate on more out of domain benchmarks, such as GSM8k_hard and GSM8k_robust and OpenLLM Leaderboard\n5. How much total training data is used. Explore the effect on other base model, such as Llama2, Mistral\n6. Explore the effect of different values of the hyperparameter β in Eq. (8). \n7. what is the change in loss of Policy model and Value model during training.\n8. What's the accurate of the Value model predictions based on the current State and Action.\n9. Insufficient introduction of related work, such as Mathematical Reasoning and MCTS.\n10. Differences reasoning ability and solution generation process across different rounds for the policy model , as well as the scoring differences by the Reward Model.\n11. How the ground truth label scores for training the first round of the Value Model are obtained, and the procedures for the second and third rounds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer EVyZ,\n\nWe greatly appreciate your detailed review and the recognition of the strengths of our paper. Due to word limit, we apologize for shortening your questions. We will address each of your comments in detail, hoping to alleviate any concerns you may have.\n\n> W1: Duplicate citations\n\nWe apologize for the error and will correct all reference errors in the revised version.\n\n> W2: Explain acc of some levels in round 3 declined\n\nThanks for highlighting this issue. In `Line 194-195`, we have discussed the selection of $K$ using the **Elbow method** on overall accuracy. The details including next round (round 4) can refer to **Table 1 in PDF file**. As we discussed in `Line 564-568`, the mixed increase and decrease performance on finer-grained levels is normal after training (overall accuracy) converges.\n\n> W3: What about CoT\n\nDue to limited time, we cannot do full experiments on CoT. For a fair comparison, we follow most SOTA methods in the setup of CoT+Code, because it has been proven (eg, [r1] or ToRA) that the performance of COT is less effective to CoT+Code, and LLMs struggle with numerical calculation. A recent famous example is that LLMs fail to compare 9.9 and 9.11.\n\n> W4: Other OOD data\n\nThanks for the suggestion. We did not use the GSM8k_hard dataset because it does not represent a truly OOD scenario. It uses the same questions of GSM8k by varying digits, and many factual errors exists, such as the negative number of people.\n\nAs introduced in Appendix C.5, the two OOD datasets we used are significantly more challenging than GSM8K and MATH, with different question sources and domains (e.g., physics). So they can better represent the true OOD scenario.\n\nFor the response purpose, we evaluate on GSM8K-robust and its corresponding method, as shown in **Table 2 in PDF file**.\n\n> W5: Training data size; Try Llama2 or Mistral\n\nIn `Line 186-187` , we noted that our raw training data only contain **15K QA pairs** from GSM8K/MATH. Using these 15k pairs, we generate **59k solution intermediate processes** by MCTS, significantly smaller than existing SOTA (**776K** in DeepseekMath-instruct). In addition, process annotation does not rely on GPT-4 or human.\n\nIn `Table 4` of Section 4.6 and `Table 5` in Appendix, we already explored the effect of other base LLMs, including general LLM **Llama3**, and SFT models (MARIO). We think Llama3 can represent the general LLMs, such as Llama2 and mistral.\n\n> W6: Hyperparameter beta in Eq(8)\n\nWe assume it refers to Eq(6). We conducted hyperparameter selection in our preliminary experiments. In this response, we would like to share our findings that $\\beta$ does not significantly impact model performance.  We list the results in **Table 3 in PDF file** for your reference.\n\n> W7: Log of training loss\n\nWe plot the loss in **Figure 1 in PDF file** for your reference.\n\n> W8: Accuracy of value model\n\nThanks for your valuable comments. It is theorectially difficult to evaluate accuracy of intermediate step values without human annotations. Even in the realm of reinforcement learning(RL), it is also a difficult task if no step reward is returned. Thus, most RL  algorithms care more about the values ranking (e.g., via argmax), given current state or action.\n\nIn `Figure 5` of Section 4.5, we have followed the convention of RL to illustrate the relationship between the value predictions of intermediate and final steps by plotting the distribution of $Q$-values against final reward/correctness. It offers a more intuitive understanding. On the testset, our value model predictions align well with expectations, particularly for correct steps.\n\nIn `Line 277-298`, we also discussed the minor flaws of the value model. As shown in Fig. 5, the value model will score $\\approx1$ for some final wrong solutions, because the final incorrectness doesn't mean all steps are wrong, and correct steps may exist for early timestamp.\n\n> W9: More related works\n\nDue to space limitation, we focused on data annotation and value model, as our title indicated. We totally agree that other fields, such as MCTS, are also important and we will revise accordingly.\n\n> W10: Performance of different rounds\n\nThanks for your insightful comments. Actually, in `Figure 3/4/6/8/9, Table 5, Appendix D`, we have performed extensive experiments to investigate the performance of different rounds, that covers your concern.\n- policy model in Figure 3/4/6/8/9, greedy decoding w.o. value\n- value model in Figure 4/6, SBS inference with value\n- case studies in Appendix D, intuitively show how Q-values changes\n- problem solving rate in Figure 3/9\n\n> W11: Labels of value model\n\nWe aim to clarify it by discussing the offline Q-learning method. Generally, the initial $Q$ values of each node in the MCTS tree are set to 0 and are subsequently updated as indicated in `Eq. (4) and $Q$ update in Line 132`.\n\n`Lines 143-147` explain how the values are calculated for the 1st round. Since the value model is randomly initialized, the value head's prediction (1st rightside term in Eq. (4)) should be $\\approx0$. Thus, $Q$ update is primarily influenced by the final reward/correctness. Theoretically, with more MCTS simulations/rollouts, the $Q$ values will gradually converge to their true underlying ground truth labels.\n\nFor other rounds, `Lines 120-132` describe the value calculation process. In 1st rightside term of Eq. (4), the value model is trained on the previously collected values, providing a more accurate prediction than $\\approx0$ in 1st round. Thus, the $Q$ update in Line 132 may rely not only on the final reward but also on the value model's prediction.\n\n[r1] Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification\n\nWe believe our explanations have clarified your concerns and offer a clearer understanding of our work. Thank you for your constructive feedback. We are eager to engage in further discussions to enhance and refine our work.\n\nSincerely,\n\nAuthors of 6802", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vCCW09YqBk", "reviewer": "Reviewer_otxC", "summary": "This paper leverages the Monte Carlo Tree Search (MCTS) to iteratively train policy and value models by automatically generating process supervision and step-level evaluation signals, eliminating the need for human-annotated process supervision data. Specifically, the method combines the inner capabilities of pre-trained LLMs with the MCTS framework to generate correct and incorrect solution paths and optimize the models based on the node evaluations along these paths. To improve inference efficiency, the paper also proposes a step-level beam search strategy that allows the value model to assist the policy model in navigating more effective reasoning paths rather than relying solely on prior probabilities while avoiding excessive time consumption. Experimental results demonstrate that even without GPT-4 or human-annotated process supervision, this paper' s method performs comparably or better than existing SOTA methods across multiple in-domain and out-domain datasets.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The work proposes a novel and efficient method that eliminates the need for human process annotations by leveraging the MCTS framework for model self-evolution.\n2. The step-level beam search strategy significantly enhances the model's reasoning capabilities while maintaining computational efficiency.", "weaknesses": "1. Although the method does not rely on human-annotated process data, it still requires actual answers as reward signals. Future work could explore fully unsupervised approaches. \n2. Although the step-level beam search improves efficiency, the MCTS method still has some limitations in terms of computational complexity, which makes its practicality indistinguishable from majority voting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer otxC,\n\nWe greatly appreciate your valuable comments and are pleased that you recognize the novelty and efficiency of our proposed method. Below, we address your comments in detail.\n\n> W1: Although the method does not rely on human-annotated process data, it still requires actual answers as reward signals. Future work could explore fully unsupervised approaches.\n\nThanks for the insightful suggestion. In `Line 499-505` of Section Future Work, we also mentioned the possibility of \"Really from Zero\", quoted as `A challenging yet profoundly meaningful future direction is to identify an appropriate reward definition that allows AlphaMath to eliminate dependence on actual answers, thereby achieving really from zero. `. \n\nA promising trial can be exploring in proof problems by leveraging the **Lean** programming language. Unlike traditional problems that require a final answer, proof problems focus on verifying both numerical computations and proof correctness. The Lean language is well-suited for this purpose, especially for proof derivation. For example, recent AlphaProof [r1] achieved 2024 IMO silver medal by applying our method-alike approach with Lean.\n\n[r1] DeepMind. AI achieves silver-medal standard solving International Mathematical Olympiad problems\n\n> W2: Although the step-level beam search improves efficiency, the MCTS method still has some limitations in terms of computational complexity, which makes its practicality indistinguishable from majority voting.\n\nThanks for pointing out this issue. We totally agree with your claim that compared with majoring voting, inference with MCTS requires larger computational complexity. This is also exactly the reason why we mainly use MCTS in training rather than inference. Value estimation with MCTS can provide more accurate learning signals than simple Monte Carlo estimate, so training with MCTS framework is most beneficial to value model.\n\nTo address the inference issue of MCTS, we further proposed to modify it and achieve a very efficient inference method--step-level beam search (SBS, note that SBS cannot be used for training). In `Line 168-172`, we  mentioned that `However, MCTS is computationally intensive for simulations, making it less viable for use in production environments. To address this, we modify the MCTS inference process by eliminating the backup operation, introducing a simplified method, which we refer to as Step-level Beam Search (SBS), detailed in Algorithm 2. This approach does not construct the entire tree; instead, it dynamically selects the best child node during node expansion.`\n\nIn `Tables 2 and 3` of our paper, we also demonstrate the significant advantages of SBS in efficiency and effectiveness compared to MCTS inference, greedy decoding or majority voting. For example, SBS ($B_1=3$) is 5 times faster than MCTS and has 1.7% better performance. Meanwhile, the efficiency of SBS is similar to that of greedy decoding, with only a difference of 0.8s per question, but the efficiency is improved by 12.1% on the MATH dataset. We believe that these aspects demonstrate the superiority of our proposed SBS.\n\nCompared to majoring voting, `Table 3` of our paper cannot fully show the efficiency difference between SBS and maj@n. However, if we scale $n$ to large numbers, from the following table, we can see SBS shows significant advantage on accuracy.\n\n|n|Maj@n|Avg time s/q|SBS ($B_1=n$)|Avg time s/q|\n|--|:--:|:--:|:--:|:--:|\n|1|||62.32||\n|2|||64.66||\n|3|56.82||65.74|2.3|\n|5|61.84|2.9|65.98|4.7|\n|10|62.58||||\n|15|63.56||||\n|20|64.16|15.9|||\n\nWe hope the above clarifications have enhanced your understanding of our work. We appreciate your valuable feedback and are eager to further discuss any additional questions or comments you may have to improve our manuscript.\n\nYour insights are invaluable in our endeavor to refine and strengthen our research. Thank you once again for your thoughtful review.\n\nSincerely,\n\nAuthors of 6802", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tZHdf9xSvI", "reviewer": "Reviewer_15WC", "summary": "The authors propose the AlphaMath method, which leverages Monte Carlo Tree Search (MCTS) to improve the mathematical reasoning capabilities of large language models (LLMs) without requiring expensive human or GPT-4 annotations for process supervision. The framework integrates a value model with the LLM to generate and evaluate reasoning steps autonomously. Experimental results demonstrate that AlphaMath achieves comparable or superior performance to baseline methods on both in-domain and out-of-domain datasets.\n\nTo AC and the authors: The detailed comments are mainly covered in the **Questions** section. Please refer to the section for details. Generally speaking, I would consider the paper as very close to the threshold. If the authors can properly address the concerns in the below section, this could be a valuable study in this area.", "questions": "1. The authors use Table 1 and corresponding statements in the main body (line 25) to claim that *the annotation of mathematical solutions primarily relies on domain experts or GPT-4 in current efforts*. This claim is inaccurate, as multiple studies employ methods such as the Monte Carlo method [1,2] or even Monte Carlo Tree Search (MCTS) [3,4,5] to automatically annotate mathematical solutions. It is totally understandable that some related studies might have been missed, especially those posted after NeurIPS submission [3,5]. However, there are studies [1,4] published last year that should have been properly considered. I recommend that the authors revisit their claims and appropriately cite and discuss works that do *annotate mathematical solutions without human intervention or GPT-4*. This paper already presents numerous promising contributions, which I am very impressed; therefore, it is advisable to carefully avoid over-claiming in minor aspects that may negatively affect the first impression of readers and researchers familiar with this field.\n2. In line 76, the authors write *the solution process can be broken into multiple reasoning steps (e.g., segmenting the solution based on distinct stages or simply on a period).* What exact step segmentation strategy do the authors use in this study?\n3. In line 118, the authors write, *we employ sampling generation with higher temperature to ensure diversity* in generating actions for a step. This method requires further elaboration. From my understanding, it is non-trivial for an LLM to generate only a single mathematical step with early stopping. If the authors generate an entire solution and truncate it to retain only the subsequent step, this approach would be computationally expensive.\n4. In line 125, the authors write *we follow a trade-off rollout strategy between AlphaGo and AlphaGo Zero. Because our tree depth is much shallower than Go games (e.g., a maximum depth of 8).* What is the \"trade-off rollout strategy between AlphaGo and AlphaGo Zero\"? I don't personally remember such a concept in the AlphaGo work series; and why your tree has a maximum depth of 8? There should have been many solutions with more steps for the MATH dataset.\n5. In line 135, *assuming that ...*. Why this assumption is valid? Please elaborate.\n6. In Eq. (6), the the loss terms for the value model is minimizing the difference between Q and V. However, in AlphaGo Zero it is minimizing the difference between V and the final reward z (1/-1 indicating the final winner at the termination state). The authors do not provide an explanation of this modification. Please elaborate.\n7. In line 164, *This selection is guided by the maximum Q-value*. In AlphaGo and AlphaGo Zero, the selection at inference time is guided by the visited count. Using Q is also a valid choice, but there should be clarification for this modification. An optional ablation study would be better.\n8. In Table 3, the SBS with $B_1=2 \\text{or} 3$ has shorter average time than $B_1=1$. This looks weird. I guess it comes from the less number of steps. But why $B_1=2 \\text{or} 3$ has less number of steps then? Also, the \"discussion of majority voting\" is really confusing. I still don't understand why they are not comparable after reading it.\n9. In Figure 5, where is the Q-value sampled from? For full solution only or also including intermediate of steps? Also why there are three curves in the right figure while there are only two legends?\n\nRef:\n\n1. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. https://arxiv.org/abs/2312.08935\n2. Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision. https://arxiv.org/abs/2402.02658\n3. Improve Mathematical Reasoning in Language Models by Automated Process Supervision. https://arxiv.org/abs/2406.06592\n4. Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training. https://arxiv.org/abs/2309.17179\n5. Step-level Value Preference Optimization for Mathematical Reasoning. https://arxiv.org/abs/2406.10858", "limitations": "The authors provide a limitations section (A.1); however, it omits several critical aspects:\n\n1. Methods that use the final answer's correctness to infer the reward or value of intermediate steps are prone to false positive and negative issues. Specifically, when the final answer is correct, all intermediate steps are rewarded to some extent. Nevertheless, false positive cases occur when models luckily guess the correct final answer despite incorrect reasoning. Previous related studies, such as Math-Shepherd and OmegaPRM, have acknowledged this limitation. It is recommended that this paper also addresses it in the limitations section.\n2. The method relies on an automatic verifier to determine the correctness of the final answer. This verifier is not always applicable to many tasks, such as open-ended text generation (e.g., translation and summarization). The authors did not clearly explain the limitations concerning the scope of tasks.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "MCTS is a powerful algorithm for complex planning. Mathematical reasoning is a very important but challenging area. Even the most advanced LLMs nowadays can easily make mistakes when solving a mathematical problem. There are many recent attempts using MCTS on mathematical tasks, and this paper is a good one among many. The empirical results demonstrate the effectiveness of the proposed method.", "weaknesses": "Many presentations in the paper are somewhat unclear, requiring further clarification and elaboration. There are also some minor over-claims in the introduction. There are many modifications comparing to the original AlphaGo and Zero paper but the authors typically do not provide explanations for such changes. Some tables and figures are confusing. Please see the **Questions** section for the details.\n\nOne thing I'd like to especially point out is, the title \"almost zero\" is kinda misleading. One would interpret it as using very little data, e.g., a few seed data. However, the method would still need the full training set of GSM8K/MATH.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 15WC,\n\nWe sincerely appreciate your efforts in evaluating our manuscript. We have carefully considered your insights and critiques, then address each of your comments.\n\n> W1: Distinctions to [1,4]\n\nThanks for raising this issue. We will avoid over-claiming in future version. Highlighted contributions and distinctions to [1,4] can refer to the `G1 in General Rebuttal`.\n\n> W2/W3: Step segmentation; How to generate/stop a step\n\nW2 and W3 are relevant, so we will address together.\n\nIn `Lines 190-192`, we already mentioned that the step is defined in the form of **LLM REACT** format and provided the examples in `Appendix F`. The REACT format step includes Thought(text analysis) / Action(python interpreter) / Action input(code snippet) / Observation(code output). In `Line 620-627`, detailed definitions and examples of the $C$-steps and $A$-steps are also provided. $C$-step is REACT step including code, and $A$-step is the last REACT step including final summary only.\n\nWe follow the REACT format to generate and stop a single step. In REACT, one can define the possible stop tokens for Observation. As shown in Appendix F, we can define the stop tokens, as \"Observation\" for round 0 and [\"</code>\", \"</step>\"] after round 0. When encountering any stop token, the LLM generation will stop. In **open-source toolkit vllm or Openai API**, stop token is an argument, thus it's not necessary to generate the entire solution and truncate it. In summary, the step-level sampling can still be efficient.\n\nMore details can refer to REACT paper or relevant API document. This is not our focus, but we will add a relevant description in future version.\n\n> W4: Explain trade-off rollout and max depth=8\n\nWe apologize for misuse of **follow** in line 125, which should be **design**. In AlphaGo, every rollout requires a light-weighted fast policy to sample hundreds steps in depth until terminal node. In AlphaGo Zero, there is no rollout at all, and the value estimation only depends on prediction of value model in previous round. Our defined value estimation $(1-\\lambda)V_{\\phi}(s_t) + \\lambda r(s_{t+1})$ has the same form as AlphaGo. Differently, we set $\\lambda$ as indicator function $I_{terminal}(s)$. `Lines 126-128` also explain it means such setting rollouts for only one-step. If the rollout reaches terminal, value adopts the only reward, otherwise, the value model prediction.\n\nAs we explained for REACT, depth=8 means 7 code steps + 1 summary step. Even for MATH, 7 code snippets are enough to solve most problems. E.g., in ToRA (ref [12] in paper) found GPT-4 can solve 83.1% MATH problems within 3 code snippets. Since our base model is weaker than GPT-4, we relax it to 7.  **Table 4 in PDF file** shows the detailed step statistics.\n\n> W5: Explain assumption\n\nOur assumption is `$Q(s_t, a_t) = r(s_t, a_t) + V(s_{t+1}) = V(s_{t+1})$ for non-terminal nodes`.\n- The 1st \"=\" is not assumption, but the definition of state-action and state values [r1,r2]. It is due to deterministic transition $s_{t+1}=cat(s_t, a_t)$ in Eq. (1) for LLM output.\n-  We mentioned why the 2nd \"=\" holds in `footnote 1`. The reward of non-terminal nodes is 0. This is a common definition in RL if the reward can only be determined until the end of the episode.\n\n[r1] Sutton & Barto's Reinforcement Learning: An Introduction\n\n[r2] The UC Berkeley course: Deep Reinforcement Learning (CS285)\n\n> W6: Explain labels are V, not z\n\nIn standard RL, most value fitting algorithms aim to learn intermediate or step-level values [r1,r2] with various value estimation techniques. Therefore, we didn't make the modification; instead, AlphaGo series did.\n\nThe reason why learning from final reward works is that AlphaGo series applied thousands of simulations or sampled trees in MCTS. In the MSE value fitting, which is theoretically a regression problem, the regression signals 1 and −1 are proportional to the number of positive and negative rewards in simulations. The ratio asymptotically approaches the true value due to the large sampling size. Thus, the overall gradient direction for MSE should be asymptotically accurate. In LLMs, we cannot afford thousands of simulations.\n\n> W7: Explain value selection in MCTS inference\n\nSelection with maximum Q-value is not a modification, but one of popular methods, as introduced by textbook [r1] Sec. 8.11 MCTS, quoted as `some mechanism that depends on the accumulated statistics in the tree, e.g., the largest action value, or action with the largest visit count)`.\n\nSimilarly, due to the large amount of simulations in AlphaGo series, using visit count is a robust estimation. In our case with much fewer simulations, we found maximum Q-value is more robust. In the submitted code **offline_inference.py**, we already implemented visit count. For the response purpose, we include the relevant result in **Table 5 of PDF file**.\n\nActually, using the variant of visit count $\\propto N^\\alpha$ in AlphaGo series is a modification. Previous works such as [4] follow the variant without considering sampling size, which may lead to suboptimal.\n\n> W8: Discuss large B1 with fewer steps, and majority voting\n\nAs we explained in previous response, each REACT code step includes text analysis, a code snippet and its execution output. If the output indicates an error message, the LLM will generate a new REACT step by rewriting the code snippet until no error occurs or max depth reaches. When B1 increases, there are more candidates for the value model to select from. Our MCTS training framework encourages the value model to choose reasoning steps that don't output error message. Thus, the average number of overall reasoning steps decreases as B1 increases.\n\nPlease kindly refer to `G2 in General Rebuttal` for the discussion of majority voting.\n\n> W9 and Limitations\n\nPlease kindly refer to `G3 in General Rebuttal`.\n\nWe sincerely appreciate your valuable feedback and look forward to address any further concerns you may have.\n\nSincerely,\n\nAuthors of 6802", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XWkFAwTTVj", "reviewer": "Reviewer_5iMV", "summary": "The paper proposes an innovative framework named AlphaMath, aimed at enhancing the mathematical reasoning capabilities of large language models (LLMs) without relying on expensive human annotations from domain experts or GPT-4. The authors leverage Monte Carlo Tree Search (MCTS) to allow a pre-trained LLM to autonomously generate process supervision and step-level evaluation signals, integrating a value model with the LLM. At the inference time, the authors propose a step-level beam search strategy, which assists the policy model in navigating more effective reasoning paths. Experimental results demonstrate that AlphaMath achieves comparable or superior results to previous state-of-the-art methods on various datasets without process supervision.\n\nThe application of MCTS in LLMs is now somewhat popular, with several highly relevant papers, such as [1] https://arxiv.org/pdf/2406.06592, [2] https://arxiv.org/pdf/2405.00451 and [3] https://arxiv.org/pdf/2309.17179] (note that [1, 2] are published after neurips deadline, so no need to perform comparison, It's just added for reference.)", "questions": "Check other sections.", "limitations": "1. The framework does not generate a reward model that can facilitate other LLMs, which potentially limits its applicability.\n2. The framework cannot be extended to tasks with open-ended outputs, limiting its potential use cases.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.The proposed method effectively eliminates the necessity for human or GPT-4 annotation, resulting in significant efficiencies in both financial and temporal aspects.\n2. The utilization of Monte Carlo Tree Search (MCTS) can enhance the performance of LLMs in reasoning tasks by facilitating the search for optimal paths. Additionally, training with trajectories derived from MCTS is a commendable approach.\n3. The introduction of step-level beam search presents a promising strategy to reduce computational costs.", "weaknesses": "1. The paper's results are all program-aided, a fact not clearly identified in the text. Consequently, the comparison with non-program-aided LLMs may not be fair enough.\n2. The benchmark only includes GPT models in the 'Proprietary Models' section. It would be beneficial to include other LLMs such as Gemini and Claude for a more robust evaluation. \n3. An image illustrating the Step-Level Beam Search (SBS) would greatly aid in explaining the algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 5iMV,\n\nWe are grateful for the time and effort you have invested in providing detailed and insightful feedback. We appreciate your recognition of the key contributions of our method, including eliminating the need for human annotation, the commendability of trajectories generated via MCTS, and proposing step-level beam search to reduce inference cost.\n\nDistinctions from other works can refer to `G1 in general rebuttal`. Then, please find our detailed responses to each of your points below.\n\n> W1: The paper's results are all program-aided, a fact not clearly identified in the text. Consequently, the comparison with non-program-aided LLMs may not be fair enough.\n\nIn `Line 208-211` of Section Experiments, we mentioned that `To ensure a fair comparison, we primarily contrast our approach with the highest-performing SFT models that utilize an external tool - a Python code interpreter. These include MAmmoTH [43], MathCoder [35], ToRA [12], MARIO [44], MathGenie [23], and DeepSeek-Math-Instruct [29]. More implementation details can be found in Appendix C.`. In addition, in the `Tool column in Table 2`, we also marked whether each method uses program.\n\nMost of the 7B SOTA methods, e.g., ToRA, DeepseekMATH-instruct, use python programs to perform mathematical reasoning. As the program can mostly address the numerical computation in math reasoning. For larger LLMs, a recent example shows that most LLMs may fail for \"9.9 and 9.11, which is bigger?\". However, with the assistance of program, LLMs can easily solve such problems [r1].\n\n[r1] Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification\n\nThe comparison with non-program-aided LLMs is for reference only, which is a convention in previous works, e.g., ToRA, MAmmoTH, etc.\n\nFurthermore, previous works relied on human or GPT-4 annotation process to supervise the data, as shown in the **Seed Data Annotation** column of Table 2. Our **almost unsupervised/weakly supervised setting** v.s. **supervised setting** in previous works is actually an unfair comparison to our method, but we still achieve very competitive results.\n\nWe hope this clarification addresses your concern, and we are open to further discussions should you have any additional questions.\n\n> W2: The benchmark only includes GPT models in the 'Proprietary Models' section. It would be beneficial to include other LLMs such as Gemini and Claude for a more robust evaluation.\n\nFirst, we include the GPT-4 results quoted from previous works for reference only, by following the convention of related works. The \"Proprietary Models\" are not fully accessible all over the world and require global credit card for the API usage. It is difficult to evaluate on other proprietary models, especially on the OOD data, but we would like to include the GSM8K/MATH results of Gemini and Claude from relevant papers in revised version.\n\nAdditionally, all \"Proprietary Models\" should be larger in size than the 7B model, which is the main focus of our related works, making it difficult to justify fairness.\n\n> W3: An image illustrating the Step-Level Beam Search (SBS) would greatly aid in explaining the algorithm.\n\nIn `Line 168-178` and `Algorithm 2`, we introduced the SBS by modifying MCTS. However, we can view the SBS from the perspective of traditional (token-level) beam search (BS).\n\nFor simplicity, we use SBS(1, 5) and max step=3 for illustration and we will add an image in our revised paper.\n1. The policy model samples 5 candidates for step 1, denoted as $a_1^1, a_1^2, \\ldots, a_1^5$. (**similar to 5 candidate tokens in BS**)\n2. The value model reranks 5 candidates, and selected the best, e.g., the top-1 is $a_1^1$. (**BS may use some variant of logprobs for reranking, such as length penalty version**)\n3. The policy model samples 5 candidates for step 2 based on prefix $a_1^1$, denoted as $(a_1^1,a_2^1), (a_1^1,a_2^2), \\ldots,( a_1^1,a_2^5)$.\n4. The value model reranks 5 candidates of first two steps, and selected the best. For example, we assume the top-1 is $(a_1^1,a_2^1)$.\n5. Repeating above process in 3 and 4 will get the final best path $(a_1^1, a_2^1, a_3^1)$.\n\n> L1: The framework does not generate a reward model that can facilitate other LLMs, which potentially limits its applicability.\n\nThank you for your helpful comments. We would like to address that our framework can also facilitate other LLMs.\n  - The step-level evaluation signals (Q-values) generated by our framework can be used to train a separate value model. Reviewer 2HCk concerned about the performance of training the value model and policy model separately. We share the experimental results here.\n\n|$B_1=1$|GSM8K|MATH|GK2023|OCW|avg. time (s/q) & GPU usage|\n|--|:--:|:--:|:--:|:--:|:--:|\n|two separate models|82.4 |61.4|46.4|29.9|4.3 / $2\\times$A100-80G|\n|AlphaMath (Ours) |81.1|62.8 |46.2|30.5| 3.1 / $1\\times$A100-80G|\n\nFrom the table, our AlphaMath has similar overall performance to the two separate models, but two separate models requires more inference time and computing resources, which is what we want to avoid.\n- Our joint model (when using value model head only) can also score other LLMs with Python Code as long as the output format of other LLMs is reorganized by following the our defined REACT format.\n- If the LLMs insist to use CoT solution, it needs to retrain the value model in the MCTS framework with step-level CoT solutions.\n\n> L2: The framework cannot be extended to tasks with open-ended outputs, limiting its potential use cases.\n\nIn `Line 518-520`, we have stated that our approach only works on tasks that are evaluated based on the actual answers in future work section. To further clarify, we will add the scope of our method in **Limitation**.\n\nWe sincerely appreciate your invaluable feedback and would be delighted to engage in further discussion to address any questions you may have, with the aim of improving and enhancing the quality of our work.\n\nSincerely,\n\nAuthors of 6802", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
> G1: Why our value model training works better?

We agree with previous works [1,4] (by Reviewer 15WC) may learn a reasonable policy model if they applied some autonomous annotation method, even they actually didn't do it. However, we respectfully argue that the value model learning in [1,4] may not favor learning from almost zero, as they require large sampling size or ORM initialization.

Our training of value model and those in [1,4] are **fundamentally different**, because we coherently integrated MCTS in both training and inference stages inspired by AlphaZero. This is the main reason why our model can successfully learn from almost zero without human annotations. Previous works utilized other reinforcement learning (RL) algorithms to learn the value model, even MCTS is used [4].

- **Value Estimation (most important)** As mentioned in `Line 91`, we didn't use Monte Carlo (MC) estimate [1,4] for value labels. MC estimate is an unbiased but high variance estimator. To fix it, variance reduction tricks are needed [r1,r2], e.g., increased sampling size, that may not work for high-dimension. Instead, we employ MCTS to construct value training labels. Theoretically, unlike MC estimate, which applies standard sampling over actions, the MCTS value estimate enables more implicit exploration. During tree expansion, MCTS prioritizes and selects non-greedy actions that may lead to better future returns. From the perspective of both practice and theory, MC and MCTS for value estimates are fundamentally different.
- **Process Supervision SFT Data** We did not utilize SFT data as [1,4]. Previous works employ other RL algorithms (e.g., PPO in [1]) that require initializing the policy model with a reference policy, typically an SFT model. Our training process adheres strictly to the AlphaZero framework, i.e., without human annotated process.
- **ORM Initialization** We did not use ORM to initialize value model (TD-$\lambda$[4] or MC estimate [1,4]). As [r1,r2] indicates, TD-$\lambda$ will introduce bias in the estimate. Tricks are needed to make the training less overfitting or degeneration. In our experience, we found a separate value model initialized from a SFT policy model can achieve more stable training and more robust predictions. Thus, we hypothesize that the ORM initialization is crucial in [1,4].
- **Network Sharing** In RL realm, it is well known [r1,r2] that two separate networks is easier to implement and more stable for training. This approach is inefficient data usage for optimization as it fails to share the internal representations, while policy / value models usually share same state input. However, shared networks are more challenging due to gradients from different losses varying in direction and scale, requiring careful tuning. Thus, stabilizing the training of shared networks is also one of our contributions, in the sense of RL algorithm. Additionally, inferencing with two separate LLMs would almost double the computational cost.
- **Inference method** Unlike previous works, we mainly use MCTS for training rather than inference. For inference, we propose the faster SBS modified from MCTS.

[r1] Sutton & Barto's Reinforcement Learning: An Introduction

[r2] The UC Berkeley course: Deep Reinforcement Learning (CS285)

> G2: Explanation between maj@n and SBS(1, n)

We use an example to illustrate the difference between maj@n and SBS(1, n) with $n=5$ and max depth=3.
- **maj@5** The final results are 5 solution paths $(a_1^j,a_2^j,a_3^j)^{j=1:5}$, where $a_t^j\ne a_t^k$ if $j\ne k$. E.g., the recurrent relation is that $a_3^j$ is sampled based on $a_{1:2}^j$ for each $j$. The sampling is from 5 different partial solution paths, and all previous candidates must be maintained.
- **SBS(1, 5)** The final result is a top-1 solution path $(a_1,a_2,a_3)$, where $a_3$ is best action selected from $a_3^{1:5}$. Note that the recurrent relation is that all $a_3^{1:5}$ are sampled based on $a_{1:2}$, where the sampling is from only 1 best partial solution path. This step-level streaming manner is more friendly to production.

Thus, in the sense of diversity sampling, maj@n has more advantage than SBS(1, n). However, SBS with the value model can reverse this advantage. For the response purpose, we also include the comprehensive comparison in `Table 6 of PDF file`.

> G3: W9 and Limitations for **Reviewer 15WC**

Due to characters limitation, we apologize for putting the response in general rebuttal.

**W9**: Q-value in Fig 5 sampled from? For full or partial steps? Missing legend?

With the Q-value update formula, MCTS assumes it is finally converged. Then we used the converged Q-values of all nodes for plotting Fig 5.

In `Lines 279-280`, we mentioned the left value plot is only for intermediate steps on trainset (the terminal value is final reward, no need to estimate). In `Lines 292-293`, we mentioned the right value plot is for both intermediate steps and terminal steps on testset.

We carefully reviewed the right figure and confirmed that it indeed contains only two curves and two legends.

**L1**: False positive / negative issues

We acknowledge that there are possible false positive / negative intermediate steps, especially for the proof (text analysis in REACT). We will include it in future version.

For calculation issue, the false cases can mostly be solved.
- Practically, in the `Solution Filtering Algorithm 3 of Appendix C.2`, python code can verify the numerical errors. Pure text reasoning (CoT in Math-Shepherd, OmegaPRM) may encounter the recent famous error $9.9<9.11$.
- Theorectially, our value estimation is based on MCTS. The Q update of an intermediate step in MCTS can achieve better estimation by considering all its children nodes in a complex weighted form.

**L2**: Not always applicable to many tasks

Thank you for your insightful comment. In `Lines 516-520`, we partially mentioned this issue. To ensure greater clarity, we will add it to **Limitations** in future version.

Meta-review: The paper tackles solving mathematical reasoning with LLMs. The key idea is to leverage Monte Carlo Tree Search (MCTS) to sample high-quality process-supervised data, which currently existing works primarily rely on expensive human annotation or GPT-4. Moreover, they propose an efficient inference strategy called step-level beam search, which leverage value models trained from the generated process supervised data. Altogether demonstrates significant improvements on various datasets, showing comparable or even superior results to existing state-of-the-art methods without relying on external process annotations.

The reviewers all gave borderline scores. Based on my assessment, most of their raised weaknesses have been addressed by the author during rebuttal. However very few engagement during and post rebuttal happened, despite multiple reminders from the AC. One of the reviewers giving a rating of 4 agreed that the paper was "very close to the threshold" and "a valuable study in this area" if the authors can properly address their concerns, which, to me, they did.

I therefore recommend accepting despite the slightly low average score.
Decision: Accept (poster)