AlphaMath Almost Zero: Process Supervision without Process

Authors: guoxin  chen
minpeng  liao
chengxi  li
kai  fan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 15:00:09

Abstract: Although recent advancements in large language models (LLMs) have significantly improved their performance on various tasks, they still face challenges with complex and symbolic multi-step reasoning, particularly in mathematical reasoning. To bolster the mathematical reasoning capabilities of LLMs, most existing efforts concentrate on seeking assistance from either domain experts or GPT-4 for high-quality process-supervised data, which is not only expensive but also labor-intensive. In our study, we propose an innovative framework, AlphaMath, that bypasses the need for process annotations (from humans or GPTs) by leveraging Monte Carlo Tree Search (MCTS). This framework focuses on unleashing the potential of a well-pretrained LLM to autonomously enhance its mathematical reasoning. Specifically, we integrate a value model with the LLM, automatically generating both process supervision and step-level evaluation signals in MCTS. Furthermore, we propose an efficient inference strategy—step-level beam search, where the value model is crafted to assist the policy model (i.e., LLM) in navigating more effective reasoning paths, rather than solely relying on prior probabilities. The experimental results on both in-domain and out-of-domain datasets demonstrate that even without GPT-4 or human-annotated process supervision, our AlphaMath framework achieves comparable or superior results to previous state-of-the-art methods.

Main paper: 
 preliminary: 
 We assume that, for any given input question q, the solution process can be broken into multiple reasoning steps (e.g., segmenting the solution based on distinct stages or simply on a period). From this perspective, we conceptualize mathematical problem solving within the context of reinforcement learning. Concretely, consider a complete solution consisting of T reasoning steps. At a given time t, we represent the partial solution as the state s t , and the subsequent reasoning step that might be taken as the action as a t . For detailed definitions and examples of our reasoning step, please refer to Appendix C.1. In this scenario, the policy model is embodied by a large language model, and the transition f (s t+1 |a t , s t ) from one state to the next is deterministically accomplished through the concatenation operation.EQUATIONOur primary goal is to develop a step-level value model, denoted as V ϕ (s), which is capable of assessing the expected returns from the current partial solution and guiding the LLM to select more reasonable subsequent reasoning steps.To train the value model, we first define the reward in the context of mathematical problem solving, by assigning the reward r = 0 to all non-terminal reasoning steps, and r = ±1 to a correct/incorrect final answer. A common method to create the training signal is to employ Monte Carlo (MC) evaluationEQUATIONt ′ >t |s t , where a (i)t ′ ≥t and s(i)t ′ >t represent the actions and states in the i-th simulation sampled by the policy model and the state transition function. r(•|s t ) means the reward of the final outcome in one simulation from state s t . Then, for any given partial solution s, we can train the step-level value model V ϕ using a regression loss defined as follows:L V ϕ (s) = V ϕ (s) -V (s) 2 .(2) 
alphamath: 
 In the above approach of MC evaluation, it requires multiple simulations from each state, which may be inefficient in practice. We propose employing the Monte Carlo Tree Search (MCTS) algorithm, which has the potential to reuse simulations and update the estimated values in a principled manner.As shown in Figure 1 , our approach employs iterative training. Before the (k + 1)-th round training, we have a value model V ϕ k and a LLM policy model π θ k , which are the same model but with different final layers in our paper. Using these models, we can construct an inference algorithm powered by MCTS. This algorithm starts with the initial state as its root, and through the synergistic use of the policy and value models, systematically grows the search tree by adding new nodes. These nodes correspond to the states deemed to have high potential based on the outcomes of simulated trajectories. Specifically within the context of mathematical problem-solving, as shown in Figure 2 , we customize the four key operations of the MCTS algorithm as follows:Selection During the i-th simulation of the MCTS, the process begins with s 0 , representing the initial state containing the input question. The algorithm then proceeds to explore the tree T k by selecting nodes according to a variant of the PUCT algorithm [30] . This selection process is mathematically represented as:EQUATIONwhere the state-action value Q(s, a) and its visiting count N (s, a) are stored in the tree and will be updated as the search progresses. N parent (a) represents the visiting count of the parent node of a.The action selection iterates until it encounters a leaf node of the current search tree. In our case, the prior π(a|s t ) is defined as the exponential of averaged log-probability of all tokens in the step a, i.e., exp 1 |a| log π(a j |a <j , s t ) .Expansion Back-tracing from the selected leaf node to the root forms a partial solution, serving as a prompt for further node expansions. In our case, given that the LLM can theoretically generate an unlimited number of potential actions (token sequence), we employ sampling generation with higher temperature to ensure diversity.Evaluation Evaluation of the leaf node or partial solution s t , identified after the selection phase, is conducted by weighted sum as introduced in [32, 33] .EQUATIONThe intermediate value estimation V in MCTS differs from the training signal V defined in preliminary section 2. The parameter λ serves to balance the contribution of the value model's estimation with the empirical reward obtained during the rollout.In our case, we follow a trade-off rollout strategy between AlphaGo [32] and AlphaGo Zero [33] .Because our tree depth is much shallower than Go games (e.g., a maximum depth of 8) and expansions can easily reach a terminal node, we set an indicator function λ = I terminal (s t ). If the expanded node is terminal, the reward is returned; otherwise, the value is predicted by the model V ϕ k .Backup We did not make any modifications to the backup. At the end of the i-th simulation, each edge (s, a) along the path from the leaf node s t to the root undergoes a backward pass update. The updates to their state-action values and visiting counts are executed according to the following rules:N (s, a) ← N (s, a) + 1 and Q(s, a) ← 1 N (s,a) i j=1 I s,a→st V (s t ) (j). Value Estimation After running N simulations with the MCTS algorithm, we obtain the final tree T k , which stores the expanded nodes and their corresponding state-action values Q(s, a). Considering that the transition function is deterministic, and assuming that Q(s t , a t ) = r(s t , a t )+V (s t+1 ) = V (s t+1 ) for non-terminal nodes 3 , we can employ the Q values as training signals. This implies that we can directly fit the state-action value of non-terminal nodes as,EQUATIONRequire: B 1 = 1, question q (s 0 ), policy / value models π θ , V ϕ , simulations N , max depth T . 1: Build the complete tree T by running MCTS π θ ,V ϕ (s 0 , N, T ). for a in T children (s t ) do ▷ Directly get children from tree 7:s t+1 = Cat [s t , a] 8: Add (s t+1 , T Q (s t , a)) to C t+1 ▷ Directly get Q-value from tree 9: C ← Top-B 1 of C t+1 return Top-1 of C ▷ Return top-1Algorithm 2Step-level Beam SearchRequire: Beam sizes B 1 , B 2 , question q (s 0 ), policy / value models π θ , V ϕ , max steps T . 1: C = [s 0 ] * B 1 , t = 0 ▷ Initialize candidatess t+1 = Cat s t , a (b) 8: Add (s t+1 , V ϕ (s t+1 )) to C t+1 ▷ V ϕ (s t+1 ) predicted by value model 9: C ← Top-B 1 of C t+1 return Top-1 of C▷ Return top-1 as the final solution pathInitialization Initially, our approach begins with a pre-trained LLM as the policy model π θ1 . We extend this model by adding an auxiliary linear layer with a tanh activation function, which works alongside the traditional softmax layer responsible for token prediction, as depicted in the rightmost panel of Figure 1 . This design implies that these two models, π θ and V ϕ , share the majority of their parameters. The parameters of the linear layer associated with V ϕ1 are randomly initialized, leading to an initial tendency of the value head to predict a value close to 0 at the first (k = 1) round of MCTS. However, as the simulations in the first round MCTS proceed, the rewards (±1) from terminal nodes are back-propagated to their parent nodes. As simulations N gradually increase, the estimated values Q of intermediate nodes converge towards the underlying true value within the range of[-1, 1].Training Method From the tree T k constructed from the k-th round of MCTS, we can sample solution paths corresponding to terminal nodes with correct and incorrect predicted answers, denoted as x + and x -, respectively, together with the value estimation of each node along these paths. We then apply a multi-task loss function to update both the policy and value models.EQUATIONwhere the first term represents the negative log-likelihood loss for next-token prediction in correct solutions, and the second term within the big brackets captures the loss in value prediction for both correct and incorrect solutions, respectively. T (x) denotes the number of steps for solution path x. β is a tunable hyper-parameter to control the weight of value loss. With the updated policy and value models π θ k+1 and V ϕ k+1 , we can advance to the next-round MCTS, iterating this training process to enhance our models further.MCTS For MCTS inference, it is necessary to set λ = 0 in the evaluation of Eq. ( 4). Unlike in board games, we cannot verify the correctness of a path during inference. Therefore, we consistently rely on the value model for node evaluation, including for terminal nodes. MCTS demands multiple simulations to update visiting counts and Q values, aiming to estimate a robust policy distribution.After the tree has been completely built, the algorithm iteratively selects the top-B 1 steps (usually B 1 = 1 in MCTS) from the root in a top-down manner. This selection is guided by the maximum Q-value stored in the child nodes of the tree. Subsequently, all child nodes from the previously selected B 1 steps are collectively re-ranked based on their Q-values, and the top-B 1 nodes from this ranking are retained for the next iteration. A summary of the algorithm can be found in Algorithm 1.Step-level Beam Search However, MCTS is computationally intensive for simulations, making it less viable for use in production environments. To address this, we modify the MCTS inference process by eliminating the backup operation, introducing a simplified method, which we refer to asStep-level Beam Search (SBS), detailed in Algorithm 2. This approach does not construct the entire tree; instead, it dynamically selects the best child node during node expansion.There are two primary technical distinctions in SBS. First, since node expansion is required on the fly, we introduce a new beam size, B 2 , to represent the maximum number of node expansions. Second, the selection criterion no longer relies on the Q-value converged after N simulations but instead uses the maximum value prediction directly from the value model. Importantly, with special case SBS B 1 = 1 as a fast approximation of MCTS, it facilitates the sequential, streaming output of steps, rendering it more suitable for practical implementation in real-world production. 
experimental setup: 
 In this study, we mainly investigate the math domain-specific language model, DeepSeekMath-Base-7B [31] , pre-trained on a substantial math-related corpus without any supervised fine-tuning (SFT), which is believed to possess necessary mathematical knowledge to tackle a wide range of mathematical problems.Training Data Generation via MCTS For the training sets, we exclusively extract question and answer pairs from GSM8K [7] and MATH [15] , omitting the human-annotated solution analysis. In total, our training set includes only 15k question-answer pairs and 0 solution process.In our setup, we utilize the MCTS framework to generate detailed solution processes equipped with the Python code interpreter. Initially, for the first round of MCTS, the prompt used for our solution generation adheres to the REACT [43] format, incorporating 2 demonstrations randomly selected from a pool of 20 prepared examples. Starting from the second round, with an already fine-tuned model from the first round, we employ a straightforward prompt in our SFT XML format without any demonstration. Two prompt examples are shown in Appendix F.Specifically, we iteratively generate data and train our policy and value models through K = 3 rounds, continuing until the enhancement observed between any two consecutive rounds is incremental. In every round, we build 10 trees for each question-answer pair and randomly sample at most 4 correct and 4 incorrect solution processes. The ratio between positive and negative examples is approximately 1:1, with the count of positive examples in each round varying between 57k and 59k.Test Data We evaluate our approach not only on GSM8K and MATH but also on out-of-distribution (OOD) datasets GaoKao2023 [21] and OCWCourses [19] . These two OOD datasets are even more challenging than MATH. Please refer to Appendix C. 5 for more details about the dataset statistics.To assess the accuracy of the predicted answers, we utilize the math evaluation toolkit [47] .Baselines We first compare our approach with strong proprietary and open-source models, including OpenAI's ChatGPT and GPT-4 [27] , Llama2 [37] , Llemma [3] . By default, we report the results obtained using Chain of Thought (CoT) prompting [40] , along with the prompting results of PAL [13] , due to its enhanced performance in mathematical reasoning.SFT models leverage high-quality seed data with process supervision derived from GPT-4 or humans to enhance their reasoning capabilities. To ensure a fair comparison, we primarily contrast our approach with the highest-performing SFT models that utilize an external tool -a Python code interpreter. These include MAmmoTH [46] , MathCoder [38] , ToRA [14] , MARIO [21] , MathGenie [25] , and DeepSeek-Math-Instruct [31] . More implementation details can be found in Appendix C. We report our in-domain and out-of-domain (OOD) results in Table 2 . Different from previous works [46, 38, 14, 21, 25] , our proposed AlphaMath does not rely on high-quality solutions annotated by humans or GPT-4, whether in the form of text analysis or code snippets. Such solutions typically bolster the model's reasoning abilities but also entail substantial costs associated with annotation. Furthermore, our method differs from prior research by not incorporating any external datasets (e.g., new questions and solutions) beyond the GSM8K and MATH datasets. The last five rows of Table 2 present our principal findings.First, we establish a baseline with the inherent mathematical reasoning ability of DeepSeekMath-Base using our designed prompt in a 2-shot setting. It's important to note that this outcome differs from the results reported for DeepSeekMath-Base (PAL) in the original study, as it utilized prompts with 8-shot and 4-shot for the GSM8K and MATH datasets, respectively. Secondly, we only evaluate the policy model with greedy decoding. In comparison to our initial study, we record an enhancement of about 20 points for challenging problems in the MATH, GaoKao2023 (GK2023), and OCWCourses (OCW) datasets, and an improvement of more than 10 points for grade school math problems.Thirdly, we delve into the role of the value model in facilitating mathematical reasoning, utilizing a computationally efficient step-level beam search (SBS) in Algorithm 2. When we increment B 1 with a default B 2 = 5 and temperature of 1.0, a corresponding gradual improvement in performance is observed. More discussion about the temperature in SBS can refer to Appendix 4.7. Ultimately, we evaluate our approach in Algorithm 1. In contrast to the training data generation, we construct a single tree with 40 simulations, a maximum of 5 child nodes, and a temperature of 0.6. While MCTS demonstrates improved performance on more challenging datasets, attributed to its expansive search space, its substantial computational demands curtail its practical applicability in real-world scenarios.In summary, our approach demonstrates that, even in the absence of high-quality GPT-4 or humanannotated solution processes, it remains competitive with or surpasses the performance of the state-of-the-art (SOTA) on 7B LLMs.We evaluate the problem-solving rate in the MATH training dataset, which categorizes each problem by difficulty level. As shown in Figure 3 , it becomes evident that MCTS achieves greater success in solving more challenging problems in subsequent rounds. In Figure 4 , our findings show a general increase in performance with additional rounds of training across all strategies, applicable to both in-domain and out-of-domain test sets. Therefore, we can conclude that the quality of our selfgenerated training data improves incrementally with each round, and this enhancement is reflected in the performance on the test set. More analysis can refer to Appendix B.3.We explore the performance of our model under various inference strategies including greedy decoding, step-level beam search, and MCTS. The results of MATH and GaoKao2023 are illustrated in Figure 4 , while the results of other datasets can be found in Appendix B.1. Specifically, for SBS, an enhancement in performance was observed with an increase in the beam size B 1 . MCTS exhibited the higher performance than its approximation SBS (B 1 = 1), but we previously noted its significant time consumption and computational inefficiency. Consequently, we provide a summary of the average problem-solving duration and the average number of intermediate steps taken on the MATH dataset in Table 3 . The results indicate that MCTS demands the longest solving time and the highest number of steps, attributable to our configuration of 40 simulations. To achieve similar accuracy, step-level beam search is more computationally friendly. Additionally, we observe an intriguing phenomenon: a larger beam size B 1 tends to reduce the average problem-solving duration. This can be attributed to the decrease in the number of average steps required when a larger B 1 is employed. Discussion of Majority Voting It is challenging to directly compare maj@5 with step-level beam search due to the inherent differences in their methodologies. Generally speaking, as Algorithm 2, SBS will eventually return the top-1 final answer based on the value model, while maj@5 will generate all 5 possible final answers and vote the majority for evaluation.From the step-level perspective, maj@5 will maintain 5 candidates for the current step to generate another 5 candidates for the next step. In contrast, the SBS (e.g., B 1 = 1, B 2 = 5) will always retain the top-1 candidate, discarding the 4 others. This provides the advantage of stepby-step streaming output in real-world production, whereas maj@5 can only output the complete solution until the voting is finalized. To sum up, their specific mechanics of candidate selection and retention differ significantly.In the left panel of Figure 5 , we plot the fitted distribution of Q-values (as defined in Eq. ( 5)) on MATH training set for intermediate steps. For correct solutions, the distribution is markedly skewed towards a value of 1. In contrast, the distribution for incorrect solutions exhibits a lower degree of skewness, albeit with the majority of the probability density leaning towards -1. This is because a correct final answer typically suggests that the entire solution process is likely accurate, whereas an incorrect final answer may still encompass some correct intermediate steps. Thus, with the backup of MCTS, the Q-values of intermediate steps in incorrect solutions may also be updated with a reward of 1 during simulations. We further investigate the potential of two other popular types of LLMs: generalpurpose pre-trained models and SFT models. These models represent the scenarios of lacking continual pre-training (CPT) in domain-specific data and supervised fine-tuning (SFT) on high-quality annotated domain data, respectively. We select Llama3 [11] and MARIO [21] as the base models and report the results in Table 4 . For a fair comparison, the MARIO is trained on DeepSeekMath-Base-7B rather than its original Llemma-7B [3] . First, although not proficient in mathematical reasoning, our Al-phaMath enhances Llama3's mathematical reasoning capabilities without any annotations, yielding an average improvement of +20 points. Secondly, AlphaMath can significantly enhance the performance of existing SFT models, enabling MARIO to be competitive with and even outperform GPT-4. We further investigate the effects of temperature during decoding on the performance of inference algorithms. For the greedy strategy, the temperature is consistently maintained at 0, whereas step-level beam search (SBS) and Monte Carlo Tree Search(MCTS) are more significantly influenced by higher temperatures. Therefore, taking steplevel beam search (B 1 = 1 and B 1 = 3) as an example, we obtained the results as illustrated in Figure 6 .First, under any temperature setting, the performance of step-level beam search significantly surpasses that of the greedy strategy. This is attributed to the value model effectively assisting the policy model in identifying more effective reasoning paths. Secondly, at lower temperatures, the performance of step-level beam search is constrained due to the lack of diversity in the generated solutions. With elevated temperatures, the value model is capable of discerning optimal paths within a more diverse set of solutions, thereby effectively enhancing reasoning performance. Finally, with a larger beam width, the model can explore more solutions. Therefore, the performance of B 1 = 3 always surpasses that of B 1 = 1. 
related works: 
 Solution Annotation in Math. Recent works [46, 38, 14, 20, 21, 31, 25, 16] on mathematical reasoning have made impressive progress empowered by process-supervised data. However, most existing efforts concentrate on seeking high-quality solutions from domain experts or formidable commercial models, such as GPT-4 [27] , which hampers the scalability of methods and escalates the associated expenses. Unlike previous work, only with the help of question-answer pairs, we focus on activating the intrinsic knowledge within LLMs to realize iterative self-evolution and strengthen their utilization of knowledge autonomously, just like humans.Value/Reward Model. Recent studies [7, 8, 22, 44, 49, 42, 41, 12] have demonstrated that process supervision can significantly enhance mathematical reasoning performance. Especially, value model [12, 23, 26] is incorporated into the decoding process, while reward model is the source of the training signal in reinforcement learning [28, 31] . However, these value/reward models require substantial annotated process-supervised data and introduce significant inference latency. In our work, we consider the state values V (s t ) from MCTS as supervision signals, which are aligned with the solutions and eliminate the annotation costs. Furthermore, we integrate the value model into the generative model to navigate more effective reasoning paths at minimal cost, thereby providing richer decoding strategies, such as step-level beam search or MCTS. 
conclusion: 
 In this work, we introduce AlphaMath, a simple iterative training paradigm for leveraging Monte Carlo Tree Search to unleash the potential of a well pre-trained large language model to autonomously enhance its mathematical reasoning capabilities. Furthermore, by applying step-level beam search, the value model can assist the policy model in selecting a more reasonable solution path, rather than solely relying on prior probabilities, which significantly enhances mathematical reasoning capabilities at minimal cost. The experimental results on both in-domain and out-of-domain datasets demonstrate that even without GPT-4 or human-annotated process supervision, AlphaMath remains competitive with or surpasses the performance of the state-of-the-art methods. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The work we have done in our paper does not involve such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have strictly adhered to the licensing and usage terms of the data and models.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: For all new assets introduced in the paper, such as data, code, and models, we provide detailed documentation in the supplementary materials. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 Recent studies have extensively explored how to improve mathematical reasoning in large language models (LLMs) [27, 1, 37, 34, 2, 35 ]. An effective approach [46, 38, 14, 21, 31, 25] is to artificially inject external knowledge into LLMs through fine-tuning on a substantial volume of high-quality, process-supervised data (i.e., solutions). As shown in Table 1 , the annotation of high-quality solutions in current efforts primarily relies on domain experts or GPT-4 [27] . However, due to trillions of training tokens and billions of parameters, existing LLMs possess a vast reservoir of knowledge, which remains underutilized in current finetuning-based approaches.To more effectively harness the intrinsic knowledge of LLMs, advanced prompting techniques, such as Program-of-Thought (PoT) [6] and Program-Aided Language (PAL) [13] , have been developed, integrating the in-context learning proficiency with external tools such as code interpreter to handle precise numerical and symbolic computation. However, these approaches have not fully unleashed the potential of LLMs and often rely on self-consistent majority voting [39] , which does not reflect the natural process by which humans solve mathematical problems. This discrepancy arises because both the PoT and PAL frameworks pursue a solution to its final answer regardless of the accuracy of intermediate steps. Unlike these approaches, humans tend to reassess and potentially alter their solution path upon encountering a mistake or dead-end in the problem-solving process. In this manner, humans iteratively enhance their self-cognition and reinforce the utilization of knowledge.In this research, we aspire for LLMs to possess the similar ability as humans to realize self-evolution and strengthen their utilization of knowledge autonomously. Notably, AlphaGo Zero [33] showcases how a neural network model can progressively evolve without human knowledge, autonomously producing the Go game training strategies. For the strategy (i.e., solution) of mathematical problems, both textual analysis [45] and code snippets [14] demand rigorous logical structuring. Consequently, most finetuning-based approaches concentrate on seeking assistance from domain experts or GPT-4 for annotated solutions, thereby overlooking the reservoir of knowledge inherent in LLMs.Instead, we hypothesize that well pre-trained LLMs already possess the necessary mathematical knowledge to generate correct reasoning; however, they require appropriate stimulation-such as an improved prompt or search strategy-to do so. In this work, solutions including both textual analysis and code snippets are autonomously generated by a well pre-trained LLM equipped with appropriate prompts and deliberately designed Monte Carlo Tree Search (MCTS) framework [4, 32] . Specifically, we integrate LLMs with the MCTS to strike a more effective balance between exploration and exploitation, enabling the generation of high-quality process-supervised solutions without professional human annotations. To enhance the efficiency of solution generation, we incorporate a value model into the same LLM by appending a linear layer. This advancement removes the necessity for timeconsuming rollouts for reward estimation. While the LLM learns to solve mathematical problems from its own annotated solutions, the value model simultaneously learns how to assess the quality of intermediate reasoning steps from the corresponding state values in MCTS, just like humans.During the inference stage, with the value model, LLMs can perform MCTS inference, which significantly enhances their reasoning capabilities but limited by efficiency. Therefore, inspired by beam search algorithm [36] , we propose a step-level beam search strategy, where the value model is crafted to assist the policy model (i.e., LLM) in navigating more effective solution paths, as opposed to relying solely on prior probabilities. Compared to the greedy or MCTS inference strategies, the step-level beam search significantly enhances the LLM's reasoning capability at a minimal cost.Empirically, we build an iterative training framework as shown in Figure 1 . Unlike in the game of Go, where the final board state directly indicates a win or loss, our methodology requires validation of the equivalence between predicted answers and actual ones. This is the fundamental reason why our training data necessarily consists of question statements and their final answers. Furthermore, we validate the applicability of our framework on three popular types of LLMs: domain-specific pretrained models [31] , general-purpose pre-trained models [11] , and supervised fine-tuned models [21] .Our contributions are summarized as follows:• We propose a novel approach that integrates a pre-trained LLM with a deliberately designed Monte Carlo Tree Search (MCTS) framework. This combination allows LLMs to autonomously generate high-quality mathematical reasoning solutions without the need for professional human annotations, leading to a more efficient utilization of their inherent knowledge. • To address the efficiency limitations of MCTS inference, we propose a step-level beam search strategy, which introduces a lightweight value model that works alongside the LLM, enabling the simultaneous assessment of the quality of intermediate reasoning steps. This method parallels human problem-solving by allowing the LLM to learn from its own solutions while also evaluating the effectiveness of its reasoning strategy, thus enhancing the overall reasoning capabilities. • Extensive experiments demonstrate that our AlphaMath can effectively stimulate the internal knowledge of LLMs, achieving better or on par task performance on both in-domain and out-ofdomain mathematical reasoning datasets, even without any process annotations. 
 
 Appendix: A.1 LimitationCompared to previous works, our AlphaMath achieves comparable or even superior results without annotated high-quality, process-supervised data. However, unlike the game of Go, where the final board configuration directly reflects winning or losing, in mathematical reasoning, we rely on the actual answer as the source of reward. This hinders us from "AlphaMath really from zero", an unsupervised algorithm. However, compared to process-supervised data, the acquisition of actual answers is considerably more straightforward. For instance, existing question-answering datasets as well as questions from examinations typically encompass the answers, yet lack annotations for process supervision.Directions for Future Work: Our research highlights several issues for further exploration:• Really from Zero: In this work, we have demonstrated that a well pre-trained large language model can unleash its potential to identify correct mathematical reasoning processes through the AlpahMath framework, independent of GPT-4 or manually annotated process-supervised datasets.A challenging yet profoundly meaningful future direction is to identify an appropriate reward definition that allows AlphaMath to eliminate dependence on actual answers, thereby achieving really from zero. Notably, this process should avoid introducing additional annotation costs, such as training a reward model to replace the actual answers. • A closed-loop self-evolution training framework: With the question-answer pairs, our Al-phaMath framework can realize iterative self-evolution in complex reasoning scenarios, just like humans. In this study, as an initial attempt, we have maintained the same set of question-answer pairs (total only 15k pairs) in each iteration, which limits the potential of AlphaMath. In the future, we will explore how to automatically obtain such question-answer pairs from the Internet, which could facilitate the development of a closed-loop self-evolution framework for AlphaMath. In this setup, the LLM automatically acquires question-answer pairs from the Internet and then autonomously enhances its reasoning capabilities through our AlphaMath framework, thereby achieving complete independence from human intervention. • Explore beyond mathematical reasoning: Since mathematical reasoning tasks involve complex, symbolic multi-step reasoning, we primarily choose them as an example to investigate the effectiveness of AlphaMath. However, our proposed AlphaMath has the potential to be broadly applied to any task that can be evaluated against actual answers. In future work, we plan to expand its application to a broader range of tasks. In this experiment, we can draw similar conclusions as in Figure 4 . With the progress of iteration, there is a significant enhancement in the model's performance, especially between the first and second rounds, as shown in Figure 7 . Furthermore, we observe that the performance of various inference strategies on the OCWCourses slightly differs from the other three datasets. This variation can be attributed to the fact that OCWCourses is a mathematical dataset in the fields of physics and chemistry. Nonetheless, our method still significantly enhances the model's reasoning capabilities on such datasets overall. In addition to the discussion regarding the value model in Section 4.5, we have also specifically analyzed the overall distribution of the predicted state values V (s) by the value model for both the intermediate and final steps in correct/incorrect solutions, as illustrated in Figure 8 . "Final step" refers to the scoring of the entire solution in the last step, representing the value model's overall assessment.In the left panel of Figure 8 , we plot the fitted distribution of the state values for both intermediate and final steps as predicted by the value model, during the process of solution generation on the training set. For correct and incorrect solutions, the value model's overall assessment is highly accurate, which is distinctly skewed towards 1 and -1, respectively. Notably, "Incorrect inter.Step" represents the intermediate steps of an incorrect solution, rather than incorrect intermediate steps. Therefore, "Incorrect inter.Step" may also contain some correct processes, which explains why its distribution crosses over 0. Overall, the distribution of the value model on the training set aligns very well with intuition, which aids in identifying higher-quality solutions in MCTS.In the right panel of Figure 8 , we plot the distribution of state value V (s) predicted by the value model on the test set. It can be clearly seen that the value model accurately distinguished between correct and incorrect solutions, which explains why the performance of step-level beam search significantly surpasses that of greedy inference. The value model aids the policy model in navigating more efficient solutions, rather than relying solely on prior probabilities. Additionally, due to the fact that incorrect solutions may contain correct steps, their distribution is primarily concentrated near 0. The intermediate steps of correct solutions exhibit a bimodal distribution, with peaks concentrated near 0 and 1. This can be attributed to the fact that even correct solution steps may contain some errors, such as coding mistakes. Therefore, in conjunction with the analysis in Section 4.5, we believe that our value model can effectively distinguish between correct and incorrect solutions, aiding the policy model in finding better solution paths.In 9a , it becomes evident that MCTS achieves greater success in solving more challenging problems in subsequent rounds. Similarly, Figure 9b indicates that, in later rounds, MCTS consistently demonstrates an improved capability to solve a broader array of problems across different subjects.For the test set depicted in Figure 10 , we include the results from round 0, which correspond to the performance of our "prompt 2-shot" in Table 2 . Unlike the training set, we observe that the improvement observed in round 3 is not consistent across different levels and subjects, even though the overall accuracy is slightly increased. In fact, for easier problems, the performance in round 3 actually declines. This is the reason we terminate our iterative training process after round 3. We further discuss the problem solving rates of different models in each round, as shown in Table 5 .First, given that the problems in the GSM8K dataset are relatively simple, the corresponding solution rates are higher, even in the first round. Despite the challenging nature of MATH, its problem-solving rate increases with more iterations, indicating continuous improvement in the model's performance.Secondly, there are also noticeable differences in the problem-solving rates between different models.Since the SFT model (MARIO) is fine-tuned on high-quality data, it exhibits the best performance. Furthermore, the math domain-specific pre-trained LLM (DeepseekMath-base-7B) significantly outperforms general-purpose pre-trained LLM (Llama3). This phenomenon is intuitive because domain-specific pre-trained LLMs possess more specialized knowledge. Finally, we note that in the third round, the solution rates of each model are quite similar. However, the final performance of the models differs significantly, as shown in Table 4 . This discrepancy can be attributed to variations in the quality of solutions generated by different models. Generally speaking, the more relevant knowledge is embedded within LLMs, the higher the quality of problem-solving solutions that will be generated. This explains why domain-specific pretrained models significantly outperform generalpurpose pretrained models. However, the advantage of our proposed AlphaMath lies in its ability to significantly enhance the performance of existing models without relying on high-quality data annotated by GPT-4 or humans. Even with a weaker general-purpose pre-trained model, AlphaMath achieves a remarkable +20 points improvement, as shown in Table 4 . Furthermore, our AlphaMath enables a domain-specific pre-trained model to achieve comparable or even superior results compared to state-of-the-art SFT models.The state s t is defined as a partial solution, consisting of the initial prompt (question) and all actions taken along the Monte Carlo tree search path from the root node to the current node, as shown in Eq. 1.Node Nodes are used to record information, such as the action (step) a t , the value predicted by the value model V ϕ , the state-action value Q from MCTS, depth, visiting counts, and etc. Each node is defined to only contain a single step.Action (Steps) Following Liao et al. [21] , we define two types of actions (steps) a t : C-steps and A-steps. Each node contains only one type of action, and A-steps typically appear at the end, as shown in Figure 14 . C-step represents code execution, which is composed of textual analysis, code snippets, and execution results. The textual analysis and code snippets are generated by the policy model (LLM), while the execution results are the outputs returned by the Python code interpreter. A-step represents the summary of the answer, which is composed of text analysis and predicted answers. Both the text analysis and predicted answers are generated by the policy model. We organize these two steps in the following XML format:<step>\n<p>\n{textual analysis}\n</p>\n<code>\n{code snippets}\n</code>\n<p>\n{code output}\n</p>\n</step> A-step <step>\n<p>\n{textual analysis}\n</p>\n<p>\nFinal Answer:{predicted answer}\n</p>\n</step>After solution generation via MCTS, we randomly sample the correct and incorrect solutions of each question for training. During this process, we found that the generated solutions might suffer from issues such as hallucinations [17] . Hence, we propose a solution filtering algorithm to optimize the solution selection.Algorithm 3 outlines the process of our solution filtering algorithm. We initially deduplicate and remove the solutions where code errors are present across all steps (Lines 3-5). As indicated in Figure 11 , the solutions that present code errors in all steps yet yield a correct final answer are if is_equiv(o, predict answer in s) then ▷ is_equiv is the evaluation toolkit [47] .12:flag ← True Add s to S c ▷ Level 3 Correct Solution evidently hallucinations. To preserve the diversity of incorrect solutions, we refrain from excessive processing, which aids the value model in being exposed to a wide variety of incorrect solutions (Lines 6-7). Furthermore, to filter out higher-quality yet diverse correct solutions, we categorize them into different levels. We consider the solutions where the predicted answer is consistent with the code output as level 1 correct solutions because the model has generated code and understood the output correctly (Lines 9-15). Then, we observe that for some questions, the quality of a solution cannot be solely gauged by the consistency between the code's output and the predicted answer (for which we provide an example in Figure 13 ). Therefore, we view the solutions with entirely correct code as level 2 correct solutions (Lines 17 -18) . Generating correct code to some extent proves that the model has a correct understanding of this question. Finally, we consider the remaining correct solutions as level 3 solutions (Line 20).In general, Algorithm 3 offers a method for swiftly filtering solutions. By categorizing solutions into different levels, we prioritize higher-level solutions for each problem. We provide several examples to intuitively illustrate the necessity and rationality of Algorithm 3, as shown in Figure 11 )) # Simplify the equation simplified_equation = simplify(factorized_equation) # Evaluate the equation evaluated_equation = eval(simplified_equation.str) # Solve the equation solution = solve(evaluated_equation, nu) # Find the sum of the solutions sum_of_solutions = sum(solution) print(sum_of_solutions) ``< /code> <output> AttributeError: 'bool' object has no attribute 'subs' </output> </step> similar error repeated in subsequent steps... <step> <p> From the result, we can see that there is an error. This is because the `subsf unction is not able to substitute the `lcm`function with the `factorintf unction. We need to find a different way to factorize the numbers in the equation. </p> <p> Final Answer: $126$ </p> </step> Figure 11 : An example of code errors persists across all steps. In this example, while the model was adjusting the code to rectify coding errors, each step of the solution generated an error. Nevertheless, in the final step, the model still produced the correct answer. This is a classic hallucination. Hence, we discard such solutions in Algorithm 3.For the solution generation via MCTS, we set c puct = 1.25, set the temperature between 0.6 to 1.2, limit the maximum tree depth to 8, and allow each node to expand 5 child nodes (B 2 = 5), In this example, the model generates the correct code and provides a correct answer based on the output of this code. We regard such solutions as dependable and categorize this type of solution as the level 1 correct solutions. and a maximum of 40 simulations (N = 40). Since MARIO [21] already has the ability to follow instructions, we set K = 2 for MARIO. For DeepseekMath-Base [31] and Llama3 [11] , we set K = 3. In each round, we build 10 trees for each question-answer pair and randomly sample at most 4 correct and 4 incorrect solution processes for training. In this setting, the ratio of positive to negative examples is approximately 1:1, and the count of positive examples varies between 57k to 59k for each round. For supervised fine-tuning, we set the learning rate of 4e-5, batch size of 1024, the weight of the value loss to 0.01 or 0.0005 (for Llama3 [11] ), and train the model for 10 epochs. We employ the AdamW optimizer [24] and the cosine learning rate scheduler with the warmup rate set to 0.03. More hyperparameter details can be found in Table 6 . For baselines, the results recorded in Table 2 come from corresponding published papers.Unlike previous work [49, 44, 12, 5] , the value model is trained separately to assist the policy model. In this study, we integrate the value model into the policy model by appending a linear layer, as illustrated in Figure 1 . Since most of the existing LLMs adhere to decode-only architecture, we utilize the last token as the representation of the entire reasoning step, similar to the role of "[CLS]" token in BERT [10] . In our case, it is typically "</step>", which ensures that the representation of the reasoning step will not be affected by the last token itself. The value model and the policy model share the majority of parameters, that is, they share the understanding of the reasoning steps. The value model assesses the expected returns based on the current reasoning step, while the policy model generates the next token. (c) An example of level 2 Correct Solution <question> How many $y$-intercepts does the graph of the parabola $x = 2y^2 -3y + 7$ have? </question> <step> <p> To find the $y$-intercepts of a parabola, we need to set $x = 0$ and solve for $y$. In this case, we have $x = 2y^2 -3y + 7$, so we need to solve the quadratic equation $2y^2 -3y + 7 = 0$. </p> <code> ```python from sympy import solve, symbols # define variables y = symbols('y') # solve the quadratic equation solution = solve(2 * y**2 -3 * y + 7, y) print(solution) ``< /code> <output> [3/4 -sqrt(47)*I/4, 3/4 + sqrt(47)*I/4] </output> </step> <step> <p> The quadratic equation $2y^2 -3y + 7 = 0$ has no real solutions, which means the parabola has no $y$-intercepts. We can also see this from the complex solutions for $y$ in the previous step. </p> <p> Final Answer: $0$ </p> </step> Figure 13 : An example of the level 2 correct solution. In this example, the consistency between the code's output and the answer does not adequately measure the quality of the solution. Therefore, we categorize solutions that are entirely correct in terms of the code as the level 2 correct solutions. In-Domain 7473 1319 MATH [15] In-Domain 7500 5000 GaoKao2023 [21] OOD -385 OCWCourses [19] OOD -272Table 7 describes the statistics of datasets in detail. The division of the training and test set follows the previous work [7, 15] . GSM8K [7] is a multi-step mathematical reasoning dataset comprising high-quality, diverse grade school math word problems, created by human problem writers. MATH [15] is a dataset of challenging competitive mathematics problems. GaoKao2023 [21] is a collection of mathematics problems from the 2023 Chinese National College Entrance Examination, the 2023 American Mathematics Competitions, and the 2023 American College Testing, while OCWCourses [19] comprises a collection of 272 STEM problems aimed at the undergraduate level, requiring multi-step reasoning for most questions.All experiments were conducted on Ubuntu 22.04 equipped with 8 * NVIDIA A100 GPUs. Our code mainly depends on Python 3.11 4 and PyTorch 2.1.2 5 . We use our customized Llama Factory [48] as the training framework and our customized vLLM [18] as the inference framework 6 . We trained all models with DeepSpeed ZeRO Stage2 [29] and Flash-Attention 2 [9] . The pre-trained language models are derived from HuggingFace7 .Solution Generation in Round 1 Figure 14 illustrates an example of solution generation on the MATH dataset via MCTS in round 1. We guide the pretrained model, such as DeepseekMath-Base [31] , to generate solutions in the form of Thought/Action/Action Input/Observation, as shown in Sec. F.1. For clarity of presentation, we only illustrate a subset of the nodes of the original Monte Carlo tree in Figure 14 . As shown in Figure 14 , the path (a) → (c) → (f ) represents a correct solution, whereas the other solutions contain errors to some degree. Node (b) attempts to solve the problem in a single step and proposes a correct thought. However, minor errors in the coding process ("k = 7 **2 * 3**3" was mistakenly written as "k = 7 * 3**3") led to mistakes in all subsequent steps. Node (d) attempts a different approach from node (c), specifically trying to solve for a first, then proceeding to solve for a 2 . Although this process is more redundant compared to that of node (c), it is nonetheless a correct approach. However, in subsequent steps, we encountered errors of various forms. Firstly, within the node (g), the model mistakenly treats the output for (d) as equivalent to a 2 , leading directly to an output. At node (h), the model opts to calculate a relying on its capabilities; however, this results in a numerical error. From a holistic perspective, we observe that, aided by MCTS, the pretrained model attempts to solve the problem through various approaches. During this process, we naturally excavate the knowledge embedded within the model, thereby reinforcing the model's understanding and application of this knowledge in subsequent training iterations. Furthermore, we collect the Q-values along the path to aid the model in enhancing its judgment of the correctness of the reasoning process.Solution Generation in Round 3 Figure 15 illustrates an example of solution generation via MCTS in round 3. Compared to round 1, the quality of the model in round 3 has significantly improved, resulting in more correct solutions. We observe that the code quality in round 3 is superior, containing more comprehensive annotations. This signifies an enhancement in the model's understanding of the question and its coding proficiency. Furthermore, as illustrated by node (b), the model attempts to resolve the question in a single step, yielding solutions that are both shorter and more efficient.<question> Suppose that $a^2$ varies inversely with $b^3$. If $a=7$ when $b=3$, find the value of $a^2$ when $b=6$. </question> <step> <p> We are given that $a^2$ varies inversely with $b^3$. This means that there is a constant $k$ such that $a^2 = \\frac{k}{b^3}$. We are also given that $a=7$ when $b=3$. We can use this information to find the value of $k$. </p> <code> ```python a = 7 b = 3 k = a**2 * b**3 print(k) ``< /code> <output> 1323 </output> </step> 𝑃 = 0.94 𝑄 = 0.30 <step> <p> To find the value of $a^2$ when $b=6$, we can use the fact that $a^2$ varies inversely with $b^3$, which means that $a^2$ is proportional to $1/b^3$. We can use the given value of $a$ when $b=3$ to find the constant of proportionality. Then, we can use the constant of proportionality to find the value of $a^2$ when $b=6$. Therefore, the value of $a^2$ when $b=6$ is approximately $2.4748^2 = 6.12293625$. </p> <p> Final Answer: $6.12293625$ </p> </step> 𝑃 = 0.87 𝑄 = -1.0 <step> <p> From the result, we can see that the value of $a^2$ when $b=6$ is $0.875$. </p> <p> Final Answer: $0.875$ </p> </step> In round 1, due to the value head being randomly initialized, the estimated values are not accurate; therefore, we have not presented the estimated value here. For the sake of clarity in our demonstration, we only display a subset of the original Monte Carlo tree and present each node in XML format (as detailed in Appendix F.2), even though the format utilized in round 1 was Thought/Action/Action Input/Observation (as detailed in Appendix F.1).EQUATIONIn this section, we conducted a comprehensive error analysis on the MATH dataset to foster future work. We randomly select 100 error solutions for analysis. We find the following three types of errors.(1) Numerical Errors (53%). This is one of the main sources of errors. Within this category of errors, the model usually possesses the correct thought to solve the problem, but an error occurs during the numerical computation. The most common types of numerical computation errors arise from incorrect code generated by the model, which we will not discuss in detail. We have identified errors caused by numerical precision issues in Python, which we refer to as "numerical precision errors". Here is an example:Question: The first two terms of a sequence are $a_1 = 1$ and $a_2 = \frac{1}{\sqrt3}$. For $n\ge 1$, \begin{align*} a_{n + 2} = \frac {a_n + a_{n + 1}}{1 -a_na_{n + 1}}. \end{align*}What is $a_{2009}$? <question> Suppose that $a^2$ varies inversely with $b^3$. If $a=7$ when $b=3$, find the value of $a^2$ when $b=6$. </question> <step> <p> To solve this problem, we need to establish an equation based on the given information. Let $k$ be the constant of variation. Since $a^2$ varies inversely with $b^3$, we have $a^2 = \\frac{k}{b^3}$. Using the given values $a=7$ and $b=3$, we can solve for $k$. </p> <code> ```python # a^2 varies inversely with b^3 # if a=7 when b=3, find the value of a^2 when b=6 # establish equation k = 7**2 * 3**3 print(k) ``< /code> <output> 1323 </output> </step> 𝑃 = 0.99 𝑄 = 0.91 𝑣𝑎𝑙𝑢𝑒 = 0.67 <step> <p> Let $k$ be the constant of variation. Since $a^2$ varies inversely with $b^3$, we have the equation $a^2 = \\frac{k}{b^3}$. We are given that $a=7$ when $b=3$, so $7^2 = \\frac{k}{3^3}$ which implies $k = 7^2 \\times 3^3$. Therefore, the equation for $a^2$ in terms of $b$ is $a^2 = \\frac{7^2 \\times 3^3}{b^3}$. The actual answer: 0The thought of LLM: To find $a_{2009}$, we need to calculate the first 2009 terms of the sequence. However, the sequence is defined recursively, so we need to find a way to calculate the next term based on the previous two.The code written by LLM: import math a1 = 1 a2 = 1 / math.sqrt(3) for i in range (2, 2009 ): an = a2 an_1 = a1 a2 = (an + an_1) / (1 -an * an_1) a1 = an print(a2)The output of above code:-2.7267039150707353In this example, the model possesses the correct thought of problem-solving and generates the correct Python code. But an incorrect answer was still produced. To ensure numerical precision, we found that the correct implementation should simplify the equation using "sympy", as shown below: import sympy as sp a1 = 1 a2 = Justification: In Appendix A, we discuss in detail the potential positive and negative social impacts of our work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Reward is 0 for non-terminal node, and reward is determined by the final answer in terminal node.https://www.python.org/https://pytorch.org/We have released our customized framework in our Github Repository.https://huggingface.co/ 


Decision: The paper tackles solving mathematical reasoning with LLMs. The key idea is to leverage Monte Carlo Tree Search (MCTS) to sample high-quality process-supervised data, which currently existing works primarily rely on expensive human annotation or GPT-4. Moreover, they propose an efficient inference strategy called step-level beam search, which leverage value models trained from the generated process supervised data. Altogether demonstrates significant improvements on various datasets, showing comparable or even superior results to existing state-of-the-art methods without relying on external process annotations.

The reviewers all gave borderline scores. Based on my assessment, most of their raised weaknesses have been addressed by the author during rebuttal. However very few engagement during and post rebuttal happened, despite multiple reminders from the AC. One of the reviewers giving a rating of 4 agreed that the paper was "very close to the threshold" and "a valuable study in this area" if the authors can properly address their concerns, which, to me, they did.

I therefore recommend accepting despite the slightly low average score.

Reviews: review: paper_summary: paper_summary: This paper utilizes Monte Carlo Tree Search (MCTS) to sample high-quality process-supervised data for iterative training, effectively reducing the dependency on GPT-4 and thus lowering the associated costs. Additionally, the authors propose an inference strategy, step-level beam search, which leverages value models to enhance the efficiency and effectiveness of the tree search process.
strength_weakness: strength_weakness: 1. The comparison in the experimental results appears to be rather unfair. Generally, more computation leads to higher performance. In Table 2, the authors compare their fine-tuned model using SBS with greedy decoding for other open-sourced models. In Table 3, they compare SBS with majority voting. It is possible that the improvement stems from the reward model rather than the SBS algorithm itself. The authors should also report the results of best-of-n or weighted majority voting using their value model to demonstrate the effectiveness of SBS.
2. The experimental results are not detailed enough in Tables 2 and 3. The authors should also include major@n with different n values. Perhaps they can plot figures with $B_1$ and $n$ as the x-axis to provide a more comprehensive view.
questions: questions: 1. As mentioned in the weaknesses, what are the results for best-of-n or weighted voting? Additionally, how does the accuracy change as n increases during majority voting?
2. How will the performance change when separating the policy model and value model as two totally different models?
limitations: limitations: Yes
paper_summary: paper_summary: The paper introduces a novel approach leveraging the Monte Carlo Tree Search (MCTS) framework to generate process supervision and step-level evaluation signals automatically, thus enhancing the mathematical reasoning capabilities of LLMs. This method bypasses the need for costly and labor-intensive manual annotations by allowing the models to train iteratively on both policy and value aspects, with an efficient inference strategy called step-level beam search. This approach demonstrates significant improvements on various datasets, showing comparable or even superior results to existing state-of-the-art methods without relying on external process annotations.
strength_weakness: strength_weakness: 1. Duplicate citations, such as [6], [7]
2. Appendix Figure 9, the reason why the performance of the third round model in MATH at Level 1 and Level 3, Counting & Probability, Number Theory and other categories has declined. Whether it will continue to decline after more rounds. 
3. The method integrates CoT and Code for cross-use. How does the performance of the model change if only CoT is used?
4. Evaluate on more out of domain benchmarks, such as GSM8k_hard and GSM8k_robust and OpenLLM Leaderboard
5. How much total training data is used. Explore the effect on other base model, such as Llama2, Mistral
6. Explore the effect of different values of the hyperparameter β in Eq. (8). 
7. what is the change in loss of Policy model and Value model during training.
8. What's the accurate of the Value model predictions based on the current State and Action.
9. Insufficient introduction of related work, such as Mathematical Reasoning and MCTS.
10. Differences reasoning ability and solution generation process across different rounds for the policy model , as well as the scoring differences by the Reward Model.
11. How the ground truth label scores for training the first round of the Value Model are obtained, and the procedures for the second and third rounds.
questions: questions: No question besides the one raised in weakness.
limitations: limitations: Yes
paper_summary: paper_summary: This paper leverages the Monte Carlo Tree Search (MCTS) to iteratively train policy and value models by automatically generating process supervision and step-level evaluation signals, eliminating the need for human-annotated process supervision data. Specifically, the method combines the inner capabilities of pre-trained LLMs with the MCTS framework to generate correct and incorrect solution paths and optimize the models based on the node evaluations along these paths. To improve inference efficiency, the paper also proposes a step-level beam search strategy that allows the value model to assist the policy model in navigating more effective reasoning paths rather than relying solely on prior probabilities while avoiding excessive time consumption. Experimental results demonstrate that even without GPT-4 or human-annotated process supervision, this paper' s method performs comparably or better than existing SOTA methods across multiple in-domain and out-domain datasets.
strength_weakness: strength_weakness: 1. Although the method does not rely on human-annotated process data, it still requires actual answers as reward signals. Future work could explore fully unsupervised approaches. 
2. Although the step-level beam search improves efficiency, the MCTS method still has some limitations in terms of computational complexity, which makes its practicality indistinguishable from majority voting.
questions: questions: N/A
limitations: limitations: N/A
paper_summary: paper_summary: The authors propose the AlphaMath method, which leverages Monte Carlo Tree Search (MCTS) to improve the mathematical reasoning capabilities of large language models (LLMs) without requiring expensive human or GPT-4 annotations for process supervision. The framework integrates a value model with the LLM to generate and evaluate reasoning steps autonomously. Experimental results demonstrate that AlphaMath achieves comparable or superior performance to baseline methods on both in-domain and out-of-domain datasets.

To AC and the authors: The detailed comments are mainly covered in the **Questions** section. Please refer to the section for details. Generally speaking, I would consider the paper as very close to the threshold. If the authors can properly address the concerns in the below section, this could be a valuable study in this area.
strength_weakness: strength_weakness: Many presentations in the paper are somewhat unclear, requiring further clarification and elaboration. There are also some minor over-claims in the introduction. There are many modifications comparing to the original AlphaGo and Zero paper but the authors typically do not provide explanations for such changes. Some tables and figures are confusing. Please see the **Questions** section for the details.

One thing I'd like to especially point out is, the title "almost zero" is kinda misleading. One would interpret it as using very little data, e.g., a few seed data. However, the method would still need the full training set of GSM8K/MATH.
questions: questions: 1. The authors use Table 1 and corresponding statements in the main body (line 25) to claim that *the annotation of mathematical solutions primarily relies on domain experts or GPT-4 in current efforts*. This claim is inaccurate, as multiple studies employ methods such as the Monte Carlo method [1,2] or even Monte Carlo Tree Search (MCTS) [3,4,5] to automatically annotate mathematical solutions. It is totally understandable that some related studies might have been missed, especially those posted after NeurIPS submission [3,5]. However, there are studies [1,4] published last year that should have been properly considered. I recommend that the authors revisit their claims and appropriately cite and discuss works that do *annotate mathematical solutions without human intervention or GPT-4*. This paper already presents numerous promising contributions, which I am very impressed; therefore, it is advisable to carefully avoid over-claiming in minor aspects that may negatively affect the first impression of readers and researchers familiar with this field.
2. In line 76, the authors write *the solution process can be broken into multiple reasoning steps (e.g., segmenting the solution based on distinct stages or simply on a period).* What exact step segmentation strategy do the authors use in this study?
3. In line 118, the authors write, *we employ sampling generation with higher temperature to ensure diversity* in generating actions for a step. This method requires further elaboration. From my understanding, it is non-trivial for an LLM to generate only a single mathematical step with early stopping. If the authors generate an entire solution and truncate it to retain only the subsequent step, this approach would be computationally expensive.
4. In line 125, the authors write *we follow a trade-off rollout strategy between AlphaGo and AlphaGo Zero. Because our tree depth is much shallower than Go games (e.g., a maximum depth of 8).* What is the "trade-off rollout strategy between AlphaGo and AlphaGo Zero"? I don't personally remember such a concept in the AlphaGo work series; and why your tree has a maximum depth of 8? There should have been many solutions with more steps for the MATH dataset.
5. In line 135, *assuming that ...*. Why this assumption is valid? Please elaborate.
6. In Eq. (6), the the loss terms for the value model is minimizing the difference between Q and V. However, in AlphaGo Zero it is minimizing the difference between V and the final reward z (1/-1 indicating the final winner at the termination state). The authors do not provide an explanation of this modification. Please elaborate.
7. In line 164, *This selection is guided by the maximum Q-value*. In AlphaGo and AlphaGo Zero, the selection at inference time is guided by the visited count. Using Q is also a valid choice, but there should be clarification for this modification. An optional ablation study would be better.
8. In Table 3, the SBS with $B_1=2 \text{or} 3$ has shorter average time than $B_1=1$. This looks weird. I guess it comes from the less number of steps. But why $B_1=2 \text{or} 3$ has less number of steps then? Also, the "discussion of majority voting" is really confusing. I still don't understand why they are not comparable after reading it.
9. In Figure 5, where is the Q-value sampled from? For full solution only or also including intermediate of steps? Also why there are three curves in the right figure while there are only two legends?

Ref:

1. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. https://arxiv.org/abs/2312.08935
2. Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision. https://arxiv.org/abs/2402.02658
3. Improve Mathematical Reasoning in Language Models by Automated Process Supervision. https://arxiv.org/abs/2406.06592
4. Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training. https://arxiv.org/abs/2309.17179
5. Step-level Value Preference Optimization for Mathematical Reasoning. https://arxiv.org/abs/2406.10858
limitations: limitations: The authors provide a limitations section (A.1); however, it omits several critical aspects:

1. Methods that use the final answer's correctness to infer the reward or value of intermediate steps are prone to false positive and negative issues. Specifically, when the final answer is correct, all intermediate steps are rewarded to some extent. Nevertheless, false positive cases occur when models luckily guess the correct final answer despite incorrect reasoning. Previous related studies, such as Math-Shepherd and OmegaPRM, have acknowledged this limitation. It is recommended that this paper also addresses it in the limitations section.
2. The method relies on an automatic verifier to determine the correctness of the final answer. This verifier is not always applicable to many tasks, such as open-ended text generation (e.g., translation and summarization). The authors did not clearly explain the limitations concerning the scope of tasks.
paper_summary: paper_summary: The paper proposes an innovative framework named AlphaMath, aimed at enhancing the mathematical reasoning capabilities of large language models (LLMs) without relying on expensive human annotations from domain experts or GPT-4. The authors leverage Monte Carlo Tree Search (MCTS) to allow a pre-trained LLM to autonomously generate process supervision and step-level evaluation signals, integrating a value model with the LLM. At the inference time, the authors propose a step-level beam search strategy, which assists the policy model in navigating more effective reasoning paths. Experimental results demonstrate that AlphaMath achieves comparable or superior results to previous state-of-the-art methods on various datasets without process supervision.

The application of MCTS in LLMs is now somewhat popular, with several highly relevant papers, such as [1] https://arxiv.org/pdf/2406.06592, [2] https://arxiv.org/pdf/2405.00451 and [3] https://arxiv.org/pdf/2309.17179] (note that [1, 2] are published after neurips deadline, so no need to perform comparison, It's just added for reference.)
strength_weakness: strength_weakness: 1. The paper's results are all program-aided, a fact not clearly identified in the text. Consequently, the comparison with non-program-aided LLMs may not be fair enough.
2. The benchmark only includes GPT models in the 'Proprietary Models' section. It would be beneficial to include other LLMs such as Gemini and Claude for a more robust evaluation. 
3. An image illustrating the Step-Level Beam Search (SBS) would greatly aid in explaining the algorithm.
questions: questions: Check other sections.
limitations: limitations: 1. The framework does not generate a reward model that can facilitate other LLMs, which potentially limits its applicability.
2. The framework cannot be extended to tasks with open-ended outputs, limiting its potential use cases.

Comments: title: Remain the rating
comment: I have read the authors' response. It addressed some of my concerns. So I will maintain my current score.
I've read the authors' responses and am satisfied with the responses to my questions. I've also read the comments from other reviewers. Based on all these, I decide to keep the rating.
Thanks for your valuable response.

> Our training method is categorized as almost unsupervised / weakly-supervised learning. The relatively small gap between an almost unsupervised method and a supervised method is noteworthy and underscores the potential of our approach.

Thanks for your suggestion about [1]. We will cite this paper in our related works, and categorize our work as bootstrapping reasoning.

> More n for maj@n. Could the authors provide more details on the experimental settings?

We directly use the open-sourced `vllm` inference framework by setting the number of generations `n=20`. The parallelism is implemented by the vllm. We think the reason why the inference time of `n=20` is $5\times$ longer than `n=5` is that the Python code execution is on CPU rather than GPU. As we mentioned, we use the LLM REACT agent, including both text+code generation (GPU parallel) and code execution (CPU parallel).

> What are the results of weighted voting and best-of-n? I believe a fair baseline should also incorporate the value/reward model during inference.

In response of W1, we mentioned that as illustrated by the SBS (step-level beam search) inference in Algorithm 2, our results are already presented as best-of-n, but on a step-level.

The value loss (the 2nd and 3rd rightside terms in Eq. (6)) indicates that the value model is trained at the step-level. During inference, for example, in a setup of 
 the SBS needs to choose the best out of 5 generated outputs at each step, using the step-level score prediction provided by the value model. In summary, the value model and SBS inference algorithm are highly integrated to improve the performance, by selecting step-level best-of-n.

MCTS is a baseline that has already incorporate the value model.

Since our **value model is step-level** in the setup of LLM REACT agent, for a fair comparison with incorporating the value model, we just implemented the step-level weighted voting based on our submitted code. For each step, we first sample 5 candidates, and select the step with largest weight summation. Then, we proceed to the next step.

|method|Accuracy on MATH|
|--|:--:|
|step-level best-of-5, i.e. SBS(1,5)|62.32|
|step-level weighted voting of 5, variant of SBS(1, 5)|61.94 (run 1), 62.36 (run 2)|

Given our step-level value model, there is not significant difference of above two setups, demonstrating the robustness of our step-level value model and SBS inference algorithm.
Thank you for you rebuttal.

> Our training method is categorized as almost unsupervised / weakly-supervised learning

It doesn't seem right to claim training on question-answer pairs as unsupervised or weakly-supervised learning, though it's a good setting for bootstrapping reasoning [1].

> The relatively small gap between an almost unsupervised method and a supervised method is noteworthy and underscores the potential of our approach.

DeepSeek-Math base model is already trained on some instruction following data and achives 66.9 GSM8K and 31.4 MATH simply by prompting. But I agree that achiving 73.5 GSM8K and 53.6 MATH after RL training on 15K question-answer pairs is still impressive.

> More n for maj@n.

Thank you for the results. I'm surprised that your $n=20$ majority voting run takes $5\times$ longer than the $n=5$ run. I would have expected the $n=20$ run to take only $2-3\times$ longer due to GPU parallelism. Could the authors provide more details on the experimental settings?

> Q1. We have addressed the relevant issue in the response for Weakness 1 and 2.

What are the results of weighted voting and best-of-n? I believe a fair baseline should also incorporate the value/reward model during inference.

> Q2: Performance of separating policy and value models.

Thanks for the results.

[1] Zelikman, Eric, et al. "Star: Bootstrapping reasoning with reasoning." Advances in Neural Information Processing Systems 35 (2022): 15476-15488.
Dear AC and reviewer:

Sorry for replying late, I have read the rebuttal and the comment. I will keep my rating. Thanks!
Thank author for the rebuttal. I've read through all the comment. I will continue discuss with AC and the other reviewers for the final decision. There's no further question for now.
Dear reviewer,

Can you let the author and us know if you've read the rebuttal, and if you have any further comments?

Thanks,

AC
Dear reviewer,

Can you let the author and us know if you've read the rebuttal, and if you have any further comments?

AC
Dear reviewer,

Can you let the author and us know if you've read the rebuttal, and if you have any further comments?

Thanks,

AC
Dear reviewer,

Can you let the author and us know if you are satisfied or not with their rebuttal, and if you have any further comments?

Thanks,

AC
Dear author,

Can you let the author and us know if you've read the rebuttal, and if you have any further comments?

Thanks,

AC
Dear Reviewer 15WC,

Although we have made responses to your W1 in `G1 of general rebuttal`, we would like to clarify again the difference between our work and the two works[1,4] you mentioned.

For **Alphazero-like[4]**, the training of policy and value models is entirely independent of MCTS. As shown in the appendix E.2 of [4], they collect multiple paths through rejection sampling, and train the value model through Monte Carlo (MC) estimate, TD-$\lambda$, or PPO. As mentioned in `G1 of general rebuttal`, their training of value model has nothing to do with MCTS. In contrast, our training falls within the true AlphaZero framework (MCTS + no annotation). As mentioned in your several questions (W6, W7), compared to AlphaZero, we have introduced some modifications to enhance the stability of AlphaMath's training. It is important to note that these adjustments are not modifications in the sense of traditional RL, but rather commonly used strategies within reinforcement learning (RL) and MCTS [r1,r2].

So the core of **Alphazero-like[4]** is to propose an independent value model to help the policy model perform better reasoning through MCTS inference, arather than training. Moreover, their MCTS inference is very inefficient. As shown in `Lines 168-172` of our paper, we have modified the MCTS inference and proposed a more efficient step-level beam search algorithm. In addition, our value model and policy model share most parameters, which significantly reduces training and inference costs.

For **Math-Shepherd[1]**, this work is even more remote from our work. The core of this work is to train a value model for reranking output. The form of collecting training step-level value siginals is Monte Carlo (MC) estimate, which is very inefficient and has high variance. They may also apply PPO for training, which is another far different RL algorithm with MCTS.

In summary, Math-Shepherd and Alphazero-like are similar works. They aim to train a value model to help the policy model to better reason (Verification ranking for Math-Shepherd, MCTS inference for Alphazero-like). However, the learning of step-level value function through MC estimate, TD-$\lambda$ or PPO, which is fundamentally different from our work.

We hope this clarification alleviates any concerns you may have had, and we look forward to engaging in further constructive discussion with you.

Sincerely,

Authors of 6802
Dear Reviewer 5iMV,

Thank you for your continued engagement.


In `Lines 199-202 and 682-684` of our paper, we provide a detailed introduction to this dataset. Specifically, OCWCourses (OCW) focuses on STEM (Science, Technology, Engineering, and Mathematics) problems, particularly emphasizing calculations in physics, chemistry, and biology, which present a high-quality out-of-domain dataset and greater challenges compared to MATH. This dataset is sourced from [1]. The dataset  [**link**](https://huggingface.co/datasets/zhangirazerbayev/ocwcourses) on huggingface hub. As the DeepSeek uses this testset and one of our main models is built upon DeepSeek pretrained base LLM, we also use this testset for a fair comparison.

We hope this clarification addresses your concerns.

Additionally, do you still have questions about your previous concerns? We would appreciate it if you could share any further questions you may have. We look forward to the opportunity to enhance your understanding of our work through continued dialogue.


[1] Lewkowycz, Aitor, et al. **Minerva: Solving quantitative reasoning problems with language models.** Advances in Neural Information Processing Systems 35 (2022): 3843-3857.

Sincerely,

Authors of 6802
Thank you for your reply!

I am curious about the OCW dataset you used, can you provide some information about it and maybe a link to its source?