Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
Authors: Ye Tian
Baolin Peng
Linfeng Song
Lifeng Jin
Dian Yu
Lei Han
Haitao Mi
Dong Yu
Venue: NeurIPS 2024 poster (2024)
Keywords: self-improving
search
large language models
Primary area: natural_language_processing

Abstract: Despite the impressive capabilities of Large Language Models (LLMs) on various tasks, they still struggle with scenarios that involves complex reasoning and planning. Self-correction and self-learning emerge as viable solutions, employing strategies that allow LLMs to refine their outputs and learn from self-assessed rewards. Yet, the efficacy of LLMs in self-refining its response, particularly in complex reasoning and planning task, remains dubious. In this paper, we introduce AlphaLLM for the self-improvements of LLMs, which integrates Monte Carlo Tree Search (MCTS) with LLMs to establish a self-improving loop, thereby enhancing the capabilities of LLMs without additional annotations. Drawing inspiration from the success of AlphaGo, AlphaLLM addresses the unique challenges of combining MCTS with LLM for self-improvement, including data scarcity, the vastness search spaces of language tasks, and the subjective nature of feedback in language tasks. AlphaLLM is comprised of prompt synthesis component, an efficient MCTS approach tailored for language tasks, and a trio of critic models for precise feedback. Our experimental results in mathematical reasoning tasks demonstrate that AlphaLLM significantly enhances the performance of LLMs without additional annotations, showing the potential for self-improvement in LLMs.

Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
Ye Tian 1,2 Baolin Peng'; Linfeng Song'; Lifeng Jin', Dian Yu', , Lei Han
Baolinn
Haitao Mi' Dong Yu'
Tencent AI Lab, Bellevue, WA
Tencent Robotics
(baolinpeng, lfsonggnnn.... wifengjin,,.. yudiann ieriiiannnnaati haitaomi, dyg)) tencent. Cyaptian, 1xhan] Steccent.coeeetco com
Abstract
Despite the impressive capabilities of Large Language Models (LLMs) on vari- ous tasks, they still struggle with scenarios that involves complex reasoning and planning. Self-correction and self-learning emerge as viable solutions, employing strategies that allow LLMs to refine their outputs and learn from self-assessed rewards. Yet, the efficacy of LLMs self-refining its response, particularly complex reasoning and planning task, remains dubious. In this paper, we introduce ALPHALLM for the reffvimproveeetss of LLMs, which integrates Monte Carlo Tree Search (MCTS) with LLMs to establish self-improving loop. thereby enhanc- ing the capabilities of LLMs without additional annotations. Drawing inspiration from the success of AlphaGo, ALPHALLM addresses the unique challenges of combining MCTS with LLM for self-improvement, including data scarcity, the vastness search spaces of language tasks, and the subjective nature of feedback in language tasks. ALPHALLM is comprised of prompt synthesis component, an efficient MCTS approach tailored for language tasks, and trio of critic models for precise feedback. Our experimental results in mathematical reasoning tasks demon- strate that ALPHALLM significantly enhances the performance of LLMs without additional annotations, showing the potential for self-improvement in LLMs. The code is available at https: /tttps://githan github....... comm/YYTTiaaJJU/Aphaa...
Introduction
LLMs, trained on trillions of tokens with billions of parameters have shown unparalleled capabilities
in a wide range of natural language processing tasks (Touvron et al. 2023b Team et al.) 2023
OpenAl 2023). Nevertheless, they continue to face challenges in scenarios requiring complex
reasoning and strategic planning (Valmeekam et al. 2022] Stechly et al., 2024). While advanced
prompting approaches such Chain, Tree, Graph-ot-Thouge (Wei al. 20022] Yao al. 2024
Besta et al. 2024; Ding et al. 2023), remains essential to fine-tune LLMs using a substantial volume of high-quality, supervised data fundamentally improve the model performance (Nye et al.)
2021 Lewkowycz et al.) [022] Chung et al. (022). This methodology is inherently limited by the scope and quality of data that humans can provide.
Considering these challenges, the concept self-correction and self-learning have been proposed promising solutions (Madaan et al. 2024 Saunders al., 2022 Chen et al.) 2024). Within these framework, LLMs typically operate by employing two main strategies: 1) they continuously refine
"Equal Contribution; iCorresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Imagination
Searching
Criticizing
Sentheslaed
Real Data
LLM
Cusco Raward
Improving
Figure I:
Imagination-Searching Srarchinn-Critciizinn seff-improvement loop: Imagination component synthe- sizes prompts new learning examples, with MCTS searching better trajectories guided by signals from critics for policy improving.
their responses based on the feedback of their past responses, and 2) they extensively sample responses then learn from preferences judged by itself as reward models with PPO or DPO (Yuan et al. 2024a bj Chen et al. 2024). However, it remains matter of ongoing research whether LLMs can effectively critique their own outputs to either enhance response quality or apply a scalar reward to indicate the quality of responses, especially in contexts demanding intricate planning and reasoning (Valmeekam
(ct al., 2022; Stechly et al.) 2024 Huang et al. 2023 Hong et al. 2023). On the other hand, advanced search algorithms such as MCTS, combined with reinforcement learning, have enabled models to learn from self-play and achieve human parity or even surpass human performance in complex tasks such the game of Go (Silver al.] 2016 2017). This naturally raises question: is it viable to leverage the strengths of MCTS alongside LLMS to inaugurate a novel paradigm of self-improving? More precisely, could the assimilation of MCTS empower LLMs more effectively explore better responses, guided by strategic signals, and subsequently optimize these responses to enhance overall performance?
To answer this question, we begin with a systematic examination of AlphaGo, identifying three critical aspects for its success: (i) The large volume of data, including self-play data. (ii) The use of tree search, which facilitates the exploration of potential moves through statistical sampling of the large search space. (iii) Accurate and unambiguous environment feedback; the direct and accurate feedback (win or loss) provided by the game of Go offers a clear and unequivocal learning signal (Silver et al. 2017). The integration of MCTS with LLMs for self- seff-improvement has several challenges: (i) Limited Data: High-quality annotated data for LLMs is generally scarce. Furthermore, how to construct of synthetic data for LLMs training, similar to AlphaGo's o's self-play data, remains unclear. (ii) Search Efficiency: The vast number of potential token combinations in natural language tasks results in an exponentially large search space, posing significant challenge to the efficiency of MCTS (Ramamurthy et al., (2022). (iii) Imperfect Feedback: In contrast to the clear win/loss feedback in Go, feedback in natural language tasks is often subjective and nuanced, without a straightforward measure of success.
In this paper, we introduce ALPHALLM, an for the self self-improvement of LLMs. ALPHALLM consists of three key components, as illustrated in Figure D First, imagination component designed to synthesize prompts, alleviating the issues of data scarcity. Second, we propose 1yMCTS tailored for efficient searching in language tasks. Particularly. it has been show that planning multiple levels of temporal abstraction is critical for RL.
framework designed
problems with a long horizon and large action space (Sutton et al. 1999b Peng et al. 2017] Luketinal al. 2019). As such, we propose formulating the text generation process as options over a Markov Decision Process (MDP) problem, where each option represents the generation of a collection of tokens for a specific subtask, similar to the concept of chains in chain-of-thougee formulation improves search efficiency by substantially reducing the search depth. Additionally, we propose the use of state merge and adaptive branching factors further enhance search efficiency by balancing the orf between search width and depth. Lastly, since accurate feedback is crucial to the success of MCTS, we introduce a trio of critic models to guide r/MCTS, including a value function for estimating expected rewards, a process reward model for assessing node correctness, and an outcome reward model for evaluating the overall trajectory. For complex tasks with which LLMs struggle assessing such as arithmetic computation and code execution, to ensure the accuracy
prompting. This

---

of feedback, we augment the critics with the capacity to make dynamic decisions on which tools to use, when to use them, and how to use them effectively. After 1MMTS stage, we collect the trajectory with the largest reward from the critic models as the training examples to improve LLMss.
The experimental results on mathematical reasoning tasks demonstrate that ALPHALLM ciently search for better responses and use them to improve LLMs' performance, forming an effective self-improving loop. Notably, based on Llama-2-70b and Moarrd---t--7--- ALPIALLM can improve its performance from 57.8 to 92.0 on GSM8K and from 20.7 to 51.0 on MATH, performing comparably to GPT-4
effi
2 Related Work
Search with LLM Effective search strategy has been shown crucial for tasks that involve complex reasoning and planning, such as go (Silver et al., 2016) and math reasoning (Cobbe et al. 2021 Hendrycks al., (021). For math reasoning tasks, various search methods have been studied. One direction of research (Zhu et al. 2024; Xie et al. 2024) designed beam search with dynamic pruning,
where beam items of low quality
pruned. Another line of work (Yao al., [024 Long 2023
Besta et al. 2024 Hao et al. 2023 Feng et al. (023) maintains a tree or a graph that represents the current, progress of solving the imput question where potential branches are iteratively expanded. Both our approach and Feng et al. (2023) are based on the MCTS algorithm, while one main difference is how to define a search step: Feng et al. (2023) fix search step to be either a token or a sentence, while our approach is more flexible on deciding steps. We have also carefully designed the MCTS process, incorporating multiple critique signals to guide the search more effectively and introducing adaptive search parameters for improved state exploration. As the result, our approach achieves much better performances.
LLM Self-improving Being key to the success of scalable oversight (Bowman et al. 2022). self-improving for LLM aims align the LLM to human preference and values mainly using the supervision from the knowledge inside the LLM Zelikman et al. 2022 2024). One crucial part of self-improving is how to obtain reliable signal of critique to distinguish between good responses from the LLM and bad ones. Initial work (Bai et al., 2022: Wang et al. (022) first asks the LLM to generate input queries of diverse tasks and the corresponding outputs. They then rely on hand-crafted heuristic rules to filter out redundant or low-quality data pairs (e.g. the query is too long or too short). Since it is non-trivial to compose effective hcuristic rule, later work (Sun et al., 2023 Li et al.) 2023 Guo et al. 2024) proposes a few general principles judging criteria and ask the LLM itself to evaluate the quality its responses based on these guidance, hoping that LLMS can automatically designate these principles into each data point to better guide data filtering. However, this requires LLMs to have strong abilities to apply these principles for each specific case and make correct judgements. Different from previous work, we propose to leverage the supervision from MCTS for LLM self-improvement taking the outputs of MCTS to continue train the LLM. This is because the outputs from MCTS are usually in much better quality then standard nucleus sampling, and the large gap ensure that the LLM can self improve.
3 Preliminaries
3.1 Problem Formulation
In this paper, we consider L.LM characterized by probability pa and denoted as policy mo. It takes a sequence x = [21, -..,... ................................ In. as input, which is typically referred as prompt, to generate the response y y= [yr,... , y. In the context of LLMs, each x; and ys represents a token from pre-defined vocabulary. The policy Te operates in an autoregressive manner, where each token is generated sequentially, relying solely on the context provided by the previously generated tokens. The policy therefore constitutes a Markov process in which the conditional probability distribution Po(y) (z) can be decomposed and expressed with the chain rule as po ())x II
po (ys [x. yei).
With this property, the text generation task can be formulated as an Markov Decision Process (MDP) problem consisting of A, T, R. 7) in which, 8, E S represents the context information of current trajectory, i.e., current status of the generation process, e.g., a partial response to a prompt; o, E A denotes a single action sampled token from the vocabulary, leading to a transition to new state
3

---

81+1, by concatenating 8 and az: rt R(s;, ai) manifest the evaluation of the generation to the prompt, reflecting the desirability or preferences of each state-action pair.
This MDP framework sets the stage for applying Reinforcement Learning (RL) methods to optimize the policy no aiming maximize the expected cumulative reward R. Base on these setups, describe the celf-improving problem. Given LLM Te and an initial dataset D9, which consists of N expert-generated prompt-response pairs { {<x) y/)
[N]], the goal of self-improving is to iteratively refine to maximize the reward. The refinement process includes learning from synthesized prompts and corresponding responses. These responses are obtained using an advanced search algorithm that navigates the space of possible responses to maximize the expected reward. The detailed process is described in Algorithm[[]] in Appendix. The primary challenges forming an effective self-improving loop lie in synthesizing suitable prompts, efficiently searching over a vast action space, and obtaining precise feedback, which will discussed in
3.2 Monte Carlo Tree Search
MCTS is a sampling caapling-based based search algorithm for policy optimization in decision decision-making problems. would iteratively build search tree, by repeating four phases: selection, expansion, evaluation, and backpropagation. In the selection phase, it would recursively select the children from the root node by Upper Confidence Bound (UCB) (Auer et al. 2002). UCB(i) = w, + C'* /2 * In N, where n, and N; the visit counts for the node and its parent respectively, C' represents hyperparameter balancing exploration and exploitation, and the W; is the average value of all descendant nodes of
ALPHALLM
4.1 Overview
The architecture of ALPHALLM is depicted in Figure comprising three key components. Firstly, the imagination component is tasked with synthesizing prompts learning examples. Secondly, an efficient search component, named r/MCTS, is proposed to search high-quality trajectories for optimizing the policy. Lastly, the search process is guided by critics specifically designed to provide reliable signals.
4.2 Data Synthesizing
Let D = {xx, y:) | E [N]] denote the initial dataset consisting of N expert-generated prompt- response pairs. The data synthesizing process aims to expand this dataset by generating a set of synthesized prompts DD = [
i E [N]]. The generation of each synthesized prompt a can be mathematically described as transformation g applied to one or more examples from DO x²) T where =...... x²) are selected examples from DD. The transformation function y controls the synthesis process, which can be a learnable function, manually defined heuristic rules, strong LLM or the policy model itself TO equipped with data synthesis instructions. The data synthesizing process aims to enrich the diversity and complexity presented for the training of policy model. Among various strategies, such as Self-instruct (Wang et al. 2022). Evol-instruct (Xu ctt al. 2023), we opt for a method akin to that described in Yu et al. (2023).
_! 9(x),
4.3 1/MCTS
4.3.1
Option-level MCTS
When applying MCTS to LLMs, it is natural to perform token-level search, where each token is considered as an action (Liu et al. 2023). However, the substantial vocabulary size typical of LLMs presents a significant challenge iee. conducting a deep search in such vast space becomes increasingly complex as the search space expands exponentially. To mitigate this, some efforts proposed a sentence-leve search, treating each sentence or step as a search node (Feng et al. (023). While this method reduces the search space, it might compromise the flexibility and effectiveness of applying MCTS to L.Mss, which is particularly true for tasks where subtle variations in token can dramatically impact the outcome, or where a more comprehensive search beyond a sentence necessary.

---

Search Node|
Example
Termination
Token-level
y/u 1/1
2/2
2/3
ys
yo
Ur
Ve
token
Sentence-level
90 1/2 + 3/
new line
Option-level you y1ya
+ 3/
yuo
termination function
Comparative illustration of token-level, sentence-level, and option-level MCTS search nodes. y denotes token sampled from the policy model. The arrow -> represents the transition from one search node to the subsequent node within the search process.
Table 1:
Inspired by SStton et al. (1999)): De Waard et al. (2016), we use the term option as a search node and propose option-level MCTS where each option represents sequence of tokens, which can range from multiple tokens to several sentences. A comparisons of different levels search is listed in Table 1 Mathematically, an option
(I, x,, where I C S is a set of initial states for the option: π : S x A + [0, 1] is a policy to generate actions, which in our case is a LLM;; and 3 S+ - [0, 1] is the termination function. Starting from state 8t, we can choose all the options for which 8, E I. Once an option is chosen, the policy π will generate actions for several steps until the option terminates according to the termination function 3. The option-level MCTS consists of stages including selection, expansion, simulation, and backpropagation. The option-level formulation offers more flexibility compared the sentence-level, new line can be treated special case of the termination function, as demonstrated in Table T Additional detailed steps of the option-level MCTS can be found in Appendix A.2
4.3.2 Importance-Band Based Adaptive Branching
In previous works related to option/sentence level tree search (Feng et al. 2023 Yao et al. 2024). was a common practice to assume that each node in the tree has the same predefined width, i.e., branching factor. This assumption was due to the fact that unlike token-level MCTS with limited action space, the sample space the option-level exceedingly large, with an unlimited number of token combinations. As result, it was necessary to set predefined maximum width for each node. However, this predefined branching factor is hard set, as an improper choice can lead to a search tree that is either too shallow or too thin, resulting in an inefficient exploration of the search space.
To quantify the error induced by the branching factor limit, we defined the branching error (t) For node with branching factor of m,, aims use the m4 child options of Dehsidren (where i E f1....,mi, m+)) to represent all possible options. Consequently, for legal option of (s;) from the option space, we can calculate the minimal value difference between it and the my existing options, which captures the error associated with representing other possible options using the m, available options. It can be formulated as (1) = BBo ((x)) [min ( of]) - vo ((s,, I]] where og is the value function which will be detailed $4.4 Here we define the importance of node 8; as I (s:) max,; |0z ([s;, of]) v (s:)). For simplicity, we assume that the value of the children nodes are uniformly distributed (a detailed analysis of the Gaussian distribution can be found in Appendix A.4 Under this assumption, we show in Appendix A.3 that E,(()= m While Esp is less than some e, we aim
I(es)
smaller total number of nodes for efficiency.
Theorem 4.1. The optimal branching factor m, in tree search set such that m,,- 1 is proportional to the node importance I(8:), under the condition I(,)) S e. Refer to Appendix A.3 for the detailed proof.
Clouse (996).. Intuitively, [(s:) captures the maximum value deviation from the current state. When this value is small, there is no need to explore further on this node, as there will not be a significant difference by rolling out on this node. Conversely, if the value is large, it is worth trying different children. We set the number of children allowed for a node n((:)) (after extracting 1) to be linear with this importance, using factor a. In practice, to avoid extreme cases of large variance of ((x)) in the early stage, we bound the number of children by depth-dependent constants Gain (1) and Gaa (az(()) m(s:) max (Cain (t), min (sa) + 1, Cuax (t))).
A similar concept has also been proposed in Taylor al. (2014): ;
5

---

4.3.3 State Merge
With ns::) determined, another issue is that options under the same node may be very similar, leading to many unnecessary sub-trees. Since we cannot directly control the o, ~ π T(8)), one strategy to mitigate this issue is to utilize the concept of move groups, as discussed in Van Eyck & Müller (2012). By merging similar nodes into the same group, we can increase the diversity among groups, thereby covering larger problem space with limited search rollouts and making the search process more efficient.
Here, we adapt the definition of node predicate P.M from Abel et al. 2018) and Fu et al. (2024) to represent whether two nodes are extremely similar. In practice, each time we generate new option from the policy, we use heuristic functions as P.M to check its similarity with all existing groups. The heuristic function can either be a faster rule-based measurement (e.g., edit distance) or a model-based method (e.g., prompting language model). Based on this, we decide whether to merge this option with a previous one or create a new group.
4.3.4 Fast Rollout with Specialized LM
The simulation operation which employs a rollout policy to project future trajectories from a given state, is crucial for an effective MCTS. This process significantly improves the efficiency of explo- ration and exploitation, and enhances the accuracy of reward estimation Estimations made the end of trajectories tend to have lower bias but higher variance; thus, simulating multiple possible trajectories yields low-bias, low-variance estimates, enabling a more informed and effective search process. Ideally, would serve as the rollout policy, yet its computational demands render it imprac- tical for the rapid simulations required by MCTS. To address this challenge, we propose the use of a smaller, specialized LM as the fast rollout policy fast Given a state 8t, the fast rollout policy n fast efficiently continues generation until reaches a termination condition, denoted as fast (=f8)(8))).
4.4 Critic
In ALPIALLMM, we design three types of critic models guide the search process.
Value Function The value function, denoted as v" (s). represents the expected return starting from state and following policy æ thereafter, given by 8² (ss)s = E=~<( s) where R(T) represents the discounted return of trajectory т. To train parameterized value function (s).
given the prompts D = {x,,, .......)))))). .) I E [N]]. for each prompt æ; we generate multiple trajectories T (a., o/2 L2 ) } by following policy for J times. A final reward r is assigned to indicate whether 7i aligns with y; y example, rewarding trajectories that contain correct answers in mathematical tasks closely follow instructions as ground truth. We then construct dataset Dralue {(5)), v E [N], t E [T],j E [J]} where 5¹2 [z; o' and 2/e = r/. The value function of is optimized by minimizing the mean squared error: Ls = -E(s.))~D,.... (v) (s) 2)?. Similar to (Feng et al. 2023), 1o is a LLM with an MLP layer on top to output a scalar on each token, using the scalar prediction at the last token of each state as the value.
PRM
The value function often struggles with credit assignment problem (Sutton 1984) and its
learning could be inefficient due to delayed and sparse rewards (Sutton & Barto 2018). 18). Therefore, we propose to incorporate PRM that introduces process supervision (Lightman et al. 2023) for direct option assessment. PRM generates intrinsic rewards (Chentanez et al. 2004) to encourage explorations of advantageous options, effectively mitigating issues of reward sparsity by providing immediate, action-specifie rewards. Given a state 8; and an option o, at time t. the PRM aims to predict the immediate reward r. that results from taking option o in state st. Formally, the PRM is a function R(s,, o,) - r While PRM ideally requires quality labels for each state (Uesato et al. 2022), due to the high cost and time involved in obtaining these, MC estimation with prefix sampling (Wang) et al. 2023) is used as a proxy, which aligns with the objective of the value function. Instead of adding a MLP layer on top of the policy model for outputting a scalar reward (Ouyang et al.) 2022). we formulate PRM as a text generation task to best leverage LLM's intrinsic knowledge
'Typically, the closer the simulation is to the termination state, the more accurate the reward estimation becomes.

---

for assessing the quality of an option. We adapt the dataset constructed for the value function as PRRM = {(, or, ) E [NT E [T]] where rt the textual description of the reward, e.g., an option can be regarded as good if Uaa is larger than certain threshold. To train PRM, we initialize it from the policy model ι and use the following prompt templates and typical language model loss.
The prompt template is shown in Appendix A.5
ORM In additional to the value function and PRM, ORM is also used to guide MCTS. ORM is designed to evaluate options sequences in their entirety. assessing the extent to which the complete trajectory aligns with the desired end goal (Uesato et al.) 2022, Lightman et al. 2023| Wang et al. 2023 Feng al. 2023). The outcome evaluation complements value function and PRM by offering comprehensive assessment of trajectories. Crucially, ORM plays a vital role in the simulation stage of MCTS by providing more accurate signals the terminal state, which in turn facilitates a more balance between exploration and exploitation strategies. ORM is formulated as a text generation task, similar to PRM. We leverage the same dataset for the value function training and construct DORM- {(2;, o a Ji E [N]]. where each instance includes a initial state or prompt æ;, a sequence of actions or options o'rr taken from that state, and a textual reward r indicating the sequence's ' success quality. Similarly, ORM is initialized from the policy model and the following prompt templates and language model loss are used for training. The prompt template is shown in Appendix A.5
The final score evaluation of state is weighted sum of the value function, PRM, and ORM: s(s) = Svalue 20 (s) + FPKM PRM(s) () + FORM Er- [RR] where T ~ x Fast (s) represents trajectories starting from under x fast, and Svalue. SPRM, SORM are hyperparameters. In practice, we found that the value function model has better precision and calibration, while PRM has superior recall (Appendix I0. Although ORM with fast rollouts provides low-bias, low-variance estimates, it still inherits some bias from fast, Thus, combining these critics yields a stronger evaluation signal.
4.5 Policy Self- Seff-Improvement
The policy improvement an iterative process with each iteration containing two main steps: data generation and policy finetuning.
Data generation In this step, we assume to have the current policy Te, and synthetic prompts Dr = (at ... at the k-th round, where each ot represents question. We obtain the corresponding training data Dz for policy To, by firstly performing 1jMCTS on DD ($4.3) and then sampling trajectory y from the corresponding tree for each question a Here we choose the trajectory that yield the highest critic score on the leaf node for each input question. Next, we filter out instances where the corresponding trajectory substandard forming D; {(z yr) fa y > 7 where represents function for quality scoring, and y indicates threshold. There can be several ways to implement the function, and here we simply use the ORM (RM($4L) ($4.4.
Policy finetuning With the obtained training data DR. we organize the data into the prompt templates shown in Appendix A.55 Then the policy Te, is finetuned using target-loss: Le. Re >tD, log no, (v. ax)]] resulting in an updated policy T0 We leave other training meth-
ods, such as DPO RRaailovv et al. 2023) or PPO (Schulman et al. 2017) in future work.
5 Experiments
5.1 Experiment Setups
ALPHALLM is generally applicable to wide spectrum tasks. As an early exploration, in this paper, we conduct experiments on mathematical reasoning problems where the learning signals are clear to define i.e., final answer is correct or wrong, We choose to evaluate on two widely used datasets GSM8K (Cobbe et al. 202T and MATH (Hendrycks et al.. 2021). For GSM8K, we utilize the whole test set while for MATH, due to computation constraints, we utilize a subset following the same procedure of Lightman et al. (2023). We evaluate the performance of predicting answers correctly for policy models. In addition, calculate the average rollouts, represented by the number of nodes in
7

---

the tree, as measure of computational efficiency. We compare the performance of ALPHALLM with suite of proprietary model, including OpenAT's GPT-4 and GPT-3.5, Anthropic's 's Claude-2, as well as Google's M-2 and the gemini model family. To ensure a fair and consistent evaluation, we employ CoT as our primary prompting method. Additionally, we conduct comparisons with strong
open-source models, including Llama-2-700 (Touvron et al. 2023a) and WizardMath-70B-VI. (Luo ict al. 2023).
We select Llama-2-70b as the policy model for the GSM8K dataset and WizardMatth70B-VII .0 for the MATH dataset. To construct the training dataset for the value function, PRM and ORM, we generate 50 trajectories for each prompt and construct the training target following Section 4.4 Both PRM and ORM are initialized using the weights from the policy model, while the value function uses a smaller Llama-2-1 2-b3b model, as we observed no performance gains from increasing the value function model size. In the design of ORM, tool usage not incorporated for GSM8K. However, for MATH, we enhance ORM by incorporating tools like python sympy to assess the quality of a trajectory, in a manner similar to that described by Gou et al. (203)). The training employ a learning rate of 1e-6 and are trained for one epoch. For the fast rollout policy model, we opt for the Abel-002-7B model (Chern) cet al. 2023) for both the GSM8K and MATH tasks for its high efficiency and superior performance. For the MCTS parameters, they are configured at different scales, as shown in Appendix A.6 We set Ovalue SPRM. and 3ORM all to 1.0.
For policy self improvinng ($4.5). we train the policy model up to 3 epochs, setting batch size to 128, learning rate x 10 and minimal learning rate to 1 10 66 Linear warm-up and decay is used with warm-up percent to be 10%. We perform early stopping based on devset held out from the training instances. For GSM8K experiments, we perform two rounds of self-improving, synthesizing 6.4k and 7.9k prompts Yu et al. 2023) respectively to obtain the corresponding MCTS outputs for training. For MATH experiments, we only perform one round of self-improving due to limited computation resources, and 5.9k prompts are synthesized.
The termination function for options can be either be learned or rule-based. In practice, for the GSM8K dataset, the termination condition occurs at the end of each line. This is based on the typical structure of this dataset, where each line represents a distinct step or point. For the MATH dataset, due to its complexity and the base model's tendency to generate many \n (n line breaks with some less meaningful content between them, termination occurs at the end of line if formula pattern is detected. During inference, if An in is encountered, we perform rule-base check for formula patterns. terminates a pattern is found or continues generating until the next \n\n.
5.2 Results
Table flsts the performance comparisons of various methods on the GSM8K and MATH datasets. Our findings reveal that ALPHALLM, based on Llama-2-70B and WizardMath- -0--- V1.0, utilizes only final answer annotations and continues to improve through training responses from m/MCSS. This comparison underscores the efficacy and broad applicability of our imagination-scarehee criticizing self-improving framework. Moreover, when our model is augmented with MMMTS decoding strategy, its performance markedly improves, achieving scores of 88.9 and 48.7 on the GSM8K and MATH datasets, respectively. Following two iterations of self celf-improvement using synthetic prompts, ALPIALLM demonstrates performance comparable to that of GPT-4 This suggests a viable approach improving LLMs' capabilities in complex problem-solving tasks in self-improving fashion, leveraging a minimal amount of labeled data. We also analyze the performance of various search methods in Appendix A.8
5.3 Ablation Study
We assess the effectiveness of each component in ALPHALLM and report the results on GSM8K in aa). Vanilla MCTS, configured with only the value function and a fixed number of children per node, achieves an accuracy of 79.5%. This serves as reference point for evaluating the incremental benefits introduced by each additional component. The use adaptive branching increae the accuracy to 84.9%. The addition of PRM improves the accuracy modestly to 85.9%, showing the effectivenss of process supervision for searching. A more significant improvement is observed with the introduction of ORM with fast rollout, which boosts the accuracy to 86.5%. Integrating state merging results in
8

---

Model
Decoding
#Annotation
RN
SYN
GSM8K
MATH
GPT-3.5
Sampling
80.8
35.5
GPT-4
Sampling
92.0
42.5
GPT-4 (PAL)
Sampling
94.2
51.8
Gemini 1.0 Pro
Sampling
77.9
32.6
Gemini 1.0 Ultra
Sampling
88.9
53.2
Gemini 1.5 Pro
Sampling
92.5
58.5
Claude-2
Sampling
85.2
32.5
PaLM-2 540B
Sampling
80.7
34.3
Llama-2-706
Greedy
0
57.8
Alama-2-700 SFT
Greedy
7.5k
x
69.3
WizardMath-70B-VI1
Greedy
96k
20.7
ALPHALLM
Greedy
7.5k/7.5k
X
v
73.7
23.6
ALPHALLM
1/MCTS
7.5k/7.5k
X
x
88.9
48.7
ALPHALLM
ryMCTS
7.5k/7.5k
92.0
51.0
Table 2: Comparison results of ALPHALLM on the GSM8K and MATH datasets. #Annotation indicates the quantity of labeled data employed for fine-tuning policy or training critic models. The annotation used for training are noted as RN for rationales and FA for final answers. SYN means models trained synthetic prompts, where trajectories were generated using m/MTS.
AB
PRM
FR-ORM
SM
LG-#Rollout
Acc
X
X
X
79.5
TA-ORM
Option
Acc
#Rollout
X
X
X
X
85.9
84.9
X
38.8
201
V
86.5
x
44.1
198
45.4
148
V
87.0
V
V
88.9
(b) Ablation study MATH
(a) Ablation study GSM8K
Table 3: (a): Ablation studies on the GSM8K test set of various components of 17MCTS, including adaptive branching, PRM, fast-rollout with ORM, state merge, and large number of rollouts. (b): Ablation studies of the impacts of augmentdd ORM and option-level formulation on MATH.
further increase in accuracy, reaching 87.0%. Finally the combined of increasing the number of rollouts with the other components yields the best performance on this task.
Table 3(b)
presents the ablation study of option formulation and the tool-augmented critic on the MATH dataset. Our proposed 1)MCTS achieves an accuracy 45.4 with 148 rollouts. When options are excluded, reverting to essentially sentence-level MCTS, the performance decreases to 44.1 with noticeable increase in the number rollouts to 198. This demonstrates that option formulation introduces enhanced flexibility MCTS, enabling better performance with fewer search efforts. Furthermore, the most significant decrease in performance observed when only intrinsic knowledge is utilized for ORM, which drops to an accuracy of 38.8. This suggests that the absence of an external tool critically impedes the ORM's capability to effectively assess challenging math problems.
Figure 2 depicts a comparative results on GSM8K of two rounds of celf-improving trained on trajectories collected using reranking and 1/MCTS. We report the performance of greedy decoding, MMCTS with relatively small number of rollouts (50-60), and 7MCTS with a larger number of rollouts (200-300) for each model. We observe that 1) Models trained on the trajectories from reranking or 1MMTTS outperform the initial policy by significant margin. In addition, the performance can be iteratively improved with training suggesting that self self-improving has the potential achieve continual performance gain. 2) While both reranking and 1/MC can generate high-quality trajectories for self-improving. rJMCTS is performant with high efficiency and better accuracy. Models trained on trajectories generated by not only exceed the performance of those trained on reranked trajectories

---

100
Greedy
(expanded #rollout)
68.9
88.3
89.5
90.8
88.4
92.0
86.0
86.4
80
73.7
Accuracy
70.5
69.3
70
64.6
60
56.8
40
n3. Reranking
w/ nMCTS
ng w/ Reranking
MMCTS
Figure 2: Empirical analysis on GSM8K of different self-improving data collection methods and number of iterations. Models are evaluated with greedy decoding, 17MCTS with small #rollout and large #rollout.
but also, when decoded with 1MMCTS, demonstrate on par performance with GPT-4, revealing that ALPHALLM is effective self-improving framework.
Method
Threshold
Acc
#Trajetory Acc
Edit distance
20
86.8
Edit distance
50
87.0
85.9
Cosine Similarity
0.7
86.3
86.5
Model-based
N/A
86.7
8
86.7
(a) Ablation on the choice of state merge func- tions.
(b) Ablation on the number of trajectories.
Table 4: (a): Ablation studies on the choice of heuristic/modellbbssee functions in state merge on GSMSK with base Llama2-70b. The model used in the model-based state merge is Llama-2-70b-chat. (b): Ablation studies of the number of rollout trajectories in fast-rollout estimation on GSM8K with base Llama2-70b.
We further analyze the impact of different hyperparameters and design choices for each component. Table shows that varying heuristic functions (with hyperparameters) for state merge has limited impact on performance. Table bb) shows that, as the number of fast-rollouts increases, there is corresponding improvement in performance. This is due to the reduction in the variance of the estimates. We used in experiments for better trade-off between performance and efficiency. Additional ablations on the choice of fast-rollout models, are provided in Appendix A.7
Conclusion
In this paper, we introduce ALPHALLM, an imagination--caaccinn---ciii criticcziriiiiiiiggggg framework designed for the self-improvement of LLMs without the necessity of additional annotations. At the heart of it is the integration of MCTS with LLMs. To tackle the inherent challenges associated with this integration, including data scarcity, the vastness of search spaces, and the subjective nature of feedback in language tasks, we introduce data synthesizer for strategic prompt synthesis, an optimized MCTS tailored for efficient search in language tasks, and a trio of critic models to provide precise feedback. Our experimental findings on mathematical reasoning tasks reveal that ALPHALLM significantly boosts the performance of L.Mss without requiring extra data annotations. Moreover, when decoded with MCTS,, ALPHALLMM performs comparably to GPT-4, highlighting the potential for self-i self-improvement in LL.Ms.
10

---

References
David Abel, Dilip Arumugam, Lucas Lehnert, and Michael Littman. State abstractions for lifelong reinforcement learning. In International Conference Machine Learning, pp. 10-19. PMLR. 2018.
Peter Auer, Nicolo Besa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235-256, 2002.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional ai: Harmlessness from ai feedback. arXiv preprini arXiv:2212.08077. 2022.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyezyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 17682-17690, 2024.
Samuel Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilé Lukošiūté, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable 2022.
oversight for large language models. arXiv preprint arXiv:221
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play inn-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024.
Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning. Advances in neural information processing systems. 17, 2004.
Ethan Chern, Haoyang Zou, Xuefeng Li, Jiewen Hu, Kehua Feng, Junlong Li, and Pengfei Liu. Generative for math: Abel. https://ggtthiie http://gittuu..cm....../. www.///I//A/A/////AA.... NLbbbbel 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetueed language models. arXiv preprint arXiv:2210.11 111111,,,,,,1,,,. 2022.
Jeffery Allen Clouse. On integrating apprentice learning and reinforcement learning. University of Massachusetts Amherst, 1996.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Maarten De Waard, Diederik M Roijers, and Sander CJ Bakkes. Monte carlo tree search with options for general video game playing. In 20/6 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2016.
Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint arXiv:231 110e22/1..440 1,04254, 2023.
Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero- tree- search can guide large language model decoding and training. arXiv preprint 1709772309..7119 2023.
Yangqing Fu, Ming Sun, Buqing Nie, and Yue Gao. Accelerating monte carlo tree search with probability tree state abstraction. Advances in Neural Information Processing Systems, 36, 2024.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint MiXiv:2309.17452 2023.
Hongyi Guo, Yuanshun Yao, Wei Shen, Jiaheng Wei, Xiaoying Zhang, Zhaoran Wang, and Yang Liu. -luman-inttruction-free Ilm self-alignment with limited samples. arXiv preprint 2rXiv:2401 06785, 2024.
11

---

Shibo Hao, Yi Gu. Haodi Ma. Joshua Hong. Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3154-8173. 2023.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
Ruixin Hong, Hongming Zhang, Xinyu Pang, Dong Yu, and Changshui Zhang. closer look at the self-verification abilities of large language models in logical reasoning. arXiv preprint arXiv:2311.07954 2023.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Xian Li. Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.06 06259, 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint TrXiv:2305.20050 2023.
Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Making ppo even better: value-guided monte-carlo tree search decoding. arXiv preprint 2rXiv:2309.15028 2023.
Jieyi Long. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.0822), 2023.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob N. Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. ArXiv, hes/1906.03926, 2019. URL https: atst::://ap..emmatticcchhoo wwwanticscoooalaatt...eeaaaaaaae org Oxg/CorpusID::. 182952502
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.095: 095883, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski. Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 14, 2021.
R OpenAI. Gpt-4 technical report. arXiv, pp.
2023.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, and Kam-Fai Wong. Composite coss-completion dialogue policy learning via hierarchical deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2017.
12

---

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305. 18290, 2023.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. ArXiv, abs/2210.01241, 2022. URL. https: /api....... www.sti org/
CorpusID:2526934055
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint wrX0:2200600880,, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06337, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwiessrr, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815. 2017.
Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. On the self-verification limitations of large language models on reasoning and planning tasks. arXiv preprint arXiv:2402.08// 00002 15, 2024.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. arXiv preprint urXiv:2305.0304 2023.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):181 1999a. ISSN 0004-3702. doi: https://doi.org/100 0t00//00000000000000000000000000 sciencedirect.
URL https: //www
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):81--11, 1999b.
Richard Stuart Sutton. Temporal credit assignment reinforcement learning. University of Mas- sachusetts Amherst, 1984.
Matthew Taylor, Nicholas Carboni, Anestis Fachantidis, Ioannis Vlahavas, and Lisa Torrey. Rein- forcement learning agents providing advice in complex video games. Connection Science, 26(1): 45-63, 2014.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et Gemini: family of highly capable multimodal models. arXiv preprint arXiv: 1805, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
13

---

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-an outcome-based feedback. arXiv preprint 1xii:22211112255. 2022.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for Ilms on planning and reasoning about change). arXiv preprint 2tti::2206.104990 2022.
Gabriel Van Eyck and Martin Müller. Revisiting move groups in monte-carlo tree search. In Ad- vances in Computer Games: 13th International Conference, ACG 2011, Tilburg, The Netherlands, November 20-22, 2011, Revised Selected Papers 13, pp. 13-23. Springer, 2012.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce ilms step-by step-by-tepp without human annotations. CoRR, 0bb/23/2.08995, 2023.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212. 100000 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thoure ohothht prompting elicits reasoning in large language models. Advances in neural information processing systems, 25:24824-24837, 2022.
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min- Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning, Advances Neural Information Processing Systems, 36, 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 2at.:22244..224,,,. 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing Iim reasoning generalists with preference trees. arXiv preprint arXiv:2404 02078, 2024a.
Weizhe Yuan, Richard Yuanzhe Pang. Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024b.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models teach themselves to think before speaking. arXiv preprini arXiv:2403.09629, 2024.
Tinghui Zhu, Kai Zhang, Jian Xie, and Yu Su. Deductive beam search: Decoding deducible rationale for chain-of-thought reasoning. arXiv preprint 4rXiv:2401.17686 2024.
14

---

Selection
Expansion
Simulation
Backpropagation
00
01
Value PRM
$1
02
ORM
Figure 3: An overview the four operations of MMMTS. A node is selected, expanded, simulated with fast rollout policy until terminal node is reached, then the signals from value function, PRM and ORM are backpropagated.
A Appendix
A.1 Imagination, Searching, Criticizing and Learning Loop
Algorithm 1: LLM self-improving loop
Input Initial dataset D {xx) y i [N]], policy model nT. reward model R, number of self-improving training loop K Output 8h
for - 1..., K do
Generate synthetic prompts [x=]] SYN(T (T DD
Collect trajectories with search algorithm, egg. MCTS guided by R.
[5 MCTS
( k-1 [x=
Construct dataset D {(x 1²}
Update policy 6 arg ming [mng DD)
end
The algorithm shown in Algorithm
A.2
Option-level MCTS
As illustrated Figure 3
option-level MCTS consists of the following operations:
Selection Starting from the root node, we iteratively select the child node based on Equation ??. Expansion Once an expandable leaf node is selected, a new node is generated by starting with the previous state of the parent node as the initial option state. The option then sampled using the policy and its completion is determined by the termination function B.
Simulation The scaled reward of the newly expanded node, well trajectories are evaluated using the feedback functions, which is discussed $4.4
simulated future
Backpropagation The average value of the newly generated node and all its ancestors is updated using the scaled reward from the evaluation step. Meanwhile, the visit counts for these nodes are also increased by one.
A.3 Importance-Basse Based Adaptive Branching Under Uniform Distribution
Let V =1U of), u (8;, ..., (5 )) be set of m, values that uniformly dis- tributed. If the maximum and minimum values from V/ are Umax and Umin the average gap between Emax Vanin,
two consecutive values is given by from new value Uhew any value from achieved when Unew consistently positioned the midpoint between two consecutive values, and it is given by 2 2(m,-1)
Mi -1 The upper bound of expected minimum distances
15

---

Since Umax
Umin (s;) for uniform distribution, we can conclude that Eg S m.-I
Theorem 4.1 The optimal branching factor my in free search set such that my 1 is proportiona the node importance (s:), under the condition I(s)) my -1 S
Proof. We can have the optimization problem as:
minimize: Σ m+
1(8))
subject to: M
S
Introduce the Lagrange multiplier A for each constraint:
L(me, As) Σ m. +
Σ Xe (e(m) 1) I(st))
Now, let's find the gradient the Lagrangian with respect to my and X and set them to zero:
Vm, = 1 cd =
VA < -1) [(st)
From the first equation, we get:
A
Substitute this value of XS into the second equation:
e(m, - 1) 1(s)) =
Solving for m,, get:
[(s:)
Thus, my 1 is proportional to the node importance [(::).
A.4 Importance-Based Adaptive Branching Under Gaussian Distribution
we assume that ([s,, and w ((se. of)) are independent and identically distributed Gaussian random variables:
w (s of)), oz ([s,, o:l) ~ NN,,
The difference Dij = ((s,, o']) v/ (1s, of)) will follow normal distribution with:
Dig ~ 2(0,202)
To find the expected minimum absolute difference between o ([st, of]) and the closest ug ([s of)). need to consider the distribution of the minimum of m, Gaussian differences.
The expected minimum value of m, absolute differences can be approximated using properties of order statistics for Gaussian distributions.
For of m independent normal random variables with variance 20 the expected minimum absolute difference, Eimingg DDjl]. can be approximated by:
0
Ea(())
m
16

---

This approximation arises from the fact that the expected minimum value of the absolute deviations of normally distributed random variables scales with the inverse of the square root of the number of samples.
Then, assume the range of the my samples R... max (1z ([s,, ofl) mén( ((st. of]), the
the expected range [[ of my samples from normal distribution can be approximated using properties of extreme values of Gaussian distributions. The range Rm can be approximated as:
Rm R a (20.9995 - 20.0005)
where 2p is the p-th percentile of the standard normal distribution, can converge to
Rm o / 2l((m)) 2 In(In(me))
4n((m))
For simplicity, we can approximate the range using the primary term, which captures the dominant behavior:
Rm o
Then we have
V2
Rm
(a(1) R
mm v mmm(())
Knowing that for all distributions,
Pm
T(s:) 2
2
We have
I(st)
Ex(t) < mm In(m)))
Then to find the optimal m. the optimization problem is
minimize:
Σ me
T(Se) Sc
subject to:
Vm, In(m))
To solve this optimization problem, we can first rewrite the constraint in terms of m
my ln(m;)) 2 (
Now, let's define a new function g(m;) = my In(m;). We want to find the minimum m, such that
g(m:) 2 (sx) To do this, we can find the derivative of g(m)) and set it to zero to find the critical points.
d g'(m:) dm,
(m, In(m))) = In(m)) + 1
Setting the derivative to zero:
In(m:) = -1
m, =
However, this critical point corresponds to minimum of the function g(ma),, and we are interested in the minimum m, that satisfies the constraint g(m+) 2 12(x.) Since the function g(m,) is increasing
2
for my > e we can find the minimum m, by setting g(m.) = 12(x)) and solving for m,:
2²
(st)
my (m,)
é2
This can not be solved directly, but we can still observe that there a positive correlation between my and (st).
17

---

GSM8K
MATH
Method
Small
Large
Small
Large
c
1.0
1.5
1.0
1.0
1.0
1.0
1.0
1.0
Cmax (0)
60
60
60
60
Omax
where 1>
10
10
10
10
Cmin
10
40
10
20
Cmin
where
2
Table 5: Parameters for MCTS. The Smal//Large means small #rollout and small #rollout
A.5
Prompt Templates
A.5.1
PRM
Y##YYou are given a math problem, followed by step-by-step reasoning process. Your task is to read the problem carefully, understand the solving steps, and check the correctness of the last reasoning step. Output "True" if the last step correct, and 'False" otherwise, Anlnittt Statein state) www.nntfftttoooooioon (ottiiiiooooniionnnnnnn option] /nnntiift
ttettul reward)
A.5.2 ORM
###tA a solution including final answer to given math problem by following below steps.In= An- Evaluate the method used for solving the problem. An- Review each calculation step for accuracy. Check for computational errors, incorrect formula applications, or arithmetic mistakes. An- The solution should use all the information provided in the question. An- Examine the final answer for correctness, considering the calculations and method used. An\n#Hi Promptin prompt] / Trajectory'n (njjectooyyntaajjeetttorr Assssmment (textual reward)
A.5.3 Policy Finetuning
For MATH experiments that take WizardMath V1.0 70B as the policy, adopt their proposed system prompt for self-improving. For GSMSK experiments taking Llama2 70B pretrain as the policy, the following system prompt.
chat between curious user and an artificial intelligence assistant.An The assistant gives helpful, detailed, and polite answers to the user's questions. An User: æ; An Assistant: y4
A.6
MCTS Details
We set the MCTS parameters in Table
A.7
Additional Ablations
Fast-rollout model Using Llama-2-70b instead of Abel-7B-002 improves performance by reducing bias from smaller model, but Abel-002-7B is faster with similar computational resources due to higher concurrency and quicker processing. The details can be found in Table 6
A.8 Search Comparison
Table presents the performance of various methods applied to different number of responses, from to 50. Our analysis confirms several key findings: 1) Reranking utilizing ORM consistently outperforms self-consistency techniques, indicating that ORM is capable of generating meaningful
18

---

Model
Acc (%)
Speed (s)
Abel-002-7B
87.0
16.8
Llama-2-700
87.3
38.1
Table 6: Ablation study over different fast-rollout models on GSM8K.
GSM8K
MATH
Method
#Responses
#Rollouts
Accuracy
#Rollouts
Accuracy
Greedy
4.6
57.8
9,9
20.7
10
46
67.4
99
22.5
Self-consistency
30
137
74.2
299
27.3
50
229
75.4
499
28.8
80.8
99
34.1
Re-ranking
30
137
86.3
299
39.0
50
229
87.7
499
42.0
55
87.0
223
45.4
n/MTS
230
88.9
341
48.7
Table 7: Comparative results of various searching method on GSM8K and MATH.
signals for searching. 2) 1MCTS demonstrates superior performance while requiring significantly fewer rollouts. For instance, on the MATH dataset, 1JMCTS achieves better results with only half the number of rollouts compared to reranking. Additionally, we evaluated the performance of BFS on GSM8K only, where requires 87.9 rollouts to achieve a score 80.6. These results suggest that our design of efficient MCTS in ALPHALLM can serve an effective policy improvement operation, enabling the search for high-quality trajectories with reduced computational cost.
A.9 Rollout Example
Consider the following GSM-like question:
Question: Sandy's 's monthly phone bill expense is equal ten times her age now. In two years, Sandy will be three times as old as Kim. If Kim is currently years old, calculate Sandy's monthly phone bill expense. InIf we know the answer the above question 340, what the value of the unknown variable x?nn
node in the second layer could have the following content:
We know that Sandy's 's monthly phone bill is 10 times her age. In two years, Sandy will be times as old as Kim. The sum of Sandy's age now and 2 years is times the sum of years. Kim's age now and years. Inaady's age now is 340/10 3340/10=34+34, In two years, Sandy's age will be 34 2 = a <3442=36x36.nn
Answer:
The parent of this node has the content:
Answer:
We know that Sandy's monthly phone bill is 10 times her age. In two years, Sandy will be times old Kim. The sum of Sandy's age now and 2 years is 3 times the sum of Kim's 's age now and two years.An
And one of its fast-rollout paths could be:
19

---

The sum of Sandy's age now and 2 years is 36. The sum of Kim's age now and two years is + 2..336 3(x+ + 2) 3x + 6ln3x = 30 10In ##### 10
A.10 Critic Performance
We evaluated the performance of the value function and PRM on the GSM8K test set. Table presents comparison of these models in terms of precision, recall, and Expected Calibration Error (ECE). Results indicate that the value function achieves higher precision and better calibration, while PRM demonstrates superior recall.
Model
Precision
Recall
ECE
Value Function PRM
0.82
0.79
0.032
0.62
0.90
0.375
Table 8: Performance comparison of the Value Function model and PRM on the GSM8K test set.
A.11
Compute Resources
Our experiments were conducted using NVIDIA A100 40GB GPUs. Serving models based on Llama-2-70B or WizardMath-70B required 4 GPUs, while serving Llama-2-7B and Abel-002-7B was possible on a single GPU. Training the 70B models required 64 GPUs.
A.12 Limitations and Future Work
Despite the promising results demonstrated by ALPHALLM in this study, there several limita- tions that requires further exploration. (i) Our current implementation employs relatively simple methods for generating synthetic prompts. Future iterations of ALPHALLM should explore advanced techniques. such as Self-Instruct, to create both diverse and model setabbiiityawwared prompts. (ii) Although ALPHALLM demonstrates improvements over base models, its performance in greedy sampling is substantially inferior to that observed when decoded with rMMTSS. This indicates that the full potential of MCTS for self-improvement in LLMs has not yet been fully realized. Two potential factors contributing to this issue have been identified: a) the self celf-improvement loop may be leveraging sufficient data; and b) the base model may be limited in its capacity for rapid learning. Addressing these concerns could lead to more significant improvemens. (iii) In our existing framework, the critic models remain static. We will explore mechanisms to continually update critic models to adapt to new policy models. This will help ensure the discriminator-genne toneeerrttoorgeeratooo gap and improve the overall training dynamics. (iv) The evaluation of ALPHALLM has been limited to mathematical reasoning tasks. To verify the generalizability and broader applicability of the framework, future research will need to extend its application to other domains.
20

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Yes the claims are accurately made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Yes discussed the limitations in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

We provide the assumptions and proofs for the Theorem 4.1. and other theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provided the hyoerparameters to reproduce the results.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification:
The code is available at http://gihhub..oomm////itttttlpp
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee 7/nips.com htte://lllllliiiiee////eeeaaaass ssiniiiiccypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Yes training and test details are mentioned.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not included in experiment results due the high computa- tional cost.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information of the compute resources we used in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Yes the research conform NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work primarily focuses on foundational research in algorithm improve- ment and, as such, does not have direct societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper has no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The datasets and models used in this paper properly cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We didn't 't release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "K2N8HTHRpE", "reviewer": "Reviewer_SX7Z", "summary": "The paper introduces a method for self-imporvement of LLMs called AlphaLLM. The method consists of three components: \nGeneration of expert trajectories\nEffective Monte-Carlo-Tree-Search over the LLM outputs (nablda-MCTS)\nA series of critics for evaluating reliable reward signal (value function, Process Reward Model, Outcome Reward Models)\n\nThe method can be used as a rollout mechanism to generate high quality outputs. To help deal with the large branching factor, the authors propose three subcomponents:\nState space is set to Options, which is neither token space nor sentence space, but a variable length sequence which terminates based on an auxiliary heuristic beta.\nStates are merged using another auxiliary function p_vM\nFast rollouts are used with a smaller LLM to help estimate. \n\nThe authors then demonstrate incredibly strong empirical results on both MATHS and GSM8K. Training with the proposed method on LLAMA-2-70b is able to achieve 51% on the MATH dataset.", "questions": "- What policy was used in the end for the fast rollout?\n- The prompt templates are missing key parts - [A detailed rubric that specifies how to evaluate a step of a task] would be helpful if shared. - In particular when comparing to GPT-4 or other model perform was an equally informative prompt used? (Or at least a prompt of equal input tokens).\n- If the synthetic data generated is used few-shot by llama2, how does your model perform?\n- I think the related work is missing some key papers such as: [https://arxiv.org/abs/2203.14465] (the STAR family of papers)", "limitations": "The paper in its current form is not replicable.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The method seems to work well!\n\nThe discussion on state merge and options helps express difficulties in applying MCTS.\n\nThe paper seems to combine many subcomponents to produce a strong algorithm. \n\nThe branching factor is well-motivated.", "weaknesses": "Lots of experimental details are missing. There is very little discussion of how AlphaLLM is trained or what base model it is (it's only mentioned in the introduction!). In particular, no information is given about the actual training of either the policy (LLM), the Critics, or the heuristic functions p_vM.\n\nThe explanation of the options is lacking - even looking at A.2 the explanation of options is unclear. Is Beta learnt as a function or generated in some other way? Are the set of options I, learnt before a rollout or generated on the fly?\n\nAblations are missing - in particular using different state methods, removing the Importance-Based Adaptive Branching, State Merge or Option-Level MCTS.\n\nThe prompts in the appendix are reduced so we can’t replicate.\n\nNo Code provided.\n\nWeird choice of baselines - surely to keep evaluations consistent with other LLMs, you should allow models to generate the same amount of output tokens with access to the PRM or ORM? I find it hard to believe COT on Claude-2 is a fair comparison, when the amount of optimisation pressure applied to multiple components of the method (including gradient updates).\n\n\nThere is no example of a rollout provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your suggestions have helped us a lot to improve the work, though we do think there might exist misunderstandings about our method. We hope the following clarifications and additional evidences would be possible for a re-evaluation.\n\n---\n\n> **[W1]** Lots of experimental details are missing. There is very little discussion of how AlphaLLM is trained or what base model it is (it's only mentioned in the introduction!). In particular, no information is given about the actual training of either the policy, the Critics, or the heuristic functions p_vM\n\nModel and training details: As detailed in **Appendix A.6** (line 609-624) in the original submission, the base models used are Llama-2-70b for GSM8K and WizardMath-70B-V1.0 for MATH, respectively. More details regarding the training (LR, warm-up, etc.), critics (data and model), and the fast-rollout are also provided in **A.6**.\n\nHeuristic functions: As mentioned in **Sec 4.3.3**, the heuristic function p_vM can either be a faster rule-based measurement (e.g. edit distance) or a model-based method (e.g. prompting a LLM). For our experiments, we used edit distance. We also include additional ablation studies demonstrating that the performance is not very sensitive to the choice of p_vM:\u000b  \nMethod | Threshold | Acc\n---|---|---\nEdit distance | 20 | 86.8\nEdit distance  | 50 | 87.0\nCosine Similarity (TF-IDF) | 0.7 | 86.3\nModel-based (Llama-2-70b-chat )| N/A     | 86.7 \n\n---\n\n> **[W2]** The explanation of the options is lacking. Is Beta learnt or generated in some other way? Are the set of options I, learnt before a rollout or generated on the fly?\n\nThe termination function $\\beta$ can be either be learnt or rule-based. In practice, for the GSM8K, we used a newline character (\\n) as the termination condition, as shown in **Table 4**. For the MATH, termination condition occurs when a line with a formula pattern is found, with line breaks indicated by '\\n\\n'. During inference, if the generated output has '\\n\\n', we check for formula patterns using rules. It terminates if a pattern is detected or continues until another '\\n\\n' appears.\n\nThe set $I \\subseteq S$  represents a set of initial states (not a set of options). It can be the state of a question or a previously terminated step.\n\n---\n\n> **[W3]** Ablations are missing - in particular using different state methods, removing the Adaptive Branching, State Merge or Option-Level MCTS.\n\nWe have included ablations for different state methods, including PRM, fast-rollout with ORM, state merge, and large number of rollouts, tool-augmented ORM and option-level MCTS in **Table 3**, with a discussion in **Sec 5.3**. Additional ablations on adaptive branching are included here:\n  - With adaptive branching: 84.9\n  - Without adaptive branching: 79.5\n\n---\n\n> **[W4]** No Code provided.\n\nWe plan to release the code after it undergoes a legal review by our entity.\n\n---\n\n> **[W5]** The prompts in the appendix are reduced so we can’t replicate.\n\nAll the prompt templates will be included with the code release. These templates are detailed in Appendix A.5 , with specific rubrics for PRM and ORM are provided in Section A in the comment below.\n\n---\n\n> **[W6]** Weird choice of baselines - surely to keep evaluations consistent with other LLMs, you should allow models to generate the same amount of output tokens with access to the PRM or ORM? I find it hard to believe COT on Claude-2 is a fair comparison, when the amount of optimisation pressure applied to multiple components of the method.\n\nOur primary focus in this paper is to explore the self-improvement capabilities of LLMs. The experimental results demonstrate that our method, AlphaLLM, can achieve significant self-improvement. Moreover, we observed that the performance of Llama-2-70b, when employs our proposed \\etaMCTS, is comparable to that of Claude-2 and GPT-4, highlighting the potential for self-improvement.\n\nWe included proprietary models mainly to illustrate the potential of our self-improvement method. The definition of a \"fair comparison\" can vary. Proprietary models, such as the latest versions of Claude-2 and GPT-4, are trained with unknown data quantities and types, often iteratively refined. By default, these models already incorporate chain-of-thought processes. It is plausible that proprietary models leverage System-2 results to enhance System-1 outputs, complicating direct comparisons.\n\nIn addition, we have demonstrated superior performance when compared with recent work [1] that also used MCTS and utilizes a similar token usage amount. For example, AlphaLLM achieved 51.0 on MATH while [1] achieved 34.3.\n\nWe hope this clarifies our approach and rationale behind the baseline choices. We welcome further suggestions to improve our evaluation methodology.\n\n---\n\n> **[W7]** There is no example of a rollout provided.\n\nPlease refer to the example in the Section B in the comment.\n\n---\n\n> **[Q1]** What policy was used in the end for the fast rollout?\n\nAs mentioned in **Appendix A.6**, the fast-rollout model is Abel-002-7B.\n\n---\n\n> **[Q2]** The prompt templates are missing key parts - In particular when comparing to GPT-4 or other model perform was an equally informative prompt used? (Or at least a prompt of equal input tokens).\n\nPlease refer to the responses for [W5] and [W6]. \n\n---\n\n> **[Q3]** If the synthetic data generated is used few-shot by llama2, how does your model perform?\n\nThe synthetic data would still have the same question format as [2] but with answer steps from MCTS. Therefore, Llama2 with few-shot prompting would exhibit similar performance to that observed on the questions in [2].\n\n---\n\n> **[Q4]** I think the related work is missing some key papers such as the STAR family of papers\n\nWe acknowledge that we missed the STaR papers and will include them in the related work section. Although we have cited some similar papers [3,4], we appreciate the importance of adding this relevant literature.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WdAn3IQ0ux", "reviewer": "Reviewer_PPNk", "summary": "The paper proposes AlphaLLM, a tree-search enhanced framework with a few improvements over Data Synthesizing, option-level MCTS, Importance-Based Adaptive Branching, state merging, fast rollout, critic function and policy improvement process. Experimental results verify the framework's effectiveness on GSM8k and MATH.", "questions": "1. What is the difference between value function training and PRM training? From the description it seems that the PRM's training is exactly the same as value function training in Monte-Carlo estimation (except you tend to take more simulations starting from a fixed state). \n\nAlso, it seems that the technique described in the critic section has almost been described in many previous works: value function training in [1], PRM training in [2, 3], and ORM in [4], the author needs to clarify more on their own contributions and make proper citations.\n\n2. One difference between your work and [1] is that they leverage the value function itself to backward (like AlphaZero) while you are using the fast model to rollout to reach the terminal state, which is like the initial version of AlphaGo (where they have a fast-move network) and [5]. What do you think of these differences and why do you make this design choices?\n\n\nReference\n[1] Feng, Xidong, et al. \"Alphazero-like tree-search can guide large language model decoding and training.\" arXiv preprint arXiv:2309.17179 (2023).\n[2] Wang, Peiyi, et al. \"Math-shepherd: A label-free step-by-step verifier for llms in mathematical reasoning.\" arXiv preprint arXiv:2312.08935 (2023).\n[3] Lightman, Hunter, et al. \"Let's verify step by step.\" arXiv preprint arXiv:2305.20050 (2023).\n[4] Uesato, Jonathan, et al. \"Solving math word problems with process-and outcome-based feedback.\" arXiv preprint arXiv:2211.14275 (2022).\n[5] Hao, Shibo, et al. \"Reasoning with language model is planning with world model.\" arXiv preprint arXiv:2305.14992 (2023).", "limitations": "See questions and weaknesses.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The technical contribution seems solid. The paper proposes a comprehensive framework and address modifications/improvements over the full pipeline of tree-search.\n2. Experimental results demonstrate great potentials for the proposed algorithms.", "weaknesses": "1. The writing is not clear enough. It is not that clear to me how the option-level is implemented to separate sentences (how the Termination function is determined).\n2. There are many different components in the pipeline design but the ablation studies are not enough to validate them one by one. For example, It at least need to involve:\n2.1 Comparison with other heuristic search methods, for example: (1) the beam-search + value function described in [1, 2]) (2) majority-vote or reranking with ORM.\n2.2 Wall time or token consumption comparison between different methods.\n2.3 The ablation studies seem to be too simple to present how different components can really influence the performance. Pure with/without results seem to be too simple. More experiments like how to control the hyperparameter for these things are required. For example, how different heuristic functions can influence the State Merge, how different fast rollout model can influence the effiency, how the thresold in ORM in the process of data generation can influence the improvement process, etc.\n\n\nReference\n[1] Feng, Xidong, et al. \"Alphazero-like tree-search can guide large language model decoding and training.\" arXiv preprint arXiv:2309.17179 (2023).\n[2] Yu, Fei, Anningzhe Gao, and Benyou Wang. \"Outcome-supervised verifiers for planning in mathematical reasoning.\" arXiv preprint arXiv:2311.09724 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  **[W1]** It is not that clear how the option-level is implemented to separate sentences\n\nThank you for your feedback regarding the clarity of our writing. To clarify, the termination function for options operates differently depending on the dataset:\n- For the GSM8K dataset, the termination condition occurs at the end of each line. This is based on the typical structure of this dataset where each line represents a distinct step or point.\n- For the MATH dataset, due to its complexity and the base model's tendency to generate many '\\n\\n' line breaks with some less meaningful content between them, termination occurs at the end of a line if a formula pattern is detected. During inference, if '\\n\\n' is encountered, we perform a rule-based check for formula patterns. It terminates if a pattern is found or continues generating until the next '\\n\\n'.\n\n\nWe will include the clarifications in the revision.\n\n---\n\n>  **[W2.1, W2.2]**  The ablation studies are not enough. 2.1 Other heuristic search methods; 2.2 Token consumption\n\nIn Table 5 of the original submission, a basic comparison of accuracy and the number of rollouts for self-consistency (majority-vote) and re-ranking methods for both GSM8K and MATH datasets was included. Following the suggestion, beam search (BFS) has also been incorporated into the experiments on GSM8K, as shown in the table below:\nMethod | #Responses | #Tokens | Acc\n---|---|---|---\nGreedy | 1 | 127 | 57.8\nMaj-Vote | 10 | 1272 | 67.4\nMaj-Vote | 30 | 3788 | 74.2\nMaj-Vote | 50 | 6332 | 75.4\nRe-ranking | 10 | 1272 | 80.8\nRe-ranking | 30 | 3788 | 86.3\nRe-ranking | 50 | 6332 | 87.7\nBFS | - | 2430 | 80.6\n\\etaMCTS | - | 1521 | 87.0\n\\etaMCTS | - | 6360 | 88.9\n\nToken consumption is estimated by tracking MCTS rollouts and multiplying by the average tokens per rollout for Llama2-70b on GSM8K. The table indicates that \\etaMCTS outperforms and is more efficient than other baselines.\n\n---\n\n> **[W2.3]** 2.3 The ablation studies seem to be too simple \n\nIn addition, we have conducted ablation studies on GSM8K with base Llama2-70b to further demonstrate the influence of various components on the performance:\n\n**State Merge**: The impact of the choice of heuristic functions (w/ hyperparameters) or model-based state merge is not very significant.\n\nMethod | Threshold | Acc\n---|---|---\nEdit distance | 20 | 86.8\nEdit distance  | 50 | 87.0\nCosine Similarity (TF-IDF) | 0.7 | 86.3\nModel-based (Llama-2-70b-chat )| N/A\t | 86.7 \n\n**Fast-rollout**:  Number of rollouts: As the number of fast-rollouts increases, there is a corresponding improvement in performance. This is due to the reduction in the variance of the estimates. We used n=4 in our experiments for better trade-off between performance and efficiency.\n#rollout| Acc\n---|---\n1|85.9\n4|86.5\n8|86.7\n\n**Fast-rollout model**: Using Llama-2-70b instead of Abel-7B-002 improves performance by reducing bias from a smaller model, but Abel-002-7B is faster with similar computational resources due to higher concurrency and quicker processing.\nModel | Acc| Speed (s)\n---|---|---\nAbel-002-7B|87.0|16.8 \nLlama-2-70b|87.3|38.1\n\n**ORM**: During training, each question is sampled multiple times, and the ORM predicts the correctness of each trajectory as True or False. The ORM's score (probability of outputting True) is used in the data generation process, along with the scores from value and PRM, eliminating the need for a threshold.\n\n**Adaptive branching**: with adaptive branching, the performance score is 84.9, whereas without it, and using a fixed number of 4 branches, the score drops to 79.5.\n\n---\n\n> **[Q1]** What is the difference between value function and PRM training? Also, please clarify your contributions and provide proper citations in the critic section.\n\nValue function training and PRM training share some similarities, but they differ in task formulation, model architecture, and training loss.\n\nValue function training focuses on estimating the expected FUTURE return of a given state or state-action pair. In contrast, PRM evaluates the quality of the current state or node. While PRM ideally requires quality labels for each state, due to the high cost and time involved in obtaining these, MC estimation is used as a proxy. \n\nThe Value Function model extends from a policy model by adding a value head and is trained using regression loss to predict expected returns, a continuous variable. PRM, trained using LM loss, evaluates the quality of states/nodes and benefits from the contextual understanding of language models. The differences in architecture and training result in distinct behaviors: the Value Function model has better precision (0.82 vs. 0.62) and calibration (ECE: 0.032 vs. 0.375), while PRM has superior recall (0.90 vs. 0.79).\n\nFinally, our contribution lies in utilizing a trio of critics, each with strengths in different aspects, to provide reliable signals for guiding MCTS. We will restructure the critic section to clarify these points further, and will also ensure to appropriately cite the references [1,2,3,4] in that section to acknowledge their contributions to this field.\n\n---\n\n> **[Q2]** Why choose to include the fast-rollout?\n\nWe use rollouts to the terminal state, because empirical evidence shows that the value network's accuracy improves with detailed information closer to the terminal state. For example, an early version of the value network had 89.5% accuracy at the terminal state and 83.2% accuracy three steps before, indicating more reliable predictions near the terminal state. Estimations at the end of rollout trajectories have lower bias but higher variance, so using the mean score from multiple rollouts offers low-bias, low-variance estimations. To mitigate bias from fast rollouts of a smaller model, we also keep value estimations at the actual progress point. Additionally, using the outcome reward model through fast rollouts provides extra signals that enhance learning, making our approach more robust and effective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kulI7FUBxi", "reviewer": "Reviewer_LHqU", "summary": "This paper proposes an imagination-searching-criticizing approach called ALPHALLM to enhance the capabilities of large language models (LLMs). ALPHALLM integrates Monte Carlo Tree Search (MCTS) with LLMs to establish a self-improving framework.", "questions": "- Have you ever tried using your value/reward function to perform RL fine-tuning with PPO?", "limitations": "None.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper introduces a novel approach to LLM self-improvement using MCTS, presenting an interesting concept.\n- The paper is well-written, and the proposed imaginative-searching-criticizing approach is clearly explained.\n- Using only 7.5k/3k final math answer annotations and after just two iterations of self-improvement, ALPHALLM achieves impressive results: 92.0 on GSM8K and 51.0 on MATH. These results are remarkable.", "weaknesses": "- Given only final math answer annotations, ALPHALLM essentially performs the final-label (reward) classification problem and tree search based on the predicted reward to generate rationales for each final answer. It is still unclear how ALPHALLM could outperform other LLM-SFTs that utilize both rationale annotations and final answer annotations.\n- ALPHALLM is a general framework. The authors should apply ALPHALLM to other tasks where learning signals are clear to demonstrate the overall effectiveness of this self-improvement framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and insightful questions. We are encouraged by your approval of the novelty and effectiveness of ALPHALLM, as well as the clarity of this approach.\n\n---\n\n> **[W1]** Given only final math answer annotations, ALPHALLM essentially performs the final-label (reward) classification problem and tree search based on the predicted reward to generate rationales for each final answer. It is still unclear how ALPHALLM could outperform other LLM-SFTs that utilize both rationale annotations and final answer annotations.\n\nOur primary motivation stems from the observation that as LLMs continue to evolve, achieving human parity in numerous tasks, the necessity and quality of explicit rationale annotations in datasets such as GSM8K and MATH become points of consideration.\n- The rationale annotations provided in datasets like GSM8K and MATH, while useful, are not necessarily optimal or superior to those that can be generated through advanced methods like MCTS guided approaches. The quality and suitability of these human-provided rationales can vary, and they might not always align with the nuanced reasoning capabilities of advanced LLMs. Our hypothesis is that the rationales generated by ALPHALLM, guided by MCTS, are potentially more robust and aligned with the model’s training process than those annotated in GSM8K and MATH.\n- ALPHALLM utilizes final math answer annotations in conjunction with tree search based on predicted rewards to derive rationales. This method allows the model to explore a wider range of reasoning paths and select the most plausible ones based on calculated rewards, rather than being confined to potentially suboptimal human-provided annotations.\n- As presented in our experimental results, ALPHALLM demonstrates superior performance compared to WizardMath, which utilizes over 96k data points for math-specific training, including both rationale and final answer annotations\n\n---\n\n> **[W2]** ALPHALLM is a general framework. The authors should apply ALPHALLM to other tasks where learning signals are clear to demonstrate the overall effectiveness of this self-improvement framework.\n\nWe appreciate the reviewer’s suggestion regarding the application of ALPHALLM to a broader range of tasks. We are actively working on extending our framework to other STEM reasoning tasks to further validate its effectiveness. Preliminary results are promising, and we are committed to sharing a comprehensive report on these additional tasks in the near future! \n\n---\n\n> **[Q1]** Have you ever tried using your value/reward function to perform RL fine-tuning with PPO?\n\nWe have not yet explored vanilla PPO but optimizing LLMs using trajectories from ALPHALLM with PPO/DPO is indeed on our roadmap for future work. To provide a comparative analysis, we included the results of Best-of-N ranked by our ORM in the current study. Best-of-N is often considered an alternative to PPO, particularly when N is sufficiently large. As demonstrated in Table 5 in the submission, ALPHALLM significantly outperforms Best-of-N, highlighting the effectiveness of our MCTS approach. This substantial margin of improvement underscores the potential of our method over traditional approaches like PPO, even before we integrate them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YMdjTN8TBD", "reviewer": "Reviewer_iEWd", "summary": "This paper introduces ALPHALLM, which is an imagination-searching-criticizing framework designed for self-improvement. Inspired by AlphaGo, authors integrate MCTS and LLMs to establish the self-improvement loop. Additionally, authors proposed eta-MCTS, which is a decoding method used to reduce the search space. The experiments show that ALPHALLM can increase LLMs’ reasoning ability obviously, especially combined with eta-MCTS decoding.", "questions": "Please refer to the part of weaknesses.", "limitations": "Limitations are sufficiently discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. It’s innovative to use tree search to do self-improvement and it’s good to identify some challenges very clearly. \n2. The experiments show that the reasoning ability of LLMs can be improved effectively.", "weaknesses": "1. It could be more clear in some parts and including some examples would be helpful. For example, in the part of 4.2 data synthesizing, I’m curious about how the synthesized data looks like. \n2. Authors identify that one of the challenges in working on LLMs’ self-improvement is the difficulty to get clear evaluations, and they proposed some method to get the evaluations, i.e. value function, PRM, ORM. But they do not discuss whether these methods are reliable to get perfect feedback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer’s insightful feedback. Your recognition of our novelty, clear identification of challenges, and the effectiveness of our results is is highly encouraging to us.\n\n---\n\n> **[W1]** It could be more clear in some parts and including some examples would be helpful. For example, in the part of 4.2 data synthesizing, I’m curious about how the synthesized data looks like.\n\nWe understand that some sections of the paper, such as the data synthesizing part in section 4.2, could benefit from further clarification and examples. \nTo address your query about the synthesized data, we used the method described in [1] to generate synthesized questions. Here's an example synthesized based on GSM8K:\n```\nQuestion: Sandy's monthly phone bill expense is equal to ten times her age now. In two years, Sandy will be three times as old as Kim. If Kim is currently x years old, calculate Sandy's monthly phone bill expense.\\nIf we know the answer to the above question is 340, what is the value of unknown variable x?\\n\n```\nWe hope this example provides a clearer understanding of how our data synthesizing process works. We are open to further discussions to improve the clarity of our work.\n\n[1]. Yu, Longhui, et al. \"Metamath: Bootstrap your own mathematical questions for large language models.\" arXiv preprint arXiv:2309.12284 (2023).\n\n---\n\n> **[W2]** Authors identify that one of the challenges in working on LLMs’ self-improvement is the difficulty to get clear evaluations, and they proposed some method to get the evaluations, i.e. value function, PRM, ORM. But they do not discuss whether these methods are reliable to get perfect feedback.\n\nWe appreciate your insightful comment regarding the reliability of the evaluation methods such as the value function, PRM, and ORM for obtaining ideal feedback in the context of LLMs' self-improvement. In our study, we have utilized a combination of these evaluation methods to ensure better calibration results.\nThe accuracy obtained on the GSM8K dataset for each critics is as follows:\n- Value: 0.832\n- PRM: 0.841\n- ORM: 0.877\n\nAs a combination of them, we have achieved the following calibration scores:\n- ECE: 0.1319\n- AUROC: 0.958\n- Brier Score: 0.0429\n\nThese results indicate that our proposed evaluation methods have provided reasonably good feedback for LLMs' self-improvement. We appreciate the suggestion and will continue to explore more reliable and robust evaluation techniques to enhance the self-improvement capabilities of LLMs in future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper draws from AlphaGo to introduce AlphaLLM, a combination of search, simulation, and critic for the self-improvements of LLMs. AlphaLLM uses Monte Carlo Tree Search (MCTS) in a self-improving loop to improve the performance of LLMs. Results on MATHS and GSM8K seem promising. In spite of the interesting results, the reviewers have been concerned with the clarity of the paper, and the presentation of methods, details of how the approach works in the main paper. We recommend the paper for acceptance as a poster conditional on the implementation of the following:
- bring appendix 6 to the main paper
- include responses to Reviewer PPNk in the main paper, including the relevant citations and the clarificaitons
- all reviewers requested the authors include more implementation details, evaluation, and the code
Thank you
Decision: Accept (poster)