Venue: NAACL 2025

LLaMA-Berry: Pairwise Optimization for Olympiad-level Mathematical Reasoning via O1-like Monte Carlo Tree Search

Di Zhang\({}^{1,2+}\) , Jianbo Wu\({}^{3*}\) , Jingdi Lei\({}^{2*{\dagger}}\) , Tong Che\({}^{4*}\) Jiatong Li\({}^{5{\dagger}}\) , Tong Xie\({}^{6}\), Xiaoshui Huang\({}^{7}\), Shufei Zhang\({}^{2}\), Marco Pavone\({}^{8}\) Yuqiang Li\({}^{2{\ddagger}}\) , Wanli Ouyang\({}^{2}\), Dongzhan Zhou\({}^{2{\ddagger}}\)

This work was done during his internship at Shanghai Artificial Intelligence Laboratory.Corresponding author

###### Abstract

This paper presents LLaMA-Berry, an advanced mathematical reasoning framework to enhance the problem-solving ability of large language models (LLMs). The framework combines Monte Carlo Tree Search with Self-Refine (SR-MCTS) to optimize the reasoning paths and utilizes a pairwise reward model to evaluate different paths globally. By leveraging the self-critique and rewriting capabilities of LLMs, our SR-MCTS overcomes the inefficiencies and limitations of conventional stepwise and greedy search algorithms, enabling a more efficient exploration of solution spaces. To guide the search process, we propose the Pairwise Preference Reward Model (PPRM), which predicts pairwise preferences between solutions through instruction-following capabilities trained by Reinforcement Learning from Human Feedback (RLHF). Finally, the Enhanced Borda Count (EBC) method is adopted to synthesize pairwise preferences into global quantile scores for evaluations. This approach mitigates the challenges of scoring variability and non-independent distributions in mathematical reasoning tasks. The framework has been tested on general and advanced benchmarks, showing superior search efficiency and performance compared to existing open-source and closed-source methods, particularly in complex Olympiad-level benchmarks, including AIME24 and AMC23.

languageresourceLanguage Resources, machine learning, ICML

## 1 Introduction

Mathematical reasoning represents a great challenge in artificial intelligence, with broad applications across automated theorem proving, mathematical problem solving, and scientific discovery Ahn et al. (2024). Recently, significant strides have been made by large language models (LLMs) like GPT-4 (Achiam et al., 2023) in general mathematical tasks involving arithmetic and geometric problem-solving Cobbe et al. (2021); Sun et al. (2024); Ying et al. (2024). However, complex mathematical reasoning remains challenging, especially at the Olympiad-level benchmarks such as AIME (MAA, 2024).

An intuitive approach to improving problem-solving is to break solutions into step-by-step reasoning paths Lightman et al. (2023); Luo et al. (2024), as demonstrated in Chain-of-Thought CoT Wei et al. (2022). While prompt-based methods can effectively facilitate the construction of such reasoning paths, they may still encounter challenges due to the lack of comprehensive feedback during the generation process, which can affect efficiency Paul et al. (2023). In contrast to stepwise generation methods, another promising line of research treats the entire solution as an independent state, employing rewriting capabilities to refine the solutions, such as in Self-Refine Madaan et al. (2023) and Reflexion Shinn et al. (2024). However, these approaches, while innovative, may occasionally face challenges like being susceptible to local optima or potentially drifting towards suboptimal solutions due to flawed feedback, which could impact their maximum potential performance.

In addition to generating reasoning paths, effective solution evaluation is crucial, with models like the outcome reward model (ORM) and process reward model (PRM) Uesato et al. (2022) serving as valuable examples. The ORM focuses on the correctness of the final answer in a reasoning path, while the PRM emphasizes the correctness of each step in the process. While both methods enable reward models to assign scalar scores, obtaining reliable labeled data for training these reward models remains a significant challenge. Moreover, the scoring standards for mathematical reasoning tasks can vary significantly, as each problem presents 

unique characteristics. This variation complicates the scaling of reward models and hinders their ability to capture local preference relations between solutions. Although trained using language models, these reward models have yet to fully leverage instruction-following capabilities, which may limit their effectiveness in handling more complex reasoning tasks (Zhang et al., 2024).

To improve the efficiency of solution search in mathematical problems, we treat a complete solution as an independent state and apply Self-Refine to optimize previous solutions in order to obtain better ones. In the Self-Refine process, feedback from critiques is utilized to make the search more efficient compared to stepwise reasoning path generation. Furthermore, we incorporate Monte Carlo Tree Search (MCTS Kocsis and Szepesvari, 2006) to replace the iterative manner in naive Self-Refine, enhancing the solution search. MCTS leverages signals from the evaluation process to assess solutions and uses the Upper Confidence Bound applied to Trees (UCT) method to balance exploration and exploitation. This approach enables the search process to effectively exploit higher-quality solutions and explore those with greater potential for improvement, while avoiding getting trapped in suboptimal local minima.

In the evaluation process, utilizing the instruction-following capabilities trained by Reinforcement Learning from Human Feedback (RLHF Christiano et al., 2017), Pairwise Preference Reward Model (PPRM) transforms the absolute rewards calculation into preferences prediction between solutions to calculate rewards. The approach reduces the variability with scoring characteristics and thus leads to a more robust and consistent evaluation of different solutions. To overcome the locality limitations inherent in pairwise comparisons, we employ the Enhanced Borda Count (EBC) method to aggregate local preference evaluations into global quantile scores, leading to more informed decision-making and, ultimately, better solutions. Combining the PPRM and EBC method not only enables the reward model to learn a more robust reward signal but also captures the global characteristics of the solution space, ensuring more reliable comparisons.

Our contributions are summarized as follows: (1) We propose **SR-MCTS**, a novel Markov Decision Process (MDP) framework that treats entire solutions as states and Self-Refine as optimization action to perform advanced solution search with MCTS. (2) **PPRM** is developed to leverage the preference relationship between solutions to evaluate their quality, which avoids the volatility of absolute scores while providing a more guided exploration of optimal paths. We adopt the EBC method to convert the local preferences into global evaluations. (3) We verify the effectiveness of LLAMA-Berry on multiple benchmarks, which outperforms baseline approaches like ToT (Yao et al., 2024) and rStar (Qi et al., 2024) in both search efficiency and accuracy. Notably, LLAMA-Berry enhances the performance of LLAMA-3.1-8B, making it comparable to proprietary models, including GPT-4 Turbo on Olympiad-level mathematical reasoning **without additional training**.

Figure 1: The main pipeline of LLAMA–Berry, where \(S_{i}\) stand for problem-solving solutions and \(C_{i}\) stands for critiques. The pipeline consists of four phases detailed in Section 2.2, including selection, expansion, evaluation, and backpropagation.

 



## 2 Methodology

### Preliminary

One of the core challenges in mathematical problem-solving is to generate and optimize reasoning paths to derive high-quality solutions. We formalize this process in a path-wise Markov Decision Process (MDP) framework, where each state \(s\) in the state space \(S\) represents a _complete solution_ to a given problem, and the action space \(A\) consists of all feasible _rewriting_ actions \(a\) that make transitions between states.

In the framework, we aim to quantify the expected reward \(Q(s,a)\) from executing action \(a\) at state \(s\), that is,

\[Q(s,a)=\mathbb{E}[R(s^{\prime})|s^{\prime}=T(s,a)], \tag{1}\]

where \(T(s,a)\) indicates the transition from \(s\) to another solution \(s^{\prime}\) via the rewriting action \(a\). Our primary objective is to identify the optimal state \(s^{*}\) that represents the best solution. We can reach \(s^{*}\) by selecting actions that maximize the reward, guiding us toward the most desirable outcome, as demonstrated in Equation 2.

\[s^{*}=\arg\max_{s^{\prime}\in S}Q(s^{\prime}) \tag{2}\]

### Self-Refine applied to MCTS

As shown in Figure 1, SR-MCTS integrates Monte Carlo Tree Search (MCTS) with the Self-Refine mechanism to continuously evaluate and optimize the solution search. This integration leverages the iterative nature of MCTS and the self-improvement capabilities of LLMs, thereby improving the search outcomes.

Monte Carlo Tree Search (MCTS) is an effective method within the Markov Decision Processes (MDP) framework, employing states, actions, and value functions through sampling. The algorithm follows four key steps: selection, expansion, evaluation, and backpropagation. In the selection phase, the root node is expanded using the Upper Confidence Bound applied to Trees (UCT) algorithm, which selects a node \(s\) by balancing exploration and exploitation:

\[a=\arg\max_{a\in A(s)}\left(Q(s,a)+c\cdot\sqrt{\frac{\ln N(s)}{N(s,a)}}\right), \tag{3}\]

where \(N(s)\) is the visitation count of node \(s\), \(N(s,a)\) is the action frequency, and \(c\) is a parameter controlling exploration. In the expansion phase, node \(s\) generates subsequent states \(s^{\prime}\), added as new nodes in the tree \(\mathcal{T}\). The evaluation phase typically uses simulations or heuristics to estimate the Q-values for these nodes. Finally, during backpropagation, the estimated Q-values are updated retroactively from the leaf nodes to the root. This iterative process allows MCTS to refine decision-making by balancing the exploration of new paths with the exploitation of known high-value paths.

**Selection phase.** The selection phase identifies a node \(s_{i}\) from the search tree \(\mathcal{T}\) for expansion, where each node represents a complete solution state. The Upper Confidence Bound applied to Trees (UCT) algorithm is employed to select the optimal node, with dynamic pruning used to avoid local optima. A node \(s_{i}\) is considered fully explored when its child nodes reach a predefined limit, and at least one child node's Q value exceeds the Q value of \(s_{i}\).

**Expansion phase.** In the expansion phase, as shown in Figure 1, the selected answer \(s_{i}\) is expanded by generating successor answers through a Self-Refine process, which includes a **Criticizing** and **Rewriting** process. The **Criticizing** process generates a critique \(c_{i}=C(s_{i})\) that identifies drawbacks (e.g., mathematical wrongs or logical faults) in the current chosen answer \(S_{i}\), and then **Rewriting** process generates a new answer \(s_{i+1}=R(s_{i},c_{i})\). In practice, to simplify the problem, we assume this process is deterministic, ensuring that the same original state of solutions \(s_{i}\) consistently produces the same successor state of solution \(s_{i+1}\). The new state of solution \(s^{\prime}\) is then added to the search tree \(\mathcal{T}\) as a new node.

**Evaluation phase.** The evaluation phase calculates the value \(Q(s^{\prime})\) of the newly generated node \(s^{\prime}\) using the Pairwise Preference Reward Model (PPRM). The evaluation involves two steps: global and local value assessments. The global value \(Q_{g}(s^{\prime})\) is determined by the quantile of \(s^{\prime}\) in a win-loss preference matrix \(\mathbf{M}\), which reflects the win-loss relationships between nodes. The local value \(Q_{l}(s^{\prime})\) is derived from comparisons with adjacent nodes in the search tree \(\mathcal{T}\). The total value \(Q(s^{\prime})\) is then computed as a weighted combination of global and local values: \(Q(s^{\prime})=\alpha Q_{g}(s^{\prime})+(1-\alpha)Q_{l}(s^{\prime})\), where \(\alpha\) controls the relative influence of each component.

**Backpropagation phase.** In the backpropagation phase, the value \(Q(s^{\prime})\) of the new node is propagated back to its parent node \(s_{i}\), updating Q value

of \(s_{i}\) as a discounted sum of its child nodes' Q values: \(Q(s_{i})=(1-\gamma)Q(s_{i})+\gamma Q(s^{\prime})\). The discount factor \(\gamma\) represents the importance of future rewards. This iterative update mechanism ensures that the values of parent nodes are progressively refined, enhancing the guidance for future selections.

Additionally, to control the growth of the search tree, the SR-MCTS method restricts the maximum number of rollouts \(N_{max}\) . The search process terminates when the restriction is reached, imposing limits on the unbounded expansion of the tree. The overarching objective of SR-MCTS is to maximize the expected highest \(Q\) value of all existing nodes \(S\), guiding us towards the most desirable outcome \(s^{*}\), ensuring that the search process efficiently converges to high-quality solutions.

### Pairwise Preference Reward Model

Reliable evaluation of different solutions plays a crucial role in mathematical problem-solving tasks as it leads to better estimation of Q-values, thereby offering better guidance. Existing reward models typically evaluate solutions by giving absolute scores, such as process reward model [4] and outcome reward model [30]. However, the score-based reward models may fall short in leveraging the instruction-following capability of LLMs or effectively handling the variations in scoring standards, especially when the differences between solutions are subtle. To address this, we propose the Pairwise Preference Reward Model (PPRM), which leverages a comprehensive preference dataset incorporating substantial samples from both PRM and ORM approaches [25, 7] to learn preference relationships among mathematical solutions.

For two solutions (\(a_{1}\) and \(a_{2}\)) to a given mathematical problem, we use \(a_{1}\succ a_{2}\) to represent the situation where \(a_{1}\) is preferred over \(a_{2}\). PPRM predicts their relative quality using a pairwise partial ordering, represented by the following probability formula:

\[P(a_{1}\succ a_{2}~{}|~{}\phi)=\frac{e^{\phi(a_{1})}}{e^{\phi(a_{1})}+e^{ \phi(a_{2})}}, \tag{4}\]

where \(P(a_{1}\succ a_{2}~{}|~{}\phi)\) denotes the probability of a partial ordering relation between solution \(a_{1}\) and \(a_{2}\), with \(\phi\) representing the parameters of the reward model. In our method, \(a_{1}\succ a_{2}\) are represented by tokens of an LLM, and \(P(a_{1}\succ a_{2}~{}|~{}\phi)\) is estimated using the logits value of tokens calculated by the LLM.

Then, inspired by advancements in Language Interface Fine-Tuning [9], we frame the training process of PPRM as a question-answering task to leverage the instructions-following capabilities of LLMs. The model is tasked with answering the question, "For Question \(Q\), is solution \(a_{1}\) better than solution \(a_{2}\)?" as shown in Figure 2. To form a robust training objective, the predicted token labels \(\hat{y}\) ('Yes' or 'No') are evaluated with ground truth label \(y\) using the indicator function **I**:

\[\textbf{I}(\hat{y},y)=\begin{cases}1,&\text{if $\hat{y}=y$}\\ 0,&\text{if $\hat{y}\neq y$}\end{cases} \tag{5}\]

Finally, a pairwise preference dataset \(\mathcal{D}\) that contains millions of mathematical problem-solving solution pairs is converted into a dataset \(\mathcal{D}^{\prime}\) suitable for a question-answering task. We employ RLHF techniques to train the model to improve its performance in the partial-order prediction question-answering task. Subsequently, the Direct Preference Optimization [6] method is utilized to find the optimal \(P_{\phi}\) by maximizing the objective \(\arg\max_{\phi}\mathbb{E}_{P}[\textbf{I}(\hat{y},y)]\). Please refer to Appendix C for details about training and inference of PPRM.

### Enhanced Borda Count Method

Although PPRM allows us to directly compare the quality of two solutions, we still need to con

Figure 2: Preference prediction process of PPRM and global quantile score based on Enhanced Borda Count method.

 

vert these local preferences into a cohesive global ranking to gain a comprehensive evaluation for the answers. This conversion process can be formalized as the global optimal ranking aggregation (GORA) problem related to Learning to Rank (LtR) methods. Further, we propose the Enhanced Borda Count (EBC) method based on the transitivity assumption of mathematical problem solutions, which integrates the naive Borda Count algorithm with a transitive closure of preferences calculated by the Floyd-Warshall [20] algorithm. For formalized discussion, please refer to Appendix H.

**Local preference calculation.** First, the PPRM generates a preference matrix \({\bf M}\in\mathbb{R}^{n\times n}\) for all \(n\) problem solutions, where \({\bf M}_{i,j}=1\) indicates that solution \(a_{i}\) is superior to solution \(a_{j}\), and \({\bf M}_{i,j}=0\) otherwise. This process can be represented as:

\[{\bf M}_{i,j}=\begin{cases}1,&\text{if $P(a_{i}\succ a_{j})\geq 0.5$}\\ 0,&\text{if $P(a_{i}\succ a_{j})<0.5$}\end{cases} \tag{6}\]

As shown in Figure 2, this matrix can be viewed as an adjacency matrix of a directed graph \({\bf G}=({\bf V},{\bf E})\), where each solution \(a_{i}\) corresponds to a vertex \(v_{i}\), and an edge \(e=(v_{i},v_{j})\) exists if \({\bf M}_{i,j}=1\), indicating that solution \(a_{i}\) is preferred over \(a_{j}\).

**Transitive closure.** To simplify the problem, we adopt the assumption of transitivity for mathematical solutions, that is, if \(v_{i}\succ v_{k}\) and \(v_{k}\succ v_{j}\), then \(v_{i}\succ v_{j}\). Under this assumption, the transitive closure \({\bf C}\) of a preference matrix can be computed through Floyd-Warshall algorithm, e.g., if \({\bf M}_{i,k}=1\) and \({\bf M}_{k,j}=1\), then \({\bf M}_{i,j}=1\).

**Borda count-based global ranking.** Next, based on the transitive closure matrix \({\bf C}\), we apply the Enhanced Borda Count method for global ranking. The Enhanced Borda Count determines the ranking of each node by calculating its out-degree, which corresponds to the number of nodes it defeats. For each node \(v_{i}\), the Borda\((v_{i})\) is defined as \(\sum_{j=1}^{n}{\bf C}_{i,j}\), like the ranked node listed in Figure 2.

Nodes with higher Borda counts are ranked higher. However, in practice, cyclic preferences can cause efficiency issues with the naive Borda Count method. To further refine the ranking, we devise a re-ranking stage, where the logits generated by the PPRM are used for soft comparison among nodes with equal Borda counts. Specifically, for two nodes \(v_{i}\) and \(v_{j}\) with equal Borda counts, the soft comparison rule can be denoted as \(v_{i}\succ v_{j}\iff P(a_{i}\succ a_{j})>P(a_{j}\succ a_{i})\). This process ensures that the ranking remains consistent and reasonable even in the presence of cycles or local ambiguities.

**Global quantile score of solutions.** Finally, the ranking is converted into a global quantile score \(Q_{g}\) for each solution \(v\) is \(Q_{g}(v)=1-\frac{\text{rank}(v)-1}{|V|-1}\), where rank\((v)\) is the position of \(v\) in the ranking based on Borda counts, and \(|V|\) is the total number of nodes. To reflect local advantages in the search tree structure, the local win rate \(Q_{l}(v)\) for a node \(v\) is calculated in \({\bf C}\) with its children nodes Children\({}_{v}\) as follows:

\[Q_{l}[v]=\frac{\sum_{u\in\text{Children}[v]}C[u,v]}{|\text{Children}[v]|} \tag{7}\]

Finally, score \(Q(v)\) for a solution is a weighted sum of local win rate \(Q_{l}(v)\) and global quantile score \(Q_{g}\).

## 3 Evaluation

### Experiment Settings

**Settings.** To better evaluate the effectiveness of our approach, we select LLaMA-3.1-8B-Instruct model [49] as the base model for SR-MCTS, **without any additional training**. We also train a Gamma2-2B-Instruct model [14] as PPRM to provide reward signals during the search process. We develop the Berry-Tree inference framework to ensure robust and efficient inference, which supports advanced features, including fault tolerance, checkpoint recovery, multi-query concurrency, and automatic multi-server load balancing. Hyper-parameter settings are detailed in Appendix A.

**Grading.** We evaluate algorithm-generated answers using the correctness evaluation standards as in Lightman et al. [20], focusing on format adherence and content accuracy. The model is provided with a prompt specifying the expected answer format. We score answers as consistent if they exactly match the ground truth, closely approximate it numerically, or are equivalent in symbolic form. To ensure a comprehensive and rigorous evaluation, we adopt **major@k**[15] and **rm@k**[26], which can be unified as the solved rate of problems [16, 17]. The evaluation methods and metric details are further outlined in Appendix B.

 



### Benchmarks

**General mathematical reasoning benchmarks.** We summarize the results on general mathematical reasoning benchmarks in Table 1, which indicates that our method significantly boosts the base model's performance. The results consistently demonstrate improvements across various levels of difficulty. Specifically, the solved rate of problems in 16 rollouts of Meta-Llama-3.1-8B-Instruct (Meta, 2024b) has been improved by more than 35% on three benchmarks. Qwen2-Math-72B-Instruct (Yang et al., 2024) exhibits the strongest mathematical reasoning capability among the competing methods, while our LLaMA-Berry, built on a base model with only 8B parameters, exceeds it in the solved rate of problems on four benchmarks. In particular, LLaMA-Berry reaches 55.1% on OlympiadBench and 68.9% on College Math, surpassing it by 11.9% and 21%, respectively.

**Cutting-edge mathematical Olympiad benchmarks.** In Table 2, we compare the performance of LLaMA-Berry with other leading models on Olympic-level benchmarks. The results demonstrate that LLaMA-Berry is highly competitive on these benchmarks, demonstrating its capability in complex reasoning. Notably, on the most challenging AIME2024 benchmark, our method boosts the base model's solving rate from 2/30 to 8/30, surpassing typical open-source models and commercial closed-source models, except for the OpenAI o1 series (OpenAI, 2024).

In addition to excelling in mathematical reasoning, our approach also excels across various science and engineering domains. For example, it achieves top performance on benchmarks such as MMLU STEM (Hendrycks et al., 2021) in Table 1 and GPQA diamond (Rein et al., 2024) in Table 2. This demonstrates the method's robustness and versatility, which highlights its potential for broader applications in both research and practical scenes.

Comparison with other tree-based or CoT methods.We compare our algorithm with other tree-based reasoning methods and CoT-based methods on GSM8K (Cobbe et al., 2021), GSMHard (Gao et al., 2022), and MATH500 (Lightman et al., 2023) which is a representative and highly challenging 10% subset of MATH (Hendrycks et al., 2021) benchmark. As shown in Table 3, the performance of RAP (Hao et al., 2023) and ToT (Yao et al., 2023) tends to degrade relative to more straightforward methods like Few-shot CoT and One-turn Self

\begin{table}
\begin{tabular}{c|c c c c c c} Model Benchmark & GSM8K & MATH & GaoFaaO23En & OlympaidBench & College Math & MMLU STEM \\ \hline Qwen2-7B-Instruct (Yang et al., 2024) & 85.7 & 52.9 & 36.4 & 21.3 & 24.5 & 68.2 \\ Meta-Llama-3.1-8B-Instruct (Meta, 2024b) & 76.6 & 47.2 & 30.1 & 15.4 & 33.8 & 60.5 \\ \hline Qwen2-72B-Instruct (Yang et al., 2024) & 93.2 & 69.0 & 58.7 & 33.2 & 43.2 & 84.4 \\ Meta-Llama-3.1-70B-Instruct (Meta, 2024) & 94.1 & 65.7 & 54.0 & 27.7 & 42.5 & 80.4 \\ \hline DeepSeeMatt-7B-R1:(Shao et al., 2024) & 88.2 & 52.4 & 43.6 & 19.0 & 37.5 & 64.8 \\ Interlmm-math-plus-7b (Vine et al., 2024) & 84.0 & 54.4 & 50.1 & 18.8 & 36.2 & 55.2 \\ Mathstral-7B-v0.1 (Martial Al, 2024) & 84.9 & 56.6 & 46.0 & 21.5 & 33.7 & 64.0 \\ NuminaMath-7B-GCT (Beeting et al., 2024) & 75.4 & 55.2 & 47.5 & 19.9 & 36.9 & 60.8 \\ Queen2-Math-P-Insortet (Yang et al., 2024) & 89.9 & 75.1 & 62.1 & 38.2 & 45.9 & 63.8 \\ \hline NominaMath-72B-CoT (Beeting et al., 2024) & 90.8 & 66.7 & 58.4 & 32.6 & 39.7 & 64.5 \\ Owen2-Math-72B-Instruct (Yang et al., 2024) & 96.7 & 84.0 & 68.3 & 43.0 & 47.9 & 79.9 \\ \hline
**Meta-Llama-3.1-8B-Instruct (Meta, 2024b)** & 89.8\({}_{\text{Paper}}^{*}\) 54.8\({}_{\text{Paper}}^{*}\) & 36.4\({}_{\text{Paper}}^{*}\) & 24.8\({}_{\text{Paper}}^{*}\) & 36.4\({}_{\text{Paper}}^{*}\) & 68.3\({}_{\text{Paper}}^{*}\) \\  +**LLAMA-Berry (Ours)@8** & 94.9\({}_{\text{Prost}}^{*}\) & 69.4\({}_{\text{rest}}^{*}\) & 61.6\({}_{\text{req}}^{*}\) & 47.2\({}_{\text{req}}^{*}\) & 63.7\({}_{\text{req}}^{*}\) & 82.9\({}_{\text{req}}^{*}\) \\ \hline
**+LLaMA-Berry (Ours)@16** & \(96.1\)\({}_{\text{req}}^{*}\) & \(75.3\)\({}_{\text{req}}^{*}\) & 68.6\({}_{\text{req}}^{*}\) & \(55.1\)\({}_{\text{req}}^{*}\) & 68.9\({}_{\text{req}}^{*}\) & 88.3\({}_{\text{req}}^{*}\) \\ \hline \end{tabular}
\end{table}
Table 1: Performance comparison of models across benchmarks of different difficulties, as represented by GaoFaaO2023En (Liao et al., 2024), College Math (Tang et al., 2024), and OlympiadBench (He et al., 2024), which range from high school to Olympiad levels. Scores denoted with subscripted notations, such as maj88, represent specific metrics, with **major@8** as an example. Scores without subscripted notations reflect the model’s greedy performance evaluated in a zero-shot CoT manner.

\begin{table}
\begin{tabular}{l|c c c}  & \multicolumn{4}{c}{Benchmarks} \\ Model & AIME24 & ACC23 & Math Obspiracy GPOA\({}_{\text{Estimation}}\) \\ \hline Clasek XOpus & 6.7 & 42.0 & 40.6 & 50.4 \\ GPT+Turo & 3.3 & – & 47.0 & 38.8 \\ GPT+Go & 13.4 & – & – & 56.1 \\ OpenAI o1 Preview & 56.7 & – & 78.3 \\ OpenAI o1 Proview & 83.3 & – & 78.0 \\ Gemini iSpo & 6.7 & – & 45.0 & – \\ Gemini Mm-Specialized 1.5 pD23.3 & – & 55.8 & – \\ Meta-Llama-3.1-8B-Instruct & 6.7 & 15.7 & 41.7 & 30.4 \\ Queue2-Math-72B-Instruct & 13.3 & 62.5 & – & – \\ NuminaMath-72B coT & 3.3 & 52.5 & – & – \\ Querai-Manj-72B-Instruct & 20.0 & 60.0 & — & – \\ \hline
**Meta-Llama-3.1-8B-Instruct** & 13.3\({}_{\text{Saper}}^{*}\) & 22.9\({}_{\text{saper}}^{*}\) 44.2\({}_{\text{saper}}^{*}\) & 39.4\({}_{\text{saper}}^{*}\) \\ +**LLAMA-Berry (Ours)@8** & 16.7\({}_{\text{rs}}^{*}\) & 48.2\({}_{\text{sssen}}^{*}\) 60.4\({}_{\text{srsen}}^{*}\) & 77.3\({}_{\text{rsen}}^{*}\) \\ \hline
**+ILAMA-Berry (Ours)@16** & \(26.7\)\({}_{\text{crsen}}^{*}\) & 54.2\({}_{\text{ragen}}^{*}\) 65.0\({}_{\text{drsen}}^{*}\) & 92.4\({}_{\text{srsen}}^{*}\) \\ \hline \end{tabular}
\end{table}
Table 2: Performance comparison across multiple olympiad benchmarks, including AIME24 (MA, 2024), AMC23 (MA, 2023), Math Odyssey (Fang et al., 2024), and GPQA Diamond (Rein et al., 2024).



Refine when the difficulty increases from GSM8K to GSMHard. We suspect the reason could be the weak self-evaluation capability of LLMs, which may guide reasoning steps to the inefficient side. Moreover, tree-based methods can incur more computational overhead than straightforward methods. In contrast, rStar Qi et al. (2024) and our method maintain a positive output performance trend, highlighting both approaches' higher search efficiency.

To make a fair comparison between the reported results on LLAMA-3-8B-instruct from rStar Qi et al. (2024), self-consistency Wang et al. (2022), and our algorithm, we also utilize the LLaMA-3-8B-instruct as the base model instead of the 3.1 version. We observe that our approach achieves on-par or even better performance with fewer rollouts. Specifically, our method achieves an accuracy of 88.1%, 31.5%, and 39.6% on GSM8K, GSMHARD, and MATH500 benchmarks, respectively, using the majority voting metric, which is among the same accuracy level as others while only consuming 1/2 of the rollout times of rStar and 1/8 of Self-consistency. This provides compelling validation of the efficacy of the EBC method and the aggressive exploration fostered by the dynamic pruning strategy.

### Ablation Study

As shown in Table 4, we conduct ablation experiments to evaluate the key components of LLAMA-Berry, using the solved rate of problems Luo et al. (2024) as a metric across benchmarks of increasing difficulty: GSM8K, MATH500 and AIME2024. Major Voting method represents the base model's reasoning capabilities. CoT-based MCTS which is a step-level MCTS is a comparison with SR-MCTS method. SR-MCTS without PPRM refers to a basic version of our method that uses self-evaluation as the reward instead of PPRM and EBC.

When comparing the two distinct tree search node expansion strategies, namely step-level CoT and Self-Refine, it is evident that the Self-refine method surpasses the step-level CoT approach across all datasets under the same number of rollouts. This advantage is further amplified when employing PPRM, as exemplified by the improvement from 37.2% to 69.0% on the MATH500 dataset.

When comparing multi-turn Self-Refine with SR-MCTS methods, especially on the GSM8K dataset, the introduction of MCTS effectively mitigates the issue of solution degradation into sub-optimal results caused by flawed critiques in iterative methods. As shown in Table 4, with rollouts of 8, SR-MCTS without PPRM improves the problem-solving rate by 3.1%, while SR-MCTS with PPRM further enhances the problem-solving rate by 16.1%. On the more challenging MATH500 dataset, the advantage of SR-MCTS with PPRM becomes even more pronounced, achieving a 28.6% improvement in the problem-solving rate. Furthermore, in comparing SR-MCTS without PPRM and SR-MCTS with PPRM, PPRM further boosts the solved rate of problems on GSM8K and MATH500 datasets, the solved rate of problems is elevated by 13% and 31.8%, respectively.

Notably, on the more challenging AIME2024 dataset, both Multi-turn Self-Refine and SR-MCTS without PPRM demonstrate limited search efficiency. However, SR-MCTS with PPRM can significantly improve the solved rate of problems from 6.7% (2/30) to 16.7% (5/30) with rollouts of 8. The results underscore the efficacy of combining the Self-Refine method with PPRM when addressing complex problems. The contrast between self

\begin{table}
\begin{tabular}{l|c c c} \hline \hline MethodBenchmark & GSM8K & GSMHARD & MATH500 \\ \hline Zero-Shot CoT & 68.4 & 14.9 & 5.8 \\ Few-shot CoT & 74.5 & 25.6 & 17.8 \\ One-turn Self-Refine & 75.7 & 26.5 & 25.0 \\ Self-Cons@8 & 78.4 & 28.5 & 30.0 \\ Self-Cons@64 & 83.2 & 30.3 & 33.0 \\ Self-Cons@128 & 84.7 & 31.2 & 33.8 \\ TGT@32 & 69.1 & 19.6 & 13.6 \\ RAP@32 & 80.6 & 29.6 & 18.8 \\ rStar@32 & 88.7a\_9022 & 33.4a\_9021 & 38.3a\_9021 \\ \hline
**LLAMA-Berry (Ours)@8** & \(86.4a\_9020 

reward and PPRM underscores the importance of designing reward mechanisms that can more effectively guide the search process. PPRM provides a more holistic incentive to the model, thus fostering more effective problem-solving strategies.

Our ablation study on LLaMA-3-1-8B-Instruct reveals that the hyper-parameter \(\alpha\), which balances local and global reward scores in PPRM, has scenario-dependent impacts. For simpler tasks like GSMHARD, global scores dominate, as evidenced by the superior performance of LLaMA-Berry when prioritizing global metrics. This suggests that holistic evaluation is sufficient for straightforward problems. In contrast, complex tasks like MATH500 benefit from a balanced combination of local and global signals--the MATH500 benchmark shows a significant improvement (from 37.2% to 69.0%) when both metrics are optimally weighted. These results indicate that while \(\alpha\) has a limited impact on simpler tasks, its tuning is crucial for complex tasks requiring both granular and global reasoning. These findings suggest that future work could focus on optimizing \(\alpha\) adaptively for different task complexities.

### Scaling Study

To explore the potentials and trends of the scaling with rollouts in inference-time, we depict the solved rate of problems with rollouts in three benchmarks with different difficulty levels. Analyzing the performance alongside Figure 3, the increment in the number of rollouts consistently enhances model performance across various benchmarks, and the extent of these improvements differs depending on the benchmark's complexity and the base model's reasoning capability. These curves underscore that the performance of the LLaMA-Berry framework benefits from scaling up rollouts during inference, similar to the observations in OpenAI [4]. However, there are ceiling limitations, as seen in the GSM8K dataset, which suggest that the base model's capabilities in both reasoning and refinement play a crucial role in determining the overall performance.

## 4 Related Works

**Reward models for reasoning.** Reliable reward models Kang et al. (2024); Wang et al. (2023); Havrilla et al. (2024); Lightman et al. (2023); Ma et al. (2023) can effectively distinguish desirable responses from undesirable ones, which is especially important in complex reasoning. The outcome reward model (ORM) are trained with the final results of the reasoning paths. As the rewards are determined by the final answers, ORM may suffer from coarse supervision and misalignment issues. In contrast, process reward model (PRM) provides step-wise reward signals that are easier to interpret and guide the models to follow the CoTs. Therefore, PRM is generally considered to be more effective Lightman et al. (2023). However, the success of PRM relies on extensive manually annotated data Luo et al. (2024); Havrilla et al. (2024), which is time-consuming and still faces the challenge of the volatility of absolute reward scores. Pairwise Preference Reward Model (PPRM) in LLaMA-Berry converts absolute scoring into preference prediction task, which then brings robust reward signals via EBC method.

**Tree search reasoning.** Sampling diverse reasoning paths Brown et al. (2024) has demonstrated its effectiveness in enhancing the probability of finding the correct answers. Self-Consistency Wang et al. (2022) samples a complete path each time while tree search methods like Tree-of-Thought (ToT) Yao et al. (2023) and Monte Carlo Tree Search (MCTS) Chen et al. (2024, 2024); Luo et al. (2024); Feng et al. (2023); Xie et al. (2024); Xu (2023); Liu et al. (2023); Tian et al. (2024); Ding et al. (2023) extend multiple steps to optimize step answers and ultimately obtain the optimal solution. Additionally, Self-Refine Madaan et al. (2023) method has become a recent focus. Self-verification Gero et al. (2023); Weng et al. (2022) and rStar Qi et al. (2024) utilize the inherent capabilities of the model to iteratively explore and refine answers. However, the performance of Self-Refine is typically constrained by the inherent capabilities of the model, especially for small language models (SLMs) with significantly weaker Self-Refine abilities Madan et al. (2023). Zhang et al. (2024) suggests that the mathematical reasoning abilities of LLMs can be enhanced by treating the refinement process as a directed acyclic graph (DAG) through multi-agent

\begin{table}
\begin{tabular}{l|c c} \hline \hline \(\alpha\)Benchmark & GSMHARD & MATH500 \\ \hline
0.0 (global scores only) & 44.9\({}_{\text{rn8}}\) & 68.2\({}_{\text{rn8}}\) \\
0.9 (default) & 43.8\({}_{\text{rn8}}\) & 69.0\({}_{\text{rn8}}\) \\
1.0 (local scores only) & 44.2\({}_{\text{rn8}}\) & 67.9\({}_{\text{rn8}}\) \\ \hline \hline \end{tabular}
\end{table}
Table 5: The results with 8 rollouts of varying the hyperparameter \(\alpha\) in the reward mechanism of PPRM.

 

collaboration. In our approach, we combine MCTS with Self-Refine to explore potential solutions and a global win-loss matrix is then constructed in the form of a directed graph to calculate the final quantile scores.

## 5 Conclusion

This research addresses challenges in complex mathematical reasoning, particularly at the Olympiad level, by enhancing the accuracy and efficiency of the search for reasoning paths. By introducing Self-Refine applied to Monte Carlo Tree Search (SR-MCTS), the LLaMA-Berry framework significantly improves the efficiency of solution generation by LLMs. Additionally, the Pairwise Preference Reward Model (PPRM) constructs preferences between solutions rather than merely scoring outcomes, calculating the final global quantile score using the enhanced Borda Count (EBC) method. Evaluation results demonstrate that LLaMA-Berry outperforms baseline approaches on benchmarks like GSM8K and MATH, and achieves competitive performance compared to closed-source models on Olympiad-level benchmarks such as AIME2024.

### Limitation

The LLaMA-Berry framework has demonstrated strong performance in reasoning tasks, but there are still some challenges in practical applications. First, methods such as Monte Carlo Tree Search (MCTS) and Self-Refine have high computational costs. These techniques demand significant computational resources, which may limit the feasibility of deployment in environments with constrained computational capacity. As for summarizer of solutions, rule-based heuristics methods like self-consistency, major voting and mutual reasoning have shown a constraint to the ceiling search performance of MCTS. Thus, we aim to develop a learning-based summarizer as OpenAI (2024) does to further enhance the search efficiency.

Furthermore, the current evaluation of the LLaMA-Berry framework has primarily focused on mathematical reasoning benchmarks, resulting in a relatively narrow assessment scope. As a result, its performance in broader domains, such as general knowledge, symbolic logic tasks, and multimodal applications, has not been sufficiently validated. In future work, we aim to improve the framework by evaluating it on a more diverse set of tasks to enhance its applicability.

Lastly, most experiments so far have utilized relatively small open-source models, with limited testing on larger or closed-source models. In future research, we plan to investigate the performance of LLaMA-Berry on larger models, particularly addressing challenges related to scaling and performance optimization.

#### Acknowledgments

This work is supported by the Shanghai Municipal Science and Technology Major Project. This work is also supported by Shanghai Artificial Intelligence Laboratory.

## References

* Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anaddat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
* Ahn et al. [2024] Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language

Figure 3: Scaling of inference-time rollouts.

 

models for mathematical reasoning: Progress and challenges. _arXiv preprint arXiv:2402.00157_.
* Anthropic (2024) Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. Technical report, Anthropic. Accessed: 2024-09-19.
* Beeching et al. (2024a) Edward Beeching, Shengyi Costa Huang, Albert Jiang, Jia Li, Benjamin Lipkin, Zihan Qina, Kashif Raussl, Ziju Shen, Roman Soletskyi, and Lewis Tunstall. 2024a. Nunnianath 72b cot. [https://huggingface.co/AI-MO/NuminaMath-72B-Cot](https://huggingface.co/AI-MO/NuminaMath-72B-Cot).
* Beeching et al. (2024b) Edward Beeching, Shengyi Costa Huang, Albert Jiang, Jia Li, Benjamin Lipkin, Zihan Qina, Kashif Raussl, Ziju Shen, Roman Soletskyi, and Lewis Tunstall. 2024b. Nunnianath 7b cot. [https://huggingface.co/AI-MO/NuminiaMath-7B-CoT](https://huggingface.co/AI-MO/NuminiaMath-7B-CoT).
* Brown et al. (2024c) Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Re, and Azalia Mirhoiseini. 2024c. Large language monkeys: Scaling inference compute with repeated sampling. _arXiv preprint arXiv:2407.21787_.
* Chen et al. (2024d) Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024d. Alphamathi almost zero: process supervision without process. _arXiv preprint arXiv:2405.03553_.
* Chen et al. (2024e) Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024e. Step-level value preference optimization for mathematical reasoning. _arXiv preprint arXiv:2406.10858_.
* Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. _Advances in neural information processing systems_, 30.
* Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reitichiro Nakano, et al. 2021. Training verifiers to solve math word problems. _arXiv preprint arXiv:2110.14168_.
* Ding et al. (2023) Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. 2023. Everything of thoughts: Defying the law of penrose triangle for thought generation. _arXiv preprint arXiv:2311.04254_.
* Dinh et al. (2022) Tuan Dinh, Yuchen Zeng, Rusiu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. 2022. Lift: Language-interfaced fine-tuning for non-language machine learning tasks. _Advances in Neural Information Processing Systems_, 35:11763-11784.
* Fang et al. (2024) Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, and Kai Zou. 2024. Mathodyssey: Benchmarking mathematical problem-solving skills in large language models using odyssey math data. _Preprint_, arXiv:2406.18321.
* Feng et al. (2023) Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. 2023. Alphazero-like tree-search can guide large language model decoding and training. _arXiv preprint arXiv:2309.17179_.
* Gao et al. (2022) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022. Pal: Program-aided language models. _arXiv preprint arXiv:2211.10435_.
* Gero et al. (2023) Zelalem Gero, Chandan Singh, Hao Cheng, Tristan Naumann, Michel Galley, Jianfeng Gao, and Hoifung Poon. 2023. Self-verification improves few-shot clinical information extraction. _arXiv preprint arXiv:2306.00024_.
* Google (2024) Google. 2024. Gema-2b-it. [https://huggingface.co/google/geema-2b-it](https://huggingface.co/google/geema-2b-it). Accessed: 2024-09-19.
* Hao et al. (2023) Shibio Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. _arXiv preprint arXiv:2305.14992_.
* Havvilla et al. (2024) Alex Havvilla, Sharath Raparthy, Christoforus Nahmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Raileanu. 2024. Glore: When, where, and how to improve llm reasoning via global and local refinements. _arXiv preprint arXiv:2402.10963_.
* He et al. (2024) Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Lang Thai, Junhao Shen, Jinyi Hu, Xu Han, Yuje Huang, Yuxiang Zhang, et al. 2024. Olymgiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. _arXiv preprint arXiv:2402.14008_.
* Hendrycks et al. (2021a) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. _Proceedings of the International Conference on Learning Representations (ICLR)_.
* Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset. _arXiv preprint arXiv:2103.03874_.
* Kang et al. (2024) Jikun Kang, Xin Zhe Li, Xi Chen, Amirreza Kazemi, Qianyi Sun, Boxing Chen, Dong Li, Xu He, Quan He, Feng Wen, Jianye Hao, and Jun Yao. 2024. Mindstar: Enhancing math reasoning in pre-trained llms at inference time. _arXiv preprint arXiv:2405.16265_.
* Kocsis and Szepesvari (2006) Levente Kocsis and Csaba Szepesvari. 2006. Bandit based monte-carlo planning. In _European conference on machine learning_, pages 282-293. Springer.
* Kuncheva (2014) Ludmila I Kuncheva. 2014. _Combining pattern classifiers: methods and algorithms_. John Wiley & Sons.

 

- a reproducible pipeline. _arXiv preprint arXiv:2401.08190_.
* Lightman et al. (2023a) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. _arXiv preprint arXiv:2305.20050_.
* Liu et al. (2023) Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannanach Hajishirzi, and Asli Celikyilmaz. 2023. Don't throw away your value model! generating more preferable text with value-guided monte-carlo tree search decoding. _arXiv preprint arXiv:2309.15028_.
* Luo et al. (2024a) Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. 2024a. Improve mathematical reasoning in language models by automated process supervision. _Preprint_, arXiv:2406.06592.
* Luo et al. (2024b) Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. 2024b. Improve mathematical reasoning in language models by automated process supervision. _arXiv preprint arXiv:2406.06592_.
* Ma et al. (2023) Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. 2023. Let's reward step by step: Step-level reward model as the navigators for reasoning. _arXiv preprint arXiv:2310.10080_.
* MAA (2023) MAA. 2023. American mathematics competitions. Online.
* MAA (2024) MAA. 2024. American invitational mathematics examination. Online.
* Madaan et al. (2023a) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimia Prabhumoye, Yiming Yang, Shahsank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023a. Self-refine: Iterative refinement with self-feedback. _arXiv preprint arXiv:2303.17651_.
* Madaan et al. (2023b) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimia Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023b. Self-refine: Iterative refinement with self-feedback. In _Advances in Neural Information Processing Systems_, volume 36, pages 46534-46594. Curran Associates, Inc.
* Meta (2024a)Meta. 2024a. Meta-llama 3.1-70b instruct. Accessed: 2024-09-03.
* Meta (2024b)Meta. 2024b. Meta-llama 3.1-8b instruct. Accessed: 2024-09-03.
* Meurer et al. (2017) Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondrej Certik, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartiq Singh, Thilingta Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Stepan Roucka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. 2017. SymPy: symbolic computing in Python. _PeerJ Computer Science_, 3:e103. Publisher: PeerJ Inc.
* Mistral (2024)Mistral AI. 2024. Mathstral. [https://mistral.ai/news/mathstat](https://mistral.ai/news/mathstat). Accessed: 2024-08-12.
* OpenAI (2024)OpenAI. 2024. Introducing openai o1-preview. Online.
* OpenAI (2024)OpenAI. 2024. Learning to reason with llms. [https://openai.com/index/learning-to-reason-with-llms/](https://openai.com/index/learning-to-reason-with-llms/). Accessed: 2024-09-19.
* Paul et al. (2023) Debjit Paul, Mete Ismayilkada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. Refiner: Reasoning feedback on intermediate representations. _arXiv preprint arXiv:2304.01904_.
* Qi et al. (2024) Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. 2024. Mutual reasoning makes smaller lms stronger problem-solvers. _arXiv preprint arXiv:2408.06195_.
* Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. _Advances in Neural Information Processing Systems_, 36.
* Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. _arXiv preprint arXiv:2403.05530_.
* Rein et al. (2024) David Rein, Betty Li Hou, Asia Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In _First Conference on Language Modeling_.
* Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. _arXiv preprint arXiv:2402.03300_.
* Shinn et al. (2024) Noah Shinn, Federico Cassano, Edward Berman, Karthik Narasimhan Ashwin Gopinath, and Shunya Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. _arXiv preprint arXiv:2303.11366_ 

Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. 2024. Easy-to-hard generalization: Scalable alignment beyond human supervision. _arXiv preprint arXiv:2403.09472_.
* Tang et al. (2024) Zhengyang Tang, Xingxing Zhang, Benyou Wan, and Furu Wei. 2024. Mathscale: Scaling instruction tuning for mathematical reasoning. _arXiv preprint arXiv:2403.02884_.
* Tian et al. (2024) Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. 2024. Toward self-improvement of llms via imagination, searching, and criticizing. _arXiv preprint arXiv:2404.12253_.
* Toshniwal et al. (2024) Shubham Toshniwal, Ivan Moshkov, Sean Narethiran, Daria Gitman, Fei Jia, and Igor Gitman. 2024. Openmathinstruct-1: A 1.8 million math instruction tuning dataset. _arXiv preprint arXiv:2402.10176_.
* Uesato et al. (2022) Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process-and outcome-based feedback. _arXiv preprint arXiv:2211.14275_.
* Wang et al. (2023) Peiyi Wang, Lei Li, Zhihong Shao, R.X. Xu, Damai Dai, Yifei Li, Deli Chen, Y.Wu, and Zhifang Sui. 2023. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. _arXiv preprint arXiv:2312.08935_.
* Wang et al. (2022) Xuezhii Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Akanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. _arXiv preprint arXiv:2203.11171_.
* Warshall (1962) Stephen Warshall. 1962. A theorem on boolean matrices. _J. ACM_, 9(1):11-12.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in neural information processing systems_, 35:24824-24837.
* Weng et al. (2022) Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2022. Large language models are better reasoners with self-verification. _arXiv preprint arXiv:2212.09561_.
* Xie et al. (2024) Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P. Lillicrap, Kenji Kawaguchi, and Michael Shieh. 2024. Monte carlo tree search boosts reasoning via iterative preference learning. _arXiv preprint arXiv:2405.00451_.
* Xu (2023) Haotian Xu. 2023. No train still gain. unleash mathematical reasoning of large language models with monte carlo tree search guided by energy function. _arXiv preprint arXiv:2309.03224_.
* Yang et al. (2021) An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinyang Lin, Kai Bang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runqi Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhao Liu, Tianyu Liu, Wenbin Ge, Xiaohua Zhou, Xinghang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Youiqon Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. 2024. Qwen2 wang, Kai Chen, and Dahna Lin. 2024. Internlm-math: Open math large language models toward verifiable reasoning. _Preprint_, arXiv:2402.06332.
* Yu et al. (2023) Fei Yu, Anninghe Gao, and Benyou Wang. 2023. Outcome-supervised verifiers for planning in mathematical reasoning. _arXiv preprint arXiv:2311.09724_ 

Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2024a. Generative verifiers: Reward modeling as next-token prediction. _arXiv preprint arXiv:2408.15240_.
* [Zhang et al.2024b] Yifan Zhang, Yang Yuan, and Andrew Chi-Chih Yao. 2024b. On the diagram of thought. _arXiv preprint arXiv:2409.10038_.

 



## Appendix A Hyper-paramerer settings

All hyper-parameter settings for Section 3 are listed in Table A1.

## Appendix B Details of Grading and Metrics

GradingWe follow the correctness evaluation method proposed in PRM800K (Lightman et al., 2023) to score the answers generated by the algorithm. For the mathematical solutions proposed by the algorithm and their corresponding ground truth, we inform the model of the expected response format in a prompt. The answer's formula or value is extracted by matching the response format with predefined rules. If the model fails to follow the expected format in the prompt and the rule-based extraction fails, the solution is directly judged as inconsistent with ground truth.

For the extracted label, we score the answer based on the following criteria. The answer is considered consistent with the ground truth label and passes the evaluation if at least one of the criteria is met:

1. The answer label string is exactly equal to the ground truth label string in terms of literal value.
2. Both the answer label and the ground truth label can be converted to floating-point numbers, and the difference between the two values is less than \(1\times 10^{-6}\).
3. Following the criterion proposed in PRM800K (Lightman et al., 2023), we use the Sympy library (Meurer et al., 2017) to simplify the difference between the expression corresponding to the answer label, denoted as \(a\), and the expression corresponding to the ground truth label, denoted as \(b\). If the simplification yields \(a-b\iff 0\), the criterion is satisfied.

MetricsTo provide a comprehensive and robust evaluation metric, we adopt **major@k** and **rm@k** as the evaluation metrics.

* **rm@k** represents reward model best-of-N among \(k\) sampled response (Yang et al., 2024).
* **major@k**(Kuncheva, 2014), also abbreviated as **maj@k**, is defined as the fraction of tasks where a majority of the top \(k\) samples generated return the same correct solution. This metric focuses on consistency across multiple generated answers. The majority weight is calculated on the solution's extracted labels.
* The **solved rate of problems**(Lightman et al., 2023; Luo et al., 2024) refers to the percentage of problems who have solutions meet the evaluation criteria.

For results without a subscript, the score represents the greedy evaluation using the default prompt of the base model. Results with a notation indicate the use of the corresponding prompt engineering technique, and those marked with **self-consistency** use the self-consistency aggregation method.

For closed-source models, we report the scores from existing official technical reports (Anthropic, 2024; Reid et al., 2024; OpenAI, 2024) or dataset-provided results, with no modifications.

## Appendix C Details of PPRM Training

### Overview

The design goal of the PPRM is to integrate the properties of both PRM and ORM, providing a more nuanced preference prediction between any two solution answers. We attempt to utilize reinforcement learning methods for training, leveraging the model's instruction-following capability to predict the relative merits of pairs of problem-solving answers. This will further enable the use of the EBC method to evaluate the global quantile scores of mathematical solutions.

### Data Collection

Our data synthesis is derived from two datasets: PRM800K (Lightman et al., 2023) and OpenMathInstruct-1 (Toshniwal et al., 2024). The PRM800K dataset, collected from the MATH dataset, comprises a substantial number of step-divided problem-solving answers, with manual quality annotations for each step. We primarily utilize this dataset to generate pairs of answers for comparative analysis based on step-wise process quality. The OpenMathInstruct-1 dataset

incorporates data from the GSM8K and MATH datasets, which have been manually annotated for outcome correctness. We use this dataset to synthesize pairs for comparative analysis based on outcome quality.

In processing PRM800K, for a given problem, we first sample steps of varying quality annotations from the step-wise dataset to construct a complete reasoning path. For pairs with the same final step annotation, paths composed of higher-quality steps are considered superior to those with lower-quality steps. In cases where the final step annotations are not identical, the reasoning path with the superior final step annotation is regarded as better.

During the processing of OpenMathInstruct-1, we exclusively utilize samples without Python interpreter calls. For the same problem, we sample pairs composed of outcomes with higher and lower quality annotations.

Notably, we filtered out any data samples from PRM800K and OpenMathInstruct-1 that may overlap with the GSM8K and MATH test sets, especially the PRM800K. Ultimately, we formed a dataset of 7,780,951 entries for training the PPRM model.

### Direct Preference Pair Construction

For all pairs, we frame the inquiry as "For Question \(Q\), is solution \(a_{1}\) better than solution \(a_{2}?\)" If solution \(a_{1}\) is deemed superior to solution \(a_{2}\), we label it as 'Yes'; otherwise, it is labeled as 'No'.

In this manner, we transform the ordinal relationship prediction task into a question-answer format, employing the Direct Preference Optimization (DPO) method for model training through reinforcement learning from human feedback (RLHF). This approach aims to enhance the model's capability to follow instructions in predicting the relative merits of pairs of problem-solving answers.

### RLHF Training

We apply the DPO method to train the Gemma2-2B-it model using RLHF which is shown in Figure A2. The loss function for DPO is structured as \(\mathcal{L}_{\mathrm{DPO}}(\pi_{g};\pi_{\mathrm{ref}})=-\mathbb{E}\left[\log \sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\mathrm{ref}}(y_{w}| x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\mathrm{ref}}(y_{l}|x)}\right)\right]\), where \(\sigma\) is the logistic function, \(\beta\) is a parameter controlling the deviation from the base reference policy \(\pi_{ref}\), namely the initial model \(\pi^{LLM}\). In practice, the language model policy \(\pi_{\theta}\) is also initialized to \(\pi^{LLM}\). For one answer, denoted as \(y_{w}\succ y_{l}|x\) where \(y_{w}\) and \(y_{l}\) denote the preferred and dispreferred completion amongst \((\hat{y},y)\) respectively.

## Appendix D Details of Berry-Tree Inference Framework

### Overview

Berry-Tree is an inference framework specifically designed for complex multi-model reasoning tasks, addressing the need to improve inference efficiency and accuracy in intricate tree search processes of LLM's mathematical reasoning. This framework is particularly suited for large-scale reasoning tasks that involve the integration of multiple models and algorithms. By incorporating advanced tree search methods especially Monte Carlo Tree Search (MCTS), robust concurrency handling, and high-performance inference engines, Berry-Tree significantly enhances inference speed and resource utilization of LLM's mathematical reasoning process. This section provides a explanation of the system architecture and key tech 

nologies of Berry-Tree, along with a preliminary performance evaluation results.

### System Architecture Overview

Figure A3 demonstrates the architecture of Berry-Tree which is divided into several layers, each handling different functional requirements. The **Data Management Layer** is responsible for the serialization and deserialization of data, ensuring efficient data read/write operations across models and systems and the ablity of recovering the search process from serialized data. The **Tree Search Methods Layer** incorporates MCTS (Monte Carlo Tree Search), ToT (Tree of Thoughts), and A* algorithms to optimize the inference process and explore multiple reasoning paths. Additionally, Berry-Tree includes a **Reliability Layer**, which ensures load balancing and failover support in highly concurrent scenarios, guaranteeing the stability of inference tasks. Finally, the **Inference Engines Layer** integrates efficient inference engines such as vLLM, FastAPI, and HuggingFace TGI to enable parallelized and efficient task processing.

### Key Technical Components

**Data Management.**Berry-Tree employs serialization and deserialization techniques, specifically using formats including JSON and CSV, to efficiently store, transfer and recover checkpoint data from tree search reasoning processes. The framework stores this data along with hash values to ensure integrity and allows for quick restoration of tree search states in memory when needed. Furthermore, Berry-Tree leverages the HuggingFace Datasets library to handle core dataset inputs for both training and inference. It supports seamless loading of benchmark datasets such as GSM8K and MATH from the HuggingFace Hub, enhancing its flexibility and ensuring compatibility with diverse reasoning tasks.

**Tree Search Methods.**Berry-Tree support multiple tree search algorithms, with MCTS being central to handling large-scale complex reasoning tasks by leveraging random simulation and statistical analysis to optimize the search space. The Tree of Thoughts (ToT) extends the exploration depth and breadth of reasoning paths, helping the system manage uncertainty. And A* can offer a efficient heuristic search capabilities.

**Reliability Design.** To ensure the stability and continuity of inference tasks, Berry-Tree incorporates load balancing and failover mechanisms. During high-concurrency operations, the load balance component distributes workloads across different inference servers, preventing server overload. The failover mechanism ensures that tasks can seamlessly recover and transition to backup servers in case of partial server failures.

**Server Architecture.** The framework's server architecture is divided into two segments: one dedicated to executing Large Language Model (LLM) inference and the other designated explicitly for handling PPRM (Pairwise Preference Reward Model) with EBC (Enhanced Borda Count) method. This modular design allows the framework to allocate computational resources flexibly, improving overall efficiency.

**Inference Engines.** Inference engines of Berry-Tree include VLLM , HuggingFace Transformers warpped by FastAPI, and HuggingFace TGI. These engines collectively enable the system to maintain high efficiency and stability while handling multi-model inference tasks, with robust support for high-concurrency demands.

 



### Preliminary Performance Evaluation

In a preliminary performance evaluation, we utilize 16 A100 GPUs to run the LLAMA3.1-8B-instruct model for large-scale inference tasks, while 4 A100 GPUs are used to run the Gemma2-2B-it model as PPRM servers. The test dataset consists of 1319 GSM8K test samples.

We conduct 16 rollouts to parallelize the inference tasks of LLaMA-Berry via Berry-Tree. The results indicate that the total inference time is 1 hour and 25 minutes, with an average inference time of approximately 3.87 seconds per sample. These results demonstrate a strong parallel inference capabilities of Berry-Tree under the given hardware configuration.

## Appendix E Scaling Study on Inference-time Token Overheads

Comparing LLaMA-3-8B-Instruct and LLaMA-3.1-8B-Instruct across GSM8K, GSMHARD, and MATH500, LLaMA-3.1-8B-Instruct consistently consumes more tokens across all categories--Solutions, Critiques, and Overall. This Figure A4 result reflects LLaMA-3.1-8B-Instruct's tendency to generate more detailed and comprehensive outputs with increased overhead. While this likely improves solution quality, it also introduces greater resource demands and variability in token usage, highlighting a trade-off between accuracy and computational resources. LLaMA-3.1-8B-Instruct is thus better suited for tasks prioritizing precision over speed. As shown in Figure A5, Token overheads during inference also scale with task difficulty across different olympiad-level benchmarks with LLaMA-3.1-8B-Instruct. AIME2024 exhibits the highest token consumption with significant variability, reflecting the complexity of its solution paths. In contrast, GPQA Diamond shows lower overall overhead, while AMC2023 falls in between, with moderate token consumption and less variability than AIME2024 but still notable.

## Appendix F Future Work

LLaMA-Berry holds significant potential for further development. First, we plan to enhance its multi-modal capabilities, enabling it to better handle complex problems like VQA and mathematic geometric problems that require visual, auditory, or even tactile perception. For instance, in tasks involving visual reasoning, such as geometric problems, LLaMA-Berry could be extended to integrate image recognition and analysis capabilities, thereby assisting in solving challenges related to spatial relationships and shape recognition. Furthermore, we aim to generalize its application across other scientific domains, improving its performance in disciplines such as physics, chemistry, and biology. In physics, for example, it could be utilized to address complex micro- and macroscopic dynamics problems; in chemistry, it may assist in molecular structure prediction and drug design; and in biology, it could potentially be used for genome analysis and disease prediction. Moreover, LLaMA-Berry can also offer technical support for other interdisciplinary fields, such as meteorology, environmental science, and

materials science, by integrating multimodal data to enhance predictive accuracy. At the same time, we will focus on how LLaMA-Berry can further enhance AI safety. For instance, LLaMA-Berry could be leveraged to design more robust safety and risk assessment mechanisms. Besides, the generation of responses could be guided by a safety-trained PPRM to produce outputs with higher safety standards. These advancements not only pave the way for broader scientific applications of LLaMA-Berry but also offer new possibilities for enhancing AI safety and promoting its widespread adoption.

## Appendix G Case Study

The prompts utilized in LLaMA-Berry for the LLaMA-3.1-8B-Instruct model are presented in Figure A6. Additionally, Figure A7 provides a detailed breakdown of problem-solving examples derived from the GSM8K dataset.

## Appendix H Convergence Analysis of the Enhanced Borda Count (EBC) Method

In this appendix, we present a formal discussion on how the quantile scores, as evaluated by the Enhanced Borda Count (EBC) method, converge to the true quantile scores of solutions within the actual quality distribution as the number of samples increases.

### Definitions and Assumptions

**Solution Set.** Let \(A=\{a_{1},a_{2},\ldots,a_{n}\}\) be a finite set of \(n\) solutions (answers).

**True Quantile Scores.** Each solution \(a_{i}\) has a true quality score \(Q^{*}(a_{i})\in\mathbb{R}\), drawn from a continuous distribution \(P_{Q}\).

**True Ranking.** The true ranking \(R^{*}\) is induced by ordering the solutions in decreasing order of their true quantile scores \(Q^{*}(a_{i})\).

**True Pairwise Preference Probabilities.** The true probability that solution \(a_{i}\) is preferred over \(a_{j}\) is defined as:

\[P^{*}(a_{i}\succ a_{j})=\\ \mathbb{P}(Q^{*}(a_{i})>Q^{*}(a_{j}))+\tfrac{1}{2}\mathbb{P}( Q^{*}(a_{i})=Q^{*}(a_{j})).\] (A1)

Given the continuity of \(P_{Q}\), we have \(\mathbb{P}(Q^{*}(a_{i})=Q^{*}(a_{j}))=0\), so:

\[P^{*}(a_{i}\succ a_{j})=\begin{cases}1,&\text{if $Q^{*}(a_{i})>Q^{*}(a_{ j})$},\\ 0,&\text{if $Q^{*}(a_{i})<Q^{*}(a_{j})$}.\end{cases}\] (A2)

**Estimated Preference Probabilities.** For \(T\) independent samples, the estimated preference probability is:

\[P_{T}(a_{i}\succ a_{j})=\frac{1}{T}\sum_{t=1}^{T}X_{t}^{(i,j)},\] (A3)

where \(X_{t}^{(i,j)}\) are independent Bernoulli random variables with success probability \(P^{*}(a_{i}\succ a_{j})\).

 



**Convergence Assumption.** As \(T\to\infty\), \(P_{T}(a_{i}\succ a_{j})\) converges almost surely to \(P^{*}(a_{i}\succ a_{j})\).

**Preference Matrix.** Construct the estimated preference matrix \(M_{T}\) and the true preference matrix \(M^{*}\) as:

\[M_{T}[i,j]=\begin{cases}1,&\text{if $P_{T}(a_{i}\succ a_{j})\geq 0.5$},\\ 0,&\text{otherwise}.\end{cases}\] (A4)

\[M^{*}[i,j]=\begin{cases}1,&\text{if $P^{*}(a_{i}\succ a_{j})=1$},\\ 0,&\text{if $P^{*}(a_{i}\sim a_{j})=0$}.\end{cases}\]

**Transitive Closure.** Let \(C_{T}\) and \(C^{*}\) be the transi

tive closures of \(M_{T}\) and \(M^{*}\), respectively.

**Borda Counts.** Compute the Borda count for each solution:

\[B_{T}(a_{i})=\sum_{j\neq i}C_{T}[i,j],\quad B^{*}(a_{i})=\sum_{j\neq i}C^{*}[i, j].\] (A5)

Estimated Ranking and Quantile Scores.

* Estimated ranking \(R_{T}\): Order solutions by decreasing \(B_{T}(a_{i})\).
* Estimated quantile score: \[Q_{g}(a_{i})=1-\frac{\text{rank}_{T}(a_{i})-1}{n-1}.\] (A6)
* True quantile score: \[Q_{g}^{*}(a_{i})=1-\frac{\text{rank}^{*}(a_{i})-1}{n-1}.\] (A7)

### Objective

To prove that as \(T\to\infty\), the estimated quantile scores \(Q_{g}(a_{i})\) converge almost surely to the true quantile scores \(Q_{g}^{*}(a_{i})\):

\[\lim_{T\to\infty}Q_{g}(a_{i})=Q_{g}^{*}(a_{i}),\quad\forall a_{i}\in A.\] (A8)

### Proof Convergence of Estimated Preference Probabilities.

By the Strong Law of Large Numbers (SLLN), since \(X_{t}^{(i,j)}\) are i.i.d. Bernoulli random variables with success probability \(P^{*}(a_{i}\succ a_{j})\), we have:

\[\lim_{T\to\infty}P_{T}(a_{i}\succ a_{j})=P^{*}(a_{i}\succ a_{j})\quad\text{ almost surely}.\] (A9)

Convergence of Preference Matrix \(M_{T}\) to \(M^{*}\).Since \(P_{T}(a_{i}\succ a_{j})\) converges to \(P^{*}(a_{i}\succ a_{j})\) and \(P^{*}(a_{i}\succ a_{j})\in\{0,1\}\), for sufficiently large \(T\), we have:

\[M_{T}[i,j]=M^{*}[i,j],\quad\forall i\neq j,\] (A10)

almost surely.

JustificationBecause \(P_{T}(a_{i}\succ a_{j})\) converges to either 0 or 1, and \(P_{T}(a_{i}\succ a_{j})\neq 0.5\) almost surely for large \(T\).

Convergence of Transitive Closure \(C_{T}\) to \(C^{*}\).The transitive closure is a deterministic function of the preference matrix. Therefore, since \(M_{T}\to M^{*}\), it follows that:

\[C_{T}\to C^{*}\quad\text{as $T\to\infty$},\] (A11)

almost surely.

Convergence of Borda Counts \(B_{T}(a_{i})\) to \(B^{*}(a_{i})\).Given that \(C_{T}\to C^{*}\), the Borda counts converge:

\[B_{T}(a_{i})=\sum_{j\neq i}C_{T}[i,j]\to B^{*}(a_{i})=\sum_{j\neq i}C^{*}[i,j],\] (A12)

almost surely.

Convergence of Estimated Ranking \(R_{T}\) to True Ranking \(R^{*}\).Since the Borda counts \(B_{T}(a_{i})\) converge to \(B^{*}(a_{i})\), and assuming that all \(Q^{*}(a_{i})\) are distinct (due to the continuity of \(P_{Q}\)), the rankings induced by \(B_{T}(a_{i})\) converge to the true rankings:

\[R_{T}\to R^{*}\quad\text{as $T\to\infty$},\] (A13)

almost surely.

Convergence of Quantile Scores \(Q_{g}(a_{i})\) to \(Q_{g}^{*}(a_{i})\).Since \(\text{rank}_{T}(a_{i})\to\text{rank}^{*}(a_{i})\), we have:

\[Q_{g}(a_{i})=1-\frac{\text{rank}_{T}(a_{i})-1}{n-1}\rightarrow\] (A14) \[Q_{g}^{*}(a_{i})=1-\frac{\text{rank}^{*}(a_{i})-1}{n-1}\]

almost surely.ConclusionTherefore, we have shown that:

\[\lim_{T\to\infty}Q_{g}(a_{i})=Q_{g}^{*}(a_{i}),\quad\forall a_{i}\in A,\] (A15)

which means that the EBC method's estimated quantile scores converge almost surely to the true quantile scores of the solutions.

### Finite Sample Analysis and Convergence Rate

Lemma 1: Hoeffding's Inequality for Preference Probability Estimates.For each pair \((a_{i},a_{j})\), \(P_{T}(a_{i}\succ a_{j})\) is the sample mean of \(T\) i.i.d. Bernoulli trials with success probability \(P^{*}(a_{i}\succ a_{j})\). By Hoeffding's inequality:

\[\mathbb{P}\left(|P_{T}(a_{i}\succ a_{j})-P^{*}(a_{i}\succ a_{j})| \geq\epsilon\right)\\ \leq 2\exp(-2T\epsilon^{2}).\] (A16)

Lemma 2: Uniform Convergence over All Pairs.Apply the union bound over all \(N=n(n-1)/2\) pairs:

\[\mathbb{P}\left(\exists(i,j):|P_{T}(a_{i}\succ a_{j})-P^{*}(a_{i }\succ a_{j})|\geq\epsilon\right)\\ \leq N\cdot 2\exp(-2Tc^{2}).\] (A17) 

Set the right-hand side equal to \(\delta\) to solve for \(T\):

\[T\geq\frac{1}{2\epsilon^{2}}\ln\left(\frac{2N}{\delta}\right).\] (A18)

**Lemma 3: Correctness of Preference Matrix with High Probability.**

Given that \(P^{*}(a_{i}\succ a_{j})\in\{0,1\}\), for any \(0<\epsilon<0.5\), if:

\[|P_{T}(a_{i}\succ a_{j})-P^{*}(a_{i}\succ a_{j})|<\epsilon,\] (A19)

then \(M_{T}[i,j]=M^{*}[i,j]\) because \(P_{T}(a_{i}\succ a_{j})\) will be greater than \(0.5\) when \(P^{*}(a_{i}\succ a_{j})=1\), and less than \(0.5\) when \(P^{*}(a_{i}\sim a_{j})=0\).

**Lemma 4: Probability of Correct Ranking.**

From Lemma 2 and 3, with probability at least \(1-\delta\), \(M_{T}=M^{*}\), and thus \(C_{T}=C^{*}\), leading to \(R_{T}=R^{*}\) and \(Q_{g}(a_{i})=Q_{g}^{*}(a_{i})\).

Convergence Rate Analysis.To achieve this with confidence level \(1-\delta\), the required number of samples per pair is:

\[T\geq\frac{1}{2\epsilon^{2}}\ln\left(\frac{n(n-1)}{\delta}\right).\] (A20)

For small \(\delta\) and \(\epsilon<0.5\), \(T\) scales logarithmically with the number of solutions \(n\).

### Addressing Practical Considerations

In practice, the true preference probabilities may not be exactly \(0\) or \(1\) due to noise or overlapping quality scores. To accommodate this:

* **Extended Preference Model**: Assume \(P^{*}(a_{i}\succ a_{j})\) is a strictly increasing function of \(\Delta Q_{ij}^{*}=Q^{*}(a_{i})-Q^{*}(a_{j})\), such as: \[P^{*}(a_{i}\succ a_{j})=F(\Delta Q_{ij}^{*}),\] (A21) where \(F\) is a cumulative distribution function (CDF).
* **Margin Condition**: Define a margin \(m>0\) such that for all \(i\neq j\): \[|P^{*}(a_{i}\succ a_{j})-0.5|\geq m.\] (A22) This ensures a minimum separation between preference probabilities.
* **Modified Sample Complexity**: With the margin condition, Hoeffding's inequality becomes: \[\mathbb{P}\left(M_{T}[i,j]\neq M^{*}[i,j]\right)\leq 2\exp(-2Tm^{2}).\] (A23) To achieve \(\mathbb{P}\left(M_{T}=M^{*}\right)\geq 1-\delta\), we need: \[T\geq\frac{1}{2m^{2}}\ln\left(\frac{n(n-1)}{\delta}\right).\] (A24)
* **Convergence under Noise**: Even with noisy preference probabilities, as long as there is a margin \(m>0\), the convergence of \(Q_{g}(a_{i})\) to \(Q_{g}^{*}(a_{i})\) still holds with high probability for sufficiently large \(T\).

### Final Conclusion

We have provided a formal discussion showing that the estimated quantile scores \(Q_{g}(a_{i})\), obtained through the EBC method, converge almost surely to the true quantile scores \(Q_{g}^{*}(a_{i})\) as the number of samples \(T\) approaches infinity. The finite sample analysis demonstrates that the convergence rate depends logarithmically on the number of solutions and is inversely proportional to the square of the margin \(m\) between preference probabilities.

## Appendix I Pseudo-code of main Algorithms

We present the process of Self-Refine applied to Monte Carlo Tree Search (SR-MCTS) Method in Algorithm 1 and provide overall pseudo-code for Enhanced Borda Count (EBC) Method in Algorithm 2 

```
Input: Initial state \(s_{0}\), search tree \(\mathcal{T}\), max nodes \(N_{max}\), exploration constant \(c\) Output: Ranked solution list \(S\)
1 Initialize search tree \(\mathcal{T}\) with root node \(s_{0}\); whilenumber of nodes \(N(\mathcal{T})<N_{max}\)do
2Selection PhaseSelect a node \(s_{i}\) which met the dynamic pruning rule from \(\mathcal{T}\) using UCT: \[a=\arg\max_{a\in A(s)}\left(Q(s,a)+c\cdot\sqrt{\frac{\ln N(s)}{N(s,a)}}\right)\]
```
5Expansion Phase Expand \(s_{i}\) by generating a successor node \(s^{\prime}\) using the rewriting process \(R(s_{i},c_{i})\), where \(c_{i}=C(s_{i})\) is a critique of the current state; Add the new node \(s^{\prime}\) to \(\mathcal{T}\);
6Evaluation PhaseCompute the value \(Q(s^{\prime})\) of the new node with Enhanced Borda Count (EBC) method: \[Q(s^{\prime})=\alpha Q_{g}(s^{\prime})+(1-\alpha)Q_{l}(s^{\prime})\] where \(Q_{g}(s^{\prime})\) is the global value from the win-loss matrix \(M\) and \(Q_{l}(s^{\prime})\) is the local value from adjacent nodes in \(\mathcal{T}\);
7Backpropagation PhasePropagate \(Q(s^{\prime})\) back to its parent node \(s_{i}\), updating \(s_{i}\)'s Q value: \[Q(s_{i})=(1-\gamma)Q(s_{i})+\gamma Q(s^{\prime})\] Check for tree growth limit if\(N(\mathcal{T})\geq N_{max}\)then break; end returnRanked solution list \(S\) ```

**Algorithm 1**Self-Refine applied to Monte Carlo Tree Search (SR-MCTS) Method 

```
Data:\(M\) (Binary Reward Matrix),\(P\) (Pairwise Preference Reward model) Result:\(Q\) (quantile rewards), \(R\) (ranked node list), \(L\) (ranked layers of nodes)
1FunctionFillTransitiveClosure(\(M\)):
2\(C\leftarrow\) all-zero matrix from size of \(M\)\(C[i,j]\leftarrow-1\) for all \(i,j\)\(C[i,j]\leftarrow\text{sign}(M[i,j]-0.5)\textbf{ if }M[i,j]\neq-1\textbf{ else }-1\)for\(k=0\)to\(|C|-1\)do
3for\(i=0\)to\(|C|-1\)do
4if\(C[i,k]=C[k,j]\)then
5\(C[i,j]\gets C[i,k]\)
6 end if
7
8 end while
9
10 end for return Updated\(C\) FunctionBordaCount(\(C\)):
11\(D\leftarrow\) outdegree of each node from \(C\)\(R\leftarrow\text{argsort}(-D)\)\(L\leftarrow\) Define layers using unique values in \(D\)return\(R\), \(L\)
12FunctionRerank(\(R\), \(L\), \(C\), \(P\)):
13\(G\leftarrow\) Group \(R\) by \(L\)\(R\leftarrow\) Sort each group in \(G\) using local comparisons computed by \(P\)\(L,C\leftarrow\) Update \(L,C\) by \(R\)return\(R\), \(L\), Updated\(C\)
14FunctionCalculateQuantileScores(\(R\), \(L\)):
15\(Q\leftarrow\emptyset\)\(S_{L}\leftarrow\{1-\frac{l}{\max(L)}:l\in L\}\)\(Q[x]\gets S_{L}[\text{layer of $x$}]\) for each \(x\in R\)return\(Q\)FunctionEnhancedBordaCount(\(M\), \(q\)):
16\(C\leftarrow\) FillTransitiveClosure(\(M\))\(R\), \(L\leftarrow\) BordaCount(\(C\))\(R\), \(L\), \(C\leftarrow\) Rerank(\(R\), \(L\), \(C\),\(P\))\(Q\leftarrow\) CalculateQuantileScores(\(R\), \(L\)) return\(Q\), \(R\), \(L\)
```

**Algorithm 2**Ehanced Borda Count (EBC) Method